A method, system and device for analyzing real-time data stream of IOT big data

Abstract

The present invention discloses a method, system and device for analyzing real-time data streams of IOT big data, and relates to the field of real-time data analysis of the Internet of Things, including obtaining basic characteristic information of real-time data streams, analyzing and processing to obtain basic characteristic check values ​​of real-time data streams; obtaining preliminary data allocation ratios of real-time data streams, and dispatching each sub-data stream to corresponding data processing locations for trial preprocessing; obtaining trial real-time load data of each edge computing node and cloud processor, and obtaining real-time operation load monitoring values ​​of each edge computing node and cloud processor; dynamically correcting the preliminary data allocation ratio of real-time data streams, obtaining data re-correction ratios, and dynamically allocating each sub-data stream. The present invention improves the efficiency of data processing and ensures the timeliness of real-time data processing by dynamically dispatching each sub-data stream to the corresponding data processing location for processing.

Description

Technical Field

[0001] The present invention relates to the field of real-time data analysis of the Internet of Things, and specifically to a method, system and device for real-time data stream analysis of IOT big data. Background Art

[0002] IOT, also known as the Internet of Things, refers to a network that connects various information sensing devices, smart devices and other objects through information carriers such as the Internet and traditional communication networks to achieve information exchange and communication between objects in order to achieve intelligent identification, positioning, tracking, monitoring and management. Due to the variety of devices connected to it and the relatively complex real-time data stream transmitted, insufficient data processing or processor overload may occur in actual processing. Therefore, it is necessary to fully analyze the real-time data stream to ensure the timeliness and smoothness of data processing.

[0003] The prior art, such as the invention patent with announcement number: CN113282568B, is an IOT big data real-time time series stream analysis application technology method, which includes the following steps: Step 1: Massive database construction: S1: Analysis of original data; S2: Data classification and structured storage; S3: Mapping data objects to database structure to establish a massive database; S4: Establishing massive database index; S5: Low real-time analysis service; Step 2: Real-time data analysis: S6: Cache the generated massive data; S7: Real-time statistical analysis service; S8: Cache analysis results.

[0004] The prior art, such as the invention patent with announcement number: CN112347094B, is a method, device and storage medium for cleaning data of the Internet of Things of industrial general equipment. The method includes: obtaining status data reported by general IoT devices and storing them in a time series database; extracting status data from the time series database and performing type identification through a preset entity recognition algorithm to obtain entity data information and data type of the data to be repaired; if the data type of the data to be repaired belongs to a preset interval quantity, using a preset interval quantity specific data inconsistency error detection and repair algorithm to clean the data to be repaired to obtain the first data to be filled; if the data type belongs to a preset instantaneous quantity, using a preset instantaneous quantity specific data inconsistency error detection and repair algorithm to clean the data to be repaired to obtain the second data to be filled; if the data type cannot be identified, it is classified as the third data to be filled; using an incomplete data algorithm to perform real-time detection on the first data to be filled, the second data to be filled and the third data to be filled, fill in the missing values ​​and store them in the preset database.

[0005] Based on the above scheme, it can be seen that for the processing of IOT real-time data, the existing technology often only analyzes and processes the data in a general and single manner. However, in actual applications, some IOT real-time data may have some problems in characteristics, such as excessive data volume or too many data types, which require targeted processing and feedback. At the same time, the existing technology often only sets a single processor to process and analyze the data, but in actual applications, massive IOT real-time data may cause a single processor to overload, resulting in data loss and hardware damage. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an IOT big data real-time data stream analysis method, system and device. To achieve the above purpose, the present invention is implemented through the following technical solutions: A method for analyzing IOT big data real-time data stream, comprising:

[0007] The basic characteristic information of the real-time data stream is obtained from the edge channel built into the IOT. Based on the basic characteristic information of the real-time data stream, the basic characteristic check value of the real-time data stream is obtained through analysis and processing.

[0008] Based on the basic feature check value analysis and processing of the real-time data stream, the preliminary data allocation ratio of the real-time data stream is obtained, and the real-time data stream is split into several independent sub-data streams. According to the preliminary data allocation ratio, each sub-data stream is dispatched to the corresponding data processing location for trial preprocessing. The data processing location includes each edge computing node and cloud processor.

[0009] Based on the monitoring agent pre-deployed at the data processing location, the trial real-time load data of each edge computing node and cloud processor is obtained, and the trial real-time load data of each edge computing node and cloud processor is transmitted to the monitoring server, and the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor are obtained through analysis and processing.

[0010] Based on the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor, the initial data allocation ratio of the real-time data flow is dynamically corrected to obtain the data re-correction ratio, and each sub-data flow is dynamically allocated based on the data re-correction ratio.

[0011] As a preferred technical solution, the basic feature information of the real-time data stream is obtained from the edge channel built into the IOT, and the basic feature check value of the real-time data stream is obtained by analysis and processing based on the basic feature information of the real-time data stream. The specific process is:

[0012] The basic characteristic information of the real-time data stream includes the number of data nesting levels, the number of data types, the number of data bytes and the number of data format types.

[0013] A preset basic feature information verification set is extracted from the database, wherein the basic feature information verification set includes a data nesting level quantity threshold, a data type quantity threshold, a data byte threshold, and a data format type quantity threshold. The basic feature information of the real-time data stream is analyzed and processed with the basic feature information verification set to obtain a basic feature verification value of the real-time data stream.

[0014] The basic characteristic check value of the real-time data stream is used to characterize the difficulty of analyzing and processing the real-time data stream.

[0015] As a preferred technical solution, the basic feature check value analysis and processing based on the real-time data stream obtains the preliminary data distribution ratio of the real-time data stream, and splits the real-time data stream into several independent sub-data streams. The specific process is:

[0016] Based on the basic feature check value of the real-time data stream, the preliminary data allocation ratio corresponding to each basic feature check value interval stored in the database is mapped and matched to obtain the preliminary data allocation ratio of the real-time data stream. The preliminary data allocation ratio of the real-time data stream is in the form of a numerical value: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the ratio of sub-data streams allocated to cloud processors.

[0017] The splitting of the real-time data stream into several independent sub-data streams specifically includes: determining several independent fields in the real-time data stream for grouping, marking the several independent fields as key values, and performing grouping operations on the real-time data stream based on the key values ​​and the stream processing framework to split it into several independent sub-data streams.

[0018] As a preferred technical solution, the real-time load data of each edge computing node and cloud processor, including:

[0019] The real-time load data of each edge computing node includes the real-time CPU usage, real-time network traffic rate, real-time disk usage and real-time temperature of each edge computing node.

[0020] The real-time load data of the cloud processor includes the real-time usage rate of network bandwidth, the real-time frequency of disk read and write operations, the real-time network packet loss rate, and the number of remaining tasks of the cloud processor.

[0021] A preset real-time load verification set of an edge computing node is extracted from a database, wherein the real-time load verification set of the edge computing node includes a CPU real-time usage threshold, a network traffic real-time rate threshold, a disk real-time usage threshold, and an edge computing node real-time maximum temperature threshold.

[0022] A preset real-time load check set of a cloud processor is extracted from a database, wherein the real-time load check set of the cloud processor includes a real-time network bandwidth usage rate threshold, a real-time disk read and write operation frequency threshold, a real-time network packet loss rate threshold, and a cloud processor remaining task quantity threshold.

[0023] As a preferred technical solution, the analysis and processing obtains the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, and the specific method includes:

[0024] ;

[0025] ;

[0026] in, For the The edge computing nodes run real-time load monitoring values, Real-time operation load monitoring value for cloud processors, For the Real-time CPU usage of edge computing nodes, For the Real-time rate of network traffic of edge computing nodes, For the Real-time disk usage of edge computing nodes, For the Real-time temperature of edge computing nodes, is the real-time usage rate of the cloud processor network bandwidth. The number of real-time read and write operations of the cloud processor disk. is the real-time packet loss rate of the cloud processor network, is the number of remaining tasks for the cloud processor, is the CPU real-time usage threshold. is the real-time rate threshold of network traffic, is the real-time disk usage threshold. is the real-time temperature threshold of the edge computing node, is the real-time usage threshold of network bandwidth. The frequency threshold of real-time disk read and write operations. is the network real-time packet loss rate threshold, is the threshold of the number of remaining tasks for the cloud processor, is the CPU real-time usage weight factor, is the real-time rate weight factor of network traffic, is the disk real-time usage weight factor, is the real-time temperature weight factor of the edge computing node, is the real-time utilization weight factor of network bandwidth, is the weight factor of the frequency of real-time disk read and write operations. is the network real-time packet loss rate weight factor, is the weight factor of the remaining tasks of the cloud processor, is the edge computing node number, , is the total number of edge computing nodes, is a natural constant.

[0027] As a preferred technical solution, the preliminary data allocation ratio of the real-time data stream is dynamically corrected to obtain a data re-correction ratio. The specific processing conditions are:

[0028] If the average value of the real-time load monitoring value of the edge computing node is greater than or equal to the real-time load monitoring value threshold of the edge computing node, the edge computing node is determined to be overloaded; if the average value of the real-time load monitoring value of the edge computing node is less than the real-time load monitoring value threshold of the edge computing node, the edge computing node is determined to be not overloaded; if the real-time load monitoring value of the cloud processor is greater than or equal to the real-time load monitoring value threshold of the cloud processor, the cloud processor is determined to be overloaded; if the real-time load monitoring value of the cloud processor is less than the real-time load monitoring value threshold of the cloud processor, the cloud processor is determined to be overloaded.

[0029] If the edge computing node is determined to be overloaded, and at the same time the cloud processor is determined to be not overloaded, a first data correction ratio is obtained by processing, and the first data correction ratio is defined as the data re-correction ratio.

[0030] If the cloud processor is determined to be overloaded and the edge computing node is determined to be not overloaded at the same time, a second data correction ratio is obtained by processing, and the second data correction ratio is defined as the data re-correction ratio.

[0031] If both the edge computing node and the cloud processor are overloaded, a system overload warning will be sent to the management terminal.

[0032] If both the edge computing node and the cloud processor are not overloaded, the preliminary data allocation ratio of the real-time data stream is used as the data re-correction ratio.

[0033] As a preferred technical solution, the first data correction ratio is specifically processed as follows:

[0034] The real-time operation load monitoring value of the edge computing node is subtracted from the real-time operation load monitoring value threshold of the edge computing node to obtain the first re-correction ratio parameter of the edge computing node. Based on the first re-correction ratio parameter, the node loss ratio value corresponding to each first re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the edge computing node, and imported into the preliminary data allocation ratio of the real-time data stream. The first data correction ratio is obtained by processing, and the first data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the first data re-correction ratio value.

[0035] The numerical form of the first data re-correction ratio is: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the proportion of sub-data streams allocated to cloud processors. It is the node loss ratio value of the edge computing node.

[0036] As a preferred technical solution, the second data correction ratio, the specific processing process is as follows:

[0037] The real-time operating load monitoring value of the cloud processor is subtracted from the real-time operating load monitoring value threshold of the cloud processor to obtain the second re-correction ratio parameter of the cloud processor. Based on the second re-correction ratio parameter, the node loss ratio value corresponding to each second re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the cloud processor, and imported into the preliminary data allocation ratio of the real-time data stream. The second data correction ratio is processed to obtain the second data correction ratio, and the second data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the second data re-correction ratio value.

[0038] The numerical form of the second data re-correction ratio is: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the proportion of sub-data streams allocated to cloud processors. It is the node loss ratio of the cloud processor.

[0039] In addition, an IOT big data real-time data stream analysis system is also provided, including:

[0040] The basic feature analysis module obtains the basic feature information of the real-time data stream from the edge channel built into the IOT, and analyzes and processes the basic feature information of the real-time data stream to obtain the basic feature check value of the real-time data stream.

[0041] The real-time data stream trial preprocessing module obtains the preliminary data allocation ratio of the real-time data stream based on the analysis and processing of the basic feature check values ​​of the real-time data stream, and splits the real-time data stream into several independent sub-data streams. According to the preliminary data allocation ratio, each sub-data stream is dispatched to the corresponding data processing location for trial preprocessing. The data processing location includes each edge computing node and cloud processor.

[0042] The real-time load monitoring module obtains the real-time load data of each edge computing node and cloud processor based on the monitoring agent pre-deployed at the data processing location, transmits the real-time load data of each edge computing node and cloud processor to the monitoring server, and analyzes and processes to obtain the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor.

[0043] The sub-data stream dynamic allocation module dynamically corrects the initial data allocation ratio of the real-time data stream based on the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, obtains the data re-correction ratio, and dynamically allocates each sub-data stream based on the data re-correction ratio.

[0044] In addition, an electronic device is also provided, including: the electronic device has one or more programs, and the one or more programs are executed by one or more processors to implement any method described in the above preferred technical solutions.

[0045] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0046] (1) The present invention provides a method for analyzing real-time data streams of IOT big data, which divides the real-time data stream into several independent sub-data streams, and then dispatches each sub-data stream to a corresponding data processing location for trial preprocessing according to the data allocation ratio. Based on the processing results obtained from the trial, the data stream is analyzed again to obtain the data re-correction ratio and then the actual sub-data stream allocation processing is performed, thereby improving the efficiency of data processing, ensuring the timeliness of real-time data processing, and meeting the application scenarios that require real-time decision-making.

[0047] (2) The present invention monitors the load of each edge computing node and cloud processor and dynamically corrects the data allocation ratio of the real-time data stream, thereby ensuring that the actually allocated sub-data streams can adapt to and match the corresponding processing locations, thereby simultaneously ensuring data security and hardware device security.

[0048] (3) The present invention analyzes the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor to obtain a re-correction ratio, thereby realizing dynamic scheduling and processing of each sub-data stream between each edge computing node and the cloud processor, thereby improving the speed of data processing and improving the flexibility and adaptability of the system. At the same time, after the secondary correction processing, it can further optimize the utilization of resources and improve the data processing quality.

[0049] Of course, any product implementing the present invention does not necessarily need to achieve all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0051] Figure 2 It is a line graph of the real-time operating load monitoring value of the edge computing node and the real-time operating load monitoring value of the cloud processor involved in the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inside", "all around" and the like indicating orientation or positional relationship are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0054] See also Figure 1 As shown, an embodiment of the present invention provides an IOT big data real-time data stream analysis method, including:

[0055] The basic characteristic information of the real-time data stream is obtained from the edge channel built into the IOT. Based on the basic characteristic information of the real-time data stream, the basic characteristic check value of the real-time data stream is obtained through analysis and processing.

[0056] The basic characteristic information of the real-time data stream is obtained from the edge channel built into the IOT, and the basic characteristic check value of the real-time data stream is obtained by analysis and processing based on the basic characteristic information of the real-time data stream. The specific process is as follows:

[0057] The IOT is the Internet of Things, which refers to a network that connects things through information carriers such as the Internet and traditional communication networks.

[0058] The basic characteristic information of the real-time data stream includes the number of data nesting levels, the number of data types, the number of data bytes and the number of data format types, and is obtained by parsing the log file of the real-time data stream captured by a log analyzer.

[0059] The number of data nesting levels refers to the depth of nesting in the data structure. In real-time data streams, the nesting level of data is crucial for correctly parsing and operating data. The number of data types refers to the types of different data types in the data stream. In real-time data stream processing, identifying and classifying different types of data helps to perform effective data processing and analysis. Data bytes refers to the size of data in the data stream. The number of data format types refers to the number of all formats that exist in the data.

[0060] Extract a preset basic feature information check set from the database, the basic feature information check set includes a data nesting level number threshold, a data type number threshold, a data byte threshold, and a data format type number threshold, analyze and process the basic feature information of the real-time data stream and the basic feature information check set, and obtain the basic feature check value of the real-time data stream:

[0061] ;

[0062] in, It is the basic characteristic check value of the real-time data stream. is the number of nested levels of data in the real-time data stream, is the number of data types in real-time data streams, is the data byte of the real-time data stream, The number of data format types for real-time data streams. is the threshold value of the number of nested levels of data, is the data type quantity threshold, is the data byte threshold, is the data format type quantity threshold, is the weight factor for the number of data nesting levels, is the weight factor of the number of data types, is the data byte weight factor, is the weight factor of the number of data format types, is a natural constant.

[0063] It should be noted that the value range of the data nesting level number weight factor, data type number weight factor, data byte weight factor and data format type number weight factor are all between 0 and 1 and meet the requirements. The data nesting level quantity weight factor is the impact factor corresponding to the basic feature check value of the real-time data stream preset in the database, which indicates the numerical value of the impact of the number of data nesting levels on the basic feature check value of the real-time data stream. The data type quantity weight factor is the impact factor corresponding to the basic feature check value of the real-time data stream preset in the database, which indicates the numerical value of the impact of the number of data types on the basic feature check value of the real-time data stream. The data byte weight factor is the impact factor corresponding to the basic feature check value of the real-time data stream preset in the database, which indicates the numerical value of the impact of the data byte on the basic feature check value of the real-time data stream. The data format type quantity weight factor is the impact factor corresponding to the basic feature check value of the real-time data stream preset in the database, which indicates the impact of the number of data format types on the basic feature check value of the real-time data stream. The numerical value of the degree can be used to directly obtain the data nesting level quantity weight factor, data type quantity weight factor, data byte weight factor and data format type quantity weight factor from the database when used, and the corresponding relationship is a pre-set mapping relationship. For example, the basic feature check value of the real-time data stream and the data nesting level quantity weight factor, data type quantity weight factor, data byte weight factor and data format type quantity weight factor corresponding to the data nesting level quantity, data type quantity, data byte weight factor and data format type quantity respectively form a mapping set, and the real-time data nesting level quantity, data type quantity, data byte and data format type quantity are input into the mapping set to obtain the data nesting level quantity weight factor, data type quantity weight factor, data byte weight factor and data format type quantity weight factor, wherein the mapping relationship is one-to-one correspondence.

[0064] It should also be noted that there is a certain correlation between the number of data nesting levels, the number of data types, the number of data bytes and the number of data format types. The more nesting levels there are, the more data types are involved, because each level usually contains different types of data, and multiple data types usually require more complex data structures to store, thereby increasing the nesting levels. More nesting levels usually lead to more complex data structures, thereby increasing the total byte size of the data. More data types usually also lead to more data format types, different data types are usually different data formats, and different data types usually occupy different storage spaces, so an increase in the number of data types usually increases the total byte size of the data. These parameters affect each other, and a change in one parameter usually leads to a change in other parameters. For example, an increase in the number of data nesting levels usually leads to an increase in the number of data types and an increase in data bytes. Therefore, this embodiment provides a comprehensive analysis of these parameters, which is more correlated than a single parameter.

[0065] The data nesting level number threshold, data type number threshold, data byte threshold and data format type number threshold are obtained by extracting from the database and used to comprehensively evaluate the difficulty of analyzing and processing the real-time data stream;

[0066] The data nesting level number threshold is processed as follows: In a specific embodiment, the data nesting level number threshold is set directly in the database during the development of the IOT big data real-time data stream analysis system involved in the embodiment of the present invention. There are many methods for setting the data nesting level number threshold, such as obtaining it through statistical analysis, including counting the number of data nesting levels of real-time data streams in a historically low processing difficulty state (for example, the processing running time of the real-time data stream in the system is less than or equal to 10 milliseconds), multiple statistical processing to obtain the corresponding number of data nesting levels, and averaging the multiple obtained numbers of data nesting levels to obtain the data nesting level number threshold, or experts judge the boundary between the number of multiple data nesting levels and the number of few data nesting levels through corresponding historical data to set the threshold, thereby adjusting the threshold. The acquisition method of other thresholds including the data type number threshold, the data byte threshold and the data format type number threshold is the same as the acquisition method of the data nesting level number threshold, which will not be repeated here.

[0067] The basic characteristic check value of the real-time data stream is used to characterize the difficulty of analyzing and processing the real-time data stream.

[0068] Based on the basic feature check value analysis and processing of the real-time data stream, the preliminary data allocation ratio of the real-time data stream is obtained, and the real-time data stream is split into several independent sub-data streams. According to the preliminary data allocation ratio, each sub-data stream is dispatched to the corresponding data processing location for trial preprocessing. The data processing location includes each edge computing node and cloud processor.

[0069] The basic characteristic check value analysis and processing based on the real-time data stream obtains the preliminary data distribution ratio of the real-time data stream, and splits the real-time data stream into several independent sub-data streams. The specific process is as follows:

[0070] Based on the basic feature check value of the real-time data stream, a mapping match is performed with the preliminary data allocation ratio corresponding to each basic feature check value interval stored in the database to obtain the preliminary data allocation ratio of the real-time data stream.

[0071] It should be noted that the basic feature check value of the real-time data stream is mapped and matched with the preliminary data allocation ratio corresponding to each basic feature check value interval stored in the database. The specific process is: based on the basic feature check value of the real-time data stream, the preliminary data allocation ratio corresponding to each basic feature check value interval stored in the database is mapped, and the preliminary data allocation ratio corresponding to the interval in which the basic feature check value of the real-time data stream is located is obtained by matching.

[0072] The splitting of the real-time data stream into several independent sub-data streams specifically includes: determining several independent fields in the real-time data stream for grouping, marking the several independent fields as key values, and performing grouping operations on the real-time data stream based on the key values ​​and the stream processing framework to split it into several independent sub-data streams.

[0073] The preliminary data allocation ratio of the real-time data stream obtained based on the basic feature check value of the real-time data stream is used to determine the number of sub-data streams allocated to each edge computing node and cloud processor. When the basic feature check value of the real-time data stream is high, it means that the real-time data stream is more difficult to process, and the focus is on allocating the sub-data streams to the cloud processor with stronger processing capabilities. When the basic feature check value of the real-time data stream is low, it means that the real-time data stream is easier to process, and the focus is on allocating the sub-data streams to the edge computing nodes with weaker processing capabilities.

[0074] The several independent fields used for grouping include device ID, geographic location information, event type and timestamp, etc. These fields are extracted in the data processing logic and set as the key of the message. These fields are specified as key values ​​in the configuration of the framework, and the grouping operation is configured according to the Apache Kafka stream processing framework.

[0075] It should be noted that by performing trial preprocessing on each sub-data stream, the processing efficiency is improved, the delay of data processing is reduced, and the fault tolerance of data processing is increased. If an edge computing node is overloaded, the sub-data stream after trial preprocessing can be quickly dispatched to other edge computing nodes or cloud processors.

[0076] Based on the monitoring agent pre-deployed at the data processing location, the trial real-time load data of each edge computing node and cloud processor is obtained, and the trial real-time load data of each edge computing node and cloud processor is transmitted to the monitoring server, and the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor are obtained through analysis and processing.

[0077] Real-time load data of each edge computing node and cloud processor, including:

[0078] The real-time load data of each edge computing node includes the real-time CPU usage, real-time network traffic rate, real-time disk usage and real-time temperature of each edge computing node.

[0079] CPU real-time usage reflects how busy the central processing unit (CPU) of the edge computing node is. High CPU usage indicates that the node is processing a large amount of data or running resource-intensive tasks.

[0080] The real-time rate of network traffic refers to the amount of data sent and received by the edge computing node per unit time. The network traffic rate can reflect the bandwidth usage and network congestion of the monitoring node.

[0081] The real-time disk usage indicates the usage of storage resources in edge computing nodes. High disk usage means that the storage space is close to the limit.

[0082] The real-time temperature of an edge computing node refers to the current temperature of the node hardware. Too high a temperature usually causes the node hardware to downgrade or be damaged. A high real-time temperature of an edge computing node also reflects that the current node load is too high.

[0083] The real-time load data of the cloud processor includes the real-time usage rate of network bandwidth, the real-time frequency of disk read and write operations, the real-time network packet loss rate, and the number of remaining tasks of the cloud processor.

[0084] The real-time network bandwidth usage shows the ratio of the network bandwidth used by the cloud processor to the total available bandwidth, reflecting the current network traffic status.

[0085] The real-time disk read and write operation frequency reflects the number of read and write operations on the cloud processor disk, usually measured in operations per second (IOPS). A high frequency of disk operations indicates that the disk is under heavy load.

[0086] The network real-time packet loss rate measures the rate at which data packets are lost during network transmission. A high packet loss rate indicates that there are network problems.

[0087] The number of remaining tasks of the cloud processor provides the current status of the processor. Too high a number of remaining tasks usually causes processor delays or freezes. Too high a number of remaining tasks of the cloud processor also reflects that the current processor load is too high.

[0088] A preset real-time load verification set of edge computing nodes is extracted from the database, wherein the real-time load verification set of the edge computing nodes includes a CPU real-time usage threshold, a network traffic real-time rate threshold, a disk real-time usage threshold, and an edge computing node real-time temperature threshold.

[0089] A preset real-time load check set of a cloud processor is extracted from a database, wherein the real-time load check set of the cloud processor includes a real-time network bandwidth usage rate threshold, a real-time disk read and write operation frequency threshold, a real-time network packet loss rate threshold, and a cloud processor remaining task quantity threshold.

[0090] The analysis and processing obtains the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, and the specific method includes:

[0091] ;

[0092] ;

[0093] in, For the The edge computing nodes run real-time load monitoring values, Real-time operation load monitoring value for cloud processors, For the Real-time CPU usage of edge computing nodes, For the Real-time rate of network traffic of edge computing nodes, For the Real-time disk usage of edge computing nodes, For the Real-time temperature of edge computing nodes, is the real-time usage rate of the cloud processor network bandwidth. The number of real-time read and write operations of the cloud processor disk. is the real-time packet loss rate of the cloud processor network, is the number of remaining tasks for the cloud processor, is the CPU real-time usage threshold. is the real-time rate threshold of network traffic, is the real-time disk usage threshold. is the real-time temperature threshold of the edge computing node, is the real-time usage threshold of network bandwidth. The frequency threshold of real-time disk read and write operations. is the network real-time packet loss rate threshold, is the threshold of the number of remaining tasks for the cloud processor, is the CPU real-time usage weight factor, is the real-time rate weight factor of network traffic, is the disk real-time usage weight factor, is the real-time temperature weight factor of the edge computing node, is the real-time utilization weight factor of network bandwidth, is the weight factor of the frequency of real-time disk read and write operations. is the network real-time packet loss rate weight factor, is the weight factor of the remaining tasks of the cloud processor, is the edge computing node number, , is the total number of edge computing nodes, is a natural constant.

[0094] It should be noted that the CPU real-time usage weight factor, network traffic real-time rate weight factor, disk real-time usage weight factor and edge computing node real-time temperature weight factor all have a value range between 0 and 1 and meet The CPU real-time usage weight factor is the impact factor corresponding to the real-time operation load monitoring value of the edge computing node preset in the database, which indicates the numerical value of the influence degree of the CPU real-time usage on the real-time operation load monitoring value of the edge computing node. The network traffic real-time rate weight factor is the impact factor corresponding to the real-time operation load monitoring value of the edge computing node preset in the database, which indicates the numerical value of the influence degree of the network traffic real-time rate on the real-time operation load monitoring value of the edge computing node. The disk real-time usage weight factor is the impact factor corresponding to the real-time operation load monitoring value of the edge computing node preset in the database, which indicates the numerical value of the influence degree of the disk real-time usage on the real-time operation load monitoring value of the edge computing node. The edge computing node real-time temperature weight factor is the impact factor corresponding to the real-time operation load monitoring value of the edge computing node preset in the database, which indicates the numerical value of the influence degree of the real-time temperature of the edge computing node on the real-time operation load monitoring value of the edge computing node. When used, the CPU real-time usage weight factor, network traffic real-time rate weight factor, disk real-time usage weight factor and edge computing node real-time temperature weight factor can be directly obtained from the database, and the corresponding relationship is a pre-set mapping relationship. For example, the real-time operation load monitoring value of the edge computing node is respectively corresponding to the CPU real-time usage rate, network traffic real-time rate, disk real-time usage rate and edge computing node real-time temperature, and the CPU real-time usage weight factor, network traffic real-time rate weight factor, disk real-time usage weight factor and edge computing node real-time temperature form a mapping set. The real-time CPU real-time usage rate, network traffic real-time rate, disk real-time usage and edge computing node real-time temperature are input into the mapping set to obtain the CPU real-time usage weight factor, network traffic real-time rate weight factor, disk real-time usage weight factor and edge computing node real-time temperature weight factor, and the mapping relationship is one-to-one correspondence.

[0095] In the edge computing environment, there is a certain correlation between the real-time CPU usage, the real-time rate of network traffic, the real-time disk usage, and the real-time temperature of the edge computing node. When the network traffic increases, the CPU is required to process more packets, which leads to an increase in CPU usage. Disk read and write operations require the CPU to participate in scheduling and processing, so an increase in disk usage usually leads to an increase in CPU usage. An increase in CPU usage leads to an increase in power consumption, which in turn leads to an increase in node temperature. The reception and transmission of network data usually involve disk read and write operations, such as caching or logging, so an increase in network traffic usually increases disk usage. The high-rate operation of the network interface usually increases power consumption, which affects the node temperature. Disk operations require the CPU to schedule and control, so an increase in disk usage usually leads to an increase in CPU usage. Network data usually needs to be written to disk, so an increase in network traffic usually increases disk usage. Frequent disk operations usually cause the disk controller and storage medium to heat up, which affects the node temperature.

[0096] As can be seen above, there is a certain correlation between these parameters. For example, increased network traffic usually leads to increased CPU and disk usage, both of which usually further lead to increased node temperature. Similarly, if the CPU is in a high utilization state for a long time, it will not only affect disk and network performance, but also usually cause node overheating.

[0097] The real-time load check set of the edge computing node includes a CPU real-time usage rate threshold, a network traffic real-time rate threshold, a disk real-time usage rate threshold, and an edge computing node real-time temperature threshold, which are extracted from a database and used to comprehensively evaluate the real-time operating load of the edge computing node;

[0098] The CPU real-time usage threshold is processed as follows: In a specific embodiment, the CPU real-time usage threshold is set directly in the database during the development of the IOT big data real-time data flow analysis system involved in the embodiment of the present invention. There are many methods for setting the CPU real-time usage threshold, such as obtaining it through statistical analysis, including statistically analyzing the CPU real-time usage of edge computing nodes under historical low-load conditions (for example, the memory occupancy rate of the CPU of the edge computing node is below 60%), multiple statistical processing to obtain the corresponding CPU real-time usage, and averaging the multiple CPU real-time usage to obtain the CPU real-time usage threshold, or experts judge the boundary between high load and low load through corresponding historical data to set, thereby adjusting the threshold. The acquisition method of other thresholds including the network traffic real-time rate threshold, the disk real-time usage threshold and the edge computing node real-time temperature threshold is the same as the acquisition method of the data nesting level number threshold, which will not be repeated here.

[0099] It should also be noted that the network bandwidth real-time usage weight factor, disk read and write real-time operation frequency weight factor, network real-time packet loss rate weight factor and cloud processor remaining task number weight factor all have a value range between 0 and 1 and meet The network bandwidth real-time usage weight factor is the impact factor corresponding to the cloud processor real-time operation load monitoring value preset in the database, which indicates the numerical value of the impact of the network bandwidth real-time usage on the cloud processor real-time operation load monitoring value. The disk read and write real-time operation frequency weight factor is the impact factor corresponding to the cloud processor real-time operation load monitoring value preset in the database, which indicates the numerical value of the impact of the disk read and write real-time operation frequency on the cloud processor real-time operation load monitoring value. The network real-time packet loss rate weight factor is the impact factor corresponding to the cloud processor real-time operation load monitoring value preset in the database, which indicates the numerical value of the impact of the network real-time packet loss rate on the cloud processor real-time operation load monitoring value. The cloud processor remaining task number weight factor is the impact factor corresponding to the cloud processor real-time operation load monitoring value preset in the database, which indicates the numerical value of the impact of the cloud processor remaining task number on the cloud processor real-time operation load monitoring value. When used, it can be directly obtained from The network bandwidth real-time usage weight factor, disk read and write real-time operation frequency weight factor, network real-time packet loss rate weight factor and cloud processor remaining task quantity weight factor are obtained from the database, and the corresponding relationship is a pre-set mapping relationship. For example, the cloud processor real-time operation load monitoring value and the network bandwidth real-time usage weight factor, disk read and write real-time operation frequency weight factor, network real-time packet loss rate weight factor and cloud processor remaining task quantity weight factor corresponding to the network bandwidth real-time usage rate, disk read and write real-time operation frequency, network real-time packet loss rate and cloud processor remaining task quantity form a mapping set, and the real-time network bandwidth real-time usage rate, disk read and write real-time operation frequency, network real-time packet loss rate and cloud processor remaining task quantity are input into the mapping set to obtain the network bandwidth real-time usage weight factor, disk read and write real-time operation frequency weight factor, network real-time packet loss rate weight factor and cloud processor remaining task quantity weight factor, wherein the mapping relationship is one-to-one correspondence.

[0100] There is a certain correlation between the real-time utilization rate of network bandwidth, the real-time operation frequency of disk read and write, the real-time network packet loss rate, and the number of remaining tasks of the cloud processor. If the network traffic is high, it is usually necessary to frequently write data to the disk or read data from the disk, thereby increasing the frequency of disk read and write. When the network bandwidth approaches or reaches the limit, it usually causes network congestion, which in turn increases the probability of data packet loss. High network bandwidth utilization usually causes the processor to process more network data, thereby increasing the workload of the processor. The increase in disk operation frequency usually causes network data processing delays, which indirectly affects the network packet loss rate. When the network bandwidth approaches or reaches the limit, the packet loss rate usually increases. Network packet loss usually leads to data integrity problems, which in turn triggers the retransmission mechanism and indirectly increases the disk operation frequency. Handling packet loss events usually requires additional processor resources.

[0101] The network bandwidth real-time usage rate threshold of the cloud processor, the disk read and write real-time operation frequency threshold, the network real-time packet loss rate threshold and the cloud processor remaining task number threshold are extracted from the database and used to comprehensively evaluate the real-time operation load of the cloud processor;

[0102] The processing method of the network bandwidth real-time usage rate threshold is as follows: In a specific embodiment, the network bandwidth real-time usage rate threshold is set directly in the database during the development of the IOT big data real-time data flow analysis system involved in the embodiment of the present invention. There are many methods for setting the network bandwidth real-time usage rate threshold, such as obtaining it through statistical analysis, including statistically analyzing the network bandwidth real-time usage rate of the cloud processor under historical low load conditions (for example, the memory occupancy rate of the CPU of the cloud processor is below 60%), multiple statistical processing to obtain the corresponding network bandwidth real-time usage rate, and averaging the network bandwidth real-time usage rate obtained multiple times to obtain the network bandwidth real-time usage rate threshold, or experts judge the boundary between high load and low load through corresponding historical data to set it, thereby adjusting the threshold. The acquisition method of other thresholds including the disk read and write real-time operation frequency threshold, the network real-time packet loss rate threshold and the cloud processor remaining task number threshold is the same as the acquisition method of the data nesting level number threshold, which will not be repeated here.

[0103] like Figure 2What is shown is a line graph of the real-time operating load monitoring value of the edge computing node and the real-time operating load monitoring value of the cloud processor involved in an embodiment of the present invention. The example graph shows the real-time operating load monitoring value of a certain edge computing node and the real-time operating load monitoring value of the cloud processor at 10 different times. In this embodiment, the horizontal axis is the time step number, and the vertical axis is the real-time operating load monitoring value. By simultaneously processing the real-time CPU utilization rate, network traffic real-time rate, disk real-time utilization rate and real-time temperature of a certain edge computing node, it is ensured that the real-time operating load monitoring value of the edge computing node fully represents the load condition of the edge computing node. Similarly, by simultaneously processing the real-time network bandwidth utilization rate, disk read and write real-time operation frequency, network real-time packet loss rate and the number of remaining tasks of the cloud processor, it is ensured that the real-time operating load monitoring value of the cloud processor fully represents the load condition of the cloud processor.

[0104] Based on the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor, the initial data allocation ratio of the real-time data flow is dynamically corrected to obtain the data re-correction ratio, and each sub-data flow is dynamically allocated based on the data re-correction ratio.

[0105] The preliminary data allocation ratio of the real-time data stream is dynamically corrected to obtain a data re-correction ratio. The specific processing conditions are:

[0106] If the average value of the real-time load monitoring value of the edge computing node is greater than or equal to the real-time load monitoring value threshold of the edge computing node, the edge computing node is determined to be overloaded; if the average value of the real-time load monitoring value of the edge computing node is less than the real-time load monitoring value threshold of the edge computing node, the edge computing node is determined to be not overloaded; if the real-time load monitoring value of the cloud processor is greater than or equal to the real-time load monitoring value threshold of the cloud processor, the cloud processor is determined to be overloaded; if the real-time load monitoring value of the cloud processor is less than the real-time load monitoring value threshold of the cloud processor, the cloud processor is determined to be overloaded.

[0107] The edge computing node real-time operation load monitoring value threshold is obtained by extracting from a database and is used to determine whether an edge computing node is overloaded in real-time operation;

[0108] In a specific embodiment, the edge computing node real-time operation load monitoring value threshold is set directly in the database during the development of the IOT big data real-time data flow analysis system involved in the embodiment of the present invention. There are many methods for setting the edge computing node real-time operation load monitoring value threshold, such as obtaining it through statistical analysis, including statistics of real-time load data of each edge computing node under a historical low load state (for example, the state where the memory occupancy rate of the CPU of the edge computing node is below 60%), including the historical CPU real-time utilization rate of each edge computing node, the historical network traffic real-time rate, the historical disk real-time utilization rate and the historical edge computing node real-time temperature, multiple statistical processing to obtain the corresponding edge computing node real-time operation load monitoring value, and the edge computing node real-time operation load monitoring value obtained multiple times is averaged to obtain the edge computing node real-time operation load monitoring value threshold, or experts judge the boundary between overload and overload through the corresponding historical data to set it, thereby adjusting the threshold.

[0109] The cloud processor real-time operation load monitoring value threshold is obtained by extracting from a database and is used to determine whether the cloud processor is overloaded in real-time operation;

[0110] In a specific embodiment, the cloud processor real-time operation load monitoring value threshold is set directly in the database during the development of the IOT big data real-time data flow analysis system involved in the embodiment of the present invention. There are many methods for setting the cloud processor real-time operation load monitoring value threshold, such as through statistical analysis, including statistics of the cloud processor's real-time load data under historical low load conditions (for example, the cloud processor's CPU memory occupancy rate is below 60%), including the cloud processor's historical network bandwidth real-time usage rate, historical disk read and write real-time operation frequency, historical network real-time packet loss rate and historical cloud processor remaining task number, multiple statistical processing to obtain the corresponding cloud processor real-time operation load monitoring value, and the cloud processor real-time operation load monitoring value obtained multiple times is averaged to obtain the cloud processor real-time operation load monitoring value threshold, or experts judge the boundary between overload and overload through the corresponding historical data to set, thereby adjusting the threshold.

[0111] The mean of the real-time operating load monitoring values ​​of the edge computing nodes is used to characterize the average load status of each edge computing node, that is, the overall load status of all edge computing nodes. The overall load status provides a global perspective, which facilitates understanding the status of the entire system, rather than just a local view.

[0112] If the edge computing node is determined to be overloaded, and at the same time the cloud processor is determined to be not overloaded, a first data correction ratio is obtained by processing, and the first data correction ratio is defined as the data re-correction ratio.

[0113] If the cloud processor is determined to be overloaded and the edge computing node is determined to be not overloaded at the same time, a second data correction ratio is obtained by processing, and the second data correction ratio is defined as the data re-correction ratio.

[0114] If both the edge computing node and the cloud processor are overloaded, the word "overload" will be sent to the management terminal and a system overload warning will be issued to the system maintenance personnel. The system maintenance personnel will close the edge channel or adjust the network bandwidth based on the real-time operating load monitoring values ​​of each edge computing node and the cloud processor to solve the overload problem.

[0115] If both the edge computing node and the cloud processor are not overloaded, the preliminary data allocation ratio of the real-time data stream is used as the data re-correction ratio.

[0116] The specific processing process of the first data correction ratio is as follows:

[0117] The real-time operation load monitoring value of the edge computing node is subtracted from the real-time operation load monitoring value threshold of the edge computing node to obtain the first re-correction ratio parameter of the edge computing node. Based on the first re-correction ratio parameter, the node loss ratio value corresponding to each first re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the edge computing node, and imported into the initial data allocation ratio of the real-time data stream. The first data correction ratio is obtained by processing, and the first data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the first data re-correction ratio value.

[0118] The numerical form of the first data re-correction ratio is: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the proportion of sub-data streams allocated to cloud processors. It is the node loss ratio value of the edge computing node.

[0119] The specific processing process of the second data correction ratio is as follows:

[0120] The real-time operating load monitoring value of the cloud processor is subtracted from the real-time operating load monitoring value threshold of the cloud processor to obtain the second re-correction ratio parameter of the cloud processor. Based on the second re-correction ratio parameter, the node loss ratio value corresponding to each second re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the cloud processor, and imported into the initial data allocation ratio of the real-time data stream. The second data correction ratio is obtained by processing, and the second data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the second data re-correction ratio value.

[0121] The numerical form of the second data re-correction ratio is: , Where, represents the ratio of sub-data streams allocated to edge computing nodes, Indicates the proportion of sub-data streams allocated to cloud processors. It is the node loss ratio of the cloud processor.

[0122] The sub-data streams are dynamically allocated. In this embodiment, the cloud processor is given priority in allocation. That is, when the number of sub-data streams cannot be allocated as an integer according to the data correction ratio, the number of sub-data streams allocated to the cloud processor is rounded up to obtain the complete number of sub-data streams.

[0123] In addition, an embodiment of the present invention further provides an IOT big data real-time data stream analysis system, specifically comprising:

[0124] The basic feature analysis module obtains the basic feature information of the real-time data stream from the edge channel built into the IOT, and analyzes and processes the basic feature information of the real-time data stream to obtain the basic feature check value of the real-time data stream.

[0125] The real-time data stream trial preprocessing module obtains the preliminary data allocation ratio of the real-time data stream based on the analysis and processing of the basic feature check values ​​of the real-time data stream, and splits the real-time data stream into several independent sub-data streams. According to the preliminary data allocation ratio, each sub-data stream is dispatched to the corresponding data processing location for trial preprocessing. The data processing location includes each edge computing node and cloud processor.

[0126] The real-time load monitoring module obtains the real-time load data of each edge computing node and cloud processor based on the monitoring agent pre-deployed at the data processing location, transmits the real-time load data of each edge computing node and cloud processor to the monitoring server, and analyzes and processes to obtain the real-time operating load monitoring value of each edge computing node and the real-time operating load monitoring value of the cloud processor.

[0127] The sub-data stream dynamic allocation module dynamically corrects the initial data allocation ratio of the real-time data stream based on the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, obtains the data re-correction ratio, and dynamically allocates each sub-data stream based on the data re-correction ratio.

[0128] An embodiment of the present invention further provides an electronic device, wherein the electronic device has one or more programs, and the one or more programs are executed by one or more processors to implement the method as described in any one of the above contents.

[0129] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0130] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that technicians in the relevant technical field can understand and use the present invention well. As long as they do not deviate from the structure of the present invention or exceed the scope defined by the present invention, they should all belong to the protection scope of the present invention.

Claims

1. A method for analyzing IOT big data real-time data stream, characterized in that: include: Obtain basic feature information of real-time data streams from the edge channel built into IOT, and analyze and process the basic feature information of real-time data streams to obtain basic feature check values ​​of real-time data streams; Based on the basic feature check value analysis and processing of the real-time data stream, a preliminary data allocation ratio of the real-time data stream is obtained, and the real-time data stream is split into a number of independent sub-data streams. According to the preliminary data allocation ratio, each sub-data stream is dispatched to a corresponding data processing location for trial preprocessing, and the data processing location includes each edge computing node and a cloud processor; Based on the monitoring agent pre-deployed at the data processing location, the trial real-time load data of each edge computing node and the cloud processor is obtained, and the trial real-time load data of each edge computing node and the cloud processor is transmitted to the monitoring server, and the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor is obtained through analysis and processing; Based on the real-time running load monitoring value of each edge computing node and the real-time running load monitoring value of the cloud processor, the preliminary data allocation ratio of the real-time data stream is dynamically corrected to obtain the data re-correction ratio, and each sub-data stream is dynamically allocated based on the data re-correction ratio; The preliminary data allocation ratio of the real-time data stream is dynamically corrected to obtain a data re-correction ratio. The specific processing conditions are: If the average value of the real-time running load monitoring value of the edge computing node is greater than or equal to the real-time running load monitoring value threshold of the edge computing node, the edge computing node is determined to be overloaded; if the average value of the real-time running load monitoring value of the edge computing node is less than the real-time running load monitoring value threshold of the edge computing node, the edge computing node is determined to be not overloaded; if the real-time running load monitoring value of the cloud processor is greater than or equal to the real-time running load monitoring value threshold of the cloud processor, the cloud processor is determined to be overloaded; if the real-time running load monitoring value of the cloud processor is less than the real-time running load monitoring value threshold of the cloud processor, the cloud processor is determined to be not overloaded; If the edge computing node is determined to be overloaded, and at the same time the cloud processor is determined to be not overloaded, a first data correction ratio is obtained by processing, and the first data correction ratio is defined as a data re-correction ratio; If the cloud processor is determined to be overloaded, and at the same time the edge computing node is determined to be not overloaded, a second data correction ratio is obtained by processing, and the second data correction ratio is defined as a data re-correction ratio; If both the edge computing node and the cloud processor are overloaded, a system overload warning will be sent to the management terminal; If both the edge computing node and the cloud processor are not overloaded, the preliminary data allocation ratio of the real-time data stream is used as the data re-correction ratio; The specific processing process of the first data correction ratio is as follows: The real-time operation load monitoring value of the edge computing node is subtracted from the real-time operation load monitoring value threshold of the edge computing node to obtain the first re-correction ratio parameter of the edge computing node, and based on the first re-correction ratio parameter, the node loss ratio value corresponding to each first re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the edge computing node, and imported into the preliminary data allocation ratio of the real-time data stream, and processed to obtain the first data correction ratio, and the first data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the first data re-correction ratio value; The numerical form of the first data re-correction ratio is: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the proportion of sub-data streams allocated to cloud processors. is the node loss ratio value of the edge computing node; The specific processing process of the second data correction ratio is as follows: The cloud processor real-time operation load monitoring value is subtracted from the cloud processor real-time operation load monitoring value threshold to obtain a second re-correction ratio parameter of the cloud processor, and based on the second re-correction ratio parameter, the node loss ratio value corresponding to each second re-correction ratio parameter interval stored in the database is mapped and matched to obtain the node loss ratio value of the cloud processor, and imported into the preliminary data allocation ratio of the real-time data stream, and processed to obtain the second data correction ratio, and the second data correction ratio is defined as the data re-correction ratio, and each sub-data stream is dynamically allocated according to the second data re-correction ratio value; The numerical form of the second data re-correction ratio is: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the proportion of sub-data streams allocated to cloud processors. It is the node loss ratio of the cloud processor.

2. According to claim 1, a method for analyzing IOT big data real-time data stream, characterized in that: The basic characteristic information of the real-time data stream is obtained from the edge channel built into the IOT, and the basic characteristic check value of the real-time data stream is obtained by analyzing and processing based on the basic characteristic information of the real-time data stream. The specific process is as follows: The basic characteristic information of the real-time data stream includes the number of data nesting levels, the number of data types, the number of data bytes and the number of data format types; Extracting a preset basic feature information check set from a database, the basic feature information check set includes a data nesting level number threshold, a data type number threshold, a data byte threshold, and a data format type number threshold, analyzing and processing the basic feature information of the real-time data stream and the basic feature information check set to obtain a basic feature check value of the real-time data stream; The basic characteristic check value of the real-time data stream is used to characterize the difficulty of analyzing and processing the real-time data stream.

3. According to claim 2, a method for analyzing IOT big data real-time data stream, characterized in that: The basic characteristic check value analysis and processing based on the real-time data stream obtains the preliminary data distribution ratio of the real-time data stream, and splits the real-time data stream into several independent sub-data streams. The specific process is as follows: Based on the basic feature check value of the real-time data stream, the preliminary data allocation ratio corresponding to each basic feature check value interval stored in the database is mapped and matched to obtain the preliminary data allocation ratio of the real-time data stream. The preliminary data allocation ratio of the real-time data stream is in the form of a numerical value: ,in Indicates the proportion of sub-data flows allocated to edge computing nodes. Indicates the ratio of sub-data streams allocated to cloud processors; The splitting of the real-time data stream into several independent sub-data streams specifically includes: determining several independent fields in the real-time data stream for grouping, marking the several independent fields as key values, and performing grouping operations on the real-time data stream based on the key values ​​and the stream processing framework to split it into several independent sub-data streams.

4. According to claim 1, a method for analyzing IOT big data real-time data stream, characterized in that: Real-time load data of each edge computing node and cloud processor, including: The real-time load data of each edge computing node includes the real-time CPU usage rate, network traffic real-time rate, disk real-time usage rate and edge computing node real-time temperature of each edge computing node; The real-time load data of the cloud processor includes the real-time usage rate of network bandwidth, the real-time frequency of disk read and write operations, the real-time packet loss rate of the network, and the number of remaining tasks of the cloud processor; Extracting a preset real-time load check set of edge computing nodes from a database, wherein the real-time load check set of the edge computing nodes includes a CPU real-time usage rate threshold, a network traffic real-time rate threshold, a disk real-time usage rate threshold, and an edge computing node real-time maximum temperature threshold; A preset real-time load check set of a cloud processor is extracted from a database, wherein the real-time load check set of the cloud processor includes a real-time network bandwidth usage rate threshold, a real-time disk read and write operation frequency threshold, a real-time network packet loss rate threshold, and a cloud processor remaining task quantity threshold.

5. According to claim 4, a method for analyzing IOT big data real-time data stream, characterized in that: The analysis and processing obtains the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, and the specific method includes: ; ; in, For the The edge computing nodes run real-time load monitoring values, Real-time operation load monitoring value for cloud processors, For the Real-time CPU usage of edge computing nodes, For the Real-time rate of network traffic of edge computing nodes, For the Real-time disk usage of edge computing nodes, For the Real-time temperature of edge computing nodes, is the real-time usage rate of the cloud processor network bandwidth. The number of real-time read and write operations of the cloud processor disk. is the real-time packet loss rate of the cloud processor network, is the number of remaining tasks for the cloud processor, is the CPU real-time usage threshold. is the real-time rate threshold of network traffic, is the real-time disk usage threshold. is the real-time temperature threshold of the edge computing node, is the real-time usage threshold of network bandwidth. The frequency threshold of real-time disk read and write operations. is the network real-time packet loss rate threshold, is the threshold of the number of remaining tasks for the cloud processor, is the CPU real-time usage weight factor, is the real-time rate weight factor of network traffic, is the disk real-time usage weight factor, is the real-time temperature weight factor of the edge computing node, is the real-time utilization weight factor of network bandwidth, is the weight factor of the frequency of real-time disk read and write operations. is the network real-time packet loss rate weight factor, is the weight factor of the remaining tasks of the cloud processor, is the edge computing node number, , is the total number of edge computing nodes, is a natural constant.

6. A system using the IOT big data real-time data stream analysis method as described in any one of claims 1 to 5, characterized in that: include: The basic feature analysis module obtains the basic feature information of the real-time data stream from the edge channel built into the IOT, and analyzes and processes the basic feature information of the real-time data stream to obtain the basic feature check value of the real-time data stream; A real-time data stream trial preprocessing module obtains a preliminary data allocation ratio of the real-time data stream based on the analysis and processing of the basic feature check values ​​of the real-time data stream, splits the real-time data stream into several independent sub-data streams, and dispatches each sub-data stream to a corresponding data processing location for trial preprocessing according to the preliminary data allocation ratio. The data processing location includes each edge computing node and a cloud processor; The real-time load monitoring module obtains the real-time load data of each edge computing node and cloud processor based on the monitoring agent pre-deployed at the data processing location, transmits the real-time load data of each edge computing node and cloud processor to the monitoring server, and analyzes and processes to obtain the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor; The sub-data stream dynamic allocation module dynamically corrects the initial data allocation ratio of the real-time data stream based on the real-time operation load monitoring value of each edge computing node and the real-time operation load monitoring value of the cloud processor, obtains the data re-correction ratio, and dynamically allocates each sub-data stream based on the data re-correction ratio.

7. An electronic device, characterized in that: include: The electronic device has one or more programs, and the one or more programs are executed by one or more processors to implement the method according to any one of claims 1 to 5.

Images