A Cloud-Edge Collaborative Adaptive Data Transmission Management Method and System Based on a Fence Mechanism

Abstract

The present invention discloses a cloud-edge collaborative adaptive data transmission management method and system based on a fence mechanism. The IoT management platform receives the service data characteristics sent by IoT terminals, generates corresponding data fence insertion strategies for the IoT terminals according to the service data characteristics, and the IoT terminals insert data fences into the data stream according to the data fence insertion strategies and output the data stream. The IoT management platform obtains the data stream, captures the data fences in the data stream, for any two adjacent data fences, counts the amount of data between the adjacent data fences, and parses the information carried by the data fences according to the corresponding data fence insertion strategies to obtain the theoretical value of the amount of data between the adjacent data fences, compares it with the counted amount of data, determines whether there is data loss, and if so, generates corresponding data transmission fault records. The present invention can achieve high-efficiency and low-occupation power big data statistics through the message fence mechanism, realize the accurate positioning of service data, and improve the efficiency of data stream analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of power Internet of Things data management, in particular to a cloud-edge collaborative adaptive data transmission management method and system based on a fence mechanism. Background Art

[0002] The power Internet of Things is the application of the Internet of Things in the smart grid. By integrating aspects such as the perception, interconnection, and control of grid infrastructure, operation and maintenance, and equipment, it improves the informatization level of the power system, improves the utilization efficiency of the existing power system infrastructure, and provides important technical support for power generation, transmission, transformation, distribution, and power consumption in the power grid.

[0003] Currently, the amount of data in the power Internet of Things is extremely large. For example, the number of edge devices connected to the Jiangsu provincial power Internet of Things platform exceeds three million, and all devices interact with the business master station at a certain frequency. In the power field, the usage scenarios of various collection devices, control devices, etc. are complex and diverse, such as analog data, digital signal quantities, streaming media data, upstream and downstream control instructions, etc.; the dimensions in which the data is located are also relatively complex, including regions, specialties, data types, etc.

[0004] Data needs to pass through a complex network structure and multiple gateways from the terminal device to the business master station. Any abnormality in the network status during this period may lead to data loss; many terminal devices are deployed in locations far from provincial and municipal companies, and even in some remote areas. Device data needs to be transmitted over a long distance and through multiple relays to reach the business system. In this process, data loss is very likely to occur due to signal attenuation, etc.

[0005] Most of the data in the cloud-edge scenario is transmitted in a streaming manner, that is, the data arrives in real time. It is impossible to obtain all the data at once, and there is no absolute start time and end time. Usually, it is necessary to batch-process the data or perform statistical analysis in the form of a sliding window (as Figure 1 shown). However, slicing the data or adding a sliding window will bring additional workload and consume more resources of the server where the program is located.

[0006] And the current statistical methods are basically to count the data in a certain period (window interval). The count method needs to traverse all the data within the statistical interval. This method usually occupies a large amount of system memory and takes a long time.

[0007] In summary, in the existing streaming processing frameworks, for the power Internet of Things scenario, there are several common problems, including time window segmentation problems, multi-stream merging problems, and data consistency problems. Solving these problems requires customized design based on the data transmission scenarios and characteristics, increasing the extra workload and potential risks. How to accurately and efficiently count the number of messages and minimize the intrusion into the core business flow has become an urgent problem to be solved in the power Internet of Things scenario. Summary of the Invention

[0008] The purpose of the present invention is to provide a cloud-edge collaborative adaptive data transmission management method and system based on a fence mechanism, which realizes high-efficiency and low-occupation power big data statistics through a message fence mechanism, realizes the accurate positioning of service data, and improves the efficiency of data flow analysis. The technical solutions adopted by the present invention are as follows.

[0009] On the one hand, the present invention provides a cloud-edge collaborative adaptive data transmission management method based on a fence mechanism, which is executed by an Internet of Things management platform. The method includes:

[0010] Receiving the service data characteristics sent by the Internet of Things terminal, generating a corresponding data fence insertion strategy for the Internet of Things terminal according to the service data characteristics, and returning it to the corresponding Internet of Things terminal; wherein, the data fence insertion strategy includes the amount of data between adjacent data fences;

[0011] Obtaining the data stream in which the Internet of Things terminal inserts data fences and transmits according to the data fence insertion strategy; the information carried by the data fences inserted into the data stream includes the reference start time of fence insertion and the order information;

[0012] Capturing the data fences in the data stream, for any two adjacent data fences, counting the amount of data between the adjacent data fences, and parsing the information carried by the data fences according to the corresponding data fence insertion strategy of the Internet of Things terminal to obtain the theoretical value of the amount of data between the adjacent data fences; comparing the counted amount of data with the theoretical value of the amount of data, and judging whether there is data loss. If there is data loss, generating a corresponding data transmission failure record.

[0013] Optionally, the method further includes: counting the number of data fences per unit time in the data stream sent by the Internet of Things terminal within a set time interval, and adjusting the data fence insertion strategy of the corresponding Internet of Things terminal according to the statistical result, so that the amount of data between adjacent data fences increases or decreases; and sending the adjusted data fence insertion strategy to the corresponding Internet of Things terminal;

[0014] The number of data fences per unit time in the data stream sent by the Internet of Things terminal within the set time interval, the formula is:

[0015]

[0016] In the formula, S is the statistical frequency, representing the number of data fences appearing per unit time within the statistical time period T, and △t a represents the amount of data between adjacent data fences, and G n represents the basic interval determined according to the characteristics of the service data, and L represents the importance coefficient of the service data.

[0017] Optionally, adjusting the data fence insertion strategy of the corresponding Internet of Things terminal according to the statistical result to increase or decrease the amount of data between adjacent data fences includes:

[0018] Comparing the statistical frequency S with the set statistical frequency threshold S0: If S < S0, and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then reduce the amount of data between adjacent data fences; if S > S0, and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then increase the amount of data between adjacent data fences; in other cases, keep the amount of data between the data fences unchanged.

[0019] Optionally, generating the data fence insertion strategy of the corresponding Internet of Things terminal according to the characteristics of the service data includes:

[0020] Determining the basic interval G according to the upload frequency of the service data n : Matching the upload frequency of the service data with multiple preset frequency intervals, and using the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy of the Internet of Things terminal;

[0021] Obtaining the importance information L of the service data;

[0022] According to the basic interval G n and the importance information L of the service data, determining the fence interval g of the data fence insertion strategy of the Internet of Things terminal, and the formula is: g = G n ×L.

[0023] Optionally, the multiple preset frequency intervals include a high upload frequency interval, a medium - high upload frequency interval, and a low upload frequency interval, and the basic intervals G h 、G m 、G l representing the amount of data between adjacent data fences are respectively preset for the three frequency intervals, and G h > G m > G l . That is, the higher the upload frequency, the more the amount of data between the data fences.

[0024] Optionally, compare the statistically obtained data volume with the theoretical data volume value to determine whether there is data loss. If there is data loss, generate a corresponding data transmission fault record, including:

[0025] Compare the statistically obtained data volume with the theoretical data volume value. If the two are the same, there is no data loss, and discard the judgment result. If the two are different, there is data loss. Generate a data transmission fault record for the data stream with data loss according to the reference start time and sequence information carried by the data fence. The data transmission fault record at least includes the IoT device ID, data stream time period information, and lost data volume information.

[0026] In a second aspect, the present invention provides a cloud-edge collaborative adaptive data transmission management system based on a fence mechanism, including an IoT management platform and IoT terminals;

[0027] The IoT terminals send service data characteristics to the IoT management platform; among them, the service data characteristics include the upload frequency of the service data.

[0028] The IoT management platform generates a data fence insertion strategy for the corresponding IoT terminals according to the received service data characteristics and returns it to the corresponding IoT terminals; among them, the data fence insertion strategy includes the data volume between adjacent data fences.

[0029] When the IoT terminals transmit data outward, insert data fences into the data to be sent according to the data fence insertion strategy; among them, the information carried by the data fence includes the reference start time and sequence information of the fence insertion.

[0030] The IoT management platform obtains the data stream transmitted by the IoT terminals, captures the data fences in the data stream, counts the data volume between any two adjacent data fences, and analyzes the information carried by the data fences according to the data fence insertion strategy corresponding to the IoT terminals to obtain the theoretical data volume value between adjacent data fences; compare the statistically obtained data volume with the theoretical data volume value to determine whether there is data loss. If there is data loss, generate a corresponding data transmission fault record.

[0031] Optionally, the IoT management platform is further configured to count the number of data fences per unit time in the data stream sent by the IoT terminals within a set time interval, and adjust the data fence insertion strategy of the corresponding IoT terminals according to the statistical result, so that the data volume between adjacent data fences increases or decreases;

[0032] And send the adjusted data fence insertion strategy to the corresponding IoT terminals.

[0033] Optionally, the number of data fences per unit time in the data stream sent by the IoT terminal within the statistical setting time interval is calculated by the formula:

[0034]

[0035] In the formula, S is the statistical frequency, representing the number of data fences per unit time within the statistical time period T, and △t a represents the data volume between adjacent data fences, and G n represents the basic interval determined according to the characteristics of business data, and L represents the importance coefficient of business data;

[0036] Adjust the data fence insertion strategy of the corresponding IoT terminal according to the statistical results to increase or decrease the data volume between adjacent data fences, including:

[0037] Compare the statistical frequency S with the set statistical frequency threshold S0: If S < S0, and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then reduce the data volume between adjacent data fences; If S > S0, and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then increase the data volume between adjacent data fences; In other cases, the data volume between data fences remains unchanged.

[0038] Optionally, the IoT management platform generates a data fence insertion strategy for the corresponding IoT terminal according to the received business data characteristics, including:

[0039] Determine the basic interval G according to the upload frequency of business data n : Match the upload frequency of business data with multiple preset frequency intervals, and use the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy for this IoT terminal;

[0040] Obtain the importance information L of business data;

[0041] According to the basic interval G n and the importance information L of business data, determine the fence interval g of the data fence insertion strategy for the IoT terminal, and the formula is: g = G n ×L.

[0042] Optionally, the multiple preset frequency intervals include a high upload frequency interval, a medium-high upload frequency interval, and a low upload frequency interval, and the basic intervals G h 、G m 、G l representing the data volume between adjacent data fences are preset for the three frequency intervals respectively, and G h > G m > G l .

[0043] As an implementation manner of obtaining the importance information of service data, the IoT management platform prestores the mapping relationship between service data types and importance levels.

[0044] The service data features sent by the IoT terminal to the IoT management platform further include service data type information, and the IoT management platform matches the corresponding importance level according to the mapping relationship from the service data type information.

[0045] In this implementation manner, the IoT management platform can pre-divide the importance levels of service data types for each IoT terminal under its management in advance, or make a prior agreement with the IoT terminal.

[0046] As another implementation manner of obtaining the importance information of service data, the importance information of service data includes an importance coefficient, which is pre-agreed by the IoT terminal and the IoT management platform; the service data features sent by the IoT terminal to the IoT management platform further include the importance coefficient of the service data. In this implementation manner, the IoT management platform can directly obtain the importance coefficient for calculating the fence interval.

[0047] Beneficial effects

[0048] The cloud-edge collaborative adaptive data transmission management method and system based on the fence mechanism of the present invention, on the one hand, realizes the adaptability of the data exhibition insertion strategy to service data features by generating a data fence insertion strategy in advance according to service data features. When the IoT terminal transmits data, after inserting data fences into the data stream according to the data fence insertion strategy, the IoT platform does not need to batch process the data stream for analysis, but only needs to count the data volume between adjacent data fences according to the data fence flag, and perform data positioning according to the information carried by adjacent data fences, and can judge whether there is a transmission failure according to the comparison between the actual data volume and the theoretical data volume, reducing the workload of service data processing and analysis on the cloud platform side and improving the analysis efficiency.

[0049] On the other hand, the present invention identifies whether there is a situation of excessive or insufficient data statistics by counting the occurrence frequency of data fences in the data stream, and then adaptively adjusts the data volume between adjacent data fences in the data fence insertion strategy to achieve efficient utilization of cloud platform resources. Description of the drawings

[0050] Figure 1 The figure shows a schematic diagram of using a sliding window for data batching processing in the prior art;

[0051] Figure 2 The figure shows a schematic diagram of a multi-terminal data stream after inserting message fences;

[0052] Figure 3 The figure shows a schematic diagram of the architecture of an embodiment of the cloud-edge collaborative adaptive data transmission management system based on the fence mechanism of the present invention;

[0053] Figure 4 The figure shows a schematic diagram of the basic interval adaptive adjustment process in the data fence insertion strategy in an embodiment of the method of the present invention. Detailed implementation manners

[0054] The following further describes in conjunction with the accompanying drawings and specific embodiments.

[0055] Embodiment 1

[0056] This embodiment introduces a cloud-edge collaborative adaptive data transmission management system based on the fence mechanism, including an IoT management platform and IoT terminals, as Figure 3 shown in the figure, where:

[0057] The IoT terminals send the service data characteristics to the IoT management platform;

[0058] The IoT management platform generates a data fence insertion strategy for the corresponding IoT terminals according to the received service data characteristics and returns it to the corresponding IoT terminals; wherein, the data fence insertion strategy includes the amount of data between adjacent data fences;

[0059] When the IoT terminals transmit data outward, they insert data fences into the data to be sent according to the data fence insertion strategy; wherein, the information carried by the data fences includes the reference start time of fence insertion and the order information;

[0060] The IoT management platform obtains the data stream transmitted by the IoT terminals, captures the data fences in the data stream, counts the amount of data between any two adjacent data fences, and analyzes the information carried by the data fences according to the data fence insertion strategy corresponding to the IoT terminals to obtain the theoretical value of the amount of data between adjacent data fences; compares the counted amount of data with the theoretical value of the amount of data, determines whether there is data loss, and if there is data loss, generates a corresponding data transmission fault record.

[0061] In the above solution content, the IoT management platform adaptively generates the data fence insertion strategies for each IoT terminal according to the service data characteristics.

[0062] Furthermore, in order to achieve efficient utilization of cloud resources and avoid the situation of excessive or insufficient data stream statistics, in the system of this embodiment: the IoT management platform is further configured to count the number of data fences per unit time in the data stream sent by the IoT terminals within a set time interval, and adjust the data fence insertion strategy of the corresponding IoT terminals according to the statistical results, so that the amount of data between adjacent data fences increases or decreases.

[0063] The following specifically introduces the content of this embodiment.

[0064] I. Generation of data fence strategy

[0065] The IoT management platform can pre-determine the importance level of the service data types corresponding to each IoT terminal and the service data upload frequency level, and then formulate corresponding data fence insertion strategies and send them to each IoT terminal.

[0066] The IoT terminal can also directly upload the importance level of the service data types and information related to the service data upload frequency to the IoT management platform. After being recognized by the IoT management platform, corresponding data fence insertion strategies are formulated.

[0067] In this embodiment, the service data characteristics uploaded by the IoT terminal to the IoT management platform include the upload frequency of the service data, and may also include information such as the IoT terminal ID representing the service data type or the data stream source end. The IoT management platform pre-sets multiple frequency intervals, including a high upload frequency interval, a medium-high upload frequency interval, and a low upload frequency interval. For the three frequency intervals, the basic intervals G h 、G m 、G l representing the data volume between adjacent data fences are pre-set respectively, and G h >G m >G l . That is, the higher the upload frequency, the larger the basic interval, which avoids the cloud platform from frequently processing the data stream intervals separated by data fences and reduces the occupation of cloud platform resources.

[0068] After receiving the service data characteristics uploaded by the IoT terminal, the IoT management platform determines the basic interval G n according to the upload frequency of the service data: matches the upload frequency of the service data with the pre-set multiple frequency intervals, and takes the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy for this IoT terminal.

[0069] Then the IoT management platform needs to obtain the importance information L of the business data. As mentioned above, as one implementation method for obtaining the importance information of the business data, the IoT management platform can pre-store the mapping relationship between the business data type and the importance; the business data features sent by the IoT terminal to the IoT management platform also include the business data type information, and the IoT management platform matches the corresponding importance from the business data type information according to the mapping relationship. As another implementation method for obtaining the importance information of the business data, the importance information of the business data includes an importance coefficient, which is pre-agreed by the IoT terminal and the IoT management platform; the business data features sent by the IoT terminal to the IoT management platform also include the importance coefficient of the business data. Under this implementation method, the IoT management platform can directly obtain the importance coefficient to calculate the fence interval.

[0070] When calculating the final fence interval, the IoT management platform uses the basic interval G n and the importance information L of the business data, the fence interval g of the IoT terminal data fence insertion strategy is determined as: g = G n ×L.

[0071] At this time, the IoT management platform sends the data fence insertion policy information corresponding to each IoT terminal to the corresponding IoT terminal, and saves it for subsequent analysis.

[0072] 2. Inserting IoT terminal data stream into data fence

[0073] After the IoT terminal obtains the data fence insertion strategy fed back by the IoT management platform, it inserts the data fence at the corresponding data interval according to the corresponding strategy when transmitting the data stream. Figure 2 and Figure 3 Each data fence carries a data fence identifier, policy-related information, and reference start time information and sequence information of fence insertion, to indicate when the data fence is inserted and which data fence it is.

[0074] An example of a fence message is shown below:

[0075]

[0076] Among them, "baseInterval" indicates the basic interval, which is set by the policy issued by the platform; "level" indicates the importance level of the message, which is set by the policy issued by the platform; "tagId" indicates the "fence" sequence number within a certain interval (auto-increment value can be used), and it is reset to zero regularly; "startTime" and "endTime" indicate the statistical interval of the "fence"; "total" indicates the estimated statistical value of the fence interval, which is used to compare with the actual value of the platform statistics.

[0077] III. Analysis of Data Stream by IoT Management Platform

[0078] After the IoT management platform receives the data stream output by the IoT terminal after inserting the data fence, it can identify the IoT terminal that emits the data stream. At the same time, it can identify the data fence according to the data fence identifier therein and can count the data volume between adjacent data fences. At this time, the IoT management platform can search for the data fence insertion strategy of the corresponding IoT terminal to obtain the theoretical data interval. By comparing the actually counted data volume between adjacent data fences with the theoretical data interval, it can be found whether there is a situation of data loss in the corresponding data stream interval. If there is no data loss, the foregoing judgment result can be discarded, or other analysis and processing can be continued. If there is data loss, a corresponding data transmission fault record can be generated at this time: according to the reference start time and sequence information of the fence insertion carried by the data fence, the data transmission fault record can include the IoT device ID, service data type information, fault data stream time period information, lost data volume information, etc.

[0079] IV. Adaptive Adjustment of Data Fence Insertion Strategy

[0080] As described above, in this embodiment, the IoT management platform can also adaptively adjust the data fence insertion strategy of the corresponding IoT terminal according to the statistical frequency of the data fences in each data stream. The specific adjustment logic is as shown in Figure 4 and includes:

[0081] Count the number of data fences per unit time in the data stream emitted by the IoT terminal within a set time interval to obtain the statistical frequency S:

[0082]

[0083] In the formula, the statistical frequency S represents the number of data fences per unit time within the statistical time period T (i.e., the statistical cycle), △t a represents the data volume between adjacent data fences, G n represents the basic interval determined according to the service data characteristics, and L represents the importance coefficient of the service data;

[0084] Compare the statistical frequency S with the set statistical frequency threshold S0: If S < S0 and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, it means that the statistical intensity of the corresponding IoT terminal is weak. The basic interval should be reduced, the insertion time △t between two adjacent "fences" should be shortened, the statistical frequency should be increased, and the statistical accuracy should be improved. Then the adjustment strategy is to reduce the amount of data between adjacent data fences; if S > S0 and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, it means that the statistical intensity of the corresponding IoT terminal is excessive. The basic interval needs to be increased, the statistical frequency should be reduced, and resource occupancy should be decreased. Then the adjustment strategy is to increase the amount of data between adjacent data fences; in other cases, the amount of data between data fences remains unchanged.

[0085] When adjusting the data fence insertion strategy, the object directly adjusted by the IoT management platform can be the basic interval G n , and each time it is adjusted in a set step size. The adjusted data fence adjustment strategy is sent to the corresponding IoT terminal, and the statistics and adjustment of the next cycle continue. The IoT terminal then changes the data fence insertion strategy each time it receives the data fence insertion strategy or related adjustment instructions.

[0086] The magnitude of the statistical frequency S depends on factors such as the data upload frequency and transmission rate. Through the above adaptive adjustment logic, when the statistical frequency changes, the data fence insertion strategy also changes accordingly. Thus, real-time statistics of big data in a high-efficiency and low-occupation power scenario can be achieved. Combining the above content, the solution of this embodiment can accurately locate the fault points of data sending and receiving for the business system, and accelerate the troubleshooting speed and repair speed caused by data loss.

[0087] Embodiment 2

[0088] Based on the same inventive concept as Embodiment 1, this embodiment introduces a cloud-edge collaborative adaptive data transmission management method based on the fence mechanism with the IoT management platform as the execution entity. It is executed by the IoT management platform, and the method includes:

[0089] Receive the business data characteristics sent by the IoT terminal, generate the data fence insertion strategy for the corresponding IoT terminal according to the business data characteristics, and return it to the corresponding IoT terminal; among them, the business data characteristics include business data upload frequency information, and the data fence insertion strategy includes the amount of data between adjacent data fences;

[0090] Obtain the data stream in which the IoT terminal inserts data fences and transmits according to the data fence insertion strategy; the information carried by the data fences inserted into the data stream includes the reference start time and sequence information of the fence insertion;

[0091] Capture data fences in the data stream. For any two adjacent data fences, count the amount of data between the adjacent data fences, and parse the information carried by the data fences according to the data fence insertion strategy corresponding to the Internet of Things (IoT) terminal to obtain the theoretical value of the amount of data between the adjacent data fences. Compare the counted amount of data with the theoretical value of the amount of data, and determine whether there is data loss. If there is data loss, generate a corresponding data transmission failure record.

[0092] Count the number of data fences per unit time in the data stream sent by the IoT terminal within a set time interval, and adjust the data fence insertion strategy of the corresponding IoT terminal according to the statistical result to increase or decrease the amount of data between adjacent data fences. Also, send the adjusted data fence insertion strategy to the corresponding IoT terminal.

[0093] The above-mentioned generation of the data fence insertion strategy for the corresponding IoT terminal according to the business data characteristics includes:

[0094] Determine the basic interval G according to the upload frequency of the business data n : Match the upload frequency of the business data with multiple preset frequency intervals, and use the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy for this IoT terminal. The multiple preset frequency intervals include a high upload frequency interval, a medium-high upload frequency interval, and a low upload frequency interval, and the basic intervals G h 、G m 、G l representing the amount of data between adjacent data fences are preset for the three frequency intervals respectively, and G h >G m >G l ;

[0095] Obtain the importance information L of the business data;

[0096] According to the basic interval G n and the importance information L of the business data, determine the fence interval g of the data fence insertion strategy for the IoT terminal. The formula is: g = G n ×L.

[0097] The above-mentioned comparison of the counted amount of data with the theoretical value of the amount of data, and determination of whether there is data loss. If there is data loss, generation of a corresponding data transmission failure record includes:

[0098] Compare the statistically obtained data volume with the theoretical value of the data volume. If the two are the same, there is no data loss, and the judgment result is discarded. If the two are different, there is data loss. According to the reference start time and sequence information of the fence insertion carried by the data fence, generate a data transmission fault record corresponding to the data stream with data loss. The data transmission fault record includes at least the IoT device ID, data stream time period information, and lost data volume information.

[0099] The number of data fences per unit time in the data stream sent by the IoT terminal within the time interval set for the above statistics, the formula is:

[0100]

[0101] In the formula, S is the statistical frequency, indicating the number of data fences per unit time within the statistical time period T, and △t a represents the data volume between adjacent data fences, and G n represents the basic interval determined according to the service data characteristics, and L represents the importance coefficient of the service data.

[0102] The above adjusts the data fence insertion strategy of the corresponding IoT terminal according to the statistical results, so that the data volume between adjacent data fences increases or decreases, including:

[0103] Compare the statistical frequency S with the set statistical frequency threshold S0: If S < S0, and the difference between the two is greater than the set difference range in consecutive multiple statistical time periods, then reduce the data volume between adjacent data fences; if S > S0, and the difference between the two is greater than the set difference range in consecutive multiple statistical time periods, then increase the data volume between adjacent data fences; in other cases, keep the data volume between data fences unchanged.

[0104] In summary, in the above embodiments, the present invention realizes segmented statistical analysis in the data stream transmission process through the data fence method, without separately segmenting the data stream, reducing the workload of data processing. At the same time, the data fence insertion strategy of the present invention is formulated according to the characteristics of service data and can be adjusted adaptively to the change of statistical frequency during the data transmission process, which can realize the efficient utilization of cloud platform resources and improve the efficiency of cloud platform data processing.

[0105] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate a means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0107] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction means that implements the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0109] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the present invention and the claims. All of these fall within the protection scope of the present invention.

Claims

1. A cloud-edge collaborative adaptive data transmission management method based on a fence mechanism, which is executed by an IoT management platform, is characterized in that the method Including: Receiving the service data characteristics sent by the Internet of Things (IoT) terminal, generating a corresponding data fence insertion strategy for the IoT terminal according to the service data characteristics, and returning it to the corresponding IoT terminal; wherein, the data fence insertion strategy includes the amount of data between adjacent data fences. Obtaining the data stream in which the IoT terminal inserts data fences and transmits according to the data fence insertion strategy; the information carried by the data fences inserted into the data stream includes the reference start time of fence insertion and the order information. Capturing the data fences in the data stream, for any two adjacent data fences, counting the amount of data between the adjacent data fences, and parsing the information carried by the data fences according to the corresponding data fence insertion strategy of the IoT terminal to obtain the theoretical value of the amount of data between the adjacent data fences; comparing the counted amount of data with the theoretical value of the amount of data, determining whether there is data loss, and if there is data loss, generating a corresponding data transmission fault record. Among them, generating a corresponding data fence insertion strategy for the IoT terminal according to the service data characteristics includes: Determine the basic interval according to the reporting frequency of service data : Match the reporting frequency of service data with multiple preset frequency intervals, and use the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy for this IoT terminal; Obtain the importance information of business data ; According to the described basic interval and the importance information of business data , determine the fence interval of the Internet of Things terminal data fence insertion strategy , the formula is: .

2. The method according to claim 1, characterized in that it further Including: Counting the number of data fences per unit time in the data stream sent by the IoT terminal within a set time interval, and adjusting the data fence insertion strategy of the corresponding IoT terminal according to the statistical result, so that the amount of data between adjacent data fences increases or decreases; and sending the adjusted data fence insertion strategy to the corresponding IoT terminal. The formula for the number of data fences per unit time in the data stream sent by the IoT terminal within the set time interval is: ， In the formula, is the statistical frequency, representing the number of data fences appearing per unit time within the statistical time period ; represents the data volume between adjacent data fences, represents the basic interval determined according to the characteristics of business data, and represents the importance coefficient of business data.

3. The method according to claim 2, wherein The adjusting the data fence insertion strategy of the corresponding IoT terminal according to the statistical result so that the amount of data between adjacent data fences increases or decreases includes: Compare the statistical frequency with a set statistical frequency threshold : If , and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then reduce the amount of data between adjacent data fences; if , and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then increase the amount of data between adjacent data fences; in other cases, keep the amount of data between the data fences unchanged.

4. The method according to claim 1, characterized in that, The preset multiple frequency ranges include a corresponding high upward transmission frequency range, a middle-high upward transmission frequency range, and a low upward transmission frequency range, and a basic interval representing the amount of data between adjacent data fences is preset for each of the three frequency ranges , and .

5. The method according to claim 1, wherein The comparing the counted amount of data with the theoretical value of the amount of data, determining whether there is data loss, and if there is data loss, generating a corresponding data transmission fault record includes: Comparing the counted amount of data with the theoretical value of the amount of data. If the two are the same, there is no data loss, and the judgment result is discarded. If the two are different, there is data loss. According to the reference start time of fence insertion and the order information carried by the data fence, generating a data transmission fault record corresponding to the data stream with data loss. The data transmission fault record at least includes the IoT device ID, the data stream time period information, and the lost data amount information.

6. A cloud-edge collaborative adaptive data transmission management system based on a fence mechanism, including an IoT management platform and IoT terminals; characterized in that: The IoT terminal sends the service data characteristics to the IoT management platform; wherein, the service data characteristics include the upload frequency of the service data. The IoT management platform generates a corresponding data fence insertion strategy for the IoT terminal according to the received service data characteristics, and returns it to the corresponding IoT terminal; wherein, the data fence insertion strategy includes the amount of data between adjacent data fences. When the Internet of Things terminal transmits data outward, it inserts a data fence into the data being sent according to the data fence insertion strategy; wherein, the information carried by the data fence includes the reference start time of fence insertion and the sequence information; The Internet of Things management platform obtains the data stream transmitted by the Internet of Things terminal, captures the data fences in the data stream, for any two adjacent data fences, counts the amount of data between the adjacent data fences, and parses the information carried by the data fences according to the data fence insertion strategy corresponding to the Internet of Things terminal to obtain the theoretical value of the amount of data between the adjacent data fences; compares the counted amount of data with the theoretical value of the amount of data, determines whether there is data loss, and if there is data loss, generates a corresponding data transmission fault record; Among them, the Internet of Things management platform generates the data fence insertion strategy for the corresponding Internet of Things terminal according to the received service data characteristics, including: Determine the basic interval according to the upload frequency of service data : Match the upload frequency of service data with multiple preset frequency intervals, and use the basic interval corresponding to the matched frequency interval as the basic interval of the data fence insertion strategy for this IoT terminal; Obtain the importance information of business data ; Based on the described basic interval and the importance level information of the service data , determine the fence interval of the Internet of Things terminal data fence insertion strategy , and the formula is: .

7. The cloud-edge collaborative adaptive data transmission management system based on the fence mechanism according to claim 6, characterized in that, The Internet of Things management platform is also used to count the number of data fences per unit time in the data stream sent by the Internet of Things terminal within a set time interval, and adjust the data fence insertion strategy of the corresponding Internet of Things terminal according to the statistical result, so that the amount of data between adjacent data fences increases or decreases; And, send the adjusted data fence insertion strategy to the corresponding Internet of Things terminal.

8. The cloud-edge collaborative adaptive data transmission management system based on the fence mechanism according to claim 7, wherein, The formula for counting the number of data fences per unit time in the data stream sent by the Internet of Things terminal within the set time interval is: ， In the formula, is the statistical frequency, representing the number of data fences appearing per unit time within the statistical time period The number of data fences appearing per unit time within the statistical time period represents the data volume between adjacent data fences, represents the basic interval determined according to the characteristics of business data, represents the importance coefficient of business data; Adjusting the data fence insertion strategy of the corresponding Internet of Things terminal according to the statistical result so that the amount of data between adjacent data fences increases or decreases includes: Compare the statistical frequency with a set statistical frequency threshold as follows: If , and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then reduce the amount of data between adjacent data fences; if , and the difference between the two is greater than the set difference range in multiple consecutive statistical time periods, then increase the amount of data between adjacent data fences; in other cases, keep the amount of data between the data fences unchanged.

9. The cloud-edge collaborative adaptive data transmission management system based on the fence mechanism according to claim 6, characterized in that, The preset multiple frequency intervals include a corresponding high upload frequency interval, a middle-high upload frequency interval, and a low upload frequency interval, and a basic interval representing the amount of data between adjacent data fences is preset for each of the three frequency intervals , and .

10. The cloud-edge collaborative adaptive data transmission management system based on the fence mechanism according to claim 9, characterized in that, The Internet of Things management platform prestores the mapping relationship between service data types and importance levels; the service data characteristics sent by the Internet of Things terminal to the Internet of Things management platform also include service data type information, and the Internet of Things management platform matches the corresponding importance level according to the mapping relationship from the service data type information; Or, The importance level information of the service data includes an importance coefficient, which is pre-agreed between the Internet of Things terminal and the Internet of Things management platform; the service data characteristics sent by the Internet of Things terminal to the Internet of Things management platform also include the importance coefficient of the service data.

Images