Multi-source heterogeneous device data unified access method based on cloud edge collaboration

Abstract

The invention relates to the technical field of data transmission, in particular to a multi-source heterogeneous device data unified access method based on cloud edge collaboration, which comprises the following steps of: determining a storage demand degree according to a network transmission interference condition at each moment in a data uploading process and a compression storage duration condition of breakpoint data at an edge gateway; obtaining a breakpoint data storage amount and a historical storage demand stability index in the edge gateway at each moment, and determining a correction compression rate of each piece of breakpoint data to be transmitted in combination with a storage demand degree; determining the correlation degree of each piece of to-be-transmitted breakpoint data according to the number of tasks and the usage ratio; determining each transmission priority index according to the correlation degree, the data size and the correction compression ratio; and sequentially transmitting the compressed breakpoint data to be transmitted to a cloud server according to the transmission priority index. According to the method, the transmission priority index of the breakpoint data is adaptively determined, so that the unified access effect of the data of the multi-source heterogeneous equipment can be enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, in particular to a multi-source heterogeneous device data unified access method based on cloud-edge collaboration. BACKGROUND

[0002] When the cloud-edge collaboration architecture matures, the unified access method based on cloud-edge collaboration emerges as the times require, which combines the powerful capabilities of centralized management, global optimization and AI training in the cloud with the real-time processing, low-latency response and data preprocessing capabilities on the edge side, and solves the pain points existing in the traditional access mode through collaborative cooperation. Among them, the ability to dynamically determine the data priority can truly release the potential of the cloud-edge collaboration architecture, and achieve the core goals of cost reduction and business agility.

[0003] The traditional method only judges the priority of the device breakpoint data according to the time of transmitting different device data to the edge gateway, but because the breakpoint data characteristics do not match the real-time demand, the network dynamics are not considered, and the demand degree of the cloud task for different breakpoint data is different, there is deviation in the priority evaluation of the breakpoint data in the multi-source heterogeneous device data unified access process. SUMMARY

[0004] In order to solve the technical problem of deviation in the priority evaluation of the breakpoint data in the multi-source heterogeneous device data unified access process, the purpose of the present application is to provide a multi-source heterogeneous device data unified access method based on cloud-edge collaboration, and the technical solution adopted is as follows: One embodiment of the present application provides a multi-source heterogeneous device data unified access method based on cloud-edge collaboration, which comprises the following steps: Obtain each to-be-transmitted breakpoint data of multi-source heterogeneous devices in a data uploading process through an edge gateway at the current time; Determine the storage demand degree of the edge gateway at each time according to the network transmission interference at each time in the data uploading process and the compression storage time length of the breakpoint data in the edge gateway; Obtain the breakpoint data storage amount in the edge gateway at each time and the historical storage demand stability index, and determine the corrected compression rate of each to-be-transmitted breakpoint data in combination with the storage demand degree of the edge gateway at each time; Determine the number of tasks of each data type in each to-be-transmitted breakpoint data and the usage proportion of each data type in the cloud processing task, and then determine the correlation degree of each to-be-transmitted breakpoint data with the cloud task according to the number of tasks and the usage proportion; Determine the transmission priority index of each to-be-transmitted breakpoint data according to the correlation degree of each to-be-transmitted breakpoint data with the cloud task, the data amount and the corrected compression rate; The compressed data from each breakpoint to be transmitted is sequentially transmitted to the cloud server according to the transmission priority index.

[0005] Furthermore, determining the storage requirements of the edge gateway at each moment based on network transmission interference at each time point during the data upload process and the compression and storage duration of breakpoint data at the edge gateway includes: Take any moment in the data upload process as the target moment, and determine the degree of network transmission interference at the target moment based on the data transmission interruption situation under the first preset time period including the target moment; The moment when the breakpoint data occurs closest to the target time is taken as the breakpoint time. Based on the time interval between the breakpoint time and the target time, the compression storage maintenance index of the breakpoint data corresponding to the target time in the edge gateway is determined. The network transmission interference level and the compressed storage maintenance index at the target time are fused to obtain the storage requirement of the edge gateway at the target time.

[0006] Furthermore, determining the network transmission interference level at the target time based on the data transmission interruption situation during a first preset time period including the target time includes: The number of transmission interruptions during a first preset time period including the target time is obtained, and the maximum time interval corresponding to every two adjacent interruption times within the first preset time period is obtained; the interruption time is the time when data breaks down within the first preset time period. The maximum time interval is inversely proportional to obtain an inverse proportional value. The inverse proportional value and the number of interruptions are then fused to determine the network transmission interference level at the target time.

[0007] Furthermore, the data fusion processing of the network transmission interference level and the compressed storage maintenance index at the target time to obtain the storage requirement of the edge gateway at the target time includes: Obtain the network transmission interference level of the time preceding the target time, calculate the difference between the network transmission interference level of the target time and the time preceding the target time, and record it as the network transmission interference level difference. The first storage requirement factor is determined based on the difference between the network transmission interference level at the target time and the network transmission interference level at the target time; and the compressed storage maintenance index at the target time is used as the second storage requirement factor. The first storage demand factor and the second storage demand factor are fused to obtain the storage demand level of the edge gateway at the target time.

[0008] Furthermore, obtain historical storage demand stability metrics, including: Set a storage demand threshold and obtain the storage demand level of the edge gateway at each moment within a second preset time period; the second preset time period is a pre-set time period located before the target time. The moments corresponding to each level of storage demand that exceeds the storage demand threshold are taken as disturbance moments, and the moments corresponding to each level of storage demand that does not exceed the storage demand threshold are taken as stable moments. Based on the difference in the number of stable moments and the number of disturbance moments, the stability index of historical storage demand in the edge gateway at the target moment is determined.

[0009] Further, determining the corrected compression ratio for each data point to be transmitted includes: Based on the breakpoint data storage volume, historical storage demand stability index, and storage demand level of the edge gateway at each time point, determine the correction compression coefficient of the data in the edge gateway at each time point. Obtain the occurrence time of each data breakpoint to be transmitted, and use the correction compression coefficient at the time closest to the occurrence time as the correction compression coefficient of the corresponding data breakpoint to be transmitted. Obtain a preset compression ratio, and use the correction compression coefficient of each data point to be transmitted to correct the preset compression ratio, thereby determining the correction compression ratio of each data point to be transmitted.

[0010] Furthermore, determining the correlation between the data at each transmission breakpoint and the cloud task based on the number of tasks and the usage ratio includes: Get the number of all tasks being processed in the cloud at the current moment and the maximum usage percentage; For each data point to be transmitted, the first correlation factor between the data point to be transmitted and the cloud task is determined based on the average number of tasks of all data types of the data point to be transmitted as a percentage of the total number of tasks. Based on the proportion of the average usage percentage of all data types of the breakpoint data to be transmitted in the maximum usage percentage, a second correlation factor between the breakpoint data to be transmitted and the cloud task is determined. The first and second correlation factors of the data to be transmitted at the breakpoint and the cloud task are fused to obtain the correlation between the data to be transmitted at the breakpoint and the cloud task.

[0011] Furthermore, the step of determining the transmission priority index for each data point to be transmitted based on its relevance to the cloud task, its data size, and the corrected compression rate includes: Obtain the time interval between the occurrence time of each data interruption point to be transmitted and the current time, and determine the urgency of transmitting each data interruption point to be transmitted based on the correlation between each data interruption point to be transmitted and the cloud task. The urgency of transmitting data at the same breakpoint, the data size, and the modified compression rate are used to perform data fusion processing to determine the transmission priority index of each breakpoint data.

[0012] Furthermore, determining the urgency of transmitting data at each interruption point includes: Calculate the ratio of the time interval of each data interruption point to the maximum time interval, and use it as the first transmission urgency factor for the corresponding interruption point data. The difference between the maximum correlation degree and the correlation degree of each data point to be transmitted is calculated and used as the second transmission urgency factor for the corresponding data point to be transmitted. The first transmission urgency factor and the second transmission urgency factor of the same data to be transmitted are fused to determine the transmission urgency of each data to be transmitted.

[0013] Furthermore, the data fusion processing of the urgency of transmitting data at the same interruption point, the data size, and the corrected compression ratio to determine the transmission priority index of each interruption point data includes: For each data point to be transmitted, the data size and the corrected compression ratio are inversely proportional to each other to obtain the inverse proportional value between the data size and the corrected compression ratio of the data point to be transmitted. The transmission priority index of the interrupted data is determined by performing data fusion processing on the urgency of transmitting the interrupted data, the data size, and the inverse proportional value of the modified compression rate.

[0014] The present invention has the following beneficial effects: This invention provides a unified access method for data from multiple heterogeneous devices based on cloud-edge collaboration. This method adaptively determines the modified compression rate of each data point to be transmitted, avoiding processing delays in the cloud while ensuring sufficient storage space for the data at the edge gateway. This facilitates more accurate determination of the transmission priority of the data at each point. By adaptively determining the correlation between each data point and cloud tasks, the method quantifies the relationship between the data and cloud processing, prioritizing the upload of data relevant to the processing tasks. The transmission priority determined by combining the correlation between the data point and cloud tasks, data size, and modified compression rate effectively overcomes the shortcomings of existing methods for prioritizing data transmission at different points, such as failing to consider the matching of data characteristics with real-time requirements, network dynamics, and varying cloud task demands for different data points. This improves the accuracy and reliability of priority assessment for data at different points during unified access to data from multiple heterogeneous devices. Furthermore, by adaptively determining the transmission priority index of each data point, tasks in the cloud can be processed promptly, increasing the processing speed and enhancing the unified access effect of data from multiple heterogeneous devices. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration is provided as an embodiment of the present invention; Figure 2 This is a flowchart illustrating the implementation of step S2 in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the implementation of step S5 in an embodiment of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] The application scenarios targeted by this invention can be: Adaptively prioritizing the transmission of breakpoint data from multi-source malfunctioning devices can improve data transmission efficiency and system response speed. Furthermore, combining this with a breakpoint data processing mechanism can ensure data integrity and continuity, enhancing overall system performance. However, current breakpoint data transmission priority is quantified solely by the timing of transmission to the edge gateway, resulting in low accuracy in prioritization and hindering the unified transmission of data from multi-source heterogeneous devices.

[0020] One embodiment of the present invention provides a method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration, such as... Figure 1 As shown, it includes the following steps: S1, obtains the data of each interruption point to be transmitted in the data upload process of the multi-source heterogeneous devices through the edge gateway at the current moment.

[0021] Here, breakpoint data refers to the location and status information of successfully transmitted data recorded and saved by the system when transmission fails due to network interruption or other reasons. Its core function is to allow the transmission task to continue from the breakpoint after recovery, rather than starting over.

[0022] Specifically, through the breakpoint resumption mechanism in the edge gateway at the current moment, the location of successfully uploaded data blocks or files is recorded during the data upload process, such as the breakpoint position and the number of bytes uploaded. For the breakpoint position, fragmented upload and a breakpoint logging file can be used to record the breakpoint position. Subsequently, the edge gateway stores the breakpoint position and breakpoint data in a cache. Therefore, through the edge gateway at the current moment, it is possible to obtain the data at each breakpoint to be transmitted during the data upload process of multi-source heterogeneous devices.

[0023] The edge gateway incorporates multiple protocol libraries and automatically parses the communication protocols of different devices using protocol parsing algorithms. A protocol conversion module transforms data from different protocols into a unified format for easier subsequent processing and transmission. A data management module records device attributes and data, ensuring data security and management.

[0024] It is worth noting that if the upload fails, it will restart the upload from the point of interruption, which can effectively avoid duplicate uploads.

[0025] It should also be noted that this embodiment takes an edge gateway at the current moment as an example to analyze the transmission priority of each data interruption point corresponding to the edge gateway.

[0026] S2. Based on the network transmission interference at each moment during the data upload process and the compression and storage time of the breakpoint data at the edge gateway, determine the storage requirements of the edge gateway at each moment.

[0027] Here, the storage requirement level refers to the edge gateway's ability and necessity to store data locally at each moment during data upload.

[0028] As an exemplary implementation, step S2 described above can be achieved through... Figure 2 The steps shown are to be implemented as follows: S21, take any moment in the data upload process as the target moment, and determine the network transmission interference level at the target moment based on the data transmission interruption situation under the first preset time period including the target moment.

[0029] Here, network transmission interference refers to the instability caused to the network by frequent interruptions in data transmission from the edge gateway. For multi-source heterogeneous devices, the empirical value of the interval between adjacent moments during data upload can be determined comprehensively based on the specific application scenario and protocol characteristics.

[0030] Network instability can lead to data transmission interruptions, which in turn can cause data backlogs that disrupt real-time applications such as video conferencing and online gaming. Therefore, to analyze network transmission interference at each moment during data upload, the frequency and interval of data transmission interruptions can be used to determine the network's reliability. The frequency of interruptions directly reflects the reliability of the network connection; a higher frequency indicates lower reliability. The interval reflects the frequency of interruptions; a smaller interval indicates more frequent interruptions and greater network transmission interference.

[0031] As an exemplary implementation, determining the network transmission interference level at the target time based on data transmission interruption during a first preset time period including the target time includes: The first step is to obtain the number of transmission interruptions during the first preset time period, which includes the target time, and to obtain the maximum time interval between every two adjacent interruption times within the first preset time period.

[0032] Here, the first preset time period can be set to 60 minutes, specifically referring to a time period calculated 60 minutes prior to the target time, or a 60-minute time period consisting of the durations on both sides of the target time, which includes the target time; the interruption time is the moment when breakpoint data occurs within the first preset time period. The value and setting method of the first preset time period can be set by the implementer according to specific circumstances, and are not specifically limited here.

[0033] The second step is to perform inverse proportional processing on the maximum time interval to obtain an inverse proportional value, and then combine the inverse proportional value with the number of interruptions to determine the degree of network transmission interference at the target time.

[0034] The more times the edge gateway data transmission is interrupted at the target time, and the smaller the maximum time interval, the more unstable the network is at the target time, and the more serious the data accumulation may be.

[0035] As an example, the formula for calculating the network transmission interference level at time t during data upload can be: In the formula, This represents the level of network transmission interference at time t. This represents the number of transmission interruptions during the first preset time period, including the t-th target time. This represents the maximum time interval between any two adjacent interruption times within the first preset time period, including the t-th target time. It represents the inverse proportional value of the maximum time interval corresponding to every two adjacent interruption times within the first preset time period containing the t-th target time.

[0036] Referring to the calculation process of network transmission interference at time t above, the network transmission interference at each time point during the data upload process can be obtained.

[0037] It is worth noting that when determining the degree of network transmission interference, in addition to the number of interruptions and the interruption interval, other factors related to network instability can also be considered, such as packet loss rate, latency jitter, or TCP retransmission rate.

[0038] S22, take the moment when the data breakpoint is closest to the target time as the breakpoint moment, and determine the compression storage maintenance index of the breakpoint data corresponding to the target time in the edge gateway based on the time interval between the breakpoint moment and the target time.

[0039] In one embodiment, the time when the breakpoint data last started to appear is obtained relative to the target time and recorded as the breakpoint time; then the time interval between the target time and the breakpoint time is determined, and the magnitude of the time interval is used as the compression storage maintenance index of the breakpoint data corresponding to the target time in the edge gateway.

[0040] S23, perform data fusion processing on the network transmission interference level and compressed storage maintenance index at the target time to obtain the storage requirement of the edge gateway at the target time.

[0041] If both the network transmission interference level and the compression storage maintenance index are high, it indicates that the network instability at the target time is high, and the data at the breakpoint has been compressed and stored in the edge gateway for a long time, indicating that the storage requirement of the edge gateway is high.

[0042] As an exemplary implementation, step S23 described above can be achieved through the following steps: The first step is to calculate the product of the network transmission interference level and the compressed storage maintenance index at the target time.

[0043] The second step is to normalize the product of the network transmission interference level and the compression storage maintenance index at the target time, and use the normalized value as the storage requirement of the edge gateway at the target time.

[0044] In one embodiment, maximum-minimum normalization can be used to normalize the product, limiting the storage requirement value to between 0 and 1. Of course, implementers can use other normalization methods, which are not specifically limited here. Preferably, step 23 above can also be achieved through the following steps: The first step is to obtain the network transmission interference level of the moment before the target time, and calculate the difference between the network transmission interference level of the target time and the moment before it, which is recorded as the network transmission interference level difference.

[0045] The edge layer needs to dynamically adjust its data storage strategy based on changes in network interference. When the level of network interference increases significantly, the gateway needs to temporarily store more data to avoid data loss due to network instability.

[0046] Here, the previous moment is a moment that is before and adjacent to the target moment. The difference in network transmission interference reflects the change in the stability of the communication link. The larger the difference in network transmission interference, the more unstable the network communication link is, and the greater the storage requirement of the edge gateway will be.

[0047] The second step is to determine the first storage demand factor based on the network transmission interference level at the target time and the difference between the network transmission interference level and the target time; and to use the compressed storage maintenance index at the target time as the second storage demand factor.

[0048] Here, the first storage demand factor is a storage indicator determined by analyzing the network instability at the target time, while the second storage demand factor is a storage indicator determined by analyzing the interval between the target time and the breakpoint time.

[0049] In one embodiment, the product of the network transmission interference level and the difference between the network transmission interference levels is used as the first storage requirement factor.

[0050] For the first storage requirement factor, the greater the network transmission interference at the target time, and the greater it is than the network transmission interference at the previous time, the more likely the edge gateway may need to maintain data storage for different access devices for a longer period of time, and the greater the storage necessity.

[0051] It should also be noted that the difference in network transmission interference levels may be zero. To avoid the extreme case where the storage demand level is zero, a non-zero constant should be added to the difference in network transmission interference levels before multiplication. An empirical value for this non-zero constant can be 0.01.

[0052] The third step is to perform data fusion processing on the first storage demand factor and the second storage demand factor to obtain the storage demand level of the edge gateway at the target time.

[0053] In one embodiment, the product of a first storage demand factor and a second storage demand factor is calculated, and the product of the first storage demand factor and the second storage demand factor is normalized. The normalized value is then used as the storage demand level of the edge gateway at the target time.

[0054] It should be noted that when merging the first and second storage demand factors, only the numerical calculation is considered, and the influence of units does not need to be taken into account. Normalization can be performed on the maximum and minimum values, which can limit the numerical range of storage demand levels to between 0 and 1.

[0055] By referring to the calculation process of the storage requirements of the edge gateway at the target time, the storage requirements of the edge gateway at each time point can be obtained.

[0056] S3: Obtain the breakpoint data storage volume and historical storage demand stability index of the edge gateway at each time point, and determine the correction compression rate of each breakpoint data to be transmitted by combining the storage demand level of the edge gateway at each time point.

[0057] When data is transmitted over the network, if the compression ratio of the breakpoint data stored at the edge gateway is too high, the cloud requires a significant amount of time to decompress the compressed breakpoint data, thus delaying task processing in the cloud. To avoid this problem while minimizing the storage space of breakpoint data in the edge gateway, it is necessary to be able to compress and store different breakpoint data to be transmitted at different compression ratios, i.e., to determine a corrected compression ratio for each breakpoint data to be transmitted.

[0058] As an exemplary implementation, the historical storage demand stability index in the edge gateway at each moment during the data upload process is obtained, including: The first step is to set the storage demand threshold and obtain the storage demand level of the edge gateway at each time point within the second preset time period.

[0059] Here, the second preset time period is a pre-set time period located before the target time, and its empirical value is 180 minutes. The value and setting method of the second preset time period can be set by the implementer according to the specific actual situation, and no specific limitation is made here.

[0060] In one embodiment, by referring to the process of determining the storage requirements of the edge gateway at the target time during the data upload process described above, the storage requirements of the edge gateway at each time point within the second preset time period can be obtained.

[0061] The second step is to designate the moments corresponding to each level of storage demand that exceeds the storage demand threshold as disturbance moments, and the moments corresponding to each level of storage demand that does not exceed the storage demand threshold as stable moments.

[0062] In one embodiment, the storage demand level ranges from 0 to 1, and the storage demand threshold can be set to 0.7. Implementers can set the size of the storage demand threshold according to specific actual requirements.

[0063] The third step is to determine the stability index of historical storage demand in the edge gateway at the target time based on the difference between the number of stable times and the number of disturbance times.

[0064] Here, the historical storage demand stability index refers to the distribution of time periods in which the storage demand level is relatively stable over a past period, that is, the distribution of time periods in which the storage demand level is lower than the storage demand threshold.

[0065] As an example, the formula for calculating the historical storage demand stability index in the edge gateway at time t can be: In the formula, This represents the stability index of historical storage demand in the edge gateway at time t. This indicates the number of stable moments within the second preset time period. This indicates the number of interference moments within the second preset time period.

[0066] In the formula for calculating the historical storage demand stability index, the greater the number of stable moments compared to the number of disturbance moments, the greater the historical storage demand stability index will be. The cumulative amount of breakpoint data after the target moment is less likely to increase significantly, and a larger compression ratio can be used to compress the breakpoint data to a greater extent. Therefore, the historical storage demand stability index and the modified compression coefficient are positively correlated.

[0067] As an exemplary implementation, determining the corrected compression ratio for each data point to be transmitted includes: The first step is to determine the corrected compression coefficient of the data in the edge gateway at each time point based on the breakpoint data storage volume, historical storage demand stability index, and storage demand level of the edge gateway at each time point.

[0068] As an example, the formula for calculating the corrected compression factor of the data in the edge gateway at time t can be: In the formula, This represents the corrected compression factor of the data in the edge gateway at time t. This represents the amount of breakpoint data stored in the edge gateway at time t. This indicates the storage requirement of the edge gateway at time t. This represents the stability index of historical storage demand in the edge gateway at time t. This represents the hyperbolic tangent function, which can be limited to a value between -1 and 1.

[0069] In the formula for calculating the compression factor, the smaller the data storage volume and storage demand of the edge gateway's breakpoint data, and the larger the stability index of historical storage demand, the smaller the cumulative amount of data transmission during data transmission recovery, and the less likely the cumulative amount of breakpoint data will increase significantly afterward. Consequently, the possibility of delays in cloud task processing due to long decompression time caused by high compression ratio is smaller. At this time, a larger compression ratio can be used to compress and store the breakpoint data to reduce the storage pressure of the breakpoint data on the edge gateway, and the larger the compression factor, the better.

[0070] The second step is to obtain the occurrence time of each data point to be transmitted, and use the correction compression coefficient at the time closest to the occurrence time as the correction compression coefficient of the corresponding data point to be transmitted.

[0071] After determining the corrected compression factor of the data in the edge gateway at each moment during the data upload process, it is necessary to determine the corrected compression factor of different data at different transmission breakpoints. This is to facilitate subsequent priority determination and compression processing of the data at these breakpoints. Each data at a transmission breakpoint has its own corresponding corrected compression factor.

[0072] The third step is to obtain the preset compression ratio, and then use the correction compression coefficient of each data point to be transmitted to correct the preset compression ratio and determine the correction compression ratio of each data point to be transmitted.

[0073] It should be noted that a larger modified compression coefficient indicates a higher compression ratio of the breakpoint data by the edge gateway. Therefore, the modified compression ratio of the breakpoint data to be transmitted can be determined, and its calculation formula is as follows: In the formula, This represents the corrected compression ratio of the data at the g-th breakpoint to be transmitted, and Y represents the preset compression ratio, which can be taken as an empirical value of 0.4. This represents the correction compression factor for the data at the g-th breakpoint to be transmitted.

[0074] Referring to the calculation process of the corrected compression ratio of the g-th data to be transmitted, the corrected compression ratio of each data point to be transmitted can be obtained.

[0075] S4. Determine the number of tasks for each data type in each data point to be transmitted and the usage ratio of each data type in the cloud processing tasks. Then, determine the correlation between each data point to be transmitted and the cloud tasks based on the number of tasks and the usage ratio.

[0076] First, it should be noted that the cloud uses different data from the edge gateway for task analysis. To avoid delays in the tasks processed by the cloud, breakpoint data related to the processing tasks should be uploaded first. Therefore, it is necessary to determine the degree of relevance between each breakpoint data to be transmitted and the cloud task.

[0077] As an exemplary implementation, the number of tasks for each data type in each data point to be transmitted and the usage percentage of each data type in the cloud processing tasks are determined, including: The first step is to count the number of tasks of each data type in the cloud when processing data from each data transmission interruption point after the data transmission from the edge gateway to the cloud is restored, as well as the total number of tasks being processed in the cloud at the current moment.

[0078] The second step involves querying specific tables and using input grouped and summarized data tables, as well as summary formulas, to output a new data table containing grouped data and calculated aggregate values. By grouping and statistically analyzing data according to fields such as DataType, the usage percentage of each data type of data at each data point to be transmitted in the cloud processing task is indirectly calculated, and the maximum usage percentage among all usage percentages is determined.

[0079] The usage percentage is equal to the total percentage of the same data type in different tasks processed in the cloud. The process of obtaining the usage percentage of each data type in a single task processed in the cloud is existing technology and will not be elaborated here.

[0080] As an exemplary implementation, the correlation between the data at each transmission breakpoint and the cloud task is determined based on the number of tasks and the usage ratio, including: The first step is to obtain the number of all tasks being processed in the cloud at the current moment and the maximum usage percentage.

[0081] In one embodiment, the total number of tasks and the maximum usage percentage have already been obtained when determining the number of tasks and the usage percentage, which are used to achieve subsequent comparative analysis.

[0082] It should be noted that by analyzing the similarity between the number of tasks with interrupted data to be transmitted and the number of tasks currently being processed in the cloud, as well as the similarity between the usage ratio and the maximum usage ratio, the correlation between the interrupted data to be transmitted and the cloud tasks can be quantitatively determined.

[0083] The second step is to determine the primary correlation factor between the data to be transmitted and the cloud tasks based on the average number of tasks of all data types of the data to be transmitted and their proportion in the total number of tasks.

[0084] In one embodiment, the average number of tasks for all data types of the breakpoint data to be transmitted is first calculated and denoted as the average number of tasks; then, the ratio of the average number of tasks to the total number of tasks is used as the first correlation factor between the breakpoint data to be transmitted and the cloud tasks.

[0085] The third step is to determine the second correlation factor between the data to be transmitted and the cloud task based on the average usage percentage of all data types of the breakpoint data in the maximum usage percentage.

[0086] In one embodiment, the average usage percentage of all data types of the breakpoint data to be transmitted is first calculated and denoted as the average usage percentage; then the ratio of the average usage percentage to the maximum usage percentage is used as the second correlation factor between the breakpoint data to be transmitted and the cloud task.

[0087] In another embodiment, the difference between the maximum usage percentage and the average usage percentage is calculated. This difference is then inverted, and the inverted difference is used as the second correlation factor between the data at the breakpoint to be transmitted and the cloud task. The difference is generally unlikely to be zero; however, in extreme cases, a non-zero constant, such as 0.01, is added to the ratio.

[0088] The fourth step is to fuse the data at the transmission breakpoint with the first and second correlation factors of the cloud task to obtain the correlation between the data at the transmission breakpoint and the cloud task.

[0089] In one embodiment, the product of the first correlation factor and the second correlation factor of the data to be transmitted at the breakpoint and the cloud task is calculated, and the product of the two correlation factors is taken as the degree of correlation between the data to be transmitted at the breakpoint and the cloud task.

[0090] As an example, the formula for calculating the relevance between the g-th data to be transmitted and the cloud task can be: In the formula, This indicates the degree of relevance between the g-th data point to be transmitted and the cloud task. This represents the average number of tasks of all data types at the g-th data transmission breakpoint, and M represents the total number of tasks being processed in the cloud at the current moment. This represents the first correlation factor between the g-th data point to be transmitted and the cloud task. Indicates the percentage of maximum usage. This represents the percentage of each data type used in processing the c-th task in the cloud for the g-th data point to be transmitted. This represents the percentage of each data type used in cloud processing tasks for the g-th data point to be transmitted. This represents the average percentage of all data types used when processing tasks in the cloud for the g-th data point to be transmitted. This represents a non-zero constant, used to avoid situations where the denominator of a fraction is zero. This represents the second correlation factor between the data at the g-th interrupt point to be transmitted and the cloud task.

[0091] It should be noted that when both the first and second correlation factors of the transmission breakpoint data and the cloud task are large, it indicates that the relationship between the transmission breakpoint data and the cloud processing task is closer. In order to ensure the timeliness of task processing in the cloud, the compressed transmission breakpoint data should be uploaded first after the cloud transmission is restored.

[0092] Referring to the process of determining the correlation between the g-th data to be transmitted and the cloud task, the correlation between each data to be transmitted and the cloud task can be obtained.

[0093] S5 determines the transmission priority index of each data point to be transmitted based on its relevance to the cloud task, data size, and compression ratio.

[0094] Here, the higher the transmission priority index, the more likely the corresponding interrupted data to be transmitted will be transmitted first.

[0095] As an exemplary implementation, step S5 described above can be achieved through... Figure 3 The steps shown are to be implemented as follows: S51, obtain the time interval between the occurrence time of each data point to be transmitted and the current time, and determine the urgency of transmitting each data point to be transmitted by combining the relevance of each data point to be transmitted to the cloud task.

[0096] To ensure the continuity of data transmission, the breakpoint data obtained earlier from the edge gateway should be transmitted with relative priority. Therefore, it is necessary to obtain the time interval between the occurrence time of each breakpoint data to be transmitted and the current time. Subsequently, the urgency of transmitting each breakpoint data can be determined based on the time interval between the occurrence time of each breakpoint data to be transmitted and the current time, as well as the degree of relevance to the cloud task.

[0097] Here, the urgency of transmission refers to the urgency of uploading the interrupted data to the cloud. The earlier the interrupted data is acquired and the higher its relevance to the cloud task, the greater the urgency of transmitting the interrupted data.

[0098] As an exemplary implementation, determining the urgency of transmitting data at each interruption point includes: The first step is to calculate the ratio of the time interval of each data point to be transmitted to the maximum time interval, which is used as the first transmission urgency factor for the corresponding data point.

[0099] The second step is to calculate the difference between the maximum correlation degree and the correlation degree of each data point to be transmitted, and use this as the second transmission urgency factor for the corresponding data point to be transmitted.

[0100] As an example, the formula for calculating the second transmission urgency factor of the g-th interrupted data can be: In the formula, This represents the second transmission urgency factor for the g-th data segment to be transmitted. Indicates the highest degree of relevance. This indicates the correlation of the data at the g-th breakpoint to be transmitted. This represents a non-zero constant, used to avoid the case where the denominator of a fraction is zero. Its empirical value can be 0.01.

[0101] In the formula for calculating the second transmission urgency factor, The smaller the value, the closer the correlation of the data at the g-th breakpoint to be transmitted is to the maximum correlation, the stronger the urgency of transmitting the data at the g-th breakpoint, and the larger the second transmission urgency factor.

[0102] Referring to the calculation process of the second transmission urgency factor of the g-th data to be transmitted, the second transmission urgency factor of each data to be transmitted can be obtained.

[0103] The third step is to perform data fusion processing on the first and second transmission urgency factors of the same data to be transmitted at the breakpoint, and determine the transmission urgency of each data to be transmitted at the breakpoint.

[0104] In one embodiment, the first transmission urgency factor and the second transmission urgency factor of the same data to be transmitted at a breakpoint are multiplied together, and the product is used as the transmission urgency of the corresponding data to be transmitted at a breakpoint.

[0105] It should be noted that the urgency of transmission is essentially a degree value. Therefore, the first and second transmission urgency factors are also degree values. When multiplying the two, the influence of units can be ignored, and only the numerical values ​​need to be calculated.

[0106] S52 performs data fusion processing on the urgency of transmission, data size, and modified compression rate of the same data to be transmitted at the breakpoint, and determines the transmission priority index of each data to be transmitted at the breakpoint.

[0107] It should be noted that the greater the urgency of transmitting a certain data interruption point, and the smaller the data volume and the smaller the compression ratio, the shorter the transmission time of the data interruption point and the shorter the decompression time in the cloud. This can avoid interruptions during transmission, and while supplementing the data required by the cloud task, it can also reduce the occurrence of multiple retransmissions of large compressed data due to network instability. The data interruption point to be transmitted should have a higher transmission priority.

[0108] As an exemplary implementation, the transmission priority index for each data interruption point to be transmitted is determined, including: The first step is to perform inverse proportional processing on the data size and the corrected compression ratio for each data point to be transmitted, so as to obtain the inverse proportional value between the data size and the corrected compression ratio of the data point to be transmitted.

[0109] The second step involves data fusion processing based on the urgency of transmitting the interrupted data, the data size, and the inverse proportional value of the compression rate to determine the transmission priority index for the interrupted data.

[0110] As an example, the formula for calculating the transmission priority index of the g-th interrupted data can be: In the formula, This represents the transmission priority index of the g-th interrupted data to be transmitted. This represents the maximum and minimum value normalization function. Since its precedence is generally a value of 1 or higher, The function is used to limit the value to between 0 and 10. This indicates the urgency of transmitting the data at the g-th interrupt point. This represents the size of the data at the g-th breakpoint to be transmitted. This represents the corrected compression ratio of the data at the g-th breakpoint to be transmitted.

[0111] It should be noted that under normal circumstances, the data volume and the compression ratio will not be zero. If there is an extreme case, a non-zero constant will be added to the denominator of the fraction, such as taking an empirical value of 0.01.

[0112] Referring to the calculation process of the transmission priority index of the g-th data to be transmitted, the transmission priority index of each data to be transmitted can be obtained.

[0113] S6 transmits the compressed data from each interruption point to the cloud server sequentially according to the transmission priority index.

[0114] In one embodiment, the data to be transmitted is first compressed according to the corrected compression ratio of each data point to be transmitted, resulting in compressed data points. The compressed data points in the edge gateway at the current moment are then transmitted to the cloud server in descending order of transmission priority.

[0115] For example, for compressed, interrupted data with the highest transmission priority, a transmission protocol can be selected as needed to encapsulate the data into a suitable transmission format. Upload parameters can then be configured to send the data to the cloud server via the selected protocol. These upload parameters can include endpoint information and authentication details.

[0116] Of course, SQL (Structured Query Language) queries can also be used to obtain the transmission priority indicators of compressed breakpoint data to be transmitted at different times in the edge gateway at the current moment, and then visualize them in the form of a table.

[0117] This completes the transmission of data from each breakpoint in the edge gateway at the current moment.

[0118] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for unified access of multi-source heterogeneous device data based on cloud-edge collaboration, characterized in that, The method comprises the following steps: obtaining each breakpoint data to be transmitted in a data uploading process of a multi-source heterogeneous device through an edge gateway at a current time point; determining a storage requirement degree of the edge gateway at each time point according to network transmission interference at each time point in the data uploading process and compression storage duration of the breakpoint data in the edge gateway; obtaining breakpoint data storage in the edge gateway at each time point and a historical storage requirement stability index, and determining a modified compression rate of each breakpoint data to be transmitted in combination with the storage requirement degree of the edge gateway at each time point; determining a task number of each data type in each breakpoint data to be transmitted and a usage proportion of each data type in a cloud processing task, and then determining a correlation degree of each breakpoint data to be transmitted and the cloud task according to the task number and the usage proportion; determining a transmission priority index of each breakpoint data to be transmitted according to the correlation degree, the data size and the modified compression rate of each breakpoint data to be transmitted and the cloud task; transmitting the compressed each breakpoint data to be transmitted to a cloud server in sequence according to the transmission priority index. 2.The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 1, characterized in that, The method of determining the storage requirement degree of the edge gateway at each time point according to the network transmission interference at each time point in the data uploading process and the compression storage duration of the breakpoint data in the edge gateway comprises: taking any time point in the data uploading process as a target time point, determining a network transmission interference degree of the target time point according to a data transmission interruption condition in a first preset time period containing the target time point; taking a time point closest to the target time point and at which breakpoint data appears as a breakpoint time point, determining a compression storage maintenance index of the breakpoint data corresponding to the target time point in the edge gateway according to a time interval between the breakpoint time point and the target time point; performing data fusion processing on the network transmission interference degree of the target time point and the compression storage maintenance index to obtain the storage requirement degree of the edge gateway at the target time point.

3. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 2, characterized in that, The method of determining the network transmission interference degree of the target time point according to the data transmission interruption condition in the first preset time period containing the target time point comprises: obtaining a number of interruptions in the first preset time period containing the target time point, and obtaining a maximum time interval corresponding to each two adjacent interruption time points in the first preset time period; the interruption time point is a time point at which breakpoint data appears in the first preset time period; performing inverse proportional processing on the maximum time interval to obtain an inverse proportional value, and performing fusion processing on the inverse proportional value and the number of interruptions to determine the network transmission interference degree of the target time point.

4. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 2, characterized in that, The method of performing data fusion processing on the network transmission interference degree of the target time point and the compression storage maintenance index to obtain the storage requirement degree of the edge gateway at the target time point comprises: obtaining the network transmission interference degree of a previous time point of the target time point, calculating a difference value between the network transmission interference degrees of the target time point and the previous time point thereof, and denoting the difference value as a network transmission interference degree difference value; determining a first storage requirement factor according to the network transmission interference degree of the target time point and the network transmission interference degree difference value; and taking the compression storage maintenance index of the target time point as a second storage requirement factor. The first storage demand factor and the second storage demand factor are subjected to data fusion processing to obtain a storage demand degree of the edge gateway at the target moment.

5. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 2, characterized in that, A historical storage demand stability index is obtained, including: A storage demand threshold is set, and a storage demand degree of the edge gateway at each moment within a second preset time period is obtained; the second preset time period is a preset time period located before the target moment; Moments corresponding to each storage demand degree greater than the storage demand threshold are taken as interference moments, and moments corresponding to each storage demand degree not greater than the storage demand threshold are taken as stable moments; According to a difference between the number of stable moments and the number of interference moments, a historical storage demand stability index of the edge gateway at the target moment is determined. 6.The cloud-edge collaboration based multi-source heterogeneous device data unified access method of claim 1, wherein, The determination of the modified compression rate of each to-be-transmitted breakpoint data includes: According to the breakpoint data storage amount in the edge gateway at each moment, the historical storage demand stability index, and the storage demand degree of the edge gateway at each moment, a modified compression coefficient of data in the edge gateway at each moment is determined; The moment of occurrence of each to-be-transmitted breakpoint data is obtained, and the modified compression coefficient at a moment closest to the moment of occurrence is taken as the modified compression coefficient of the corresponding to-be-transmitted breakpoint data; A preset compression rate is obtained, and the modified compression coefficient of each to-be-transmitted breakpoint data is used to modify the preset compression rate to determine the modified compression rate of each to-be-transmitted breakpoint data.

7. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 1, characterized in that, The determination of the relevance of each to-be-transmitted breakpoint data to cloud tasks according to the number of tasks and the usage proportion includes: The number of all tasks being processed in the cloud at the current moment and the maximum usage proportion are obtained; For each to-be-transmitted breakpoint data, a first correlation factor of the to-be-transmitted breakpoint data to cloud tasks is determined according to the proportion of the average value of the number of tasks of all data types of the to-be-transmitted breakpoint data in the number of all tasks; A second correlation factor of the to-be-transmitted breakpoint data to cloud tasks is determined according to the proportion of the average value of the usage proportion of all data types of the to-be-transmitted breakpoint data in the maximum usage proportion; The first correlation factor and the second correlation factor of the to-be-transmitted breakpoint data to cloud tasks are subjected to fusion processing to obtain the relevance of the to-be-transmitted breakpoint data to cloud tasks. 8.The cloud-edge collaboration based multi-source heterogeneous device data unified access method of claim 1, wherein, The determination of the transmission priority index of each to-be-transmitted breakpoint data according to the relevance of each to-be-transmitted breakpoint data to cloud tasks, the data amount, and the modified compression rate includes: The time interval between the moment of occurrence of each to-be-transmitted breakpoint data and the current moment is obtained, and the transmission urgency of each to-be-transmitted breakpoint data is determined in combination with the relevance of each to-be-transmitted breakpoint data to cloud tasks; The transmission urgency, the data amount, and the modified compression rate of the same to-be-transmitted breakpoint data are subjected to data fusion processing to determine the transmission priority index of each to-be-transmitted breakpoint data.

9. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 8, characterized in that, The determination of the transmission urgency of each to-be-transmitted breakpoint data includes: The ratio of the time interval of each to-be-transmitted breakpoint data to the maximum time interval is calculated as a first transmission urgency factor of the corresponding to-be-transmitted breakpoint data; The ratio of the data amount of each to-be-transmitted breakpoint data to the maximum data amount is calculated as a second transmission urgency factor of the corresponding to-be-transmitted breakpoint data; The ratio of the modified compression rate of each to-be-transmitted breakpoint data to the maximum modified compression rate is calculated as a third transmission urgency factor of the corresponding to-be-transmitted breakpoint data; The first transmission urgency factor, the second transmission urgency factor, and the third transmission urgency factor of the same to-be-transmitted breakpoint data are subjected to fusion processing to determine the transmission urgency of each to-be-transmitted breakpoint data. calculate a difference between the maximum correlation degree and the correlation degree of each to-be-transmitted breakpoint data as a second transmission urgency factor corresponding to the to-be-transmitted breakpoint data; perform data fusion processing on the first transmission urgency factor and the second transmission urgency factor of the same to-be-transmitted breakpoint data to determine a transmission urgency degree of each to-be-transmitted breakpoint data.

10. The cloud-edge collaboration based multi-source heterogeneous device data unified access method according to claim 8, characterized in that, The data fusion processing on the transmission urgency degree, the data size and the modified compression rate of the same to-be-transmitted breakpoint data to determine a transmission priority index of each to-be-transmitted breakpoint data comprises: respectively for each to-be-transmitted breakpoint data, inversely proportionally process the data size and the modified compression rate of the to-be-transmitted breakpoint data to obtain inversely proportional values of the data size and the modified compression rate of the to-be-transmitted breakpoint data; perform data fusion processing on the transmission urgency degree, the inversely proportional values of the data size and the modified compression rate of the to-be-transmitted breakpoint data to determine a transmission priority index of the to-be-transmitted breakpoint data.

Images