Cloud edge collaboration based multi-source heterogeneous device data unified access method

Abstract

The application relates to the technical field of data transmission, in particular to a multi-source heterogeneous device data unified access method based on cloud edge cooperation, which comprises the following steps: determining a storage demand degree according to network transmission interference conditions at each moment in a data uploading process and compression storage time length conditions of breakpoint data in an edge gateway; acquiring breakpoint data storage capacity in the edge gateway and historical storage demand stability indexes at each moment, combining the storage demand degree to determine a correction compression rate of each to-be-transmitted breakpoint data; determining a correlation degree of each to-be-transmitted breakpoint data according to the number of tasks and the proportion of the amount used; determining each transmission priority index according to the correlation degree, the data amount and the correction compression rate; and transmitting the compressed to-be-transmitted breakpoint data to a cloud server in sequence according to the transmission priority index. The application can enhance the unified access effect of multi-source heterogeneous device data by adaptively determining the transmission priority index of the breakpoint data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data transmission, and in particular to a multi-source heterogeneous device data unified access method based on cloud-edge collaboration. BACKGROUND

[0002] When the cloud-edge collaboration architecture matures, the unified access method based on cloud-edge collaboration emerges as the times require, which combines the powerful capabilities of centralized management, global optimization and AI training in the cloud with the real-time processing, low-latency response and data preprocessing capabilities on the edge side, and solves the pain points existing in the traditional access mode through collaborative cooperation. Among them, the ability to dynamically determine the data priority can truly release the potential of the cloud-edge collaboration architecture, and achieve the core goals of cost reduction and business agility.

[0003] The traditional method only judges the priority of the device breakpoint data according to the time of transmitting different device data to the edge gateway, but because the breakpoint data characteristics do not match the real-time demand, the network dynamics are not considered, and the demand degree of the cloud task for different breakpoint data is different, there is deviation in the priority evaluation of the breakpoint data in the multi-source heterogeneous device data unified access process. SUMMARY

[0004] In order to solve the technical problem of deviation in the priority evaluation of the breakpoint data in the multi-source heterogeneous device data unified access process, the purpose of the present application is to provide a multi-source heterogeneous device data unified access method based on cloud-edge collaboration, and the technical solution adopted is as follows:

[0005] One embodiment of the present application provides a multi-source heterogeneous device data unified access method based on cloud-edge collaboration, which comprises the following steps:

[0006] Obtain each to-be-transmitted breakpoint data of the multi-source heterogeneous device in the data uploading process through the edge gateway at the current time;

[0007] Determine the storage demand degree of the edge gateway at each time according to the network transmission interference at each time in the data uploading process and the compressed storage time length of the breakpoint data in the edge gateway;

[0008] Obtain the breakpoint data storage amount and the historical storage demand stability index of the edge gateway at each time, and determine the corrected compression rate of each to-be-transmitted breakpoint data in combination with the storage demand degree of the edge gateway at each time;

[0009] Determine the number of tasks of each data type in each to-be-transmitted breakpoint data and the usage proportion of each data type in the cloud processing task, and then determine the correlation degree of each to-be-transmitted breakpoint data and the cloud task according to the number of tasks and the usage proportion;

[0010] According to the relevance of each to-be-transmitted breakpoint data to the cloud task, the data size, and the correction compression rate, a transmission priority index of each to-be-transmitted breakpoint data is determined.

[0011] According to the transmission priority index, the compressed to-be-transmitted breakpoint data is transmitted to the cloud server in sequence.

[0012] Further, the storage demand degree of the edge gateway at each moment is determined according to the network transmission interference at each moment during the data uploading process and the compression storage duration of the breakpoint data at the edge gateway, including:

[0013] Taking any moment during the data uploading process as a target moment, the network transmission interference degree of the target moment is determined according to the data transmission interruption during a first preset time period containing the target moment.

[0014] Taking a moment closest to the target moment and at which the breakpoint data appears as a breakpoint moment, the compression storage maintenance index of the breakpoint data corresponding to the target moment at the edge gateway is determined according to the time interval between the breakpoint moment and the target moment.

[0015] The network transmission interference degree of the target moment and the compression storage maintenance index are subjected to data fusion processing to obtain the storage demand degree of the edge gateway at the target moment.

[0016] Further, the network transmission interference degree of the target moment is determined according to the data transmission interruption during a first preset time period containing the target moment, including:

[0017] The number of interruptions during the transmission of the first preset time period containing the target moment is obtained, and the maximum time interval corresponding to each two adjacent interruption moments within the first preset time period is obtained; the interruption moment is a moment at which breakpoint data appears within the first preset time period.

[0018] The maximum time interval is subjected to inverse proportional processing to obtain an inverse proportional value, and the inverse proportional value and the number of interruptions are subjected to fusion processing to determine the network transmission interference degree of the target moment.

[0019] Further, the network transmission interference degree of the target moment and the compression storage maintenance index are subjected to data fusion processing to obtain the storage demand degree of the edge gateway at the target moment, including:

[0020] The network transmission interference degree of a previous moment of the target moment is obtained, and a difference value between the network transmission interference degrees of the target moment and the previous moment thereof is calculated, which is denoted as a network transmission interference degree difference value.

[0021] determine a first storage demand factor according to the network transmission interference degree of the target moment and the network transmission interference degree difference; and take the compression storage maintenance index of the target moment as a second storage demand factor;

[0022] perform data fusion processing on the first storage demand factor and the second storage demand factor to obtain a storage demand degree of the edge gateway at the target moment.

[0023] Further, a historical storage demand stability index is obtained, including:

[0024] A storage demand threshold is set, and a storage demand degree of the edge gateway at each moment in a second preset period is obtained; the second preset period is a preset period located before the target moment;

[0025] Moments corresponding to each storage demand degree greater than the storage demand threshold are taken as interference moments, and moments corresponding to each storage demand degree not greater than the storage demand threshold are taken as stable moments;

[0026] According to the number difference between the stable moments and the interference moments, a historical storage demand stability index of the edge gateway at the target moment is determined.

[0027] Further, the determination of the correction compression rate of each to-be-transmitted breakpoint data includes:

[0028] According to the breakpoint data storage amount in the edge gateway at each moment, the historical storage demand stability index, and the storage demand degree of the edge gateway at each moment, a correction compression coefficient of data in the edge gateway at each moment is determined;

[0029] The occurrence moment of each to-be-transmitted breakpoint data is obtained, and the correction compression coefficient at a moment closest to the occurrence moment is taken as the correction compression coefficient of the corresponding to-be-transmitted breakpoint data;

[0030] A preset compression rate is obtained, the preset compression rate is corrected by using the correction compression coefficient of each to-be-transmitted breakpoint data, and the correction compression rate of each to-be-transmitted breakpoint data is determined.

[0031] Further, the determination of the correlation degree of each to-be-transmitted breakpoint data and the cloud task according to the number of tasks and the usage proportion includes:

[0032] The number of all tasks being processed in the cloud at the current moment and the maximum usage proportion are obtained;

[0033] For each to-be-transmitted breakpoint data, a first correlation factor of the to-be-transmitted breakpoint data and the cloud task is determined according to the proportion of the average value of the number of tasks of all data types of the to-be-transmitted breakpoint data in the number of all tasks.

[0034] determine a second correlation factor of the to-be-transmitted breakpoint data and the cloud task according to a proportion of an average value of a proportion of a usage of all data types of the to-be-transmitted breakpoint data in the maximum usage proportion in the maximum usage proportion;

[0035] fuse the first correlation factor and the second correlation factor of the to-be-transmitted breakpoint data and the cloud task to obtain a correlation degree of the to-be-transmitted breakpoint data and the cloud task.

[0036] Further, the determining of the transmission priority index of each to-be-transmitted breakpoint data according to the correlation degree, the data amount size and the modified compression rate of each to-be-transmitted breakpoint data comprises:

[0037] obtaining a time interval between an occurrence time of each to-be-transmitted breakpoint data and a current time, and determining a transmission urgency degree of each to-be-transmitted breakpoint data in combination with the correlation degree of each to-be-transmitted breakpoint data and the cloud task;

[0038] fusing the transmission urgency degree, the data amount size and the modified compression rate of the same to-be-transmitted breakpoint data to determine the transmission priority index of each to-be-transmitted breakpoint data.

[0039] Further, the determining of the transmission urgency degree of each to-be-transmitted breakpoint data comprises:

[0040] calculating a ratio of the time interval of each to-be-transmitted breakpoint data to a maximum time interval as a first transmission urgency factor of the corresponding to-be-transmitted breakpoint data;

[0041] calculating a difference between the maximum correlation degree and the correlation degree of each to-be-transmitted breakpoint data as a second transmission urgency factor of the corresponding to-be-transmitted breakpoint data;

[0042] fusing the first transmission urgency factor and the second transmission urgency factor of the same to-be-transmitted breakpoint data to determine the transmission urgency degree of each to-be-transmitted breakpoint data.

[0043] Further, the fusing of the transmission urgency degree, the data amount size and the modified compression rate of the same to-be-transmitted breakpoint data to determine the transmission priority index of each to-be-transmitted breakpoint data comprises:

[0044] respectively performing inverse proportional processing on the data amount size and the modified compression rate of each to-be-transmitted breakpoint data to obtain inverse proportional values of the data amount size and the modified compression rate of the to-be-transmitted breakpoint data;

[0045] The transmission urgency of the to-be-transmitted breakpoint data, the data size and the inverse proportional value of the modified compression rate are fused to determine a transmission priority index of the to-be-transmitted breakpoint data.

[0046] The present application has the following advantages:

[0047] The present application provides a cloud-edge collaborative multi-source heterogeneous device data unified access method, which can avoid task processing delay in the cloud by adaptively determining the modified compression rate of each to-be-transmitted breakpoint data, while ensuring the storage space of the breakpoint data in the edge gateway, which is conducive to determining the transmission priority of the breakpoint data more accurately; by adaptively determining the relevance of each to-be-transmitted breakpoint data and the cloud task, the relevance of the to-be-transmitted breakpoint data and the cloud processing task can be quantified, which is conducive to uploading the breakpoint data related to the processing task; the transmission priority determined by the relevance of the to-be-transmitted breakpoint data and the cloud task, the data size and the modified compression rate can effectively overcome the defects that the existing breakpoint data transmission priority evaluation does not consider the matching of breakpoint data characteristics and real-time demand, network dynamics and the demand degree of different breakpoint data by the cloud task, and improve the accuracy and reliability of the breakpoint data priority evaluation in the process of multi-source heterogeneous device data unified access; by adaptively determining the transmission priority index of each to-be-transmitted breakpoint data, the tasks in the cloud can be processed in time, the rate of cloud task processing is improved, and the unified access effect of multi-source heterogeneous device data is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0049] Figure 1 A flow chart of a cloud-edge collaborative multi-source heterogeneous device data unified access method provided by an embodiment of the present application;

[0050] Figure 2 An implementation flow chart of step S2 in the embodiment of the present application;

[0051] Figure 3 An implementation flow chart of step S5 in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to further clarify the technical means and effects taken by the present application to achieve the predetermined object of the application, the specific implementation, structure, features and effects of the technical solutions proposed according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0054] The application scenario to which the present application is directed can be:

[0055] Adaptive acquisition of the transmission priority of breakpoint data of multi-source heterogeneous devices can improve data transmission efficiency and system response speed. At the same time, combined with the breakpoint data processing mechanism, the data integrity and continuity can be ensured, and the overall performance of the system can be improved. However, the existing breakpoint data transmission priority is only quantitatively analyzed by the time of transmission to the edge gateway, which makes the transmission priority situation judgment accuracy low, which is not conducive to the unified transmission of multi-source heterogeneous device data.

[0056] One embodiment of the present application provides a multi-source heterogeneous device data unified access method based on cloud edge collaboration, as shown in Figure 1 The method comprises the following steps:

[0057] S1, obtaining each to-be-transmitted breakpoint data of the multi-source heterogeneous devices in the data uploading process through the edge gateway at the current time.

[0058] Here, the breakpoint data refers to the data position and state information successfully transmitted by the system and saved when the transmission fails due to network interruption or other reasons in the transmission process. Its core role is to allow the transmission task to continue from the breakpoint after recovery, rather than to start again.

[0059] Specifically, through the breakpoint continuation mechanism in the edge gateway at the current time, the successfully uploaded data block or file position, such as the breakpoint position and the number of uploaded bytes, is recorded in the data uploading process. For the breakpoint position, the breakpoint position can be recorded by using the file recording breakpoint position and the subsequent edge gateway will store the breakpoint position and the breakpoint data to the cache. Therefore, through the edge gateway at the current time, each to-be-transmitted breakpoint data of the multi-source heterogeneous devices in the data uploading process can be obtained.

[0060] The edge gateway is built-in with multiple protocol libraries, and the communication protocols of different devices are automatically parsed through a protocol parsing algorithm. The data of different protocols is converted into a unified format through a protocol conversion module, facilitating subsequent processing and transmission. The device attributes and data are recorded through a data management module to ensure data security and management.

[0061] It is worth noting that during the data uploading process, if the uploading fails, the unfinished part will be uploaded from the breakpoint position, which can effectively avoid repeated uploading.

[0062] It should be further noted that the present embodiment takes one edge gateway at the current time as an example to analyze the transmission priority of each breakpoint data corresponding to the edge gateway.

[0063] S2, according to the network transmission interference situation at each time during the data uploading process and the compression storage time length of the breakpoint data in the edge gateway, determine the storage requirement degree of the edge gateway at each time.

[0064] Here, the storage requirement degree refers to the ability and necessity of the edge gateway to store data locally at each time during the data uploading.

[0065] As an exemplary embodiment, the above step S2 can be implemented by the steps shown in the following table: Figure 2

[0066] S21, any time during the data uploading process is taken as a target time, and the network transmission interference degree of the target time is determined according to the data transmission interruption situation in the first preset time period containing the target time.

[0067] Here, the network transmission interference degree refers to the instability degree caused by the interruption frequency of the edge gateway data transmission. The interval experience value between adjacent times during the data uploading process of the multi-source heterogeneous device can be determined according to the specific application scenario and protocol characteristics.

[0068] The instability of the network can cause the interruption of data transmission, which in turn can affect the real-time requirement of applications such as video conferencing and online gaming due to the accumulation of a large amount of data. Therefore, in order to analyze the transmission interference situation of the network at each time during the data uploading process, the number of interruptions and the interval size within the time period can be determined. The interruption frequency can directly reflect the reliability of the network connection, and the larger the interruption frequency, the worse the reliability of the network connection; while the interval size can reflect the frequency of interruption, and the smaller the interval size, the more frequent the interruption, and the greater the network transmission interference.

[0069] As an exemplary embodiment, the network transmission interference degree of the target time is determined according to the data transmission interruption situation in the first preset time period containing the target time, which includes:​

[0070] In a first step, the number of interruptions of data transmission in a first preset time period containing the target time is obtained, and the maximum time interval corresponding to each two adjacent interruption times in the first preset time period is obtained.

[0071] Here, the first preset time period can be set to 60 minutes, specifically, a time period of 60 minutes calculated from the target time, or a time period of 60 minutes formed by obtaining the time length on both sides of the target time as the center, which contains the target time; the interruption time is the time when the breakpoint data appears in the first preset time period. The numerical size and setting method of the first preset time period can be set by the implementer according to the specific actual situation, which is not limited here.

[0072] In a second step, the maximum time interval is inversely proportional to the interruption number to obtain an inverse proportional value, and the inverse proportional value and the interruption number are fused to determine the network transmission interference degree of the target time.

[0073] The greater the number of interruptions of data transmission corresponding to the target time, the smaller the maximum time interval, the more unstable the network of the target time, and the more serious the data accumulation.

[0074] As an example, the calculation formula of the network transmission interference degree of the tth time in the data uploading process can be:

[0075] In the formula, the network transmission interference degree of the tth time, the number of interruptions of data transmission in the first preset time period containing the tth target time, the maximum time interval corresponding to each two adjacent interruption times in the first preset time period containing the tth target time, the inverse proportional value of the maximum time interval corresponding to each two adjacent interruption times in the first preset time period containing the tth target time.

[0076] Referring to the calculation process of the network transmission interference degree of the tth time, the network transmission interference degree of each time in the data uploading process can be obtained.

[0077] It is worth noting that in determining the network transmission interference degree, in addition to the interruption number and the interruption interval, other factors related to network instability, such as packet loss rate, delay jitter, or TCP retransmission rate, can also be combined.

[0078] S22, the time when the breakpoint data appears closest to the target time is taken as the breakpoint time, and the compression storage maintenance index of the breakpoint data corresponding to the target time in the edge gateway is determined according to the time interval between the breakpoint time and the target time.

[0079] In one embodiment, a time point of a last time when breakpoint data starts to appear is obtained relative to the target time point, and is recorded as a breakpoint time point; and then a time interval between the target time point and the breakpoint time point is determined, and a numerical value of the time interval is taken as the compression storage maintenance index of the breakpoint data corresponding to the target time point in the edge gateway.

[0080] S23, data fusion processing is performed on the network transmission interference degree and the compression storage maintenance index of the target time point to obtain a storage requirement degree of the edge gateway at the target time point.

[0081] The network transmission interference degree and the compression storage maintenance index are both large, which indicates that the network at the target time point has a large instability degree, and the compression storage of the breakpoint data in the edge gateway has been maintained for a long time, and the storage requirement of the edge gateway is large.

[0082] As an exemplary embodiment, the step S23 can be implemented by the following steps:

[0083] Firstly, a product of the network transmission interference degree and the compression storage maintenance index of the target time point is calculated.

[0084] Secondly, normalization processing is performed on the product of the network transmission interference degree and the compression storage maintenance index of the target time point, and a normalized value obtained is taken as the storage requirement degree of the edge gateway at the target time point.

[0085] In one embodiment, the maximum-minimum value normalization can be used to normalize the product, so that the value range of the storage requirement degree is limited to 0 to 1. Of course, the implementer can also use other normalization means, which are not specifically limited here

[0086] Preferably, the step 23 can also be implemented by the following steps:

[0087] Firstly, the network transmission interference degree of a previous time point of the target time point is obtained, and a difference value between the network transmission interference degree of the target time point and that of the previous time point is calculated, which is recorded as a network transmission interference degree difference value.

[0088] The edge layer needs to dynamically adjust the data storage strategy according to the network interference change. When the network interference degree significantly increases, the gateway needs to temporarily store more data to avoid data loss caused by network instability .

[0089] Here, the previous time point is one time point located before the target time point and adjacent to the target time point. The network transmission interference degree difference value reflects the stability change of the communication link. The larger the network transmission interference degree difference value is, the more unstable the change of the network communication link is, and the larger the storage requirement of the edge gateway is.

[0090] Secondly, according to the network transmission interference degree of the target time and the network transmission interference degree difference, a first storage demand factor is determined; and the compression storage maintenance index of the target time is taken as a second storage demand factor.

[0091] Here, the first storage demand factor is a storage index determined by analyzing the network instability of the target time, and the second storage demand factor is a storage index determined by analyzing the interval between the target time and the breakpoint time.

[0092] In one embodiment, the product of the network transmission interference degree and the network transmission interference degree difference is taken as the first storage demand factor.

[0093] For the first storage demand factor, the greater the network transmission interference degree of the target time and the greater than the network transmission interference degree of the previous time, the longer the edge gateway may need to maintain the data storage of different access devices, and the greater the storage necessity.

[0094] It should be noted that the network transmission interference degree difference has the possibility of being zero, in order to avoid the extreme case of the storage demand degree being zero, the network transmission interference degree difference is added with a non-zero constant before multiplication calculation. The non-zero constant can be 0.01 if an empirical value is taken.

[0095] Thirdly, the first storage demand factor and the second storage demand factor are data fusion processed to obtain the storage demand degree of the edge gateway at the target time.

[0096] In one embodiment, the product of the first storage demand factor and the second storage demand factor is calculated, and the product of the first storage demand factor and the second storage demand factor is normalized to obtain the normalized value as the storage demand degree of the edge gateway at the target time.

[0097] It should be noted that when the first storage demand factor and the second storage demand factor are fused, only the numerical value is considered, and the dimension influence is not considered. The normalization processing can be maximum minimum value normalization, which can limit the numerical value range of the storage demand degree to 0 to 1.

[0098] Referring to the calculation process of the storage demand degree of the edge gateway at the target time, the storage demand degree of the edge gateway at each time can be obtained.

[0099] S3, the breakpoint data storage amount and the historical storage demand stability index in the edge gateway at each time are obtained, and the storage demand degree of the edge gateway at each time is combined to determine the correction compression rate of each to-be-transmitted breakpoint data.

[0100] When the network is transmitting, if the compression rate of the breakpoint data stored in the edge gateway is too large, the cloud needs a larger decompression time to decompress the compressed breakpoint data, thereby causing a delay in the task processing in the cloud. To avoid the above problems while ensuring that the storage space of the breakpoint data in the edge gateway is small, it is necessary to compress and store different compression rates for different to-be-transmitted breakpoint data, that is, to determine the modified compression rate of each to-be-transmitted breakpoint data.

[0101] As an exemplary embodiment, the historical storage demand stability index in the edge gateway at each time during the data uploading process is obtained, including:

[0102] Firstly, a storage demand threshold is set, and the storage demand degree of the edge gateway at each time in a second preset time period is obtained.

[0103] Here, the second preset time period is a preset time period located before the target time, and the experience value is 180 minutes. The numerical size and setting method of the second preset time period can be set by the implementer according to the specific actual situation, which is not limited here.

[0104] In an embodiment, referring to the determination process of the storage demand degree of the edge gateway at the target time during the data uploading process, the storage demand degree of the edge gateway at each time in the second preset time period can be obtained.

[0105] Secondly, the time corresponding to each storage demand degree greater than the storage demand threshold is taken as an interference time, and the time corresponding to each storage demand degree not greater than the storage demand threshold is taken as a stable time.

[0106] In an embodiment, the value range of the storage demand degree is between 0 and 1, and the storage demand threshold can be set to 0.7. The implementer can set the size of the storage demand threshold according to the specific actual requirements.

[0107] Thirdly, according to the difference between the number of stable times and interference times, the historical storage demand stability index in the edge gateway at the target time is determined.

[0108] Here, the historical storage demand stability index refers to the time distribution of the storage demand degree in the past period, that is, the time distribution of the storage demand degree lower than the storage demand threshold in the past.

[0109] As an example, the calculation formula of the historical storage demand stability index in the edge gateway at the tth time can be:

[0110] In the formula, denotes the historical storage demand stability index in the edge gateway at the tth time, a number of stable time instants within a second preset time period, a number of interference time instants within the second preset time period.

[0111] In the calculation formula of the historical storage demand stability index, the greater the number of stable time instants than the number of interference time instants, the greater the historical storage demand stability index, and the less likely the accumulation of breakpoint data after the target time instant to increase greatly, so a larger compression rate can be used to compress the breakpoint data to a greater extent, so the historical storage demand stability index and the correction compression coefficient present a positive correlation.

[0112] As an exemplary embodiment, the correction compression rate of each to-be-transmitted breakpoint data is determined, including:

[0113] First, according to the breakpoint data storage amount in the edge gateway at each time instant, the historical storage demand stability index, and the storage demand degree of the edge gateway at each time instant, the correction compression coefficient of the data in the edge gateway at each time instant is determined.

[0114] As an example, the calculation formula of the correction compression coefficient of the data in the edge gateway at the tth time instant can be:

[0115] In the formula, the correction compression coefficient of the data in the edge gateway at the tth time instant, the breakpoint data storage amount in the edge gateway at the tth time instant, the storage demand degree of the edge gateway at the tth time instant, the historical storage demand stability index in the edge gateway at the tth time instant, the hyperbolic tangent function, which can limit the value to between -1 and 1.

[0116] In the calculation formula of the correction compression coefficient, the smaller the breakpoint data storage amount and the storage demand degree of the edge gateway, and the greater the historical storage demand stability index, the smaller the transmission accumulation when the data transmission is resumed, and the less likely the accumulation of breakpoint data after that to increase greatly, and thus the possibility of cloud task processing delay due to large compression rate and long decompression time is smaller, at which time a larger compression rate can be used to compress the breakpoint data to reduce the storage pressure of the breakpoint data in the edge gateway, and the correction compression coefficient is greater.

[0117] Second, the occurrence time of each to-be-transmitted breakpoint data is obtained, and the correction compression coefficient at the time instant closest to the occurrence time is taken as the correction compression coefficient of the corresponding to-be-transmitted breakpoint data.

[0118] After the modified compression coefficient of the data in the edge gateway at each moment in the data uploading process is determined, the modified compression coefficient of different to-be-transmitted breakpoint data needs to be determined, so as to facilitate subsequent transmission priority determination and compression processing of the to-be-transmitted breakpoint data. Each to-be-transmitted breakpoint data has a corresponding modified compression coefficient.

[0119] In the third step, the preset compression rate is obtained, the modified compression coefficient of each to-be-transmitted breakpoint data is used to modify the preset compression rate, and the modified compression rate of each to-be-transmitted breakpoint data is determined.

[0120] It should be noted that the greater the modified compression coefficient, the greater the compression rate of the edge gateway to the breakpoint data, and thus the modified compression rate of the to-be-transmitted breakpoint data can be determined, and the calculation formula can be:

[0121] In the formula, Yg represents the modified compression rate of the gth to-be-transmitted breakpoint data, Y represents the preset compression rate, and the experience value can be 0.4, Yg represents the modified compression coefficient of the gth to-be-transmitted breakpoint data.

[0122] Referring to the calculation process of the modified compression rate of the gth to-be-transmitted breakpoint data, the modified compression rate of each to-be-transmitted breakpoint data can be obtained.

[0123] In S4, the number of tasks of each data type in each to-be-transmitted breakpoint data and the usage proportion of each data type in the cloud processing task are determined, and then the correlation degree of each to-be-transmitted breakpoint data and the cloud task is determined according to the number of tasks and the usage proportion.

[0124] First of all, it should be noted that the cloud will use different data transmitted by the edge gateway for task analysis, in order to avoid delay of the tasks processed by the cloud, the breakpoint data related to the processing task should be uploaded in priority, so the correlation degree of each to-be-transmitted breakpoint data and the cloud task needs to be determined.

[0125] As an exemplary embodiment, the number of tasks of each data type in each to-be-transmitted breakpoint data and the usage proportion of each data type in the cloud processing task are determined, including:

[0126] In the first step, after the data transmission of the edge gateway to the cloud is restored, the number of tasks of each data type in the cloud processing each to-be-transmitted breakpoint data, and the number of all tasks currently being processed in the cloud are counted.

[0127] Second, by querying a specific table and using input grouping and summary data table, and summary formula, output new data table containing grouped data and calculated aggregate values, etc. operations, according to the DataType field, etc. Grouping statistics, indirectly calculate the proportion of each data type in the cloud processing task of each to-be-transmitted breakpoint data, and determine the maximum proportion of all proportion of use.

[0128] Wherein, the proportion of use is equal to the sum of the proportion of the same data type in the cloud processing of different tasks, and the proportion of use of each data type in the cloud processing of a single task is obtained by the prior art, which is not described here.

[0129] As an exemplary embodiment, the degree of relevance of each to-be-transmitted breakpoint data to the cloud task is determined according to the number of tasks and the proportion of use, which includes:

[0130] First, obtain the number of all tasks being processed in the cloud at the current time and the maximum proportion of use.

[0131] In one embodiment, the number of all tasks and the maximum proportion of use have been obtained when determining the number of tasks and the proportion of use, which are used to realize subsequent comparative analysis.

[0132] It should be noted that by analyzing the similarity of the number of tasks of the to-be-transmitted breakpoint data and the number of tasks being processed in the cloud, and the similarity of the proportion of use and the maximum proportion of use, the degree of relevance of the to-be-transmitted breakpoint data to the cloud task can be quantitatively determined.

[0133] Second, for each to-be-transmitted breakpoint data, according to the proportion of the average value of the number of tasks of all data types of the to-be-transmitted breakpoint data in all the number of tasks, determine the first correlation factor of the to-be-transmitted breakpoint data to the cloud task.

[0134] In one embodiment, the average value of the number of tasks of all data types of the to-be-transmitted breakpoint data is calculated first, which is denoted as the average value of the number of tasks; Then the ratio of the average value of the number of tasks to all the number of tasks is used as the first correlation factor of the to-be-transmitted breakpoint data to the cloud task.

[0135] Third, according to the proportion of the average value of the proportion of use of all data types of the to-be-transmitted breakpoint data in the maximum proportion of use, determine the second correlation factor of the to-be-transmitted breakpoint data to the cloud task.

[0136] In one embodiment, the average value of the proportion of use of all data types of the to-be-transmitted breakpoint data is calculated first, which is denoted as the average value of the proportion of use; Then the ratio of the average value of the proportion of use to the maximum proportion of use is used as the second correlation factor of the to-be-transmitted breakpoint data to the cloud task.

[0137] In another embodiment, the difference between the maximum usage ratio and the average usage ratio is calculated, the difference is inverted, and the inverted difference is taken as the second correlation factor of the to-be-transmitted breakpoint data and the cloud task. In general, the difference cannot be zero, and if there is an extreme case, a non-zero constant, such as 0.01, is added to the difference.

[0138] In a fourth step, the first correlation factor and the second correlation factor of the to-be-transmitted breakpoint data and the cloud task are fused to obtain the correlation degree of the to-be-transmitted breakpoint data and the cloud task.

[0139] In one embodiment, the product of the first correlation factor and the second correlation factor of the to-be-transmitted breakpoint data and the cloud task is calculated, and the product of the two correlation factors is taken as the correlation degree of the to-be-transmitted breakpoint data and the cloud task.

[0140] As an example, the calculation formula of the correlation degree of the gth to-be-transmitted breakpoint data and the cloud task can be:

[0141] In the formula, g represents the gth to-be-transmitted breakpoint data, and represents the correlation degree of the gth to-be-transmitted breakpoint data and the cloud task, represents the average number of tasks of all data types of the gth to-be-transmitted breakpoint data, and M represents the number of all tasks being processed in the cloud at the current moment, represents the first correlation factor of the gth to-be-transmitted breakpoint data and the cloud task, represents the maximum usage ratio, represents the usage ratio of each data type of the gth to-be-transmitted breakpoint data in processing the cth task in the cloud, represents the usage ratio of each data type of the gth to-be-transmitted breakpoint data in processing a task in the cloud, represents the average usage ratio of all data types of the gth to-be-transmitted breakpoint data in processing a task in the cloud, represents a non-zero constant, which is used to avoid the case that the denominator of the fraction is zero, represents the second correlation factor of the gth to-be-transmitted breakpoint data and the cloud task.

[0142] It should be noted that when the first correlation factor and the second correlation factor of the to-be-transmitted breakpoint data and the cloud task are both large, it indicates that the relationship between the to-be-transmitted breakpoint data and the task being processed in the cloud is closer. In order to ensure the timeliness of the task processing in the cloud, after the transmission of the breakpoint data is resumed, the compressed transmission breakpoint data should be preferentially uploaded.

[0143] Referring to the determination process of the correlation degree of the gth to-be-transmitted breakpoint data and the cloud task described above, the correlation degree of each to-be-transmitted breakpoint data and the cloud task can be obtained.

[0144] S5, determining the transmission priority index of each to-be-transmitted breakpoint data according to the relevance of each to-be-transmitted breakpoint data to the cloud task, the data size, and the correction compression rate.

[0145] Here, the greater the transmission priority index, the more likely the corresponding to-be-transmitted breakpoint data is to be transmitted preferentially.

[0146] As an exemplary embodiment, the above step S5 can be implemented by the steps shown in FIG. 5. Figure 3

[0147] S51, determining the transmission urgency of each to-be-transmitted breakpoint data according to the time interval between the occurrence time of each to-be-transmitted breakpoint data and the current time, and the relevance of each to-be-transmitted breakpoint data to the cloud task.

[0148] To ensure the continuity of data transmission, the breakpoint data acquired earlier in the edge gateway should be transmitted preferentially, so it is necessary to obtain the time interval between the occurrence time of each to-be-transmitted breakpoint data and the current time, and subsequently determine the transmission urgency of each to-be-transmitted breakpoint data according to the time interval between the occurrence time of each to-be-transmitted breakpoint data and the current time and the relevance of each to-be-transmitted breakpoint data to the cloud task.

[0149] Here, the transmission urgency refers to the urgency of uploading the to-be-transmitted breakpoint data to the cloud, and the earlier the acquisition time of the to-be-transmitted breakpoint data, and the higher the relevance to the cloud task, the greater the transmission urgency of the to-be-transmitted breakpoint data.

[0150] As an exemplary embodiment, determining the transmission urgency of each to-be-transmitted breakpoint data comprises:

[0151] First, calculating the ratio of the time interval of each to-be-transmitted breakpoint data to the maximum time interval as the first transmission urgency factor of the corresponding to-be-transmitted breakpoint data.

[0152] Second, calculating the difference between the maximum relevance and the relevance of each to-be-transmitted breakpoint data as the second transmission urgency factor of the corresponding to-be-transmitted breakpoint data.

[0153] As an example, the calculation formula of the second transmission urgency factor of the gth to-be-transmitted breakpoint data can be:

[0154] ; in the formula, denotes the second transmission urgency factor of the gth to-be-transmitted breakpoint data, denotes the maximum relevance, denotes the relevance of the gth to-be-transmitted breakpoint data, ​represents a non-zero constant, which is used to avoid the situation that the denominator of the fraction is zero, and its empirical value can be 0.01.

[0155] In the calculation formula of the second transmission urgency factor, The smaller the value is, the more the correlation degree of the gth to-be-transmitted breakpoint data fits the maximum correlation degree, the stronger the urgency of the gth to-be-transmitted breakpoint data for data transmission is, and the greater the second transmission urgency factor is.

[0156] Referring to the calculation process of the second transmission urgency factor of the gth to-be-transmitted breakpoint data, the second transmission urgency factors of all to-be-transmitted breakpoint data can be obtained.

[0157] In the third step, the first transmission urgency factor and the second transmission urgency factor of the same to-be-transmitted breakpoint data are fused to determine the transmission urgency degree of each to-be-transmitted breakpoint data.

[0158] In one embodiment, the first transmission urgency factor and the second transmission urgency factor of the same to-be-transmitted breakpoint data are multiplied to obtain a product as the transmission urgency degree of the corresponding to-be-transmitted breakpoint data.

[0159] It should be noted that the transmission urgency degree is essentially a degree value, so the first transmission urgency factor and the second transmission urgency factor are also degree values, and the units can not be considered when the two are multiplied, only the numerical values are calculated.

[0160] In the third step, the first transmission urgency factor and the second transmission urgency factor of the same to-be-transmitted breakpoint data are fused to determine the transmission urgency degree of each to-be-transmitted breakpoint data.

[0161] It should be noted that the greater the transmission urgency degree of a to-be-transmitted breakpoint data is, and the smaller the data size and the modified compression rate are, the shorter the transmission time of the to-be-transmitted breakpoint data is, and the shorter the decompression time in the cloud is, which can avoid interruption during transmission, and at the same time, the data required by the cloud task can be supplemented, and the situation of retransmission of large compressed data caused by network instability can be reduced, so the to-be-transmitted breakpoint data should have a greater transmission priority.

[0162] As an exemplary embodiment, determining the transmission priority index of each to-be-transmitted breakpoint data includes:

[0163] In the first step, for each to-be-transmitted breakpoint data, the data size and the modified compression rate of the to-be-transmitted breakpoint data are inversely proportional processed respectively to obtain the inverse proportional values of the data size and the modified compression rate of the to-be-transmitted breakpoint data.

[0164] Secondly, the transmission urgency, the data size and the inverse value of the modified compression rate of the to-be-transmitted breakpoint data are fused to determine the transmission priority index of the to-be-transmitted breakpoint data.

[0165] For example, the calculation formula of the transmission priority index of the gth to-be-transmitted breakpoint data can be as follows:

[0166] In the formula, the transmission priority index of the gth to-be-transmitted breakpoint data is represented by P(g), the transmission urgency of the gth to-be-transmitted breakpoint data is represented by U(g), the data size of the gth to-be-transmitted breakpoint data is represented by S(g), the modified compression rate of the gth to-be-transmitted breakpoint data is represented by C(g), the maximum minimum value normalization function is represented by F, the inverse value of the modified compression rate is represented by C'(g), the maximum value of the inverse value of the modified compression rate is represented by C'max, the minimum value of the inverse value of the modified compression rate is represented by C'min, the function for limiting the value between 0 and 10 is represented by L, and the constant is represented by a. The transmission priority index of the gth to-be-transmitted breakpoint data is represented by P(g), The maximum minimum value normalization function is represented by F, since the priority is generally a value greater than 1, and the inverse value of the modified compression rate is represented by C'(g). The function for limiting the value between 0 and 10 is represented by L, The transmission urgency of the gth to-be-transmitted breakpoint data is represented by U(g), The data size of the gth to-be-transmitted breakpoint data is represented by S(g), The modified compression rate of the gth to-be-transmitted breakpoint data is represented by C(g).

[0167] It should be noted that the data size and the modified compression rate are generally not zero, and if there is an extreme case, a non-zero constant is added at the denominator position of the fraction, such as 0.01.

[0168] According to the calculation process of the transmission priority index of the gth to-be-transmitted breakpoint data, the transmission priority index of each to-be-transmitted breakpoint data can be obtained.

[0169] S6, the compressed to-be-transmitted breakpoint data is transmitted to the cloud server in sequence according to the transmission priority index.

[0170] In one embodiment, the to-be-transmitted breakpoint data is compressed according to the modified compression rate of each to-be-transmitted breakpoint data to obtain compressed to-be-transmitted breakpoint data; and the compressed to-be-transmitted breakpoint data in the edge gateway at the current time is transmitted to the cloud server in sequence according to the transmission priority index from large to small.

[0171] For example, for the compressed to-be-transmitted breakpoint data with the largest transmission priority index, the transmission protocol can be selected according to the requirements, the data is encapsulated into a format suitable for transmission, and the upload parameters are configured to send the data to the cloud server through the selected protocol. The upload parameters can be an endpoint and authentication information.

[0172] Of course, the transmission priority index of the to-be-transmitted compressed breakpoint data corresponding to different occurrence times in the edge gateway at the current time can also be obtained by using the query statement of SQL (Structured Query Language), and visualized in the form of a table.

[0173] Thus, the embodiment completes the transmission of each breakpoint data in the edge gateway at the current time.

[0174] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration, characterized in that, Includes the following steps: The edge gateway at the current moment acquires data from various points of interruption during the data upload process of multi-source heterogeneous devices. Based on the network transmission interference at each moment during the data upload process and the compression and storage duration of the breakpoint data at the edge gateway, the storage requirements of the edge gateway at each moment are determined. Obtain the breakpoint data storage volume and historical storage demand stability index in the edge gateway at each time point, and determine the correction compression rate of each breakpoint data to be transmitted by combining the storage demand level of the edge gateway at each time point; Determine the number of tasks for each data type in each data point to be transmitted and the usage percentage of each data type in the cloud processing tasks, and then determine the correlation between each data point to be transmitted and the cloud tasks based on the number of tasks and the usage percentage. Based on the relevance of each data point to be transmitted to the cloud task, the data volume, and the corrected compression rate, determine the transmission priority index for each data point to be transmitted. The compressed data of each data point to be transmitted is transmitted to the cloud server in sequence according to the transmission priority index. This includes: first, compressing the data of each data point to be transmitted according to the corrected compression ratio to obtain the compressed data of each data point to be transmitted; then, for the compressed data of each data point to be transmitted in the edge gateway at the current moment, transmitting it to the cloud server in sequence according to the transmission priority index from large to small. Based on the network transmission interference at each moment during the data upload process and the compression storage duration of the interrupted data at the edge gateway, the storage requirement of the edge gateway at each moment is determined. This includes: taking any moment during the data upload process as the target moment, determining the network transmission interference level at the target moment based on the data transmission interruption situation during a first preset time period including the target moment; taking the moment closest to the target moment where the interrupted data occurs as the interrupted moment, determining the compression storage maintenance index of the interrupted data at the edge gateway corresponding to the target moment based on the time interval between the interrupted moment and the target moment; and performing data fusion processing on the network transmission interference level and compression storage maintenance index at the target moment to obtain the storage requirement of the edge gateway at the target moment. Determine the corrected compression ratio for each data point to be transmitted, including: determining the corrected compression coefficient of the data in the edge gateway at each time point based on the data storage volume of the data point at each time point, the historical storage demand stability index, and the storage demand level of the edge gateway at each time point; obtaining the occurrence time of each data point to be transmitted, and using the corrected compression coefficient at the time point closest to the occurrence time as the corrected compression coefficient of the corresponding data point to be transmitted; obtaining the preset compression ratio, and using the corrected compression coefficient of each data point to be transmitted to correct the preset compression ratio to determine the corrected compression ratio of each data point to be transmitted. Based on the relevance of each data point to be transmitted to the cloud task, the data size, and the modified compression ratio, the transmission priority indicators for each data point to be transmitted are determined. This includes: obtaining the time interval between the occurrence time of each data point to be transmitted and the current time; combining the relevance of each data point to be transmitted to the cloud task to determine the urgency of transmitting each data point to be transmitted; and performing data fusion processing on the urgency of transmitting the same data point to be transmitted, the data size, and the modified compression ratio to determine the transmission priority indicators for each data point to be transmitted.

2. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, Based on the data transmission interruption situation during a first preset time period including the target time, determine the degree of network transmission interference at the target time, including: Obtain the number of transmission interruptions during the first preset time period containing the target time, and obtain the maximum time interval between every two adjacent interruption times within the first preset time period; the interruption time is the moment when data breaks down within the first preset time period. The maximum time interval is inversely proportional to obtain an inverse proportional value. The inverse proportional value and the number of interruptions are then fused to determine the network transmission interference level at the target time.

3. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, Data fusion processing is performed on the network transmission interference level and compressed storage maintenance index at the target time to obtain the storage requirements of the edge gateway at the target time, including: Obtain the network transmission interference level of the time preceding the target time, calculate the difference between the network transmission interference level of the target time and the time preceding it, and record it as the network transmission interference level difference. The first storage requirement factor is determined based on the network transmission interference level at the target time and the difference between the network transmission interference level and the target time; and the compressed storage maintenance index at the target time is used as the second storage requirement factor. The first storage demand factor and the second storage demand factor are fused to obtain the storage demand level of the edge gateway at the target time.

4. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, Obtain historical storage demand stability metrics, including: Set a storage demand threshold and obtain the storage demand level of the edge gateway at each time point within a second preset time period; the second preset time period is a pre-set time period located before the target time. The moments corresponding to each level of storage demand exceeding the storage demand threshold are taken as disturbance moments, and the moments corresponding to each level of storage demand not exceeding the storage demand threshold are taken as stable moments. Based on the difference in the number of stable and disruptive moments, determine the stability index of historical storage demand in the edge gateway at the target moment.

5. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, The correlation between the data at each transmission breakpoint and the cloud task is determined based on the number of tasks and the usage percentage, including: Get the number of all tasks being processed in the cloud at the current moment and the maximum usage percentage; For each data point to be transmitted, the first correlation factor between the data point to be transmitted and the cloud task is determined based on the average number of tasks of all data types of the data point to be transmitted as a percentage of the total number of tasks. Based on the proportion of the average usage percentage of all data types of the breakpoint data to be transmitted in the maximum usage percentage, determine the second correlation factor between the breakpoint data to be transmitted and the cloud task. The first and second correlation factors of the data to be transmitted at the breakpoint and the cloud task are fused to obtain the correlation between the data to be transmitted at the breakpoint and the cloud task.

6. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, Determine the urgency of transmitting data at each interruption point, including: Calculate the ratio of the time interval of each data point to be transmitted to the maximum time interval, and use it as the first transmission urgency factor for the corresponding data point to be transmitted. The difference between the maximum correlation degree and the correlation degree of each data point to be transmitted is calculated and used as the second transmission urgency factor for the corresponding data point to be transmitted. Data fusion processing is performed on the first and second transmission urgency factors of the same data to be transmitted at the breakpoint to determine the transmission urgency of each data to be transmitted at the breakpoint.

7. The method for unified access to data from multiple heterogeneous devices based on cloud-edge collaboration according to claim 1, characterized in that, Data fusion processing is performed on the urgency, data size, and modified compression ratio of data at the same transmission breakpoint to determine the transmission priority indicators for each transmission breakpoint, including: For each data point to be transmitted, the data size and the corrected compression ratio are inversely proportional to each other to obtain the inverse proportional value between the data size and the corrected compression ratio of the data point to be transmitted. Data fusion processing is performed on the inverse proportion of the urgency of transmitting interrupted data, data size, and modified compression rate to determine the transmission priority index of interrupted data.

Images