Data Information Transmission Method Based on Cloud-Edge Collaboration

Abstract

The present invention relates to the field of cloud-edge collaborative data processing technology, and specifically to a data information transmission method based on cloud-edge collaboration. The present invention first obtains historical data of text information data in the central cloud; further constructs a shared dictionary based on the historical data, and transmits the shared dictionary to the edge end as an edge dictionary; further based on the usage characteristics of real-time data for the target edge dictionary, combined with the compression effect score of the target edge end on the real-time data, the target edge dictionary is iteratively updated, the text information data is compressed and transmitted synchronously in real time. The present invention constructs a shared dictionary for each edge node, and dynamically adjusts the edge dictionary content according to the changes in the real-time data stream on the basis of the shared dictionary, so that the edge dictionary can meet the compression requirements of the edge end, while limiting the update frequency of the edge dictionary, thereby improving the stability of the cloud-edge collaborative system for real-time synchronous transmission of text information data.

Description

Technical Field

[0001] The present invention relates to the field of cloud-edge collaborative data processing technology, and in particular to a data information transmission method based on cloud-edge collaboration. Background Art

[0002] Cloud-edge collaboration is a distributed computing architecture that combines the advantages of cloud computing and edge computing. By deploying computing resources at the edge of the network, it performs preliminary processing of real-time data, reducing the workload of the central cloud and the pressure on data transmission. The current diversified development of networks has led to an increasing variety of data. The demand for processing more data has brought huge challenges to network load and transmission consumption, and has placed stricter requirements on network data.

[0003] During the real-time synchronous transmission of text information data in a cloud-edge collaborative system, the data is compressed at the edge and then transmitted to the central cloud. In the process of compressing data using existing dictionary compression algorithms, a dictionary needs to be constructed based on the text information data to achieve the purpose of compression. However, in the dictionary construction process, when the data differences between different edge ends are large, there will be large differences between the dictionaries. In addition, the text information data is constantly changing, and the dictionary of the edge end needs to be constantly updated when compressing the data, which easily generates a large number of compression dictionaries. A large number of compression dictionaries not only requires a large amount of storage space, but also consumes a large amount of transmission resources. If the same shared dictionary is used for compression and transmission, although dictionary updates and transmission are not required, the shared dictionary may no longer be suitable for text information data compression as the real-time data changes, resulting in the system transmitting and storing invalid text information data. In addition, an overly large dictionary will reduce the compression efficiency and cannot meet the needs of the cloud-edge collaborative system. Summary of the Invention

[0004] In order to solve the technical problem that when transmitting text information data based on existing dictionary-based compression algorithms, the unreasonable compression dictionary update method leads to excessive consumption of system resources, the purpose of the present invention is to provide a data information transmission method based on cloud-edge collaboration. The technical solutions adopted are as follows:

[0005] A data information transmission method based on cloud-edge collaboration, the method comprising:

[0006] Obtain historical data of text information data in the central cloud; construct a shared dictionary based on the data distribution characteristics of each type of data in the historical data; and transmit the current shared dictionary to the edge end as an edge dictionary;

[0007] Select any edge end as the target edge end, select the target edge dictionary corresponding to the target edge end; obtain real-time data of the target edge end; analyze the distribution characteristics of each data in the target edge dictionary based on the real-time data, and obtain the common index of each data in the target edge dictionary based on the difference in distribution characteristics between the target edge dictionary and the same data in the current shared dictionary; obtain the compression effect score of the target edge end on the real-time data using the target edge dictionary, and iteratively update the target edge dictionary based on the common index of each data in the target edge dictionary.

[0008] Furthermore, the method for constructing a shared dictionary includes:

[0009] Obtain the frequency of occurrence of each type of data in the historical data; obtain the preference of each type of data in the historical data based on the frequency of occurrence of each type of data in the historical data; the frequency of occurrence of each type of data is positively correlated with the preference of each type of data; filter data based on the preference to obtain the shared dictionary.

[0010] Furthermore, the method for obtaining the commonly used index includes:

[0011] Obtaining the frequency of occurrence of each data type in the target edge dictionary in the real-time data; obtaining a correction coefficient for each data type in the target edge dictionary based on a difference between the frequency of occurrence of each data type in the target edge dictionary and the frequency of occurrence of the same data type in the current shared dictionary;

[0012] According to the occurrence frequency of each data in the target edge dictionary and the correction coefficient of each data in the target edge dictionary, the common index of each data in the target edge dictionary is obtained; the occurrence frequency and the correction coefficient of each data in the target edge dictionary are both positively correlated with the common index.

[0013] Furthermore, the method for obtaining the correction coefficient includes:

[0014] Select any data in the target edge dictionary as target data; obtain a frequency difference parameter according to the difference between the appearance frequency of the target data in the target edge dictionary and the appearance frequency of the same data in the current shared dictionary;

[0015] Obtaining a frequency change rate of the target data in the target edge dictionary based on the frequency difference parameter corresponding to the target data and the time interval between the current acquisition time of the real-time data and the time of building the current shared dictionary;

[0016] When the frequency of occurrence of the target data in the target edge dictionary is greater than or equal to the frequency of occurrence in the current shared dictionary, obtaining the correction coefficient of the target data according to the frequency difference parameter and the rate of change of the frequency of occurrence; the frequency difference parameter and the rate of change of the frequency of occurrence are both positively correlated with the correction coefficient;

[0017] When the frequency of occurrence of the target data in the target edge dictionary is less than the frequency of occurrence in the current shared dictionary, the correction coefficient of the target data is obtained based on the frequency difference parameter and the frequency change rate of occurrence; the frequency difference parameter and the frequency change rate of occurrence are both negatively correlated with the correction coefficient.

[0018] Furthermore, the method for obtaining the compression effect score includes:

[0019] Obtain the memory usage of the real-time data after compression; obtain the compression time of the real-time data compression; obtain the compression effect score of the target edge end on the real-time data based on the memory usage and the compression time; the memory usage and the compression time are both negatively correlated with the compression effect score.

[0020] Furthermore, the method for iteratively updating the target edge dictionary includes:

[0021] Among the data currently in the target edge dictionary, when the common index is less than a first preset threshold and the compression effect score is less than a second preset threshold, deleting the corresponding data from the current target edge dictionary;

[0022] For data in the real-time data that is not in the current target edge dictionary, the preference corresponding to the data is obtained, and when the preference is greater than a third preset threshold, the corresponding data is added to the current target edge dictionary.

[0023] Furthermore, after iteratively updating the target edge dictionary, the method further includes:

[0024] Iteratively updating the shared dictionary and retransmitting it to the edge end according to similar features between the latest target edge dictionary and the current shared dictionary, thereby updating the target edge dictionary;

[0025] The latest target edge dictionary is used to compress text information data and transmit it synchronously in real time.

[0026] Furthermore, the method for iteratively updating the shared dictionary includes:

[0027] Acquire the data in the current shared dictionary that is identical to the current target edge dictionary as the data to be analyzed; and use the ratio of the amount of the data to be analyzed to the amount of all data in the current shared dictionary as a first similarity parameter;

[0028] Obtaining, at the target edge end, a frequency of occurrence of each of the to-be-analyzed data from a data set consisting of all data obtained between receiving the current shared dictionary and the current moment; and obtaining a second similarity parameter based on a concentrated feature of the frequencies of occurrence of all of the to-be-analyzed data.

[0029] According to the first similarity parameter and the second similarity parameter, obtaining an update determination parameter in the current shared dictionary and the current target edge dictionary; the first similarity parameter and the second similarity parameter are both positively correlated with the update determination parameter;

[0030] When the update judgment parameter is less than a fourth preset threshold, all text information data collected by all edge ends between the current shared dictionary update time and the current time are selected to form an updated data set; the priority of each data in the updated data set is obtained, and the data is filtered according to the priority to obtain the shared dictionary.

[0031] Furthermore, the method for obtaining the preference degree includes:

[0032] The frequency of occurrence of each data in the corresponding historical data is normalized to obtain the corresponding preference of each data.

[0033] Furthermore, the method of filtering data according to the priority and obtaining the shared dictionary includes:

[0034] All data with a preference greater than a preset preference threshold are selected as entries of the shared dictionary to obtain the shared dictionary.

[0035] The present invention has the following beneficial effects:

[0036] The present invention first obtains historical data of text information data in the central cloud to provide a data basis; further constructs a shared dictionary based on the historical data, and transmits the shared dictionary to the edge end as an edge dictionary to provide a basis for subsequent dictionary updates; further analyzes the distribution characteristics of each data in the target edge dictionary based on real-time data, and combines the differences in distribution characteristics of the same data in the target edge dictionary and the current shared dictionary to obtain a common index for each data in the target edge dictionary, reflecting the usage of real-time data for each dictionary entry in the target edge dictionary, and providing a basis for subsequent target edge dictionary updates; further obtains the compression effect score of the target edge end on the real-time data, and explains the adaptability of the target edge dictionary to real-time data from the perspective of compression effect. As a supplement to the common index, it provides more basis for the target edge dictionary update; further combines the common index and compression effect score of each data in the target edge dictionary to update the target edge dictionary to ensure that the target edge dictionary can meet the changes in real-time data, while limiting the update frequency of the target edge dictionary and reducing the dictionary's consumption of system resources. The present invention constructs a shared dictionary for each edge node, and dynamically adjusts the edge dictionary content according to changes in real-time data streams based on the shared dictionary, so that the edge dictionary can meet the compression requirements of the edge end, while limiting the update frequency of the edge dictionary and improving the stability of the cloud-edge collaborative system in real-time synchronous transmission of text information data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A flowchart of a data information transmission method based on cloud-edge collaboration provided by one embodiment of the present invention;

[0039] Figure 2 A flowchart of a method for obtaining a correction coefficient provided by one embodiment of the present invention;

[0040] Figure 3 A flowchart of iteratively updating a shared dictionary is provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, characteristics and effects of a data information transmission method based on cloud-edge collaboration proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0043] The following describes in detail a specific solution of a data information transmission method based on cloud-edge collaboration provided by the present invention with reference to the accompanying drawings.

[0044] See also Figure 1 , which shows a flow chart of a data information transmission method based on cloud-edge collaboration provided by an embodiment of the present invention, specifically including:

[0045] Step S1: Obtain historical data of text information data in the central cloud; construct a shared dictionary based on the data distribution characteristics of each type of data in the historical data; and transmit the current shared dictionary to the edge end as an edge dictionary.

[0046] In an embodiment of the present invention, it is first determined that text information data needs to be collected, which in one embodiment of the present invention at least includes text information data to be transmitted.

[0047] In one embodiment of the present invention, a shared dictionary is first constructed based on historical data, and the shared dictionary is transmitted to the edge end as an edge dictionary. Then, the edge dictionary is updated according to the real-time data collected by the edge end. Then, the edge dictionary is compared with the shared dictionary. When the difference between the edge dictionary and the shared dictionary is too large, the shared dictionary is updated and retransmitted to the edge end. The edge dictionary is updated, and the latest edge dictionary is used to compress the text information data and transmit it synchronously in real time, thereby improving the compression effect of the algorithm on the real-time data of the edge end, reducing the dictionary storage pressure caused by the difference in edge data, improving the compression capability and compression effect of all real-time data of the edge end, and ensuring that the text information data can be transmitted synchronously in real time in the cloud-edge collaborative system.

[0048] Considering that the distribution characteristics of data in historical data reflect the distribution law of data in historical data and the inherent characteristics and structure of historical data, an initial shared dictionary is constructed according to the data distribution characteristics of each data in historical data, providing a basis for subsequent dictionary updates.

[0049] Preferably, in one embodiment of the present invention, considering that the more frequently data appears, the more likely it is to reappear in the future, high-frequency data is placed in a shared dictionary, so that the shared dictionary can contain common data items, thereby using less space to represent these data during the compression process, and reducing the update frequency and transmission overhead of the dictionary.

[0050] Based on this, the frequency of occurrence of each data in the historical data is obtained; based on the frequency of occurrence of each data in the historical data, the preference of each data in the historical data is obtained; the frequency of occurrence of each data is positively correlated with the preference of each data; the data is filtered according to the preference to obtain a shared dictionary.

[0051] As an example, the frequency of occurrence of each data in the corresponding historical data is normalized and used as the priority corresponding to each data.

[0052] In another embodiment of the present invention, considering that different data occupy different memory resources, an overly large dictionary may increase memory usage and search time, so it is necessary to balance the frequency of data occurrence and memory usage.

[0053] As an example, the ratio of the frequency of occurrence of data in the corresponding historical data to the memory occupied is normalized and used as the priority corresponding to each data.

[0054] It should be noted that in one embodiment of the present invention, when a shared dictionary is first constructed, the historical data is all text information data before the shared dictionary is first constructed, and all data with a preference greater than a preset preference threshold is selected as entries in the shared dictionary to obtain the shared dictionary. As an example, the preset preference threshold is 0.5.

[0055] It should be noted that the normalization method is a technical means well known to those skilled in the art. In one embodiment of the present invention, all normalizations adopt the existing maximum and minimum value normalization method, which will not be described in detail here.

[0056] Step S2: Select any edge end as the target edge end, and select the target edge dictionary corresponding to the target edge end; obtain the real-time data of the target edge end; analyze the distribution characteristics of each data in the target edge dictionary based on the real-time data, and obtain the common index of each data in the target edge dictionary based on the difference in distribution characteristics between the target edge dictionary and the same data in the current shared dictionary; obtain the compression effect score of the target edge end on the real-time data using the target edge dictionary, and iteratively update the target edge dictionary based on the common index of each data in the target edge dictionary.

[0057] First, select any edge end as the target edge end, select the target edge dictionary corresponding to the target edge end, and analyze it one by one; obtain the real-time data of the target edge end to provide a basis for subsequent updating of the target edge dictionary.

[0058] Considering that the introduction of real-time data streams may bring new data items or change the attributes of existing data items, as real-time data is continuously updated, the dictionary also needs to be dynamically adjusted to ensure compression effect, reduce storage space requirements and energy consumption during transmission;

[0059] Taking into account that the usage of the target edge dictionary by real-time data may change continuously, the distribution characteristics of each data in the target edge dictionary are analyzed by real-time data in the target edge dictionary to reflect the usage of each dictionary entry in the target edge dictionary by real-time data; and considering that the target edge dictionary is continuously updated based on the shared dictionary, the difference in distribution characteristics of the same data in the target edge dictionary and the current shared dictionary can also reflect the usage of each dictionary entry in the target edge dictionary by real-time data, so the two are integrated to obtain the common index of each data in the target edge dictionary, which provides a basis for subsequent target edge dictionary updates.

[0060] Preferably, in one embodiment of the present invention, considering that the frequency of occurrence of each type of data in the target edge dictionary corresponding to the real-time data reflects the usage of each dictionary entry in the target edge dictionary by the real-time data, the frequency of occurrence of each type of data in the target edge dictionary in the real-time data is first obtained; considering the difference between the frequency of occurrence of the real-time data in the target edge dictionary and the frequency of occurrence of the same data in the current shared dictionary, the usage trend of the real-time data for the entries in the target edge dictionary can be reflected, thereby further correcting the frequency of occurrence of each type of data in the target edge dictionary corresponding to the real-time data to obtain a more accurate common index;

[0061] Based on this, the frequency of occurrence of each data in the target edge dictionary in the real-time data is obtained; according to the difference between the frequency of occurrence of each data in the target edge dictionary and the frequency of occurrence of the same data in the current shared dictionary, the correction coefficient of each data in the target edge dictionary is obtained.

[0062] According to the frequency of occurrence of each data in the target edge dictionary and the correction coefficient of each data in the target edge dictionary, the common index of each data in the target edge dictionary is obtained; considering the frequency of occurrence of each data in the target edge dictionary, it means that the greater the frequency of use of the real-time data for the corresponding data in the target edge dictionary, the more commonly used the corresponding data in the target edge dictionary is; and the larger the correction coefficient is, the more commonly used the change trend is, so the frequency of occurrence and correction coefficient of each data in the target edge dictionary are positively correlated with the common index.

[0063] As an example, the product of the occurrence frequency of each data corresponding to the real-time data in the target edge dictionary and the corresponding correction coefficient is normalized and used as the common index of each data in the target edge dictionary.

[0064] It should be noted that, in other embodiments of the present invention, the implementer may also adjust the sensitivity of the correction coefficient or the frequency of occurrence by using a positive correlation mapping function, for example, by using a natural constant Exponential function with base , taking the frequency of occurrence of data as the independent variable pass After function mapping, the sensitivity of the frequency of occurrence of data is amplified. When the frequency of occurrence of data changes, the impact on the commonly used index is greater.

[0065] It should be noted that when the real-time data does not contain a certain data in a certain dictionary, the occurrence frequency of this data in the corresponding dictionary is 0; when the shared dictionary does not contain a certain data in the target edge dictionary, the occurrence frequency of this data in the shared dictionary is 0.

[0066] It should be noted that when the target edge dictionary is updated, the shared dictionary may not be updated. In this case, the shared dictionary may not contain certain data in the target edge dictionary.

[0067] Preferably, in one embodiment of the present invention, the method for obtaining the correction coefficient includes:

[0068] See also Figure 2 , which shows a flow chart of a method for obtaining a correction coefficient provided by an embodiment of the present invention, specifically comprising:

[0069] Step S201: Select any data in the target edge dictionary as target data; obtain a frequency difference parameter according to the difference between the appearance frequency of the target data in the target edge dictionary and the appearance frequency of the same data in the current shared dictionary.

[0070] As an example, the absolute value of the difference between the frequency of occurrence of the target data in the target edge dictionary and the frequency of occurrence of the same data in the current shared dictionary is used as the frequency difference parameter. The absolute value of the difference between the different frequencies of occurrence is used to represent the difference between the frequency of occurrence of the target data in the target edge dictionary and the frequency of occurrence of the same data in the current shared dictionary, reflecting the intensity of the change in the frequency of occurrence and providing a basis for the subsequent acquisition of the correction coefficient.

[0071] It should be noted that the frequency of occurrence of target data in the target edge dictionary is the frequency of occurrence of target data in the target edge dictionary in real-time data; the frequency of occurrence of the same data in the shared dictionary is the frequency of occurrence of target data in historical data when constructing the shared dictionary.

[0072] Step S202: Obtain the occurrence frequency change rate of the target data in the target edge dictionary according to the frequency difference parameter corresponding to the target data and the time interval between the acquisition time of the current real-time data and the time of building the current shared dictionary.

[0073] As an example, the ratio of the frequency difference parameter to the time interval is used as the occurrence frequency change rate of the corresponding target data; wherein the time interval is not zero.

[0074] The frequency change rate is used to characterize the intensity of the change in the frequency of occurrence of data, providing more basis for the subsequent acquisition of the correction coefficient.

[0075] Step S203: When the frequency of occurrence of the target data in the target edge dictionary is greater than or equal to the frequency of occurrence in the current shared dictionary, a correction coefficient of the target data is obtained according to the frequency difference parameter and the occurrence frequency change rate; both the frequency difference parameter and the occurrence frequency change rate are positively correlated with the correction coefficient.

[0076] Considering that when the frequency of occurrence of the target data in the target edge dictionary is greater than or equal to the frequency of occurrence in the current shared dictionary, it means that the real-time data uses the dictionary entry corresponding to the target data in the target edge dictionary more frequently, the common index needs to be larger, and the correction coefficient needs to be larger; and the larger the frequency difference parameter is, the greater the frequency increase of the dictionary entry corresponding to the target data used by the real-time data is, and the greater the frequency change rate is, the greater the rate of increase of the frequency of the dictionary entry corresponding to the target data used by the real-time data is, so the frequency difference parameter and the frequency change rate are both positively correlated with the correction coefficient.

[0077] As an example, the product of the frequency difference parameter and the occurrence frequency change rate corresponding to the target data is added with a constant 1 to serve as the correction coefficient of the target data; at this time, the frequency difference parameter and the occurrence frequency change rate are both positively correlated with the correction coefficient, which conforms to the correlation relationship. At the same time, after adding the constant 1, the correction coefficient must be greater than or equal to 1, ensuring that the correction coefficient can play an amplifying role.

[0078] As another example, the product of the frequency difference parameter corresponding to the target data and the frequency change rate is mapped to a positive correlation exponential function with a base value greater than 1, for example, mapped to In the function, since the frequency difference parameter and the frequency change rate are both non-negative, they are mapped to The number after the function is greater than or equal to 1. If the target data in the real-time data appears more frequently than the target data in the shared dictionary, the common index of the target data is larger.

[0079] Step S204: When the frequency of occurrence of the target data in the target edge dictionary is less than the frequency of occurrence in the current shared dictionary, a correction coefficient of the target data is obtained according to the frequency difference parameter and the occurrence frequency change rate; both the frequency difference parameter and the occurrence frequency change rate are negatively correlated with the correction coefficient.

[0080] Considering that when the frequency of occurrence of the target data in the target edge dictionary is less than the frequency of occurrence in the current shared dictionary, it means that the frequency of real-time data using the dictionary entries corresponding to the target data in the target edge dictionary is smaller, showing a real-time situation in which the usage rate of the dictionary entries corresponding to the target data is reduced, the common index needs to be smaller, and the correction coefficient needs to play a corrective role in reducing the frequency of occurrence of the target data. Moreover, the larger the frequency difference parameter, the greater the frequency decrease of the dictionary entries corresponding to the target data used by the real-time data. At the same time, the greater the frequency change rate, the greater the rate of decrease of the frequency of the dictionary entries corresponding to the target data used by the real-time data. Therefore, the frequency difference parameter and the frequency change rate are negatively correlated with the correction coefficient.

[0081] As an example, add a constant of 1 to the product of the frequency difference parameter and the frequency change rate corresponding to the target data, then take the reciprocal of the sum and use the reciprocal as the correction coefficient for the target data. In this case, both the frequency difference parameter and the frequency change rate are negatively correlated with the correction coefficient, conforming to the correlation relationship. Furthermore, after adding the constant of 1 and taking the reciprocal, the correction coefficient is guaranteed to be less than or equal to 1, ensuring that the correction coefficient can effectively reduce the error.

[0082] As another example, after adding a constant 1 to the product of the frequency difference parameter corresponding to the target data and the frequency change rate, The function is used to map the data and the mapped value is used as the correction coefficient of the target data.

[0083] In an embodiment of the present invention, the compression effect of the target edge end on the real-time data is also taken into consideration, which also reflects the adaptability of the target edge dictionary to the real-time data. The better the compression effect, the better the adaptability of the dictionary to the real-time data, and the less need for dictionary update. Therefore, the compression effect score of the target edge end on the real-time data is obtained to characterize the compression effect of the dictionary of the target edge end on the real-time data.

[0084] Preferably, in one embodiment of the present invention, considering that the shorter the compression time used for real-time data compression, the less memory the compressed data compression package occupies, which means that the edge end has a better compression effect on the real-time data, so the memory occupancy after real-time data compression is obtained; the compression time of real-time data compression is obtained; based on the memory occupancy and compression time, the compression effect score of the target edge end on the real-time data is obtained; the memory occupancy and compression time are both negatively correlated with the compression effect score.

[0085] As an example, the reciprocal of the product of memory usage and compression time is used as the compression effect score.

[0086] As another example, the product of memory usage and compression time is used After the negative correlation mapping function performs mapping, the mapping value is used as the compression effect score.

[0087] It should be noted that, in one embodiment of the present invention, the dictionary used to compress real-time data is the latest real-time target edge dictionary of the target edge end, and the compression algorithm is the well-known LZ4 data compression algorithm.

[0088] In other embodiments of the present invention, the compression effect score can also be evaluated from the perspective of the accuracy of the compressed data, and the quality loss ratio after real-time compression can be obtained, and the compression effect score can be obtained together with the compression time and memory usage; wherein the quality loss ratio is negatively correlated with the compression effect score.

[0089] After obtaining the evaluation basis of the compression effect of the target edge end on the real-time data, and obtaining the common index representing the usage characteristics of the real-time data for each data in the target edge dictionary, the target edge dictionary can be updated based on the compression effect score and the common index of each data in the target edge dictionary to ensure that the target edge dictionary can meet the changes in real-time data.

[0090] Preferably, in one embodiment of the present invention, considering that the smaller the common index corresponding to a certain data in the target edge dictionary is, the less frequently the dictionary entry corresponding to the data in the target edge dictionary is used, the less likely it is to be used, and the more it needs to be removed from the dictionary to improve the compression efficiency of the dictionary; at the same time, the lower the compression effect score is, the lower the adaptability of the target edge dictionary to real-time data is, and the more the dictionary needs to be updated. In order to eliminate the dimensionality effect and limit the data range, the common index and the compression effect score are both normalized; for the data already in the current target edge dictionary, when the common index is less than a first preset threshold and the compression effect score is less than a second preset threshold, the corresponding data is deleted from the current target edge dictionary;

[0091] It is also taken into consideration that there may be data in the real-time data that is not included in the target edge dictionary. These data may need to be included in the target edge dictionary. At the same time, when constructing a corresponding shared dictionary, the data to be included in the dictionary is filtered by preference. Here, preference is also obtained for filtering. Therefore, for data in the real-time data that is not in the current target edge dictionary, the preference corresponding to the data is obtained. When the preference is greater than the third preset threshold, the corresponding data is added to the current target edge dictionary.

[0092] As an example, the first preset threshold is 0.3; the second preset threshold is 0.6; and the third preset threshold is 0.3.

[0093] It should be noted that, in one embodiment of the present invention, for data in real-time data that is not in the current target edge dictionary, when obtaining the corresponding preference of the data, the data set is the text information data at the target edge end between the second-to-last moment of updating the shared dictionary in the historical records and the moment of real-time data collection, and the preference of the data in real-time data that is not in the current target edge dictionary is calculated in this data set. By limiting the data set to text information data of a more recent time, it adapts to the constantly changing text information data, thereby better compressing the text information data and transmitting it synchronously, reducing resource consumption. For records of shared dictionary updates with only one time in the historical records, all historical data between the moments of real-time data collection are used as the data set.

[0094] In other embodiments of the present invention, the implementer can change the starting time of the data in the data set by adjusting the shared dictionary update time in the selected historical record, for example, adjusting the data set to the text information data at the target edge end between the time of the third-to-last shared dictionary update in the historical record and the time of real-time data collection.

[0095] Preferably, in one embodiment of the present invention, considering that the method of iteratively updating the edge dictionary based only on the real-time data of the edge end is too simple, and considering that as new real-time data is continuously collected and the target edge dictionary is gradually updated, the similar features between the target edge dictionary and the shared dictionary may become less and less obvious, so the shared dictionary can be iteratively updated and retransmitted to the edge end based on the similar features between the latest target edge dictionary and the current shared dictionary to update the target edge dictionary; more update methods are provided for the iterative update of the edge dictionary, and the computing resources of the central cloud are used to assist the edge end in updating the edge dictionary, thereby improving the flexibility of the system in updating the edge dictionary and enhancing the adaptability of the cloud-edge collaborative system.

[0096] Utilize the latest edge dictionary on the edge to compress text information data and transmit it synchronously in real time.

[0097] It should be noted that each edge dictionary update does not necessarily cause a shared dictionary update. When the shared dictionary does not need to be updated, the updated edge dictionary is re-integrated with real-time data for iterative update until the shared dictionary needs to be updated. Only then will the latest shared dictionary be transmitted to the edge as a way to update the edge dictionary. Each edge dictionary and shared dictionary has a unique version number to distinguish different versions of the dictionary.

[0098] Preferably, in one embodiment of the present invention, the method for iteratively updating the shared dictionary includes:

[0099] See also Figure 3, which shows a flowchart of iteratively updating a shared dictionary provided by an embodiment of the present invention, specifically including:

[0100] Step S301: obtaining data in the current shared dictionary that is identical to the current target edge dictionary as data to be analyzed; and taking the ratio of the amount of the data to be analyzed to the amount of all data in the current shared dictionary as a first similarity parameter.

[0101] Considering that the more data that is identical between the shared dictionary and the target edge dictionary, the more the shared dictionary can adapt to the changes in real-time data needs, the data that is identical to the current target edge dictionary in the current shared dictionary is obtained as the data to be analyzed; the ratio of the number of data to be analyzed to the total number of data in the current shared dictionary is used as the first similarity parameter; and the similarity characteristics between the shared dictionary and the target edge dictionary are measured from the perspective of dictionary term similarity.

[0102] Step S302: obtaining the occurrence frequency of each data to be analyzed from a data set consisting of all data obtained between receiving the current shared dictionary and the current moment at the target edge; obtaining a second similarity parameter based on the concentrated feature of the occurrence frequencies of all data to be analyzed.

[0103] Since the purpose is to analyze whether the shared dictionary can meet the compression requirements of new real-time data after being updated, the analysis is limited to the data set consisting of all data obtained from the target edge end between receiving the current shared dictionary and the current moment; considering that the frequency of occurrence of the data to be analyzed reflects the usage of the corresponding dictionary entries in the shared dictionary by the data to be analyzed, the more the frequency of occurrence of all data to be analyzed is concentrated in the high-frequency part, the higher the concentration frequency, which means that the data to be analyzed uses the shared dictionary more frequently, the higher the similarity between the shared dictionary and the target edge dictionary, and the less the shared dictionary needs to be updated.

[0104] As an example, the average value of the occurrence frequencies of all the data to be analyzed is used as the second similarity parameter.

[0105] As another example, the mode, median and mean of the occurrence frequencies of all the data to be analyzed are weighted and summed, and the sum value is used as the second similarity parameter; the weighted weights may be 0.3, 0.3 and 0.4.

[0106] Step S303: obtaining an update determination parameter of the current shared dictionary and the current target edge dictionary according to the first similarity parameter and the second similarity parameter; both the first similarity parameter and the second similarity parameter are positively correlated with the update determination parameter; and updating the shared dictionary according to the update determination parameter.

[0107] After obtaining the first similarity parameter and the second similarity parameter, the two can be combined to obtain an update determination parameter to provide a determination basis for updating the shared dictionary.

[0108] As an example, the product of the first similarity parameter and the second similarity parameter is normalized and used as the update determination parameter. The larger the first similarity parameter and the second similarity parameter, the larger the update determination parameter, which means that the shared dictionary does not need to be updated.

[0109] In other embodiments of the present invention, the implementer may also adjust the sensitivity of the first similarity parameter or the second similarity parameter by using a positive correlation mapping function, for example, by using a natural constant Exponential function with base The sensitivity of amplifying the first similarity parameter or the second similarity parameter is not described in detail here.

[0110] Preferably, in one embodiment of the present invention, considering that a larger update determination parameter indicates that the shared dictionary does not need to be updated, the update determination parameter is normalized. When the update determination parameter is less than a fourth preset threshold, it is determined that the shared dictionary needs to be updated, specifically including:

[0111] Selecting all text information data collected by all edge devices between the time the shared dictionary was updated and the current time to form an updated dataset helps the updated shared dictionary capture new data patterns and trends, eliminate interference from outdated data, and better improve compression. Obtaining the preference of each data type in the updated dataset, filtering the data based on the preference to obtain the shared dictionary. As an example, the fourth preset threshold is 0.7.

[0112] It should be noted that the method of obtaining the priority and using the priority to filter data and obtain the shared dictionary has been described in step S1 and will not be repeated here.

[0113] It should be noted that, in other embodiments of the present invention, all historical data before the current moment may also be selected to form an updated data set to reconstruct the shared dictionary.

[0114] It should be noted that, in one embodiment of the present invention, the real-time data recorded by the edge end is compressed using the LZ4 data compression algorithm based on the latest edge dictionary of each edge end in the system, and the compressed data is encapsulated into a format suitable for network transmission, which may include adding metadata such as packet headers, indexes, and checksums. Subsequently, an encryption algorithm such as AES is used to encrypt the encapsulated data, and a suitable network protocol such as MQTT, HTTP, or WebSocket is selected for data transmission based on the data characteristics and network environment. The central cloud service receives the data from the edge end, decrypts, decapsulates, and decompresses it, and then stores or further processes it as needed to complete the real-time synchronous transmission of text information data based on cloud-edge collaboration. The technologies involved are all technical means well known to those skilled in the art, and will not be elaborated here.

[0115] It should be noted that in another embodiment of the present invention, the implementer may also periodically update the shared dictionary using an edge dictionary version interval, thereby updating the edge dictionary. As an example, the shared dictionary is updated once every 100 versions of the edge dictionary at all edge ends are updated.

[0116] In summary, in order to solve the technical problem that when transmitting text information data based on existing dictionary-based compression algorithms, the unreasonable update method of the compression dictionary leads to excessive consumption of system resources, the present invention provides a data information transmission method based on cloud-edge collaboration. The present invention first obtains the historical data of the text information data in the central cloud; further constructs a shared dictionary based on the historical data, and transmits the shared dictionary to the edge end as an edge dictionary; further based on the usage characteristics of the target edge dictionary for real-time data, combined with the compression effect score of the target edge end on the real-time data, the target edge dictionary is updated, the text information data is compressed and transmitted synchronously in real time. The present invention constructs a shared dictionary for each edge node, and dynamically adjusts the edge dictionary content according to the changes in the real-time data stream on the basis of the shared dictionary, so that the edge dictionary can meet the compression requirements of the edge end, while limiting the update frequency of the edge dictionary, thereby improving the stability of the cloud-edge collaborative system for real-time synchronous transmission of text information data.

[0117] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A data information transmission method based on cloud-edge collaboration, characterized in that: The method comprises: Obtain historical data of text information data in the central cloud; construct a shared dictionary based on the data distribution characteristics of each type of data in the historical data; and transmit the current shared dictionary to the edge end as an edge dictionary; Select any edge end as the target edge end, select the target edge dictionary corresponding to the target edge end; obtain real-time data of the target edge end; analyze the distribution characteristics of each data in the target edge dictionary based on the real-time data, and obtain the common index of each data in the target edge dictionary based on the difference in distribution characteristics between the target edge dictionary and the same data in the current shared dictionary; obtain the compression effect score of the target edge end on the real-time data using the target edge dictionary, and iteratively update the target edge dictionary based on the common index of each data in the target edge dictionary.

2. The data information transmission method based on cloud-edge collaboration according to claim 1 is characterized in that: The method for constructing a shared dictionary includes: Obtain the frequency of occurrence of each type of data in the historical data; obtain the preference of each type of data in the historical data based on the frequency of occurrence of each type of data in the historical data; the frequency of occurrence of each type of data is positively correlated with the preference of each type of data; filter data based on the preference to obtain the shared dictionary.

3. The data information transmission method based on cloud-edge collaboration according to claim 2 is characterized in that: The method for obtaining the commonly used index includes: Obtaining the frequency of occurrence of each data type in the target edge dictionary in the real-time data; obtaining a correction coefficient for each data type in the target edge dictionary based on a difference between the frequency of occurrence of each data type in the target edge dictionary and the frequency of occurrence of the same data type in the current shared dictionary; According to the occurrence frequency of each data in the target edge dictionary and the correction coefficient of each data in the target edge dictionary, the common index of each data in the target edge dictionary is obtained; the occurrence frequency and the correction coefficient of each data in the target edge dictionary are both positively correlated with the common index.

4. The data information transmission method based on cloud-edge collaboration according to claim 3 is characterized in that: The method for obtaining the correction coefficient includes: Select any data in the target edge dictionary as target data; obtain a frequency difference parameter according to the difference between the appearance frequency of the target data in the target edge dictionary and the appearance frequency of the same data in the current shared dictionary; Obtaining a frequency change rate of the target data in the target edge dictionary based on the frequency difference parameter corresponding to the target data and the time interval between the current acquisition time of the real-time data and the time of building the current shared dictionary; When the frequency of occurrence of the target data in the target edge dictionary is greater than or equal to the frequency of occurrence in the current shared dictionary, obtaining the correction coefficient of the target data according to the frequency difference parameter and the rate of change of the frequency of occurrence; the frequency difference parameter and the rate of change of the frequency of occurrence are both positively correlated with the correction coefficient; When the frequency of occurrence of the target data in the target edge dictionary is less than the frequency of occurrence in the current shared dictionary, the correction coefficient of the target data is obtained based on the frequency difference parameter and the frequency change rate of occurrence; the frequency difference parameter and the frequency change rate of occurrence are both negatively correlated with the correction coefficient.

5. The data information transmission method based on cloud-edge collaboration according to claim 1 is characterized in that: The method for obtaining the compression effect score includes: Obtain the memory usage of the real-time data after compression; obtain the compression time of the real-time data compression; obtain the compression effect score of the target edge end on the real-time data based on the memory usage and the compression time; the memory usage and the compression time are both negatively correlated with the compression effect score.

6. The data information transmission method based on cloud-edge collaboration according to claim 2 is characterized in that: The method for iteratively updating the target edge dictionary includes: Among the data currently in the target edge dictionary, when the common index is less than a first preset threshold and the compression effect score is less than a second preset threshold, deleting the corresponding data from the current target edge dictionary; For data in the real-time data that is not in the current target edge dictionary, the preference corresponding to the data is obtained, and when the preference is greater than a third preset threshold, the corresponding data is added to the current target edge dictionary.

7. The data information transmission method based on cloud-edge collaboration according to claim 6 is characterized in that: After iteratively updating the target edge dictionary, the method further includes: Iteratively updating the shared dictionary and retransmitting it to the edge end according to similar features between the latest target edge dictionary and the current shared dictionary, thereby updating the target edge dictionary; The latest target edge dictionary is used to compress text information data and transmit it synchronously in real time.

8. The data information transmission method based on cloud-edge collaboration according to claim 7 is characterized in that: The method for iteratively updating the shared dictionary includes: Acquire the data in the current shared dictionary that is identical to the current target edge dictionary as the data to be analyzed; and use the ratio of the amount of the data to be analyzed to the amount of all data in the current shared dictionary as a first similarity parameter; Obtaining, at the target edge end, a frequency of occurrence of each of the to-be-analyzed data from a data set consisting of all data obtained between receiving the current shared dictionary and the current moment; and obtaining a second similarity parameter based on a concentrated feature of the frequencies of occurrence of all of the to-be-analyzed data. According to the first similarity parameter and the second similarity parameter, obtaining an update determination parameter in the current shared dictionary and the current target edge dictionary; the first similarity parameter and the second similarity parameter are both positively correlated with the update determination parameter; When the update judgment parameter is less than a fourth preset threshold, all text information data collected by all edge ends between the current shared dictionary update time and the current time are selected to form an updated data set; the priority of each data in the updated data set is obtained, and the data is filtered according to the priority to obtain the shared dictionary.

9. The data information transmission method based on cloud-edge collaboration according to claim 8 is characterized in that: The method for obtaining the preference degree includes: The frequency of occurrence of each data in the corresponding historical data is normalized to obtain the corresponding preference of each data.

10. The data information transmission method based on cloud-edge collaboration according to claim 9 is characterized in that: The method of filtering data according to the priority and obtaining the shared dictionary includes: All data with a preference greater than a preset preference threshold are selected as entries of the shared dictionary to obtain the shared dictionary.

Images