Data information transmission method based on cloud edge collaboration

Abstract

The invention relates to the technical field of cloud edge collaboration data processing, in particular to a data information transmission method based on cloud edge collaboration. The method comprises the steps of firstly obtaining historical data of text information data of a central cloud; further constructing a shared dictionary based on historical data, and transmitting the shared dictionary to an edge end as an edge dictionary; and further based on the use characteristics of the real-time data for the target edge dictionary, combining with the compression effect score of the target edge end for the real-time data, carrying out iterative updating on the target edge dictionary, compressing the text information data and carrying out real-time synchronous transmission. According to the method, the shared dictionary of each edge end node is constructed, and the content of the edge dictionary is dynamically adjusted according to the change of the real-time data stream on the basis of the shared dictionary, so that the edge dictionary can meet the compression requirement of the edge end, and meanwhile, the updating frequency of the edge dictionary is limited; and the stability of real-time synchronous transmission of the character information data by the cloud edge coordination system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud-edge collaborative data processing, and particularly to a data information transmission method based on cloud-edge collaboration. Background Art

[0002] Cloud-edge collaboration is a distributed computing architecture that combines the advantages of cloud computing and edge computing. By deploying computing resources on the edge side of the network, it preliminarily processes real-time data, reducing the workload of the central cloud and the data transmission pressure. The current diversified development of the network has led to an increasing variety of data, and the processing requirements of a large amount of data pose a huge challenge to network load and transmission consumption. At this time, stricter requirements are put forward for network data.

[0003] During the real-time synchronous transmission in the cloud-edge collaborative system of text information data, the data is compressed at the edge side and the compressed data is transmitted to the central cloud. In the process of compressing the data using the existing dictionary-based compression algorithm, a dictionary needs to be constructed according to the text information data to achieve the purpose of compression. However, in the process of constructing the dictionary, when the data between different edge sides varies greatly, there will be a large difference between the dictionaries. Moreover, since the text information data is constantly changing, the dictionary for compressing data at the edge side needs to be updated continuously, which easily generates a large number of compression dictionaries. A large number of compression dictionaries not only require a large amount of storage space but also consume a large amount of transmission resources. If the same shared dictionary is used for compression transmission, although there is no need for dictionary update and transmission, the shared dictionary is very likely to become inapplicable to the compression of text information data as the real-time data changes, resulting in the system transmitting and storing invalid text information data. In addition, an overly large dictionary will lead to a reduction in compression efficiency and cannot meet the requirements of the cloud-edge collaborative system. Summary of the Invention

[0004] In order to solve the technical problem that the unreasonable update method of the compression dictionary leads to excessive consumption of system resources when transmitting text information data based on the existing dictionary-based compression algorithm, the purpose of the present invention is to provide a data information transmission method based on cloud-edge collaboration, and the specific technical solution adopted is as follows: A data information transmission method based on cloud-edge collaboration, the method includes: Obtain the historical data of the text information data of the central cloud; construct a shared dictionary according to the data distribution characteristics of each type of data in the historical data; and transmit the current shared dictionary to the edge side as the edge dictionary; Select any edge end as the target edge end, and select the target edge dictionary corresponding to the target edge end; obtain the real-time data of the target edge end; analyze the distribution characteristics of each type of data in the target edge dictionary based on the real-time data, and combine the distribution characteristic differences of the same data in the target edge dictionary and the current shared dictionary to obtain the usage index of each type of data in the target edge dictionary; obtain the compression effect score of the real-time data by the target edge end using the target edge dictionary, and iteratively update the target edge dictionary in combination with the usage index of each type of data in the target edge dictionary.

[0005] Further, the method for constructing the shared dictionary includes: Obtain the occurrence frequency of each type of data in the historical data; according to the occurrence frequency of each type of data in the historical data, obtain the preference degree of each type of data in the historical data; the occurrence frequency of each type of data is positively correlated with the preference degree of each type of data; screen the data according to the preference degree to obtain the shared dictionary.

[0006] Further, the method for obtaining the usage index includes: Obtain the occurrence frequency of each type of data in the target edge dictionary in the real-time data; according to the difference between the occurrence frequency of each type of data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary, obtain the correction coefficient of each type of data in the target edge dictionary; According to the occurrence frequency of each type of data in the target edge dictionary, in combination with the correction coefficient of each type of data in the target edge dictionary, obtain the usage index of each type of data in the target edge dictionary; both the occurrence frequency and the correction coefficient of each type of data in the target edge dictionary are positively correlated with the usage index.

[0007] Further, the method for obtaining the correction coefficient includes: Select any data in the target edge dictionary as the target data; according to the difference between the occurrence frequency of the target data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary, obtain the frequency difference parameter; According to the frequency difference parameter corresponding to the target data, in combination with the time interval between the acquisition time of the current real-time data and the time of constructing the current shared dictionary, obtain the occurrence frequency change rate of the target data in the target edge dictionary; When the occurrence frequency of the target data in the target edge dictionary is greater than or equal to the occurrence frequency in the current shared dictionary, obtain the correction coefficient of the target data according to the frequency difference parameter and the occurrence frequency change rate; both the frequency difference parameter and the occurrence frequency change rate are positively correlated with the correction coefficient; When the occurrence frequency of the target data in the target edge dictionary is less than that in the current shared dictionary, obtain the correction coefficient of the target data according to the frequency difference parameter and the occurrence frequency change rate; both the frequency difference parameter and the occurrence frequency change rate are negatively correlated with the correction coefficient.

[0008] Further, the method for obtaining the compression effect score includes: Obtain the memory occupancy after compressing the real-time data; obtain the compression time of the real-time data compression; according to the memory occupancy and the compression time, obtain the compression effect score of the target edge device for the real-time data; both the memory occupancy and the compression time are negatively correlated with the compression effect score.

[0009] Further, the method for iteratively updating the target edge dictionary includes: Among the existing data in the current target edge dictionary, when the common index is less than the first preset threshold and the compression effect score is less than the second preset threshold, delete the corresponding data in the current target edge dictionary; For the data in the real-time data that is not in the current target edge dictionary, obtain the corresponding preference degree, and when the preference degree is greater than the third preset threshold, add the corresponding data to the current target edge dictionary.

[0010] Further, after iteratively updating the target edge dictionary, it further includes: According to the similarity features of the latest target edge dictionary and the current shared dictionary, iteratively update the shared dictionary and re-transmit it to the edge device to update the target edge dictionary; Use the latest target edge dictionary to compress text information data and perform real-time synchronous transmission.

[0011] Further, the method for iteratively updating the shared dictionary includes: Obtain the data in the current shared dictionary that is the same as the current target edge dictionary as the data to be analyzed; take the proportion of the number of the data to be analyzed in all the data in the current shared dictionary as the first similarity parameter; In the data set composed of all the data obtained by the target edge device from receiving the current shared dictionary to the current moment, obtain the occurrence frequency of each data to be analyzed; according to the concentration characteristics of the occurrence frequencies of all the data to be analyzed, obtain the second similarity parameter; Obtain the update determination parameter of the current target edge dictionary in the current shared dictionary according to the first similarity parameter and the second similarity parameter; both the first similarity parameter and the second similarity parameter are positively correlated with the update determination parameter. When the update determination parameter is less than the fourth preset threshold, select all the text information data collected by all edge devices between the current shared dictionary update time and the current time to form an update data set; obtain the preference degree of each data in the update data set, screen the data according to the preference degree, and obtain the shared dictionary.

[0012] Further, the method for obtaining the preference degree includes: After normalizing the occurrence frequency of each data in the corresponding historical data, use it as the preference degree corresponding to each data.

[0013] Further, the method for screening data according to the preference degree and obtaining the shared dictionary includes: Select all the data with preference degrees greater than the preset preference threshold as the entries of the shared dictionary to obtain the shared dictionary.

[0014] The present invention has the following beneficial effects: The present invention first obtains the historical data of the text information data in the central cloud, providing a data basis; further constructs a shared dictionary based on the historical data and transmits the shared dictionary to the edge device as an edge dictionary, providing a basis for subsequent dictionary updates; further analyzes the distribution characteristics of each data in the target edge dictionary based on real-time data, combines the distribution characteristic differences of the same data in the target edge dictionary and the current shared dictionary, obtains the usage index of each data in the target edge dictionary, reflecting the usage of each dictionary entry in the target edge dictionary by real-time data, providing a basis for subsequent target edge dictionary updates; further obtains the compression effect score of the target edge device for real-time data, explaining the adaptability of the target edge dictionary to real-time data from the perspective of compression effect, as a supplementary item to the usage index, providing more bases for target edge dictionary updates; further combines the usage index and compression effect score of each data in the target edge dictionary to update the target edge dictionary to ensure that the target edge dictionary can meet the changes of real-time data, while restricting the update frequency of the target edge dictionary and reducing the consumption of system resources by the dictionary. The present invention constructs a shared dictionary for each edge device node, and dynamically adjusts the content of the edge dictionary according to the changes of the real-time data stream on the basis of the shared dictionary, so that the edge dictionary can meet the compression requirements of the edge device, while restricting the update frequency of the edge dictionary and improving the stability of the cloud-edge collaborative system for real-time synchronous transmission of text information data. Description of the Drawings

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 Flowchart of a data information transmission method based on cloud-edge collaboration provided by an embodiment of the present invention; Figure 2 Flowchart of a method for obtaining a correction coefficient provided by an embodiment of the present invention; Figure 3 Flowchart of iteratively updating a shared dictionary provided by an embodiment of the present invention. Detailed implementation manners

[0017] To further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in combination with the drawings and preferred embodiments, details the specific implementation manners, structures, features, and effects of a data information transmission method based on cloud-edge collaboration proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0019] The following specifically describes the specific solution of a data information transmission method based on cloud-edge collaboration provided by the present invention in combination with the drawings.

[0020] Please refer to Figure 1 , which shows the flowchart of a data information transmission method based on cloud-edge collaboration provided by an embodiment of the present invention, specifically including: Step S1: Obtain the historical data of the text information data of the central cloud; construct a shared dictionary according to the data distribution characteristics of each type of data in the historical data; and transmit the current shared dictionary to the edge side as an edge dictionary.

[0021] In the embodiment of the present invention, first, it is determined that text information data needs to be collected. In one embodiment of the present invention, it at least includes; text information data to be transmitted.

[0022] In an embodiment of the present invention, first, a shared dictionary is constructed based on historical data, and the shared dictionary is transmitted to the edge side as an edge dictionary. Then, the edge dictionary is updated according to the real-time data collected by the edge side. Next, the edge dictionary is compared with the shared dictionary. When the difference between the edge dictionary and the shared dictionary is too large, the shared dictionary is updated and re-transmitted to the edge side to update the edge dictionary. Using the latest edge dictionary, the text information data is compressed and synchronously transmitted in real time, thereby improving the compression effect of the algorithm on the real-time data of the edge side, reducing the dictionary storage pressure caused by the data difference of the edge side, improving the compression ability and compression effect on the real-time data of all edge sides, and ensuring that the text information data can be synchronously transmitted in the cloud-edge collaboration system in real time.

[0023] Considering that the data distribution characteristics in historical data reflect the distribution law of data in historical data and the internal characteristics and structure of historical data, an initial shared dictionary is constructed according to the data distribution characteristics of each type of data in historical data, providing a basis for subsequent dictionary updates.

[0024] Preferably, in an embodiment of the present invention, considering that the higher the occurrence frequency of data, the more likely it is to reappear in the future, putting high-frequency data into the shared dictionary can make the shared dictionary contain common data items, so that these data can be represented with less space during the compression process and reduce the dictionary update frequency and transmission overhead.

[0025] Based on this, obtain the occurrence frequency of each type of data in historical data; according to the occurrence frequency of each type of data in historical data, obtain the preference degree of each type of data in historical data; the occurrence frequency of each type of data is positively correlated with the preference degree of each type of data; screen data according to the preference degree to obtain the shared dictionary.

[0026] As an example, after normalizing the occurrence frequency of each type of data in the corresponding historical data, it is used as the preference degree corresponding to each type of data.

[0027] In another embodiment of the present invention, it is also considered that different data occupy different memory resources, and an overly large dictionary may increase memory usage and search time, so it is necessary to balance the occurrence frequency of data and memory occupancy.

[0028] As an example, after normalizing the ratio of the occurrence frequency of data in the corresponding historical data to the occupied memory, it is used as the preference degree corresponding to each type of data.

[0029] It should be noted that, in an embodiment of the present invention, when initially constructing the shared dictionary, the historical data is all the text information data before the initial construction of the shared dictionary. Select all data with a preference degree greater than the preset preference threshold as the entries of the shared dictionary to obtain the shared dictionary. As an example, the preset preference threshold is 0.5.

[0030] It should be noted that the normalization method is already a well-known technical means in the art. In an embodiment of the present invention, all normalizations adopt the existing maximum-minimum normalization method, and will not be elaborated here.

[0031] Step S2: Select any edge end as the target edge end, and select the target edge dictionary corresponding to the target edge end; obtain the real-time data of the target edge end; analyze the distribution characteristics of each type of data in the target edge dictionary based on the real-time data, and combine the distribution characteristic differences of the same data in the target edge dictionary and the current shared dictionary to obtain the common index of each type of data in the target edge dictionary; obtain the compression effect score of the real-time data by the target edge dictionary using the target edge dictionary, and iteratively update the target edge dictionary in combination with the common index of each type of data in the target edge dictionary.

[0032] First, select any edge end as the target edge end, and select the target edge dictionary corresponding to the target edge end for convenient analysis one by one; obtain the real-time data of the target edge end, which provides a basis for the subsequent update of the target edge dictionary.

[0033] Considering that the introduction of real-time data streams may bring new data items or change the attributes of existing data items, as the real-time data is continuously updated, the dictionary also needs to be dynamically adjusted to ensure the compression effect, reduce the storage space requirements and energy consumption during the transmission process; Considering that the usage situation of real-time data for the target edge dictionary may change continuously, so analyze the distribution characteristics of each type of data in the target edge dictionary through the real-time data, which reflects the usage situation of the real-time data for each dictionary entry in the target edge dictionary; and considering that the target edge dictionary is continuously updated on the basis of the shared dictionary, so the distribution characteristic differences of the same data in the target edge dictionary and the current shared dictionary can also reflect the usage situation of the real-time data for each dictionary entry in the target edge dictionary. Therefore, fuse the two to jointly obtain the common index of each type of data in the target edge dictionary, providing a basis for the subsequent update of the target edge dictionary.

[0034] Preferably, in an embodiment of the present invention, considering that the occurrence frequency of each type of data corresponding to the real-time data in the target edge dictionary reflects the usage situation of the real-time data for each dictionary entry in the target edge dictionary, so first obtain the occurrence frequency of each type of data in the target edge dictionary in the real-time data; considering that the difference between the occurrence frequency of the real-time data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary can reflect the usage change trend of the real-time data for the entries in the target edge dictionary, thereby further correcting the occurrence frequency of each type of data corresponding to the real-time data in the target edge dictionary to obtain a more accurate common index; Based on this, obtain the occurrence frequency of each type of data in the target edge dictionary in the real-time data; according to the difference between the occurrence frequency of each type of data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary, obtain the correction coefficient of each type of data in the target edge dictionary.

[0035] According to the occurrence frequency of each type of data in the target edge dictionary, combined with the correction coefficient of each type of data in the target edge dictionary, obtain the common index of each type of data in the target edge dictionary; considering the occurrence frequency of each type of data in the target edge dictionary, it shows that the greater the usage frequency of the real-time data for the corresponding data in the target edge dictionary, the more common the corresponding data in the target edge dictionary; and the greater the correction coefficient, it shows that the change trend presents a more common trend. Therefore, both the occurrence frequency and the correction coefficient of each type of data in the target edge dictionary are positively correlated with the common index.

[0036] As an example, after normalizing the product of the occurrence frequency and the corresponding correction coefficient of each type of data corresponding to the real-time data in the target edge dictionary, it is used as the common index of each type of data in the target edge dictionary.

[0037] It should be noted that in other embodiments of the present invention, the implementer can also adjust the sensitivity of the correction coefficient or the occurrence frequency through a positive correlation mapping function. For example, through the exponential function with the natural constant as the base, taking the occurrence frequency of the data as the independent variable after mapping through the function, the sensitivity of the occurrence frequency of the data is amplified, and when the occurrence frequency of the data changes, the impact on the common index is greater.

[0038] It should be noted that when the real-time data does not contain a certain data in a certain dictionary, the occurrence frequency of this data in the corresponding dictionary is 0; when the shared dictionary does not contain a certain data in the target edge dictionary, the occurrence frequency of this data in the shared dictionary is 0.

[0039] It should be noted that when the target edge dictionary is updated, the shared dictionary may not be updated. At this time, the shared dictionary may not contain a certain data in the target edge dictionary.

[0040] Preferably, in an embodiment of the present invention, the method for obtaining the correction coefficient includes: Please refer to Figure 2 , which shows a flowchart of a method for obtaining a correction coefficient provided by an embodiment of the present invention, specifically including: Step S201: Select any data in the target edge dictionary as the target data; according to the difference between the occurrence frequency of the target data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary, obtain the frequency difference parameter.

[0041] As an example, the absolute value of the difference between the occurrence frequency of the target data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary is used as the frequency difference parameter. By using the absolute value of the difference in different occurrence frequencies, the difference between the occurrence frequency of the target data in the target edge dictionary and the occurrence frequency of the same data in the current shared dictionary is represented, reflecting the change intensity of the occurrence frequency, and providing a basis for obtaining the correction coefficient in the subsequent steps.

[0042] It should be noted that the occurrence frequency of the target data in the target edge dictionary is the occurrence frequency of the target data in the target edge dictionary in the real-time data; the occurrence frequency of the same data in the shared dictionary is the occurrence frequency of the target data in the historical data when the shared dictionary is constructed.

[0043] Step S202: According to the frequency difference parameter corresponding to the target data, combined with the time interval between the acquisition time of the current real-time data and the time when the current shared dictionary is constructed, obtain the change rate of the occurrence frequency of the target data in the target edge dictionary.

[0044] As an example, the ratio of the frequency difference parameter to the time interval is used as the change rate of the occurrence frequency of the corresponding target data; where the time interval is not zero.

[0045] The change intensity of the occurrence frequency of the data is characterized by the change rate of the occurrence frequency, providing more basis for obtaining the correction coefficient in the subsequent steps.

[0046] Step S203: When the occurrence frequency of the target data in the target edge dictionary is greater than or equal to the occurrence frequency in the current shared dictionary, obtain the correction coefficient of the target data according to the frequency difference parameter and the change rate of the occurrence frequency; both the frequency difference parameter and the change rate of the occurrence frequency are positively correlated with the correction coefficient.

[0047] Considering that when the occurrence frequency of the target data in the target edge dictionary is greater than or equal to the occurrence frequency in the current shared dictionary, it indicates that the real-time data uses the dictionary entry corresponding to the target data in the target edge dictionary more frequently. The more frequently used index needs to be larger, and the correction coefficient needs to be larger; and the larger the frequency difference parameter, the more the frequency of using the dictionary entry corresponding to the target data in the real-time data increases. At the same time, the larger the change rate of the occurrence frequency, the greater the rate of increase in the frequency of using the dictionary entry corresponding to the target data in the real-time data. Therefore, both the frequency difference parameter and the change rate of the occurrence frequency are positively correlated with the correction coefficient.

[0048] As an example, the product of the frequency difference parameter corresponding to the target data and the occurrence frequency change rate is added with a constant 1, and the result is used as the correction coefficient of the target data. At this time, both the frequency difference parameter and the occurrence frequency change rate are positively correlated with the correction coefficient, meeting the correlation relationship. At the same time, after adding the constant 1, the correction coefficient must be greater than or equal to 1, ensuring that the correction coefficient can play a magnifying role.

[0049] As another example, the product of the frequency difference parameter corresponding to the target data and the occurrence frequency change rate is mapped into a positive correlation exponential function with a base greater than 1. For example, it is mapped into the function. Since both the frequency difference parameter and the occurrence frequency change rate are non-negative numbers, after being mapped into the function, the result is a number greater than or equal to 1, corresponding to the feature that the occurrence frequency of the target data in the real-time data is more frequent than that of the target data in the shared dictionary, and the common index of the target data is larger.

[0050] Step S204: When the occurrence frequency of the target data in the target edge dictionary is less than that in the current shared dictionary, obtain the correction coefficient of the target data according to the frequency difference parameter and the occurrence frequency change rate. Both the frequency difference parameter and the occurrence frequency change rate are negatively correlated with the correction coefficient.

[0051] Considering that when the occurrence frequency of the target data in the target edge dictionary is less than that in the current shared dictionary, it indicates that the real-time data uses the dictionary entry corresponding to the target data in the target edge dictionary less frequently, showing the real-time situation of the reduced usage rate of the dictionary entry corresponding to the target data. The common index needs to be smaller, and the correction coefficient needs to play a role in reducing the occurrence frequency of the target data. Moreover, the larger the frequency difference parameter, the more the frequency of the real-time data using the dictionary entry corresponding to the target data decreases, and at the same time, the larger the occurrence frequency change rate, the greater the rate of decrease in the frequency of the real-time data using the dictionary entry corresponding to the target data. Therefore, both the frequency difference parameter and the occurrence frequency change rate are negatively correlated with the correction coefficient.

[0052] As an example, the product of the frequency difference parameter corresponding to the target data and the occurrence frequency change rate is added with a constant 1, and then the sum value is taken as the reciprocal, and the reciprocal is used as the correction coefficient of the target data. At this time, both the frequency difference parameter and the occurrence frequency change rate are negatively correlated with the correction coefficient, meeting the correlation relationship. At the same time, after adding the constant 1 and then taking the reciprocal, the correction coefficient must be less than or equal to 1, ensuring that the correction coefficient can play a reducing role.

[0053] As another example, the product of the frequency difference parameter corresponding to the target data and the occurrence frequency change rate is added with a constant 1, and then mapped through the function, and the mapped value is used as the correction coefficient of the target data.

[0054] In the embodiments of the present invention, considering that the compression effect of the target edge device on real-time data also reflects the fitness of the target edge dictionary for real-time data, the better the compression effect, the better the adaptability of the dictionary to real-time data, and the less the need for dictionary update. Therefore, the compression effect score of the target edge device on real-time data is obtained to represent the compression effect of the dictionary of the target edge device on real-time data.

[0055] Preferably, in an embodiment of the present invention, considering that the shorter the compression time used for compressing real-time data, the less memory the compressed data packet occupies, indicating that the compression effect of the edge device on real-time data is better. Therefore, the memory occupancy after compressing real-time data is obtained; the compression time of real-time data compression is obtained; according to the memory occupancy and the compression time, the compression effect score of the target edge device on real-time data is obtained; both the memory occupancy and the compression time are negatively correlated with the compression effect score.

[0056] As an example, the reciprocal of the product of the memory occupancy and the compression time is used as the compression effect score.

[0057] As another example, after mapping the product of the memory occupancy and the compression time using a negatively correlated mapping function, the mapped value is used as the compression effect score.

[0058] It should be noted that, in an embodiment of the present invention, the dictionary used for compressing real-time data is the real-time latest target edge dictionary of the target edge device, and the compression algorithm is the known LZ4 data compression algorithm.

[0059] In other embodiments of the present invention, the compression effect score can also be evaluated from the perspective of the accuracy rate of the compressed data. The quality loss ratio after real-time compression is obtained, and together with the compression time and the memory occupancy, the compression effect score is obtained; wherein the quality loss ratio is negatively correlated with the compression effect score.

[0060] After obtaining the evaluation basis for the compression effect of the target edge device on real-time data, and obtaining the common index representing the usage characteristics of each type of data in the target edge dictionary for real-time data, the target edge dictionary can be updated by combining the compression effect score and the common index of each type of data in the target edge dictionary to ensure that the target edge dictionary can meet the changes in real-time data.

[0061] Preferably, in an embodiment of the present invention, considering that the smaller the common index corresponding to a certain data in the target edge dictionary, the lower the frequency of use of the dictionary entry corresponding to the certain data in the target edge dictionary, the less likely it is to be used, and the more necessary it is to remove it from the dictionary to improve the compression efficiency of the dictionary; at the same time, the lower the compression effect score, the lower the fitness of the target edge dictionary to real-time data, and the more necessary it is to update the dictionary. To eliminate the influence of dimension and limit the data range, both the common index and the compression effect score are normalized. For the data existing in the current target edge dictionary, when the common index is less than the first preset threshold and the compression effect score is less than the second preset threshold, the corresponding data is deleted from the current target edge dictionary; It is also considered that there may be data in the real-time data that is not included in the target edge dictionary, and these data may need to be incorporated into the target edge dictionary. At the same time, when constructing the shared dictionary correspondingly, the data incorporated into the dictionary is screened by the preference degree. Here, the preference degree is also obtained for screening. Therefore, for the data in the real-time data that is not in the current target edge dictionary, the preference degree corresponding to the data is obtained. When the preference degree is greater than the third preset threshold, the corresponding data is added to the current target edge dictionary.

[0062] As an example, the first preset threshold is 0.3; the second preset threshold is 0.6; the third preset threshold is 0.3.

[0063] It should be noted that, in an embodiment of the present invention, for the data in the real-time data that is not in the current target edge dictionary, when obtaining the preference degree corresponding to the data, the data set is the text information data of the target edge end between the moment of the penultimate update of the shared dictionary in the historical record and the moment of real-time data acquisition. The preference degree of the data in the real-time data that is not in the current target edge dictionary is calculated in this data set. By limiting the data set to the text information data in a relatively recent time to adapt to the constantly changing text information data, the text information data can be better compressed and synchronously transmitted, reducing resource consumption. For the historical record with only one record of updating the shared dictionary, all the historical data between the moment of real-time data acquisition is used as the data set.

[0064] In other embodiments of the present invention, the implementer can change the start time of the data in the data set by adjusting the selected shared dictionary update time in the historical record. For example, the data set is adjusted to the text information data of the target edge end between the moment of the third-to-last update of the shared dictionary in the historical record and the moment of real-time data acquisition.

[0065] Preferably, in an embodiment of the present invention, considering that the method of iteratively updating the edge dictionary only based on the real-time data at the edge side is too single, and considering that as new real-time data is continuously collected and the target edge dictionary is gradually updated, the similar features between the target edge dictionary and the shared dictionary may become less and less obvious. Therefore, the shared dictionary can be iteratively updated according to the similar features between the latest target edge dictionary and the current shared dictionary, and then re-transmitted to the edge side to update the target edge dictionary; this provides more update methods for the iterative update of the edge dictionary, and uses the computing resources of the central cloud to assist the edge side in updating the edge dictionary, improving the flexibility of the system to update the edge dictionary and enhancing the adaptability of the cloud-edge collaborative system.

[0066] Use the latest edge dictionary at the edge side to compress the text information data and synchronize it in real time.

[0067] It should be noted that each update of the edge dictionary does not necessarily cause an update of the shared dictionary; when the shared dictionary does not need to be updated, the updated edge dictionary is re-iteratively updated in combination with the real-time data until the shared dictionary needs to be updated, and then the latest shared dictionary is transmitted to the edge side as the update method of the edge dictionary; each edge dictionary and shared dictionary has a unique version number to distinguish different versions of the dictionary.

[0068] Preferably, in an embodiment of the present invention, the method for iteratively updating the shared dictionary includes: Please refer to Figure 3 , which shows a flowchart of a method for iteratively updating a shared dictionary provided by an embodiment of the present invention, specifically including: Step S301: Obtain the data in the current shared dictionary that is the same as the current target edge dictionary as the data to be analyzed; take the ratio of the number of the data to be analyzed to the number of all data in the current shared dictionary as the first similarity parameter.

[0069] Considering that the more data in the shared dictionary is the same as that in the target edge dictionary, it means that the shared dictionary can better adapt to the changing needs of real-time data. Therefore, obtain the data in the current shared dictionary that is the same as the current target edge dictionary as the data to be analyzed; take the ratio of the number of the data to be analyzed to the number of all data in the current shared dictionary as the first similarity parameter; measure the similar features between the shared dictionary and the target edge dictionary from the perspective of the similarity of dictionary entries.

[0070] Step S302: In the data set obtained from all the data received at the target edge side from the time of receiving the current shared dictionary to the current time, obtain the occurrence frequency of each data to be analyzed; according to the concentration characteristics of the occurrence frequencies of all the data to be analyzed, obtain the second similarity parameter.

[0071] Since the purpose is to analyze whether the updated shared dictionary can meet the compression requirements of new real-time data, the analysis is limited to the data set composed of all the data obtained from the reception of the current shared dictionary to the current moment at the target edge side. Considering that the occurrence frequency of the data to be analyzed reflects the usage of the corresponding dictionary entries in the shared dictionary, the more concentrated the occurrence frequencies of all the data to be analyzed are in the high-frequency part and the higher the concentration frequency is, the more frequently the data to be analyzed is used in the shared dictionary, the higher the similarity between the shared dictionary and the target edge dictionary, and the less the shared dictionary needs to be updated.

[0072] As an example, the average value of the occurrence frequencies of all the data to be analyzed is used as the second similarity parameter.

[0073] As another example, the mode, median and average value of the occurrence frequencies of all the data to be analyzed are weighted and summed, and the sum value is used as the second similarity parameter; the weighting weights can be 0.3, 0.3 and 0.4.

[0074] Step S303: Obtain the update determination parameter of the current shared dictionary and the current target edge dictionary according to the first similarity parameter and the second similarity parameter; both the first similarity parameter and the second similarity parameter are positively correlated with the update determination parameter; update the shared dictionary according to the update determination parameter.

[0075] After obtaining the first similarity parameter and the second similarity parameter, the two can be fused to obtain the update determination parameter, providing a determination basis for the update of the shared dictionary.

[0076] As an example, the product of the first similarity parameter and the second similarity parameter is normalized and used as the update determination parameter. The larger the first similarity parameter and the second similarity parameter are, the larger the update determination parameter is, indicating that the shared dictionary needs to be updated less.

[0077] In other embodiments of the present invention, the implementer can also adjust the sensitivity of the first similarity parameter or the second similarity parameter through a positive correlation mapping function, for example, by using the exponential function with the natural constant as the base to amplify the sensitivity of the first similarity parameter or the second similarity parameter, which will not be elaborated here.

[0078] Preferably, in an embodiment of the present invention, considering that the larger the update determination parameter is, the less the shared dictionary needs to be updated, after the update determination parameter is normalized, when the update determination parameter is less than the fourth preset threshold, it is determined that the shared dictionary needs to be updated, specifically including: Select all the text information data collected by all edge devices between the current shared dictionary update time and the current time to form an update data set, which helps the updated shared dictionary capture new data patterns and trends, exclude the interference of outdated data, and better improve the compression effect; obtain the preference of each type of data in the update data set, and filter the data according to the preference to obtain the shared dictionary. As an example, the fourth preset threshold is 0.7.

[0079] It should be noted that the method of obtaining the preference and using the preference to filter the data to obtain the shared dictionary has been described in step S1, and will not be elaborated here.

[0080] It should be noted that in other embodiments of the present invention, all historical data before the current time can also be selected to form an update data set to reconstruct the shared dictionary.

[0081] It should be noted that in an embodiment of the present invention, according to the latest edge dictionary of each edge device in the system, the LZ4 data compression algorithm is used to compress the real-time data recorded by the edge device, and the compressed data is encapsulated into a format suitable for network transmission, which may include adding metadata such as packet headers, indexes, and checksums. Subsequently, an encryption algorithm such as AES is used to encrypt the encapsulated data, and a suitable network protocol such as MQTT, HTTP, or WebSocket is selected according to the data characteristics and network environment for data transmission. The central cloud service receives the data from the edge device, decrypts, unpacks, and decompresses it, and then stores or further processes it as needed to complete the real-time synchronous transmission of text information data based on cloud-edge collaboration. The technologies involved are all well-known technical means to those skilled in the art and will not be elaborated here.

[0082] It should be noted that in another embodiment of the present invention, the implementer can also adopt the method of the edge dictionary version interval to update the shared dictionary regularly, thereby updating the edge dictionary. As an example, whenever the edge dictionaries of all edge devices are updated by 100 versions, the shared dictionary is updated once.

[0083] In summary, in view of the technical problem that when transmitting text information data based on a dictionary-based compression algorithm, the unreasonable update method of the compression dictionary leads to excessive consumption of system resources, the present invention provides a data information transmission method based on cloud-edge collaboration. The present invention first obtains historical data of text information data in the central cloud; further constructs a shared dictionary based on the historical data, and transmits the shared dictionary to the edge side as an edge dictionary; further updates the target edge dictionary based on the usage characteristics of the target edge dictionary for real-time data, combined with the compression effect score of the target edge side for real-time data, compresses the text information data and synchronously transmits it in real time. By constructing a shared dictionary for each edge-side node, and dynamically adjusting the content of the edge dictionary according to the changes in the real-time data stream on the basis of the shared dictionary, the present invention enables the edge dictionary to meet the compression requirements of the edge side, while restricting the update frequency of the edge dictionary, and improving the stability of the cloud-edge collaboration system for real-time synchronous transmission of text information data.

[0084] It should be noted that the above sequence of embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A data information transmission method based on cloud-edge collaboration, characterized in that, The method includes: Obtaining historical data of text information data from the central cloud; constructing a shared dictionary according to the data distribution characteristics of each type of data in the historical data; transmitting the current shared dictionary to the edge side as an edge dictionary. Selecting any edge side as the target edge side and selecting the target edge dictionary corresponding to the target edge side; obtaining the real-time data of the target edge side; analyzing the distribution characteristics of each type of data in the target edge dictionary based on the real-time data, and combining the distribution characteristic differences of the same type of data between the target edge dictionary and the current shared dictionary to obtain the common index of each type of data in the target edge dictionary; obtaining the compression effect score of the real-time data by the target edge side using the target edge dictionary, and iteratively updating the target edge dictionary in combination with the common index of each type of data in the target edge dictionary.

2. The data information transmission method based on cloud-edge collaboration according to claim 1, wherein The method for constructing the shared dictionary includes: Obtaining the occurrence frequency of each type of data in the historical data; obtaining the preference degree of each type of data in the historical data according to the occurrence frequency of each type of data in the historical data; the occurrence frequency of each type of data is positively correlated with the preference degree of each type of data; screening data according to the preference degree to obtain the shared dictionary.

3. A data information transmission method based on cloud-edge collaboration according to claim 2, characterized in that, The method for obtaining the common index includes: Obtaining the occurrence frequency of each type of data in the real-time data in the target edge dictionary; obtaining the correction coefficient of each type of data in the target edge dictionary according to the difference between the occurrence frequency of each type of data in the target edge dictionary and the occurrence frequency of the same type of data in the current shared dictionary. Obtaining the common index of each type of data in the target edge dictionary according to the occurrence frequency of each type of data in the target edge dictionary and combining the correction coefficient of each type of data in the target edge dictionary; both the occurrence frequency and the correction coefficient of each type of data in the target edge dictionary are positively correlated with the common index.

4. A data information transmission method based on cloud-edge collaboration according to claim 3, characterized in that, The method for obtaining the correction coefficient includes: Selecting any data in the target edge dictionary as the target data; obtaining the frequency difference parameter according to the difference between the occurrence frequency of the target data in the target edge dictionary and the occurrence frequency of the same type of data in the current shared dictionary. Obtaining the change rate of the occurrence frequency of the target data in the target edge dictionary according to the frequency difference parameter corresponding to the target data and combining the time interval between the acquisition time of the current real-time data and the time of constructing the current shared dictionary. When the occurrence frequency of the target data in the target edge dictionary is greater than or equal to the occurrence frequency in the current shared dictionary, obtaining the correction coefficient of the target data according to the frequency difference parameter and the change rate of the occurrence frequency; both the frequency difference parameter and the change rate of the occurrence frequency are positively correlated with the correction coefficient. When the occurrence frequency of the target data in the target edge dictionary is less than the occurrence frequency in the current shared dictionary, obtaining the correction coefficient of the target data according to the frequency difference parameter and the change rate of the occurrence frequency; both the frequency difference parameter and the change rate of the occurrence frequency are negatively correlated with the correction coefficient.

5. A data information transmission method based on cloud-edge collaboration according to claim 1, characterized in that The method for obtaining the compression effect score includes: Obtaining the memory occupancy after compressing the real-time data; obtaining the compression time of the real-time data compression; according to the memory occupancy and the compression time, obtaining the compression effect score of the real-time data by the target edge device; both the memory occupancy and the compression time are negatively correlated with the compression effect score.

6. A data information transmission method based on cloud-edge collaboration according to claim 2, characterized in that, The method for iteratively updating the target edge dictionary includes: In the data existing in the current target edge dictionary, when the common index is less than the first preset threshold and the compression effect score is less than the second preset threshold, delete the corresponding data in the current target edge dictionary; For the data in the real-time data that is not in the current target edge dictionary, obtain the corresponding preference degree of the data, and when the preference degree is greater than the third preset threshold, add the corresponding data to the current target edge dictionary.

7. A data information transmission method based on cloud-edge collaboration according to claim 6, characterized in that, After iteratively updating the target edge dictionary, it further includes: According to the similarity features between the latest target edge dictionary and the current shared dictionary, iteratively update the shared dictionary and re-transmit it to the edge device to update the target edge dictionary; Use the latest target edge dictionary to compress the text information data and synchronously transmit it in real time.

8. A data information transmission method based on cloud-edge collaboration according to claim 7, characterized in that, The method for iteratively updating the shared dictionary includes: Obtain the data in the current shared dictionary that is the same as the current target edge dictionary as the data to be analyzed; take the proportion of the number of the data to be analyzed in all the data in the current shared dictionary as the first similarity parameter; In the data set composed of all the data obtained by the target edge device from receiving the current shared dictionary to the current moment, obtain the occurrence frequency of each data to be analyzed; according to the concentration characteristics of the occurrence frequencies of all the data to be analyzed, obtain the second similarity parameter; According to the first similarity parameter and the second similarity parameter, obtain the update determination parameter of the current shared dictionary with respect to the current target edge dictionary; both the first similarity parameter and the second similarity parameter are positively correlated with the update determination parameter; When the update determination parameter is less than the fourth preset threshold, select all the text information data collected by all edge devices between the update time of the current shared dictionary and the current moment to form an update data set; obtain the preference degree of each data in the update data set, and filter the data according to the preference degree to obtain the shared dictionary.

9. A data information transmission method based on cloud-edge collaboration according to claim 8, characterized in that, The method for obtaining the preference degree includes: After normalizing the occurrence frequency of each data in the corresponding historical data, use it as the preference degree corresponding to each data.

10. A data information transmission method based on cloud-edge collaboration according to claim 9, characterized in that The method for filtering data according to the preference degree to obtain the shared dictionary includes: Select all the data with a preference degree greater than the preset preference threshold as the entries of the shared dictionary to obtain the shared dictionary.

Images