Data compression transmission method and data decompression method

Abstract

The application provides a data compression transmission method and a data decompression method, and belongs to the technical field of data compression. The method comprises the following steps: preprocessing to-be-sent data to obtain original data, wherein the original data comprises multiple groups of training data, each group of training data in the multiple groups of training data comprises collected data and a label corresponding to the collected data; training a preset network model based on the original data to obtain a prediction model; and sending the prediction model to a receiving end; or receiving the prediction model sent by the sending end, inputting the label into the prediction model, and obtaining prediction data output by the prediction model. The application learns the characteristics of to-be-sent data through a network model to obtain a prediction model. The prediction model file is small in size. When data is transmitted, only the learned prediction model needs to be transmitted, the bandwidth required for data transmission is reduced, the transmission speed is improved, the delay and cost of data transmission are reduced, and resource utilization is optimized.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and in particular to a data compression and transmission method and a data decompression method. Background Technology

[0002] Driven by the powerful force of digital transformation, various institutions, such as scientific research, financial services, IT and cloud services, and manufacturing, are processing massive amounts of data every day. This data not only drives intelligent production and refined management but is also crucial for improving market responsiveness, optimizing production processes, and enhancing competitiveness. Fast and secure data transmission is particularly critical for real-time monitoring, data analysis, and customer service, but it also faces challenges such as bandwidth bottlenecks, latency, and security. Effective data compression algorithms can significantly reduce data volume, alleviate these problems, improve transmission efficiency and resource utilization, and ensure data security.

[0003] Traditional lossless and lossy compression techniques, such as ZIP and JPEG, directly encode data based on statistical, dictionary, or transform methods, relying on identifying data redundancy to reduce file size. While these methods are effective in their respective application scenarios, they are insufficient to capture the complex interrelationships within the data, resulting in limited compression ratios, high bandwidth requirements for data transmission, high data transmission costs, and low resource utilization.

[0004] Therefore, how to reduce the bandwidth required for data transmission has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides a data compression and transmission method and a data decompression method to solve the problem of high bandwidth requirements for data transmission in the prior art.

[0006] In a first aspect, the present invention provides a data compression and transmission method applied at a sending end, comprising:

[0007] The data to be sent is preprocessed to obtain the original data, which includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data.

[0008] Based on the original data, the pre-set network model is trained to obtain the prediction model;

[0009] The prediction model is sent to the receiving end.

[0010] According to a data compression and transmission method provided by the present invention, after training a preset network model based on the original data to obtain a prediction model, the method further includes:

[0011] Based on the prediction model, the original data is predicted to obtain the predicted data;

[0012] Based on the original data and the predicted data, a prediction error is determined, wherein the prediction error represents the error of the predicted data compared to the original data;

[0013] The prediction error is sent to the receiving end.

[0014] According to a data compression and transmission method provided by the present invention, determining the prediction error based on the predicted data and the original data includes:

[0015] Calculate the verification matrix based on the difference between the original data and the predicted data;

[0016] Based on the verification matrix, the prediction error is determined.

[0017] According to a data compression and transmission method provided by the present invention, determining the prediction error based on the parity check matrix includes:

[0018] The parity check matrix is ​​sparsified to obtain a sparse parity check matrix;

[0019] Based on a preset algorithm, the sparse parity-check matrix is ​​compressed to obtain the prediction error.

[0020] According to a data compression and transmission method provided by the present invention, the step of sparsifying the parity check matrix to obtain a sparse parity check matrix includes:

[0021] For each element in the parity check matrix, a zeroing step is performed to obtain a sparse parity check matrix;

[0022] The zeroing step includes:

[0023] If the absolute value of the element is less than a preset threshold, then the value of the element is updated to 0;

[0024] If the absolute value of the element is greater than or equal to the preset threshold, then the value of the element remains unchanged.

[0025] Secondly, the present invention provides a data decompression method, applied at a receiving end, comprising:

[0026] The predictive model sent by the receiving end;

[0027] The labels are input into the prediction model to obtain the prediction data output by the prediction model.

[0028] Thirdly, the present invention also provides a data compression and transmission apparatus, comprising:

[0029] The output processing module is used to preprocess the data to be sent to obtain the original data. The original data includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data.

[0030] The model training module is used to train a pre-set network model based on the original data to obtain a prediction model;

[0031] The model sending module is used to send the prediction model to the receiving end.

[0032] Fourthly, the present invention also provides a data decompression apparatus, comprising:

[0033] The data receiving module is used to receive the prediction model sent by the sending end;

[0034] The data restoration module is used to input the labels into the prediction model and obtain the prediction data output by the prediction model.

[0035] Fifthly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the data compression and transmission methods described above, or to implement the steps of any of the data decompression methods described above.

[0036] In a sixth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the data compression and transmission methods described above, or implements the steps of any of the data decompression methods described above.

[0037] In a seventh aspect, the present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of any of the data compression and transmission methods described above, or implements the steps of any of the data decompression methods described above.

[0038] This invention provides a data compression and transmission method and a data decompression method. The method involves preprocessing the data to be transmitted to obtain raw data, which includes multiple sets of training data. Each set of training data includes collected data and corresponding labels. Based on the raw data, a pre-set network model is trained to obtain a prediction model, which is then sent to a receiving end. Alternatively, the receiving end receives the prediction model, inputs the labels into the prediction model, and obtains the prediction data output by the prediction model. This solution learns the features of the data to be transmitted through a network model to obtain a prediction model. The prediction model file size is small, and only the learned prediction model needs to be transmitted during data transmission. The receiving end can calculate the prediction data based on the prediction model, thereby reducing the bandwidth required for data transmission, increasing transmission speed, reducing data transmission latency and cost, and optimizing resource utilization. Furthermore, transmitting the prediction model also encrypts the data to a certain extent, enhancing data privacy and security, making it more difficult to interpret and steal during transmission, reducing the loss of data directly exposed to insecure environments and thus protecting data confidentiality. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the data compression and transmission method provided by the present invention;

[0041] Figure 2 This is a schematic diagram of the temperature distribution data for the 186th day provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the model loss function values ​​for meteorological data compression and transmission provided by the present invention;

[0043] Figure 4 This is a flowchart illustrating the data decompression method provided by the present invention;

[0044] Figure 5 This is a schematic diagram of the temperature forecast data for the 186th day provided by the present invention;

[0045] Figure 6 This is a schematic diagram illustrating the difference between the collected data and the predicted data provided by this invention;

[0046] Figure 7 This is a schematic diagram of the decompression data provided by the present invention;

[0047] Figure 8 This is a schematic diagram illustrating the difference between the decompressed data and the collected data provided by this invention;

[0048] Figure 9 This is a schematic diagram of the data compression and transmission system provided by the present invention;

[0049] Figure 10 This is a schematic diagram of the process by which the sending end compresses and transmits the data to be sent, as provided by the present invention.

[0050] Figure 11 This is a schematic diagram of the process by which the receiving end decompresses received data, as provided by the present invention.

[0051] Figure 12 This is a schematic diagram of the data compression and transmission device provided by the present invention;

[0052] Figure 13 This is a schematic diagram of the data decompression device provided by the present invention;

[0053] Figure 14 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] It should be noted that in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0056] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0057] The following is combined Figures 1-14 This invention describes the data compression and transmission method and the data decompression method provided in the embodiments of the present invention.

[0058] Figure 1 This is a flowchart illustrating the data compression and transmission method provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps:

[0059] S110, Preprocess the data to be sent to obtain the original data, the original data including multiple sets of training data, each set of training data including collected data and the label corresponding to the collected data;

[0060] S120, Based on the original data, the preset network model is trained to obtain the prediction model;

[0061] S130, the prediction model is sent to the receiving end.

[0062] It should be noted that the execution subject of the data compression and transmission method provided in this application embodiment is the sending end, which can be a server, computer equipment, such as mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc.

[0063] For ease of understanding, the embodiments of the present invention use the compression and transmission of meteorological data as examples for auxiliary illustration. It is understood that the application of the data compression and transmission method and the data decompression method provided by the present invention is not limited to the compression and decompression of meteorological data, but can be applied to the compression of any text data, image data, audio data, video data, etc.

[0064] In S110, the sending end organizes the data to be transmitted, determining its format, size, unit, and tags. If this information was not previously provided to the receiving end, the sending end sends it to the receiving end so that the receiving end can reconstruct the compressed data based on this information. It's understandable that the required information varies depending on the data type and is not limited to, or may not include, the information listed above. For example, assuming the data to be transmitted is the global average daily temperature from 1981 to 2010 for 365 days, and the data is in matrix format with latitude and longitude divided in 2.5° increments, then the size of the data to be transmitted is 365 × 73 × 144. Here, 365 represents 365 days, 73 represents latitude divided in 2.5° increments, 144 represents longitude divided in 2.5° increments, and the unit of the data to be transmitted is Kelvin (°K). Figure 2 This is a schematic diagram of the temperature distribution data for day 186 provided by the present invention, as shown below. Figure 2 As shown, the temperature distribution data for a day is a collection data point, and the label corresponding to each collection data point is the label corresponding to the temperature distribution data for each day. Here, the label corresponding to each collection data point is its corresponding date and latitude and longitude.

[0065] Optionally, a large amount of data to be sent can be transmitted in batches to reduce the amount of data transmitted each time.

[0066] In S120, a suitable machine learning network model can be selected based on the data characteristics, including but not limited to Convolutional Neural Networks (CNN), Fully Convolutional Networks (FCN), Generative Adversarial Networks (GAN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Residual Networks (ResNet), and Variational Autoencoders (VAE). Appropriate training hyperparameters can also be selected, including but not limited to the number of neural network layers, the number of neurons per layer, the activation function, the optimizer, the learning rate, the loss function, the batch size, and the number of iterations. All collected data and their corresponding labels are then fed into the selected network model for training, allowing the network model to learn the internal patterns of the data. In the example of meteorological data compression and transmission, an FCN network model with two hidden layers is selected. The number of neurons in the input layer is 10, the number of neurons in the hidden layer is 100, the number of neurons in the output layer is 1, the activation function is SELU, the optimizer is Adam, the learning rate is 0.01, the loss function is mean squared error (MSE), the batch size is 64, and the number of iterations is 500.

[0067] In S130, the prediction model is sent to the receiving end for use in predicting and recovering data. The prediction model learns the inherent characteristics and patterns of the data, enabling compressed data transmission that focuses on the meaning of the transmitted data rather than the data itself, thus efficiently transmitting the most critical information.

[0068] In some embodiments, the sender directly sends the prediction model to the receiver;

[0069] In other embodiments, the sending end and the receiving end pre-agree on the model to be used, and the sending end sends the model parameters of the prediction model to the receiving end, thereby reducing the amount of data transmission.

[0070] The data compression and transmission method provided in this invention preprocesses the data to be transmitted to obtain raw data. The raw data includes multiple sets of training data, each set including collected data and corresponding labels. Based on the raw data, a pre-set network model is trained to obtain a prediction model, which is then sent to the receiving end. This solution learns the features of the data to be transmitted through a network model to obtain a prediction model. The prediction model file size is small, and only the learned prediction model needs to be transmitted during data transmission. The receiving end can calculate the prediction data based on the prediction model, thereby reducing the bandwidth required for data transmission, increasing transmission speed, reducing data transmission latency and cost, and optimizing resource utilization. Furthermore, transmitting the prediction model also encrypts the data to a certain extent, enhancing data privacy and security, making it more difficult to interpret and steal during transmission, reducing the loss of data directly exposed to insecure environments and thus protecting data confidentiality.

[0071] In an optional embodiment, after training the preset network model based on the original data to obtain the prediction model, the method further includes:

[0072] Based on the prediction model, the original data is predicted to obtain the predicted data;

[0073] Based on the original data and the predicted data, a prediction error is determined, wherein the prediction error represents the error of the predicted data compared to the original data;

[0074] The prediction error is sent to the receiving end.

[0075] Specifically, after training the prediction model, the labels of all the collected data in the original data are input into the prediction model again for prediction to obtain the prediction data. Based on the collected data and the prediction data, the prediction error is determined and sent to the receiving end so that the receiving end can verify the restored data based on the prediction error.

[0076] Figure 3 This is a schematic diagram of the model loss function value for meteorological data compression and transmission provided by this invention. In the meteorological data compression and transmission example, the metadata is input into the FCN again. The FCN is trained based on the date and latitude / longitude labels of all collected data, and calculates the loss function value based on the collected data. The changes in the model loss function value in the first 100 iterations are as follows: Figure 3 As shown, the decreasing and converging trend of the loss function value indicates that network training can effectively discover the internal patterns of the data.

[0077] The data compression and transmission method provided in this invention determines the prediction error, which characterizes the error of the prediction model, based on the original data and the predicted data. This allows the receiving end to verify the predicted data obtained by restoring the prediction model based on the prediction error after receiving the prediction error, thereby reducing the error between the final data obtained by the receiving end and the actual data.

[0078] In an optional embodiment, determining the prediction error based on the predicted data and the original data includes:

[0079] Calculate the verification matrix based on the difference between the original data and the predicted data;

[0080] Based on the verification matrix, the prediction error is determined.

[0081] Specifically, the verification matrix equals the difference between the collected data and the predicted data. In the example of meteorological data compression and transmission, this means the difference between the actual temperature distribution data and the predicted temperature distribution data for a given day.

[0082] Alternatively, the verification matrix can be directly sent to the receiving end as the prediction error.

[0083] It should be understood that because neural networks capture patterns or trends between data, their predictions are difficult to be 100% accurate. The difference between the actual value and the predicted value is used to determine the prediction error, which allows the receiving end to correct the predicted value based on the prediction error.

[0084] Further, determining the prediction error based on the verification matrix includes:

[0085] The parity check matrix is ​​sparsified to obtain a sparse parity check matrix;

[0086] Based on a preset algorithm, the sparse parity-check matrix is ​​compressed to obtain the prediction error.

[0087] It should be understood that the parity check matrix obtained by directly subtracting is not a sparse matrix, but it has a large number of local minima, especially for decimals with a long number of decimal places. These data have little impact on data recovery, but their data size is large, increasing bandwidth consumption. Therefore, by sparsifying the parity check matrix and removing these data, the memory size occupied by the parity check matrix is ​​reduced, thereby reducing the bandwidth occupied during transmission, improving transmission speed, and reducing data transmission latency and cost.

[0088] Here, the method used for sparsification of the parity check matrix is ​​not limited, nor is the method used for compression of the sparse parity check matrix.

[0089] Optionally, considering matrix structure characteristics, efficiency, and storage space requirements, the sending end selects an effective method (including but not limited to compressed sparse row CSR / column CSC, elliptic coordinate format ELL, singular value decomposition SVD, principal component analysis PCA, low-rank matrix approximation, etc.) to compress the sparse parity check matrix, and then derives the compressed sparse parity check matrix, i.e., the prediction error.

[0090] Optionally, the sending end also derives a compressed model of the compressed matrix and sends the compressed model along with the prediction error to the receiving end.

[0091] In the example of meteorological data compression transmission, a variant of singular value decomposition is used, truncated singular value decomposition (TSVD) is used to further compress the sparse parity-check matrix, only the 10 largest singular values ​​are retained, and the compressed sparse parity-check matrix and compressed TSVD model are derived; the prediction model, compressed sparse parity-check matrix and compressed TSVD model are sent to the receiving end.

[0092] The data compression and transmission method provided in this invention further reduces the storage space occupied by prediction errors by compressing sparse matrices, effectively saving storage resources and bandwidth occupied during data transmission, and improving transmission speed.

[0093] In an optional embodiment, the step of sparsifying the parity-check matrix to obtain a sparse parity-check matrix includes:

[0094] For each element in the parity check matrix, a zeroing step is performed to obtain a sparse parity check matrix;

[0095] The zeroing step includes:

[0096] If the absolute value of the element is less than a preset threshold, then the value of the element is updated to 0;

[0097] If the absolute value of the element is greater than or equal to the preset threshold, then the value of the element remains unchanged.

[0098] Specifically, by setting the absolute value of each parity check matrix to 0, a sparse parity check matrix with many zero values ​​can be obtained. In the meteorological data compression and transmission example, the preset threshold is 1, and more than half of the values ​​in the resulting sparse parity check matrix are 0.

[0099] The data decompression method provided in the embodiments of this application is described below. The data decompression method described below can be referred to in correspondence with the data compression and transmission method described above.

[0100] Figure 4 This is a flowchart illustrating the data decompression method provided by the present invention, as shown below. Figure 4 As shown, including but not limited to the following steps:

[0101] S410, the prediction model sent by the receiving end;

[0102] S420, Input the label into the prediction model to obtain the prediction data output by the prediction model.

[0103] It should be noted that the data compression and transmission method provided in this application is executed by a receiving end, which can be a server, computer equipment, such as a mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0104] Specifically, the receiving end first inputs the known set of labels into the received prediction model, and then performs predictions and data recovery through the prediction model.

[0105] It should be understood that the data recovered through the predictive model is the predicted data.

[0106] Figure 5 This is a schematic diagram of the temperature forecast data for the 186th day provided by the present invention;

[0107] Figure 6 This is a schematic diagram illustrating the difference between the collected data and the predicted data provided by this invention;

[0108] In the example of meteorological data compression and transmission, the predicted temperature data for day 186, as predicted by the forecasting model, is as follows: Figure 5 As shown, the difference between the predicted data and the collected data (actual data) is as follows: Figure 6 As shown, the predicted data obtained by the receiver at this time still has an error of no more than ±20°K compared with the collected data, especially in locations where the data changes drastically.

[0109] Optionally, if the receiving end receives a prediction error, it verifies and corrects the prediction data based on the prediction error; if the receiving end receives both the prediction error and the compressed model, it decompresses the prediction error through the inverse transformation of the model, and then verifies and corrects the prediction data based on the decompressed matrix.

[0110] In the example of meteorological data compression and transmission, the receiving end decompresses the prediction error to obtain a sparse parity-check matrix, and adds the sparse parity-check matrix to the prediction data to obtain the final decompressed data result.

[0111] Figure 7 This is a schematic diagram of the decompression data provided by the present invention;

[0112] Figure 8 This is a schematic diagram illustrating the difference between the decompressed data and the collected data provided by this invention;

[0113] The temperature of day 186 obtained after decompression is as follows Figure 7 As shown, the difference between it and the actual temperature on day 186 is as follows: Figure 8 As shown, the error between the decompressed data obtained by the receiving end and the actual data does not exceed ±4°K.

[0114] The data compression and transmission system provided in the embodiments of this application is described below. The data compression and transmission system described below can be referred to in correspondence with the data compression and transmission method and data decompression method described above.

[0115] Figure 9 This is a schematic diagram of the data compression and transmission system provided by the present invention, as shown below. Figure 9 As shown, it includes: a network trainer, a network predictor, a matrix compressor, and a matrix decompressor. The sending end only needs to activate and use the network trainer and the matrix compressor, while the receiving end only needs to activate and use the network predictor and the matrix decompressor.

[0116] Specifically, the network trainer is mainly responsible for training the network on the original data. That is, after the sending end processes the data, the network trainer sets up a network model to discover the inherent information of the data, adjusts the training hyperparameters, and inputs the labels and collected data for training. The network trainer outputs the trained prediction model and the prediction data predicted by the prediction model.

[0117] The network predictor recovers the main data through the prediction model and label information. That is, the network predictor is responsible for loading the received prediction model, inputting the known labels into the prediction model, and obtaining the prediction data.

[0118] The matrix compressor provides compression functionality for sparse parity-check matrices. It calculates the parity-check matrix based on the predicted and collected data, sets an appropriate threshold to simplify the parity-check matrix into a sparse parity-check matrix, compresses the sparse parity-check matrix using an appropriate compression method according to the compression requirements, and outputs the compressed sparse parity-check matrix and the matrix compression model (if needed).

[0119] The matrix decompressor provides decompression functionality for compressed sparse parity-check matrices, that is, it decompresses the parity-check matrix through inverse model transformation based on the received compressed sparse parity-check matrix and the matrix compression model (if necessary).

[0120] The receiving end adds the predicted data to the decompressed parity matrix to obtain the final decompressed data.

[0121] It should be understood that Figure 9A schematic diagram of an example architecture is shown. For descriptive purposes, the architecture depicted is merely an example for a suitable environment and does not impose any limitations on the scope or functionality of this application. Nor should this system be construed as an extension of... Figure 9 Any component shown or a combination thereof has any dependencies or requirements.

[0122] Figure 10 This is a schematic diagram of the process by which the sending end compresses and transmits the data to be sent, as provided by the present invention. Figure 10 As shown, firstly, the sending end collects data from different stages such as monitoring, office work, production, and operations, and performs preliminary preprocessing on this data. Next, in the network trainer, labels are assigned to the data, and appropriate network models and training hyperparameters are selected based on the characteristics of the data. All data is then fed into the network for deep learning training to explore and extract the inherent patterns in the data. After network training is complete, on the one hand, all data is re-input into the prediction model to obtain prediction results, and these results are transmitted to the matrix compressor. On the other hand, the trained prediction model is directly sent to the receiving end. In the matrix compressor, a verification matrix is ​​generated by comparing the differences between the predicted data and the actual data. Then, a threshold is set according to specific needs, and the verification matrix is ​​processed into a sparse format and compressed using appropriate matrix compression techniques and parameters. Finally, the sending end sends the generated compressed sparse verification matrix and, if necessary, the matrix compression model to the receiving end.

[0123] Figure 11 This is a schematic diagram of the process by which the receiving end decompresses received data, as provided by the present invention. Figure 11 As shown, the receiving end first imports the obtained prediction model, compressed sparse parity-check matrix, and compressed matrix model into the network predictor and matrix decompressor, respectively. By inputting a predefined set of labels into the prediction model, preliminary data prediction is performed to obtain a set of preliminary prediction data. Next, in the matrix decompressor stage, the received compressed sparse parity-check matrix is ​​decompressed according to the compressed matrix model. After decompression, the decompressed parity-check matrix is ​​added to the preliminary prediction data generated by the network predictor to obtain the final decompressed data. This process ensures that the integrity and accuracy of the data are restored. Finally, the receiving end distributes these restored data to various storage or application devices to support various application scenarios such as intelligent production, refined management, and digital operation, thereby playing an important role in practical operation.

[0124] In summary, the data compression and transmission method and data decompression method provided by this invention can uncover deep patterns in data, improve data representation capabilities, represent semantics with models, transmit models and key parameters at the sending and receiving ends, and decompress the data at the receiving end based on the models and key parameters to obtain the original data, thereby achieving efficient transmission.

[0125] Specifically:

[0126] For the transmission of digital data, different network models and matrix compression algorithms are used according to the characteristics of the data, which greatly reduces the data size and achieves lossy compression with acceptable error. In the example of meteorological data compression and transmission, the original temperature data is 13231K in size, the size of the fully connected network model to be transmitted is 175K, the TSVD compression model is 412K, and the compression sparse error matrix is ​​15K. The total data compression is approximately 4.5499% of the original data. Using different models and parameters can achieve even better compression results. It can be seen that this scheme greatly reduces the bandwidth required for data transmission, increases the transmission speed, reduces the latency and cost of data transmission, and optimizes resource utilization.

[0127] This solution provides a lossy compression scheme designed to maintain the integrity and accuracy of information as much as possible while reducing the compression ratio. Through network model design, it ensures that the compressed data maintains a small deviation from the original data in terms of overall numerical range, distribution characteristics, and trends, thus maximizing fidelity. Furthermore, by introducing a check matrix, it can not only effectively correct large deviations at individual data points and enhance the richness of data details, but also further correct outliers in the prediction results, improving data quality. In the meteorological data compression and transmission example, the network model's mean absolute error (MAE) is 1.0539, indicating that the average temperature difference between each measurement point and the original data is 1.0539°K, with an average relative error (RE) not exceeding 0.005. After correction using the check matrix, the MAE of the final data obtained at the receiving end is reduced to 0.5126°K, with an average RE not exceeding 0.0025. By selecting different models and parameter configurations, the fidelity of the numerical values ​​can be further optimized.

[0128] Different network models, training parameters, and matrix compression algorithms have their applicable scenarios and advantages and disadvantages. The selection of appropriate network models, training parameters, and matrix compression algorithms for different transmission scenarios and needs depends on the degree of information loss that both parties can accept, network conditions, compression ratio requirements, decompression speed, cost, and performance. This solution allows for the adjustment of multiple network models and compression transmission methods, making the method more flexible and providing a wider range of choices, suitable for data compression transmission needs in various scenarios.

[0129] Transmitting large amounts of data in smaller batches reduces the amount of data transmitted each time, which can improve data fluency, adapt to low-bandwidth environments, and make management and control easier. At the same time, batch transmission can better adapt to dynamic transmission environments, improving the stability and flexibility of data transmission.

[0130] The compression process also achieves a certain degree of protocol-based encryption of the data, enhancing data privacy and security, making it more difficult to interpret and steal during transmission, reducing the loss of data directly exposed to insecure environments and thus protecting data confidentiality.

[0131] The data compression and transmission apparatus provided in the embodiments of this application is described below. The data compression and transmission apparatus described below can be referred to in correspondence with the data compression and transmission method described above.

[0132] Figure 12 This is a schematic diagram of the data compression and transmission device provided by the present invention, as shown below. Figure 12 As shown, the data compression and transmission device may include:

[0133] The output processing module 1210 is used to preprocess the data to be sent to obtain the original data. The original data includes multiple sets of training data. Each set of training data includes collected data and a label corresponding to the collected data.

[0134] The model training module 1220 is used to train a preset network model based on the original data to obtain a prediction model;

[0135] The model sending module 1230 is used to send the prediction model to the receiving end.

[0136] It should be noted that the data compression and transmission device provided in this embodiment of the invention can execute the data compression and transmission method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0137] The data decompression apparatus provided in the embodiments of this application is described below. The data decompression apparatus described below can be referred to in correspondence with the data decompression method described above.

[0138] Figure 13 This is a schematic diagram of the data decompression device provided by the present invention, as shown below. Figure 13 As shown, the data decompression device may include:

[0139] The data receiving module 1310 is used to receive the prediction model sent by the sending end;

[0140] The data restoration module 1320 is used to input the labels into the prediction model and obtain the prediction data output by the prediction model.

[0141] It should be noted that the data decompression apparatus provided in this embodiment of the invention can execute the data decompression method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0142] Figure 14 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 14 As shown, the electronic device may include: a processor 1410, a communications interface 1420, a memory 1430, and a communication bus 1440, wherein the processor 1410, the communications interface 1420, and the memory 1430 communicate with each other via the communication bus 1440. The processor 1410 can call logical instructions in the memory 1430 to execute a data compression and transmission method, which includes:

[0143] The data to be sent is preprocessed to obtain the original data, which includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data.

[0144] Based on the original data, the pre-set network model is trained to obtain the prediction model;

[0145] Sending the prediction model to the receiving end; or, including:

[0146] The predictive model sent by the receiving end;

[0147] The labels are input into the prediction model to obtain the prediction data output by the prediction model.

[0148] Furthermore, the logical instructions in the aforementioned memory 1430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the data compression and transmission method provided in the above embodiments, the method comprising:

[0150] The data to be sent is preprocessed to obtain the original data, which includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data.

[0151] Based on the original data, the pre-set network model is trained to obtain the prediction model;

[0152] Sending the prediction model to the receiving end; or, including:

[0153] The predictive model sent by the receiving end;

[0154] The labels are input into the prediction model to obtain the prediction data output by the prediction model.

[0155] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data compression and transmission method provided in the above embodiments, the method comprising:

[0156] The data to be sent is preprocessed to obtain the original data, which includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data.

[0157] Based on the original data, the pre-set network model is trained to obtain the prediction model;

[0158] Sending the prediction model to the receiving end; or, including:

[0159] The predictive model sent by the receiving end;

[0160] The labels are input into the prediction model to obtain the prediction data output by the prediction model.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data compression and transmission method, characterized in that, Applied to the sending end, including: The data to be sent is preprocessed to obtain the original data, which includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data. Based on the original data, the pre-set network model is trained to obtain the prediction model; The prediction model is sent to the receiving end; After training the pre-set network model based on the original data to obtain the prediction model, the method further includes: Based on the prediction model, the original data is predicted to obtain the predicted data; Based on the original data and the predicted data, a prediction error is determined, wherein the prediction error represents the error of the predicted data compared to the original data; The prediction error is sent to the receiving end; The step of determining the prediction error based on the predicted data and the original data includes: Calculate the verification matrix based on the difference between the original data and the predicted data; Based on the verification matrix, the prediction error is determined; The determination of prediction error based on the verification matrix includes: The parity check matrix is ​​sparsified to obtain a sparse parity check matrix; Based on a preset algorithm, the sparse parity-check matrix is ​​compressed to obtain the prediction error; The process of sparsifying the parity-check matrix to obtain a sparse parity-check matrix includes: For each element in the parity check matrix, a zeroing step is performed to obtain a sparse parity check matrix; The zeroing step includes: If the absolute value of the element is less than a preset threshold, then the value of the element is updated to 0; If the absolute value of the element is greater than or equal to the preset threshold, then the value of the element remains unchanged.

2. A data decompression method, characterized in that, Applied to the receiving end, including: Receive the prediction model sent by the sender as described in claim 1; The labels are input into the prediction model to obtain the prediction data output by the prediction model.

3. A data compression and transmission device, characterized in that, The data compression and transmission method as described in claim 1 includes: The output processing module is used to preprocess the data to be sent to obtain the original data. The original data includes multiple sets of training data. Each set of training data includes collected data and the label corresponding to the collected data. The model training module is used to train a pre-set network model based on the original data to obtain a prediction model; The model sending module is used to send the prediction model to the receiving end.

4. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data compression and transmission method as described in claim 1, or the steps of the data decompression method as described in claim 2.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data compression and transmission method as described in claim 1, or the steps of the data decompression method as described in claim 2.

6. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the data compression and transmission method as described in claim 1, or the steps of the data decompression method as described in claim 2.

Images