Road structure information monitoring data lossless compression method and device

Abstract

The invention discloses a lossless compression method and device for road structure information monitoring data, solves the problem that collected high-frequency massive road structure information monitoring data is difficult to transmit and store, and belongs to the field of road engineering. The method comprises the following steps: intercepting acquired original road structure information monitoring data, and storing the intercepted monitoring data and starting and ending timestamps in a file f; performing first-order difference on the monitoring data in the file f to obtain f1; expanding the monitoring data after the first-order difference, and removing a zero position and a decimal point position in the monitoring data to obtain a file f2; and performing Huffman coding processing on the monitoring data in the f2, and in the Huffman coding processing, taking each monitoring data as a single value, coding each single value according to the occurrence frequency of the single value to obtain a binary file f3, and completing compression. When the monitoring data needs to be restored, according to the characteristic that an encoding algorithm is a symmetric algorithm, inverse process processing is conducted on the binary file f3, a decoding file f4 is obtained, and data restoration is completed.

Description

Technical Field

[0001] The invention relates to a method for lossless compression of highway structure information monitoring data, belonging to the field of road engineering. Background Art

[0002] Highway structural monitoring (SHM) is an important means to ensure the safety of road structures and improve road maintenance efficiency. Long-term highway performance observation can explore the mechanical response and material parameter changes of asphalt pavement under environmental and traffic loads in real time. It is an effective way to guide pavement structure design, optimize road service performance, and increase road service life. With the continuous upgrading of traffic monitoring technology, the widespread deployment of sensors, cameras and other equipment has led to a massive increase in monitoring data. The storage and transmission of this data is a huge challenge for urban traffic management systems.

[0003] The previous data transmission and storage methods were inefficient. In the previous pavement structure monitoring, monitoring data was often collected in real time through sensor elements, then stored locally at the monitoring site, and regularly copied by data analysts for data analysis and management. However, with the trend of scientific and data-centralized long-term performance observation networks, this manual copying transmission, storage and analysis method is relatively inefficient and can no longer meet the processing needs of massive high-frequency highway pavement dynamic monitoring data. In addition, the previous data transmission and storage methods were costly. For the long-term performance observation of the roadbed and pavement, a large number of sensor elements buried inside the pavement structure are required to carry out real-time collection of multiple observation indicators such as vehicle load, stress, strain, temperature inside the structure, humidity inside the structure, etc., and because the sampling frequency of some sensor elements is high, a steady stream of massive high-frequency data will accumulate at the observation point for a long time, resulting in excessive storage and processing costs.

[0004] The most ideal data transmission and storage architecture is to transmit the road surface dynamic monitoring data collected by the sensor element to the cloud with stronger computing and storage capabilities through the network, and then store and process the data. In actual projects, the difficulty in implementing this architecture lies in the high cost of transmitting and storing the high-frequency and massive data collected, and the huge amount of data makes the requirements for network, power and other conditions extremely demanding. Summary of the invention

[0005] In view of the problem that high-frequency and massive highway structure information monitoring data collected is difficult to transmit and store, the present invention provides a method and device for lossless compression of highway structure information monitoring data.

[0006] A method for lossless compression of highway structure information monitoring data of the present invention comprises:

[0007] intercepting the original highway structure information monitoring data collected by the sensor, recording the start and end timestamps, and storing the intercepted monitoring data and the start and end timestamps in the file f;

[0008] Perform first-order difference processing on the monitoring data in file f, and then update file f to file f1;

[0009] Expand the monitoring data after the first-order difference processing, remove the zero and decimal points in the monitoring data, and then update the file f1 to file f2;

[0010] The monitoring data in file f2 is encoded to obtain binary file f3, and compression is completed.

[0011] Preferably, the values ​​of the monitoring data in the first line of file f are retained, and the values ​​of the monitoring data in other lines are replaced by the difference between the next line and the current line.

[0012] Preferably, the monitoring data in the file f2 is subjected to Huffman coding, and in the Huffman coding, each monitoring data is regarded as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

[0013] Preferably, the method further comprises:

[0014] When it is necessary to restore the monitoring data, according to the characteristic that the encoding algorithm is a symmetric algorithm, the binary file f3 is processed inversely to obtain the decoded file f4 to complete the data restoration.

[0015] The present application also includes a device for lossless compression of highway structure information monitoring data, comprising:

[0016] A data selection module is used to intercept the original highway structure information monitoring data collected by the sensor, record the start and end timestamps, and store the intercepted original highway structure information monitoring data and the start and end timestamps in a file f;

[0017] A first-order difference processing module is used to perform first-order difference processing on the monitoring data in the file f, and then update the file f to the file f1;

[0018] An expansion processing module is used to expand the data after the first-order difference processing, remove the zeros and decimal points in the data, and then update the file f1 to the file f2;

[0019] The encoding module is used to encode the data in the file f2 to obtain the binary file f3 and complete the compression.

[0020] Preferably, the encoding module performs Huffman encoding on the data in the file f2, and in the Huffman encoding, each monitoring data is regarded as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

[0021] Preferably, the device further comprises:

[0022] The restoration module is used to perform inverse processing on the binary file f3 when data restoration is needed, according to the characteristic that the encoding algorithm is a symmetric algorithm, to obtain the decoded file f4 and complete data restoration.

[0023] The beneficial effects of the present invention are as follows: the present invention processes highway structure information monitoring data through the method. On the one hand, the original data can be compressed by several dozen times and stored as a binary file, effectively reducing data transmission and storage costs and improving efficiency; on the other hand, when the application end needs to apply the data, the compressed binary file can be completely restored through the designed restoration algorithm, without data loss, and the restoration algorithm has high efficiency and can meet most restoration needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the principle of transmission and storage of highway structure information monitoring data;

[0025] Figure 2 It is a schematic diagram of the principle of the lossless compression method of highway structure information monitoring data;

[0026] Figure 3 Schematic diagram of the principle of first-order difference processing. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0030] Figure 1In the process, the buried sensor transmits the collected highway structure information monitoring data to the data collection end, the data collection end transmits the data to the data center and stores it, and the data center finally transmits the data to the user end according to the demand. Among them, all the transmission and storage steps need to reduce the order of data to reduce costs and improve efficiency. The core idea of ​​the method of this embodiment is to analyze the characteristics of highway structure information monitoring data, design compression and restoration methods for this type of data, and solve the problems of low efficiency and high cost in the storage and transmission of highway structure information monitoring data at the level of data volume. According to the analysis and processing of the characteristics of this type of data, it is found that the non-critical parts of the highway structure information monitoring data have similarities and redundancy, but the non-critical part of the data still has value in transmission and storage. Therefore, according to its characteristics, adhering to the concept of lossless compression without losing data, based on the first-order difference method and Huffman coding algorithm, the two are evolved and combined for road domain data, and the design of the highway structure information monitoring data compression algorithm is completed. In order to solve the problems of high storage and transportation costs and low efficiency of highway structure information monitoring data collected by existing sensors, this embodiment provides a lossless compression method for highway structure information monitoring data, such as Figure 2 As shown, including:

[0031] Step 1: Preprocess the raw data collected by the sensor:

[0032] Intercepting the original highway structure information monitoring data collected by the sensor: preliminarily sorting out the original data collected by the sensor, and splicing or splitting the monitoring data to obtain data blocks suitable for compression, while recording the start and end timestamps of each data block, checking the timestamps and corresponding to the data block, and storing a data block and the corresponding start and end timestamps in the file f, so as to facilitate subsequent compression processing; the original highway structure information monitoring data of this implementation mode is wavelength;

[0033] Furthermore, in order to facilitate updating, transmitting, downloading or processing files, each compressed file has a size limit. Since the size of the compressed highway structural health monitoring file is unknown before data compression, this step sets a file f size constraint for the highway structural health monitoring data block, and the data block will be compressed into file f. In addition, high-speed network transmission causes the data block size of each timestamp to be different. Therefore, the selection of data blocks is related to both the size constraint of file f and the size of each data block. It is necessary to select a suitable data block size to improve compression efficiency. In this embodiment, the file f size is set to 500mb.

[0034] Step 2: Perform first-order difference processing on the monitoring data in file f, and then update file f to file f1;

[0035] Since the dynamic response ratio of the highway structural health monitoring data is about 1:200, and the structural health monitoring data without dynamic response has a small change range, the highway structural health monitoring data fluctuates less, that is, the difference between two adjacent values ​​in the highway structural health monitoring data is not large. In this implementation, the first-order difference can be added to filter the redundant values, and the difference between two adjacent values ​​can be recorded, such as Figure 3 As shown, the storage space of each structural health monitoring data file is reduced;

[0036] The first-order difference process includes: initializing an empty set f1 to store the compressed code of f. Then, obtaining the number and length of attributes of file f. Figure 3 As shown, the first line of monitoring data in file f is retained, and for the values ​​of monitoring data in other lines, the values ​​of monitoring data in other lines are replaced by the difference between the next line and the current line. The first-order difference increases the frequency of numbers close to 0 to facilitate subsequent encoding.

[0037] Step 3: Expand the monitoring data after the first-order difference processing, remove the zero and decimal points in the monitoring data, and then update the file f1 to file f2;

[0038] After the first-order difference, some monitoring data are rewritten with decimals and multiple zeros. In order to further compress the storage space, the data is expanded, the zeros and decimal points in the data are removed, and the data is compressed in decimal, generally multiplied by 1000.

[0039] Step 4: Encode the data in file f2 to obtain binary file f3, complete compression, and perform storage and transmission;

[0040] The purpose of encoding is to shorten the length of the compressed code of the highway structural health monitoring data. Since the frequency distribution of the structural health monitoring data is uneven, the Huffman coding algorithm is a suitable coding strategy. This implementation constructs a Huffman tree to store the compressed structural health monitoring data, ensuring that the encoding pre-order position of each data does not contain the encoding of other data, thereby reducing the length of the code.

[0041] The existing Huffman coding algorithm extracts all the constituent numbers and symbols of the data and encodes each number or symbol according to the frequency of occurrence. In the Huffman coding process of this embodiment, each monitoring data is regarded as a single value and each single value is encoded according to the frequency of occurrence of the single value.

[0042] Step 5: When the data needs to be restored for analysis, according to the characteristic of the encoding algorithm being a symmetric algorithm, f3 is processed inversely to obtain a decoded file f4, which is a processed file that is exactly the same as f1, completing the data restoration step.

[0043] This implementation process processes sensor data and presets the data volume, improves the accuracy of data and corresponding timestamps, and records timestamps for subsequent compression processing; based on the analysis of the characteristics of highway structure information monitoring data, combined with data processing methods, first-order difference methods and Huffman coding, the pre-processed highway structure information monitoring data is binary-encoded and stored to achieve a higher compression efficiency than traditional methods; based on the coding method, the inverse process is processed to obtain the pre-encoded pre-processed data and complete the decoding of the binary file. The data is efficiently restored without loss.

[0044] This embodiment also provides a device for lossless compression of highway structure information monitoring data, including a data selection module, a first-order difference processing module, an expansion processing module, an encoding module and a restoration module;

[0045] A data selection module is used to intercept the original highway structure information monitoring data collected by the sensor, record the start and end timestamps, and store the intercepted original highway structure information monitoring data and the start and end timestamps in a file f;

[0046] The first-order difference processing module is used to perform first-order difference processing on the monitoring data in file f, and then update file f to file f1; specifically, the first line of monitoring data in file f is retained, and for the values ​​of monitoring data in other lines, the values ​​of monitoring data in other lines are replaced by the difference between the next line and the current line. The first-order difference increases the frequency of numbers close to 0 to facilitate subsequent encoding;

[0047] An expansion processing module is used to expand the data after the first-order difference processing, remove the zeros and decimal points in the data, and then update the file f1 to the file f2;

[0048] The encoding module is used to perform Huffman encoding on the data in the file f2. In the Huffman encoding process, each monitoring data is regarded as a single value, and each single value is encoded according to the frequency of occurrence of the single value to obtain a binary file f3, thereby completing compression;

[0049] The restoration module is used to perform inverse processing on the binary file f3 when data restoration is needed, according to the characteristic that the encoding algorithm is a symmetric algorithm, to obtain the decoded file f4 and complete data restoration.

[0050] For newly-incoming highway structural health monitoring data and given file size constraints, a data block selection module is first added to generate a to-be-compressed file consisting of data blocks. Then, a first-order difference module is added to reduce the storage space of highway structural health monitoring data. The first-order difference result of each file is expanded, and a Huffman encoding module is added to output the corresponding Huffman encoding as the final compressed file. Under these modules, the compression device of this embodiment retains the key information in the highway structural health monitoring data and ensures satisfactory compression performance.

[0051] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A method for lossless compression of highway structure information monitoring data, characterized in that: The method comprises: intercepting the original highway structure information monitoring data collected by the sensor, recording the start and end timestamps, and storing the intercepted monitoring data and the start and end timestamps in the file f; Perform first-order difference processing on the monitoring data in file f, and then update file f to file f1; Expand the monitoring data after the first-order difference processing, remove the zero and decimal points in the monitoring data, and then update the file f1 to file f2; The monitoring data in file f2 is encoded to obtain binary file f3, and compression is completed.

2. The method for lossless compression of highway structure information monitoring data according to claim 1, characterized in that: Keep the value of the first line of monitoring data in file f, and replace the values ​​of monitoring data in other lines with the difference between the next line and this line.

3. The method for lossless compression of highway structure information monitoring data according to claim 1, characterized in that: The monitoring data in the file f2 are subjected to Huffman coding. In the Huffman coding, each monitoring data is regarded as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

4. The method for lossless compression of highway structure information monitoring data according to claim 1, characterized in that: The method further comprises: When it is necessary to restore the monitoring data, according to the characteristic that the encoding algorithm is a symmetric algorithm, the binary file f3 is processed inversely to obtain the decoded file f4 to complete the data restoration.

5. A computer-readable storage device storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for lossless compression of highway structure information monitoring data as claimed in any one of claims 1 to 4 are implemented.

6. A device for lossless compression of highway structure information monitoring data, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, characterized in that: The processor executes the computer program to implement the steps of the method for lossless compression of highway structure information monitoring data as claimed in any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for lossless compression of highway structure information monitoring data as claimed in any one of claims 1 to 4 are implemented.

8. A device for lossless compression of highway structure information monitoring data, characterized in that: include: A data selection module is used to intercept the original highway structure information monitoring data collected by the sensor, record the start and end timestamps, and store the intercepted original highway structure information monitoring data and the start and end timestamps in a file f; A first-order difference processing module is used to perform first-order difference processing on the monitoring data in the file f, and then update the file f to the file f1; An expansion processing module is used to expand the data after the first-order difference processing, remove the zeros and decimal points in the data, and then update the file f1 to the file f2; The encoding module is used to encode the data in the file f2 to obtain the binary file f3 and complete the compression.

9. The method for lossless compression of highway structure information monitoring data according to claim 8, characterized in that: The encoding module performs Huffman encoding processing on the data in the file f2, and in the Huffman encoding processing, each monitoring data is regarded as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

10. The device for lossless compression of highway structure information monitoring data according to claim 8, characterized in that: The device also includes: The restoration module is used to perform inverse processing on the binary file f3 when data restoration is needed, according to the characteristic that the encoding algorithm is a symmetric algorithm, to obtain the decoded file f4 and complete data restoration.

Images