Highway structure information monitoring data lossless compression method and device

Abstract

The application discloses a lossless compression method and device for highway structure information monitoring data, and solves the problem of difficult transmission and storage of high-frequency massive highway structure information monitoring data, and belongs to the field of road engineering.The application comprises the following steps: intercepting original highway structure information monitoring data, storing the intercepted monitoring data and start-stop time stamp into a file f; performing first-order difference on the monitoring data in the file f to obtain f1; expanding the monitoring data after the first-order difference to remove zero bits and decimal point bits in the monitoring data to obtain a file f2; performing Huffman coding processing on the monitoring data in the f2, wherein each monitoring data is taken as a single value, each single value is coded according to the frequency of the single value, a binary file f3 is obtained, and the compression is completed.When it is necessary to restore the monitoring data, the binary file f3 is processed in a reverse process according to the characteristic of a symmetric algorithm, a decoding file f4 is obtained, and the data restoration is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to a highway structure information monitoring data lossless compression method, belonging to the field of road engineering. BACKGROUND

[0002] Highway structure information monitoring (SHM) is an important means to ensure road structure safety and improve road maintenance efficiency. Long-term performance observation of highways can real-time tap the mechanical response and material parameter changes of asphalt pavement under the action of environment and traffic load, which is an effective way to guide pavement structure design, optimize road service performance and increase road service life. With the continuous upgrading of traffic monitoring technology, the widespread deployment of sensors, cameras and other equipment has led to a massive increase in monitoring data. The storage and transmission of these data pose a huge challenge to urban traffic management systems.

[0003] The previous data transmission and storage method is low in efficiency. In the previous pavement structure monitoring, the monitoring data is usually collected by a sensing element in real time, then stored locally at the monitoring site, and then copied regularly by a data analyst for data analysis and management. However, with the trend of scientific and data-centric long-term performance observation network, this manual copying transmission, storage and analysis method is low in efficiency and cannot meet the processing needs of massive high-frequency highway pavement dynamic monitoring data. In addition, the previous data transmission and storage method is high in cost. For long-term performance observation of roadbed pavement, a large number of sensing elements buried in the pavement structure collect real-time data on vehicle load, stress, strain, internal temperature, internal humidity and other observation indicators. Due to the high sampling frequency of some sensing elements, massive high-frequency data will accumulate at the observation point over a long period of time, resulting in high storage and processing costs.

[0004] The most ideal data transmission and storage architecture is to transmit the highway pavement dynamic monitoring data collected by the sensing element to the cloud with stronger computing power and storage capacity through the network, and then store and process the data. In practical engineering, the difficulty of implementing this architecture lies in the high cost of transmitting and storing the collected high-frequency massive data, and the extremely high requirements for network and power conditions. SUMMARY

[0005] To solve the problem of transmitting and storing the collected high-frequency massive highway structure information monitoring data, the present application provides a highway structure information monitoring data lossless compression method and device.

[0006] The highway structure information monitoring data lossless compression method of the present application comprises:

[0007] The original road structure information monitoring data collected by the sensor is intercepted, start and end time stamps are recorded, and the intercepted monitoring data and the start and end time stamps are stored in a file f;

[0008] The monitoring data in the file f is subjected to first-order difference processing, and the file f is updated to a file f1;

[0009] The monitoring data after the first-order difference processing is subjected to expansion processing to remove zero bits and decimal point bits in the monitoring data, and the file f1 is updated to a file f2.

[0010] The monitoring data in the file f2 is encoded to obtain a binary file f3, and compression is completed.

[0011] As a preferred, the value of the first row of monitoring data in the file f is retained, and the values of the monitoring data in other rows are replaced by the difference between the next row and the current row.

[0012] As a preferred, the monitoring data in the file f2 is subjected to Huffman encoding processing, and in the Huffman encoding processing, each monitoring data is taken as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

[0013] As a preferred, the method further comprises:

[0014] When it is necessary to restore the monitoring data, the binary file f3 is subjected to inverse process processing according to the characteristics of the symmetric algorithm of the encoding algorithm to obtain a decoded file f4, and data restoration is completed.

[0015] The application also includes a lossless compression device for road structure information monitoring data, comprising:

[0016] A data selection module is configured to intercept the original road structure information monitoring data collected by the sensor, record start and end time stamps, and store the intercepted original road structure information monitoring data and the start and end time stamps in a file f;

[0017] A first-order difference processing module is configured to subject the monitoring data in the file f to first-order difference processing, and update the file f to a file f1;

[0018] An expansion processing module is configured to subject the data after the first-order difference processing to expansion processing to remove zero bits and decimal point bits in the data, and update the file f1 to a file f2.

[0019] An encoding module is configured to encode the data in the file f2 to obtain a binary file f3, and complete compression.

[0020] As a preferred, the encoding module subjects the data in the file f2 to Huffman encoding processing, and in the Huffman encoding processing, each monitoring data is taken as a single value, and each single value is encoded according to the frequency of occurrence of the single value.

[0021] As preferred, the device further comprises:

[0022] The restoring module is used for inversely processing the binary file f3 according to the characteristics of the symmetric algorithm of the encoding algorithm to obtain a decoding file f4 and complete data restoration when data restoration is required.

[0023] The present application has the advantages that the highway structure information monitoring data is processed by the method, the original data can be compressed by tens of times in data quantity and stored as a binary file, the data transmission and storage cost is effectively reduced and the efficiency is improved, the compressed binary file can be completely restored by the designed restoring algorithm when the data is required to be applied at the application end, there is no data loss, the restoring algorithm has high efficiency and can meet most restoring requirements. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Fig. 1 is a schematic diagram of the principle of transmission and storage of highway structure information monitoring data;

[0025] Figure 2 Fig. 2 is a schematic diagram of the principle of the lossless compression method of highway structure information monitoring data;

[0026] Figure 3 Fig. 3 is a schematic diagram of the principle of first-order difference processing. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0028] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0029] The present application will be further described below with reference to the drawings and specific embodiments, but is not limited by the present application.

[0030] Figure 1In the method, the buried sensor transmits the collected highway structure information monitoring data to the data acquisition end, the data acquisition end transmits the data to the data center and stores the data, and the data center finally transmits the data to the user end according to the requirement. All the transmission and storage steps need to reduce the magnitude of the data to reduce the cost and improve the efficiency. The core idea of the method is to analyze the characteristics of the highway structure information monitoring data, design the compression and restoration method for the data, and solve the problem of low efficiency and high cost of the highway structure information monitoring data in the storage and transmission process in the aspect of data quantity. According to the analysis and processing of the characteristics of the data, it is found that the non-critical part of the highway structure information monitoring data has similarity and redundancy, but the non-critical part of the data still has the value of transmission and storage, so according to the characteristics, the lossless compression concept is adhered, the first-order difference method and Huffman coding algorithm are combined to complete the design of the highway structure information monitoring data compression algorithm. In order to solve the problem of high cost and low efficiency of the highway structure information monitoring data collected by the existing sensor, the embodiment provides a lossless compression method for highway structure information monitoring data, as shown in Figure 2 , comprising:

[0031] Step 1, preprocessing the original data collected by the sensor:

[0032] The original highway structure information monitoring data collected by the sensor is intercepted: the original data collected by the sensor is preliminarily arranged, and the monitoring data is spliced or split to obtain a data block suitable for compression, and the start and end time stamps of each data block are recorded, the time stamps are checked and corresponded with the data block, a data block and the corresponding start and end time stamps are stored in a file f, which is convenient for subsequent compression processing; the original highway structure information monitoring data of the embodiment is a wavelength;

[0033] Further, in order to conveniently update, transmit, download or process the file, each compressed file has a size limit. Since the size of the compressed highway structure health monitoring file is unknown before data compression, the file f size constraint is set for the highway structure health monitoring data block in this step, and the data block will be compressed into the file f. In addition, the highway network transmission causes the size of each time stamp data block to be different. Therefore, the selection of the data block is related to both the size constraint of the file f and the size of each data block, and the appropriate data block size needs to be selected to improve the compression efficiency, and the file f size of the embodiment is set to 500mb.

[0034] Step 2, first-order difference processing is performed on the monitoring data in the file f, and then the file f is updated to the file f1;

[0035] Since the dynamic response ratio of the highway structure health monitoring data is about 1:200, and the change range of the structure health monitoring data without dynamic response is small, the highway structure health monitoring data fluctuates little, that is, the difference between two adjacent values in the highway structure health monitoring data is small. The embodiment can add a first-order difference to filter redundant values, and record the difference between two adjacent values, as shown in Figure 3 The storage space of each structure health monitoring data file is reduced.

[0036] The process of the first-order difference includes: initializing an empty set f1 to store the compressed code of f. Then, the number and length of the attributes of the file f are obtained. As shown in Figure 3 The first row of monitoring data of the file f is reserved, and for the values of other rows of monitoring data, the values of the monitoring data in other rows are replaced by the difference between the next row and the current row. The first-order difference increases the frequency of numbers close to 0, which facilitates subsequent coding.

[0037] Step 3, the monitoring data after the first-order difference processing is enlarged, the zero and decimal point positions in the monitoring data are removed, and then the file f1 is updated to the file f2;

[0038] After the first-order difference, some monitoring data is rewritten as a decimal number with multiple zeros. In order to further compress the storage space, the data is enlarged to remove the zero and decimal point positions in the data, and the data is compressed under the decimal system, which can generally be multiplied by 1000.

[0039] Step 4, the data in the file f2 is coded to obtain a binary file f3, the compression is completed, and the storage and transmission are performed;

[0040] The coding is to shorten the length of the compressed code of the highway structure health monitoring data. Since the frequency distribution of the structure health monitoring data is uneven, the Huffman coding algorithm is a suitable coding strategy. The embodiment constructs a Huffman tree to store the compressed structure health monitoring data, ensures that each data corresponds to a pre-coding position that does not contain the coding of other data, and reduces the length of the code.

[0041] The existing Huffman coding algorithm extracts all constituent numbers and symbols of the data, and codes each number or symbol according to the frequency of occurrence. In the Huffman coding process of the embodiment, each monitoring data is regarded as a single value, and each single value is coded according to the frequency of occurrence of the single value.

[0042] Step 5, when the data needs to be restored for analysis, the inverse process of f3 is performed according to the characteristics of the symmetric algorithm, and the decoded file f4 is obtained. The f4 is the same as the processed file f1, and the data restoration step is completed.

[0043] The embodiment processes sensor data and presets data volume, improves the accuracy of data and corresponding time stamp, records the time stamp, and facilitates subsequent compression processing; based on analysis of highway structure information monitoring data characteristics, combining data processing method, first-order difference method and Huffman coding, the preprocessed highway structure information monitoring data is binary coded and stored, achieving higher compression efficiency than traditional methods; based on the encoding method, the inverse process is processed to obtain the preprocessed data before encoding, and the binary file is decoded. Efficient lossless data restoration without data loss.

[0044] The embodiment also provides a highway structure information monitoring data lossless compression device, comprising a data selection module, a first-order difference processing module, an expansion processing module, an encoding module and a restoration module.

[0045] The data selection module is used to intercept the original highway structure information monitoring data collected by the sensor, record the start and end time stamps, and store the intercepted original highway structure information monitoring data and the start and end time stamps into a file f.

[0046] The first-order difference processing module is used to perform first-order difference processing on the monitoring data in the file f, and then update the file f to a file f1; specifically, the first row of monitoring data of the file f is retained, and the values of the monitoring data in other rows are replaced by the difference between the next row and the current row. First-order difference increases the frequency of numbers close to 0, which facilitates subsequent coding;

[0047] The expansion processing module is used to perform expansion processing on the data after first-order difference processing, remove the zero and decimal point positions in the data, and then update the file f1 to a file f2.

[0048] The encoding module is used to perform Huffman coding processing on the data in the file f2, and in the Huffman coding processing, each monitoring data is taken as a single value, each single value is coded according to the frequency of the single value, a binary file f3 is obtained, and compression is completed.

[0049] The restoration module is used to perform inverse process on the binary file f3 according to the characteristics of the symmetric algorithm to obtain a decoded file f4 and complete data restoration when data restoration is needed.

[0050] For the newly incoming highway structure health monitoring data and given file size constraint, firstly, a data block selection module is added to generate the to-be-compressed file composed of data blocks. Then, a first-order difference module is added to reduce the storage space of the highway structure health monitoring data. The first-order difference result of each file is processed by expansion, and a Huffman coding module is further added to output the corresponding Huffman code as the final compressed file. Under these modules, the compression device of the embodiment not only retains the key information in the highway structure health monitoring data, but also ensures satisfactory compression performance.

[0051] While the application has been described with reference to particular embodiments thereof, it should be understood that these are merely illustrative examples of the principles and application of the application. It will thus be appreciated that those skilled in the art will be able to devise numerous modifications that, although not explicitly described herein, will fall within the spirit and scope of the application as defined in the following claims. It will be understood that, in general, the head number of dependent claims referred to in the preceding description shall not be construed as limiting the claim in question to the particular embodiment in which that head number is first mentioned. It will be understood that the features of the preceding description can be used in any combination of one or more embodiments.

Claims

1. A method for lossless compression of highway structure information monitoring data, characterized in that, The method comprises: intercepting original highway structure information monitoring data collected by a sensor, recording start and end time stamps, and storing the intercepted monitoring data and the start and end time stamps into a file f; performing first-order difference processing on the monitoring data in the file f, and updating the file f into a file f1; the first-order difference processing comprises: retaining the value of the monitoring data in the first row of the file f, and replacing the values of the monitoring data in other rows with the difference between the next row and the current row; performing expansion processing on the monitoring data after the first-order difference processing, removing zero bits and decimal point bits in the monitoring data, compressing the monitoring data in the decimal system, and updating the file f1 into a file f2; performing Huffman coding processing on the monitoring data in the file f2, and in the Huffman coding processing, each monitoring data is taken as a single value, each single value is coded according to the frequency of occurrence of the single value, a binary file f3 is obtained, and compression is completed.

2. The method for lossless compression of highway structure information monitoring data according to claim 1, characterized in that, The method further comprises: when it is necessary to restore the monitoring data, performing inverse process processing on the binary file f3 according to the characteristic that the coding algorithm is a symmetric algorithm, obtaining a decoded file f4, and completing data restoration.

3. A computer-readable storage device storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 2. The computer program is executed by the processor to realize the steps of the highway structure information monitoring data lossless compression method according to any one of claims 1 to 2. 4.A device for lossless compression of highway structure information monitoring data, comprising a storage device, a processor, and a computer program stored in the storage device and executable on the processor, wherein, The processor executes the computer program to realize the steps of the highway structure information monitoring data lossless compression method according to any one of claims 1 to 2.

5. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the highway structure information monitoring data lossless compression method according to any one of claims 1 to 2.

6. Highway structure information monitoring data lossless compression device, characterized in that, comprises: a data selection module configured to intercept original highway structure information monitoring data collected by a sensor, record start and end time stamps, and store the intercepted original highway structure information monitoring data and the start and end time stamps into a file f; a first-order difference processing module configured to perform first-order difference processing on the monitoring data in the file f, and update the file f into a file f1; the first-order difference processing comprises: retaining the value of the monitoring data in the first row of the file f, and replacing the values of the monitoring data in other rows with the difference between the next row and the current row; an expansion processing module configured to perform expansion processing on the data after the first-order difference processing, remove zero bits and decimal point bits in the data, compress the monitoring data in the decimal system, and update the file f1 into a file f2; an encoding module configured to perform Huffman coding processing on the data in the file f2, and in the Huffman coding processing, each monitoring data is taken as a single value, each single value is coded according to the frequency of occurrence of the single value, a binary file f3 is obtained, and compression is completed.

7. The apparatus for lossless compression of highway structure information monitoring data according to claim 6, characterized in that, The device further comprises: a restoration module configured to, when it is necessary to restore the data, perform inverse process processing on the binary file f3 according to the characteristic that the coding algorithm is a symmetric algorithm, obtain a decoded file f4, and complete data restoration.

Images