[0047] The technical solutions of the present invention will be further described below in conjunction with the drawings and embodiments.
[0048] See figure 1 , Is a block diagram of the overall structure of the present invention. It is a device that adopts the method of online real-time data compression and decompression of the present invention. It includes real-time data reading interface device 11, real-time data caching device 12, real-time data compression execution device 13, real-time process database device 14, data Query device 15, decompression device 16, data reading device 17;
[0049] The real-time data reading interface device 11 is mainly responsible for the acquisition of data during the operation of the process monitoring system. It reads from the real-time data management module 10 of the field equipment according to the pre-set (which can be set during configuration) the list of tag numbers that need to be read Read the corresponding tag information list and its data;
[0050] The real-time data caching device 12 is used to temporarily store the tag information acquired by the real-time data reading interface device 11 for reading during compression execution. Its characteristic is that the real-time data acquisition and online compression do not interfere with each other, and ensure that the real-time data acquisition and online compression do not interfere with each other. Abnormal termination of the recoverable system data;
[0051] The real-time data compression execution module device 13 is characterized by reading the data to be compressed from the real-time data buffer device 12, and using a certain compression algorithm (such as the more mature LZW algorithm) to compress the read data, and at the same time generate a compressed package ;
[0052] The real-time process database device 14 is used to store compressed data, and is characterized in that it is indexed according to the corresponding relationship between the compressed package and the time range, so that it can be read with high efficiency;
[0053] The data query device 15 mainly queries the data in the real-time process database device 14, and indexes the compressed package address according to the request condition of the query data;
[0054] The decompression device 16 restores the original data from the compressed data according to the corresponding relationship during compression and encoding;
[0055] The data reading device 17 provides the decompressed data to the inquiring user.
[0056] See figure 2 , Is the temporary data file structure in the real-time data caching device 12 of the present invention. In addition to the file identifier 21, the temporary data file also includes a tag data information area 22, a first data area 23, and a second data area 24. The bit number data information area 22 records the status of the first data area 23 and the second data area 24, such as real-time writing, full writing and waiting for compression, compressing, and idle. Two data areas are allocated in the temporary data, so that the temporary data writing and the reading of the compressed data can alternately use the two data areas without interfering with each other. See its workflow Figure 5 , The process is as follows:
[0057] Step S51, real-time collection and recording of one-digit data;
[0058] Step S52: After collecting the tag number data, judge the status of the two data areas of the tag number in the file, and find out the data areas whose status is marked as "real-time write" in the two blocks. If both data areas are "free" "Then, go to step S53, otherwise go directly to step S54;
[0059] Step S53: Take the first block of data area and mark it with "real-time writing" status (except for these two blocks are in idle state, the other mechanism of the present invention guarantees that there will be no other possibility that no block is "real-time writing". In the state of "write", for example, both blocks are to be compressed or are being compressed);
[0060] Step S54, then read the real-time data from the real-time data interface module, and record it in the data area whose status is "real-time writing";
[0061] Step S55, compressed submission; when the data area with the status of "real-time writing" has completed a predetermined amount of data reading and recording, the data area is modified to the status of "full writing and waiting for compression";
[0062] Step S56, judge whether the other data area is "idle", if it is not "idle", go to step S57 and wait for it to be free before performing related operations; if the other data area is "idle", go to step S58;
[0063] Step S58, changing the state of the data area to "real-time writing state", and submitting the data area in the state of "full and waiting for compression" to the compression thread for data compression;
[0064] Step S59, data compression; after receiving the compressed data request, the compression thread puts it into the compression queue. The thread executes the compression request as early as the request time in each work cycle. After the compression is completed, modify the state of the data area to "free".
[0065] See image 3 , Is the compressed data file structure of the present invention. The compressed data file of the present invention is characterized in that it includes several compressed data files, and each compressed data file is set according to the time segment, that is, according to the preset time range of each compressed data file to record data, each new When the time segment is selected, the compression module will create a new compressed data file in the real-time process database module to store the compressed data of the next time segment, which is convenient for historical data query. In addition to the file identifier 31, the compressed data file format also includes the number of bits 32, the starting address of the index area 33, the starting address of the data area 34, the maximum recording time of the file 35, the last recording time of the file 36, and the last recording in the information area. Bit number position 37, file information area 38, index area 39, data area 310 and isolation area 311. The number of bit numbers 32, the starting address of the index area 33, the starting address of the data area 34, and the maximum recording time 35 of the file are preset according to actual needs (can be set during configuration); the last recording time of the file 36, information area The last bit number position 37 is dynamically modified during the compression execution process in the record; the file information area 38 is used to store the bit number information of the data to be compressed, including their ID, range, unit, and description; the index area 39 is used to store each The bit number is based on the correspondence between the data compressed packet and the recording time. The most basic unit of the index area 39 is the index block. The index block contains the time and address range of the compressed data and the corresponding relationship; the data area 310 is used to store the compressed data The data of each bit number; the space allocated for each bit number in the file index area and the data area is determined by other parameters in the bit number information, such as the maximum time of a single data file record and the minimum data record collection period.
[0066] The process of writing and reading compressed data in the index area 39 and the data area 310 is as follows:
[0067] Data writing process:
[0068] After a certain bit number is compressed into a data packet for a period of time, the most recently used index block is found in the index block area allocated for the bit number, and a new index block is created thereafter, and the newly created index block records the packet The time range of the data, and the starting range of the absolute physical address of the data area to be written; after the index area is recorded, the data packet is written in the data area according to the address expected by the index block.
[0069] Data reading process:
[0070] When reading a certain number in the specified time range, the data file that matches the time period is found in the historical data file group (that is, the following Figure 4 instruction of). Afterwards, find the corresponding index block allocated for the bit number in the index area of the data file according to the address range, read the corresponding compressed data packet in the data area according to the address recorded in the index block, and decompress it into a data group, So as to complete the data reading.
[0071] See Figure 4 , Is the index file structure of the present invention. The index file of the present invention is a file that manages all compressed data files and provides a query mode. In addition to the file identifier 41, the file format also includes the number of recorded files 42 and the corresponding index 43 of the data file and the time. . The present invention uses a multi-level index file format to obtain historical data, and the method for obtaining historical data will be described in detail below.
[0072] The online compression and decompression method for real-time data of the present invention includes the following steps:
[0073] a. Read real-time data,
[0074] Set the real-time data reading as an independent thread. When the thread is initialized, the bit number information that needs to be recorded is first read through the real-time data reading interface, and then each collection cycle (for example, 1 second) is read from the bit number information list. The real-time data management module of the field device (the real-time data management module is a functional module that is set in the field device with real-time update data and provides a read and write data interface) to read real-time data and read it. The data is temporarily stored in the real-time data cache module; the real-time data cache module can provide shared read and write operations. There is a mark in the implementation file of the module to indicate the current data status (real-time write, full-write waiting for compression, Compression and idle), this flag makes the real-time data reading and data compression distinguish, thereby providing the possibility for online compression.
[0075] b. Perform data compression,
[0076] In order to make the compression of data and the reading of real-time data not affect each other, the compression execution module of real-time data also uses an independent thread. The compression process of this thread adopts a queue method, and several are removed from the compression queue in each execution cycle. The bit number to be compressed uses a certain compression algorithm to generate compressed data packets. The use of compressed message queues makes the compression process proceed smoothly, avoiding the impact of CPU occupancy peaks on other modules of the system.
[0077] c. Carry out data storage,
[0078] The compressed data is written and saved in the real-time process database module. The data storage of this module adopts a file group structure, including several compressed data files. The time range of the compressed data file to record data can be defined during system configuration. Compressed data The creation of the file is completed by the compression module. The compression thread first checks whether the time limit is reached in each execution cycle and needs to create a new compressed data file. If the conditions are met, the newly created data file is called and the newly created data file is written in the index file, otherwise it is still Operate on the original compressed data file.
[0079] The compressed data package is stored in the data area of the compressed data file, and the absolute address of the compressed package in the file is recorded in the index area of the file. The index area not only records the file address of each compressed package of each bit number, but also includes the data time Wait for data query. In the data area, several compressed packets of a bit number form a data strip (a data strip is a one-time address range allocated for a certain bit number), and the data strips are allocated according to the principle of first-come first allocation and continuous address allocation. In this way, the files gradually grow during the operation of the system, and the storage space is effectively used, and it can run stably even when the hard disk space is tight.
[0080] When the compression thread detects the upper limit of the execution cycle time, it will prepare to call the new data file, but before calling the new data file, all data in the real-time data cache module that still belongs to the previous cycle is compressed and written into the previous cycle. In the data file, in order to ensure the integrity of the data compressed according to the cycle time.
[0081] In addition, considering that even if compression is used, data files cannot be recorded and saved unlimitedly. Therefore, the data storage time can be set in advance (defined when configurable). When the number of data files exceeds, the newly created data file will be overwritten. The oldest file to reach the limit of the number of data files.
[0082] d. Obtain historical data,
[0083] The acquisition of historical data includes data query, decompression and reading. The historical data acquisition module uses dynamic link library technology to facilitate program call. The present invention adopts the file form of multi-level index to obtain historical data, please refer toFigure 6 , The indexing steps are:
[0084] d1. Read the index file first, and obtain a list of compressed data files from the file;
[0085] d2. Query index files and filter these files according to the submitted data time range;
[0086] d3. Open the data file that meets the time requirements and obtain the tag information recorded in the file;
[0087] d4. Search for the tag to be read from the list of tag information obtained, if there is no record of the tag information, return zero, if there is, get the index information of the tag in the file, that is, the information in the corresponding index block ;
[0088] d5. Obtain the data area address range of the bit number according to the obtained index block information to read and filter the data that meets the requirements;
[0089] d6. After obtaining the compressed data packet, gradually restore the compressed data according to the corresponding relationship code table implicitly generated in the compression process of the compressed data as described in the LZW algorithm;
[0090] d7. The data reading module reads the decompressed data and displays it to the user.
[0091] If the time range of the data to be queried is newer than the latest data time in the index file, the data will be queried, filtered and read in the temporary file according to the latest tag information after reading the data file.
[0092] The LZW (Lempel-Ziv-Welch, abbreviation of the three inventors) algorithm used in the present invention is a relatively efficient compression algorithm. The basic principle is: LZW encodes each string that appears for the first time with a numeric value, and then restores this numeric value to the original string in the restoration program. For example: replace the string "abccddeee" with the value 0x100, and replace it with 0x100 whenever the string appears, which has a compression effect. As for the correspondence between 0x100 and character strings, it is dynamically generated during the compression process, and this correspondence is implicit in the compressed data. As the decompression progresses, this code table will be gradually restored from the compressed data. The compressed data will generate more correspondences based on the correspondences generated by the previous data until the end of the compressed file. Since the LZW compression algorithm is lossless, that is, it can be completely restored, so it does not affect the data content.
[0093] Due to the above technical solution, the present invention has the following advantages: the present invention adopts the more mature LZW algorithm, the compression process is aimed at the data structure, and has a higher compression ratio; the compressed data storage of the present invention adopts the file structure form, which realizes the data storage Online compression and storage can ease compression and reading during simultaneous reading; the data storage of the present invention also adopts a multi-level index method, which greatly improves the efficiency of historical data query.
[0094] The above-mentioned embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above-mentioned embodiments without departing from the inventive idea of the present invention. The protection scope of the present invention is not limited by the above-mentioned embodiments, but should be the maximum scope that meets the innovative features mentioned in the claims.
