A method for compressing and storing industrial big data while maintaining its effectiveness and usability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RES INST OF NUCLEAR POWER OPERATION
- Filing Date
- 2026-01-14
- Publication Date
- 2026-05-26
Smart Images
Figure CN122086853A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of big data processing technology, specifically relating to a method for compressing and storing industrial big data to maintain its effectiveness and usability. Background Technology
[0002] Data acquisition in nuclear power plants involves collecting data from numerous sensors, typically on a per-second basis. Over years, this results in a massive dataset. The sheer volume of data is so large that even opening it with common office software is a laborious and time-consuming task, let alone performing specialized analysis such as searching, reading, and computation. Furthermore, the sheer volume of data can lead to a continuous increase in the cost of data storage hardware. Therefore, there is an urgent need to find an effective method to optimize data structures, reduce the amount of data stored, and maintain the validity and usability of the data itself.
[0003] After collecting this data, it will be exported and stored within the planned timeframe. Over the years, this data will accumulate and grow larger. As the data volume increases, both the economic cost of storage and the difficulty of using the data later become increasingly significant.
[0004] Taking nuclear power data acquisition as an example, during nuclear power plant operation, a large number of sensors automatically collect data at a frequency of once per second. These sensors are divided into two categories: one is analog-digital signals (i.e., real digital signals with upper and lower limit ranges), and the other is trigger signals (i.e., switching signals with only 0 and 1). Therefore, the data acquired by nuclear power plants is processed in two ways, as follows: One type is for trigger signal processing. Trigger signal data consists of only two states: 0 and 1. Therefore, all data consists of these two values and their corresponding time. The processing method is to retain the value of the first changing bit. For example, a data recording format like 1, 0, 0, 0, 1 for consecutive times becomes 1, 0, 1. In this case, consecutive identical data will not be recorded; only when a data change occurs will it be recorded.
[0005] Another type is for analog-to-digital signal processing. Analog-to-digital signals are real digital signals with upper and lower limit ranges. The processing methods are as follows: First, for values at the upper and lower limit ranges, similar to trigger-type signal processing, only the initial value is recorded. For special values, such as low current values used for protection, only the first current value is recorded until the current value exceeds the low protection current limit. Similarly, for values with large oscillations that show a stable range, only the first stable value is recorded until a change occurs and recording begins again.
[0006] Using the above method, a lot of data can be compressed. If you need to search, read, or process the data, you only need to follow the above rules. Summary of the Invention
[0007] The purpose of this invention is to provide a compressed storage method that maintains the effectiveness and usability of industrial big data, which can solve the problems of excessive hardware costs and difficulties in data use caused by the large amount of data collected and stored.
[0008] The technical solution of the present invention is as follows: A method for compressed storage that maintains the effectiveness and usability of industrial big data, comprising the following steps: Step 1: Determine the initial data file; Step 2: Develop a data compression program to process and refine the initial data file; Step 3: Compress the data file. After processing by the data compression program, the file is saved to the data storage in the format required by the user. Step 4: Use or apply the data file, use general viewing tools to view the information, or develop data application based on the data.
[0009] Step 2 includes: Step 21: Read the file path, identify the file type and file name; Step 22: Data processing method, reading and analyzing data; Step 23: Data deduplication.
[0010] Step 21 includes storing the read file content into maps, where maps contain the read file content; reading the data and performing data validity processing based on the data physical conditions, specifically calling data processing methods including deduplication of the data.
[0011] Step 21 includes a specific execution method of calling a data deduplication logic method. The principle of processing is that the first piece of data does not need to be deduplicated, and the time and value are retained. If the subsequent data is the same as the first one, it is deduplicated. If the value is different, the value and time are saved. The deduplicated data is saved to lists, which are used to store the deduplicated data information. The data is written and saved as a new file, and the data information in lists is written to the new file.
[0012] Step 22 includes: skipping when a time column is identified; obtaining field data values when a data column is identified; if the field data value is not empty, starting data processing; converting the data into numbers and performing logical judgment to determine whether it is within a valid range; if it is within a valid range, recording the value; if it is not within a valid range, taking the valid boundary value and performing deduplication; and using a Resultlist to store the processed and returned data information.
[0013] Step 23 includes: if the data is empty, jump to step 3, obtain the column values except for the time column; assign the processed result to Resuit storage, where Resuit is the storage space for storing the processed data information; process the column values, the first data does not need to be deduplicated, and the time is retained.
[0014] Step 23 includes adding a first row of time data, adding a first row of value data, deduplicating the subsequent data, clearing the data in the row if the subsequent value is the same as the first value, retaining the value and corresponding time in the row if they are different, skipping the value if the previous value is the same as the current value, and adding the value to the result information to be returned if they are different.
[0015] The beneficial effect of this invention is that the amount of data stored will be greatly reduced after using this data compression and storage method. For example... Figure 1-4 The figure shows the effect of the data volume change after adopting this data compression and storage method. Figure 1-4 The four images show a comparison between data compression and storage without this method. By comparing the amount of data collected in 27 sets over a simple one-minute period, approximately 20% of storage space can be saved. Attached Figure Description
[0016] Figure 1 This represents the amount of data that was not compressed and stored using this method. Figure 2 The amount of data to be compressed and stored using this data compression method; Figure 3 For data structures that do not use this data compression and storage method; Figure 4 The data structure for using this data compression and storage method; Figure 5 This is a logic control diagram of the method used in nuclear power plant data storage. The diagram shows the technical process relationships of using this method. Detailed Implementation
[0017] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0018] A method for compressing and storing industrial big data while maintaining its effectiveness and usability includes the following steps: Step 1: Determine the initial data file Taking nuclear power data acquisition as an example, during the operation of a nuclear power plant, a large number of sensors will automatically collect data at a frequency of once per second. These sensors are divided into two categories: one is analog-digital signals (that is, real digital signals with upper and lower limit ranges), and the other is trigger signals (that is, switching signals with only two types, 0 and 1). These data form the nuclear power acquisition database, which is called the initial data file.
[0019] Industrial big data consists of status records collected by equipment or sensors, typically stored on a hard drive-like storage device, similar to a black box on an airplane. Data is extracted from this black box... Figure 3 The time and data shown do not include the Chinese information in the first column; that information was added later. Whenever the storage device is nearly full, staff export and store the data. Over time, the stored data accumulates, forming the initial data. This method is used to process this data.
[0020] Step 2: Data Compression Program Development The initial data file is very large and inconvenient to use. The purpose of the data compression program is to process the initial data file, making it more concise. The specific implementation process is as follows: Step 21: Read the file path, identify the file type and file name. The read file content is stored in maps, where maps contain the read file content; Read the data and perform data validity processing based on the physical conditions of the data. The specific processing calls the data processing methods, including data deduplication. The specific execution method calls the data deduplication logic method. The processing principle is that the first piece of data does not need to be deduplicated, and the time and value are retained. If the subsequent data is the same as the first one, it is deduplicated. If the value is different, the value and time are saved.
[0021] Save the deduplicated data to lists. Lists stores the deduplicated data. Write the data to and save it to a new file named New.xls.
[0022] Step 22: Data processing method, reading and analyzing data. Skip when a time column is detected; when a data column is detected, retrieve the field data value; if the field data value is not empty, start processing the data; convert the data into numbers and perform logical judgment to determine whether it is within the valid range; if it is within the valid range, record the value; if it is not within the valid range, take the valid boundary value and perform deduplication; use Resultlist to store the processed and returned data information.
[0023] Step 23: Data Deduplication If the data is empty, proceed to step 3 to obtain the column values excluding the time column; assign the processed result to Resuit storage, which is the storage space for storing the processed data; process the column values, the first data does not need to be deduplicated, retain the time. Add the first row of time data, add the first row of value data, and deduplicate the subsequent data. If a subsequent value is the same as the first value, the data in that row is cleared; if they are different, the row value and the corresponding time are retained. If the value in the previous row is the same as the value in the current row, skip that value; if they are different, add them to the result information to be returned.
[0024] Step 3: Compress the data file. After processing by the data compression program, the file is saved to the data storage in the format required by the user, so that the user can save the database data with the smallest possible data space.
[0025] The compression process involves taking the integer data that repeats 0-1, keeping only the first and last integer data, and converting the measurement data that is below or above the valid interval into minimum and maximum boundary values. Then, the measurement data that remains unchanged is taken only from the first and last measurement data.
[0026] Step 4: Use or apply the data file, use general viewing tools to view the information, or develop data application based on the data.
Claims
1. A method for compressing and storing industrial big data while maintaining its effectiveness and usability, characterized in that, Includes the following steps: Step 1: Determine the initial data file; Step 2: Develop a data compression program to process and refine the initial data file; Step 3: Compress the data file. After processing by the data compression program, the file is saved to the data storage in the format required by the user. Step 4: Use or apply the data file, use general viewing tools to view the information, or develop data application based on the data.
2. The method for compressing and storing industrial big data to maintain its effectiveness and usability as described in claim 1, characterized in that, Step 2 includes: Step 21: Read the file path, identify the file type and file name; Step 22: Data processing method, reading and analyzing data; Step 23: Data deduplication.
3. The method for compressing and storing industrial big data while maintaining its effectiveness and usability as described in claim 2, characterized in that: Step 21 includes storing the read file content into maps, where maps are the read file content; Read the data and process its validity based on its physical conditions. Specific processing methods include deduplication.
4. The method for compressing and storing industrial big data to maintain its effectiveness and usability as described in claim 3, characterized in that: Step 21 includes a specific execution method of calling a data deduplication logic method. The principle of processing is that the first piece of data does not need to be deduplicated, and the time and value are retained. If the subsequent data is the same as the first one, it is deduplicated. If the value is different, the value and time are saved. The deduplicated data is saved to lists, which are used to store the deduplicated data information. The data is written and saved as a new file, and the data information in lists is written to the new file.
5. The method for compressing and storing industrial big data while maintaining its effectiveness and usability as described in claim 2, characterized in that: Step 22 includes: skipping when a time column is detected; obtaining field data values when a data column is detected; and starting data processing if the field data values are not empty. Convert the data into numbers and perform logical checks to determine if it is within a valid range. If it is within a valid range, record the value. If it is not within a valid range, take the valid boundary value and perform deduplication. Use a Resultlist to store the processed data.
6. The method for compressing and storing industrial big data to maintain its effectiveness and usability as described in claim 2, characterized in that: Step 23 includes: if the data is empty, jump to step 3, obtain the column values except for the time column; assign the processed result to Resuit storage, where Resuit is the storage space for storing the processed data information; process the column values, the first data does not need to be deduplicated, and the time is retained.
7. The method for compressing and storing industrial big data while maintaining its effectiveness and usability as described in claim 6, characterized in that: Step 23 includes adding a first row of time data, adding a first row of value data, deduplicating the subsequent data, clearing the data in the row if the subsequent value is the same as the first value, retaining the value and corresponding time in the row if they are different, skipping the value if the previous value is the same as the current value, and adding the value to the result information to be returned if they are different.