Multi-channel high-performance data storage and processing method based on structured file system

Abstract

This invention discloses a multi-channel high-performance data storage and processing method based on a structured file system, belonging to the field of data processing technology. It is applicable to application scenarios such as testing benches and metallurgy where data acquisition density is high, storage frequency is fast, continuous storage time is long, and single file data volume is large. This invention constructs a structured file system for big data storage, which consists of a data index file, a data description file, and binary data files. Only one data index file and one data description file are generated in a single storage process, while one or more binary data files can be generated as needed. This invention also provides an efficient method for reading this file system, enabling fast and accurate access to large-scale data.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular to a multi-channel high-performance data storage and processing method based on a structured file system. Background Technology

[0002] In industrial measurement and control fields such as testing benches and metallurgy, data storage and processing are core and essential components. Data storage in this field typically exhibits characteristics such as dense acquisition channels, high storage frequency, long continuous storage periods, and large single-file data volumes. After data storage is completed, it needs to support frequent subsequent analysis and processing operations, requiring efficient and convenient retrieval of target data from the stored files. However, directly loading large files not only significantly reduces data retrieval speed but may also cause program crashes; furthermore, if the stored file is corrupted, the entire batch of data will be permanently lost, resulting in serious losses. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a multi-channel high-performance data storage and processing method based on a structured file system. This method can efficiently address the characteristics of "multi-channel, high-frequency storage, and long-term storage" in industrial control fields such as test benches and metallurgy, and achieve efficient management of big data.

[0004] The technical problem solved by this invention is achieved through the following technical solution: A multi-channel high-performance data storage and processing method based on a structured file system includes a data storage process and a data reading process. First, the data is stored in a structured file system, and then the data is loaded and read from the structured file system.

[0005] Furthermore, the data storage process includes the following steps: Step 1.1: The system automatically generates a data description file based on the channel information of the data to be stored; Step 1.2: Configure the storage core parameters according to the data description file; Step 1.3: The system organizes the data points to be stored into a data array according to the channels defined in the data description file, initializes the data cache queue, and adds the organized array as a queue element to the cache queue. Step 1.4: The data cache queue adopts a producer-consumer collaborative working mode to decouple data acquisition and storage; Step 1.5: During the process of step 4, the size of the current binary data file is monitored in real time. When the file size reaches the single file size threshold set in step 1.2, the system automatically creates a new binary data file to continue storage and synchronously updates the index information of the new file name, data start time and estimated end time to the data index file to ensure the consistency between the index information and the stored file. Step 1.6: Scan all binary data files according to the set cycle. At the same time, based on the cold data judgment criteria set in Step 1.2, identify cold data files that meet the conditions and automatically perform compression processing. After compression is completed, immediately update the corresponding file's cold data mark and compressed file size information in the data index file to ensure the accuracy of cold data management.

[0006] Furthermore, the data description file in step 1.1 fully records the core attributes of each data channel, including the channel name, the data type of the corresponding channel, and the size of the data in that channel in bytes.

[0007] Furthermore, the core storage parameters in step 1.2 include: determining that the storage format of the data file is binary, setting a size threshold for a single binary data file, and clarifying the criteria for judging cold data.

[0008] Moreover, the producer-consumer collaborative working mode in step 1.4 is as follows: the producer module is responsible for organizing the real-time collected data into an array according to the rules in step 1.3 and continuously adding it to the cache queue; the consumer module asynchronously extracts data elements from the queue and writes the data into a binary data file in binary format, thereby decoupling data collection and storage and improving storage efficiency.

[0009] Furthermore, the data reading process includes the following steps: Step 2.1: The user inputs the time range of the data to be read, including the start time and end time. The system quickly matches and determines all binary data files containing data within this time range by parsing the data index file. At the same time, it determines whether these files are cold data based on the markers in the index file. If they are cold data, they are first decompressed and preprocessed to ensure that the data can be parsed normally; otherwise, no decompression is required and the binary data file is loaded directly.

[0010] Step 2.2: The system loads the data description file of the corresponding storage task, extracts and confirms all storage channel information of the data to be read, including the name, data type and byte size of each channel, providing a key basis for data parsing and restoration; Step 2.3: The system allocates an appropriate amount of memory space based on the total amount of data to be read, and then loads the binary data files into memory in batches according to the set rules. After the data is parsed, it is displayed.

[0011] The advantages and positive effects of this invention are: The storage and processing method proposed in this invention can efficiently address the characteristics of "multi-channel, high-frequency, and long-term storage" in industrial control fields such as test benches and metallurgy, achieving efficient big data management. The core effects of this invention are as follows: 1. An automatic cold data compression mechanism (with an adjustable 30-day threshold) significantly reduces storage space and alleviates hardware pressure; 2. The "index file + description file + binary file" architecture enables rapid data location, solving the problem of time-consuming multi-channel data reading; 3. Adapting to the needs of industrial control scenarios, the description file ensures orderly storage of multi-source data, while the binary format and dynamic file splitting mechanism support high-frequency, long-term stable storage. Attached Figure Description

[0012] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0013] The present invention will be further described in detail below with reference to the accompanying drawings.

[0014] Multi-channel high-performance data storage and processing methods based on structured file systems, such as Figure 1 As shown, the process includes data storage and data retrieval. First, the data is stored in a structured file system, and then the structured file system is loaded and retrieved.

[0015] The data storage process includes the following steps: Step 1.1: Create a data description file: The system automatically generates a data description file based on the channel information of the data to be stored.

[0016] The data description file fully records the core attributes of each data channel, including the channel name, the data type of the corresponding channel, and the size of the data in bytes for that channel.

[0017] Step 1.2: Configure storage core parameters: Configure storage core parameters according to the data description file.

[0018] The core storage parameters include: determining that the data file is stored in binary format (to achieve storage space compression), setting the size threshold for a single binary data file (which can be customized by the user according to storage needs), and clarifying the criteria for judging cold data (the default threshold is "30 days from the current storage time", which can be adjusted by the user based on business scenarios).

[0019] Step 1.3: Construct a data cache queue: The system organizes the data points to be stored into a data array according to the channels defined in the data description file, initializes the data cache queue, and adds the organized array as a queue element to the cache queue.

[0020] Step 1.4: Data storage based on producer-consumer model: The data cache queue adopts a producer-consumer collaborative working model to decouple data acquisition and storage.

[0021] The producer-consumer collaborative working mode is as follows: the producer module is responsible for organizing the real-time collected data into an array according to the rules in step 1.3 and continuously adding it to the cache queue; the consumer module asynchronously extracts data elements from the queue and writes the data into a binary data file in binary format, thereby decoupling data collection and storage and improving storage efficiency.

[0022] Step 1.5, Dynamic File Splitting and Index Update: During Step 4, the size of the current binary data file is monitored in real time. When the file size reaches the single file size threshold set in Step 1.2, the system automatically creates a new binary data file to continue storage and synchronously updates the index information of the new file's name, data start time, and estimated end time to the data index file to ensure the consistency between the index information and the stored file.

[0023] Step 1.6, Cold Data Timed Compression Processing: Scan all binary data files at a set period, and in conjunction with the cold data judgment criteria set in Step 1.2, identify cold data files that meet the conditions and automatically perform compression processing; after compression is completed, immediately update the corresponding file's cold data mark and compressed file size information in the data index file to ensure the accuracy of cold data management.

[0024] The data reading process includes the following steps: Step 2.1: Locate the target file and preprocess cold data: The user inputs the time range of the data to be read, including the start time and end time. The system quickly matches and identifies all binary data files containing data within that time range by parsing the data index file. At the same time, it determines whether these files are cold data based on the markers in the index file. If they are cold data, they are first decompressed and preprocessed to ensure that the data can be parsed normally; otherwise, no decompression is required and the binary data file is loaded directly.

[0025] Step 2.2: Loading Data Description Information: The system loads the data description file corresponding to the storage task, extracts and confirms all storage channel information of the data to be read, including the name, data type and byte size of each channel, providing a key basis for data parsing and restoration.

[0026] Step 2.3: Load and display data in batches: The system allocates an appropriate amount of memory space based on the total amount of data to be read, and then loads the binary data files into memory in batches according to the set rules. After the data is parsed, it is displayed.

[0027] Parsing Process: Batch Reading Optimization Mechanism To prevent program crashes caused by excessively large data reads in a single session, this invention provides two switchable batch reading modes, which users can choose according to their needs: 1. Loading mode based on data volume: The system presets a fixed data volume threshold for each load. After the user triggers a read operation, the program first loads the data within the threshold range, completes the display, and then loads the next batch of data. This process is repeated until all target data has been read.

[0028] 2. Loading mode by time slice: The program divides the time range to be read into multiple consecutive time slices according to the time granularity of data storage (such as seconds, minutes, hours), and loads the corresponding data in the order of time slices. After the data of one time slice is displayed, the data of the next time slice is loaded, until the data of all time slices has been read.

[0029] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical solutions of this invention are also within the scope of protection of this invention.

Claims

1. A multi-channel high-performance data storage and processing method based on a structured file system, characterized in that: It includes data storage process and data retrieval process. First, the data is stored in a structured file system, and then the structured file system is loaded and retrieved.

2. The multi-channel high-performance data storage and processing method based on a structured file system according to claim 1, characterized in that: The data storage process includes the following steps: Step 1.1: The system automatically generates a data description file based on the channel information of the data to be stored; Step 1.2: Configure the storage core parameters according to the data description file; Step 1.3: The system organizes the data points to be stored into a data array according to the channels defined in the data description file, initializes the data cache queue, and adds the organized array as a queue element to the cache queue. Step 1.4: The data cache queue adopts a producer-consumer collaborative working mode to decouple data acquisition and storage; Step 1.5: During the process of step 4, the size of the current binary data file is monitored in real time. When the file size reaches the single file size threshold set in step 1.2, the system automatically creates a new binary data file to continue storage and synchronously updates the index information of the new file name, data start time and estimated end time to the data index file to ensure the consistency between the index information and the stored file. Step 1.6: Scan all binary data files according to the set cycle. At the same time, based on the cold data judgment criteria set in Step 1.2, identify cold data files that meet the conditions and automatically perform compression processing. After compression is completed, immediately update the corresponding file's cold data mark and compressed file size information in the data index file to ensure the accuracy of cold data management.

3. The multi-channel high-performance data storage and processing method based on a structured file system according to claim 2, characterized in that: In step 1.1, the data description file fully records the core attributes of each data channel, including the channel name, the data type of the corresponding channel, and the size of the data in that channel in bytes.

4. The multi-channel high-performance data storage and processing method based on a structured file system according to claim 2, characterized in that: The core storage parameters in step 1.2 include: determining that the storage format of the data file is binary, setting a size threshold for a single binary data file, and clarifying the criteria for judging cold data.

5. The multi-channel high-performance data storage and processing method based on a structured file system according to claim 2, characterized in that: In step 1.4, the producer-consumer collaborative working mode is as follows: the producer module is responsible for organizing the real-time collected data into an array according to the rules in step 1.3 and continuously adding it to the cache queue; the consumer module asynchronously extracts data elements from the queue and writes the data into a binary data file in binary format, thereby decoupling data collection and storage and improving storage efficiency.

6. The multi-channel high-performance data storage and processing method based on a structured file system according to claim 1, characterized in that: The data reading process includes the following steps: Step 2.1: The user inputs the time range of the data to be read, including the start time and end time. The system quickly matches and determines all binary data files containing data within this time range by parsing the data index file. At the same time, it determines whether these files are cold data based on the marks in the index file. If they are cold data, they are first decompressed to ensure that the data can be parsed normally; otherwise, no decompression is required and the binary data file is loaded directly. Step 2.2: The system loads the data description file of the corresponding storage task, extracts and confirms all storage channel information of the data to be read, including the name, data type and byte size of each channel, providing a key basis for data parsing and restoration; Step 2.3: The system allocates an appropriate amount of memory space based on the total amount of data to be read, and then loads the binary data files into memory in batches according to the set rules. After the data is parsed, it is displayed.

Images