Method and device for fusing, storing and managing massive small files and data records

Abstract

The invention discloses a method and device for fusing, storing and managing massive small files and data records, and the method comprises the steps: building an external metadatabase which is used for managing a hierarchical relation of scientific observation data sources and a mapping relation between an aggregation file and an original file of scientific observation data; according to a preset aggregation rule based on data product categories and timelines, the multiple original files are aggregated and packaged in a target file container in a column storage format, and the content of the original files and structured data records obtained by analyzing the original files are stored in the target file container at the same time; and based on the external metadatabase and the target file container, providing file-level access aiming at the original file content and record-level retrieval service aiming at the structured data record. According to the method, the aspects of system simplification, performance optimization, space saving and service capability are remarkably improved, and a new way with high expandability is provided for efficient storage and multi-level service of space weather scientific data.

Description

Technical Field

[0001] This invention belongs to the field of scientific data storage, and in particular relates to a method and apparatus for integrating storage management of massive small files and data records. Background Technology

[0002] With the continuous development of high technology and the increasing reliance on it, space weather has become increasingly significant in its impact on socio-economic development and national defense security. After long-term research and exploration of various space weather phenomena such as coronal mass ejections, solar flares and geomagnetic storms in the Sun-Earth space environment, the Sun-Earth space environment is gradually being studied as a whole system. At the same time, many countries and international organizations have begun to build comprehensive space environment monitoring systems.

[0003] With the enhancement of monitoring capabilities, expansion of monitoring scope, and improvement of data timeliness in the second phase of the Meridian Project, the data from the second phase project exhibits characteristics of more complex data file types and a larger quantity compared to the first phase project: the total amount of data and storage volume have both increased by two orders of magnitude.

[0004] In terms of storage, the Meridian Project Phase II employs a cold and hot data storage mechanism of "data record repository + data file repository" to store data files uploaded by monitoring equipment. The "data file repository" is implemented using a relational database and file system for storing all data files; the "data record repository" is based on HBase for storing and retrieving massive amounts of space weather scientific data records. Regarding querying, the core data service of the Meridian Project Phase II currently only supports data file-based services and has not yet implemented data record-based data retrieval services, resulting in a limited range of external data services.

[0005] During the second phase of the Meridian Project, the number of observation stations and equipment density significantly increased, and the observation frequency was continuously optimized. Statistical data shows that the annual data growth of the second phase has reached over 30 million files, amounting to 7 PB. The vast majority of these files range in size from several hundred KB to several GB, and the data formats include CSV, CDF, HDF5, and binary streams. Specifically, the Circular Array Solar Radio Imaging Telescope, the Meter-Wave and Ten-Meter-Wave Radio Heliograph, the Centimeter-Decimeter-Wave Radio Heliograph (MUSER), and the Meter-Wave Solar Radio Spectrometer produced a total of 966,500 data files annually, totaling 6.29 PB. The Mid-Latitude High-Frequency Radar and the Circular Array Solar Radio Imaging Telescope produced a total of 463,000 data files annually, totaling 0.053 PB. In contrast, other monitoring data produced a staggering 32.265 million data files annually, but only 0.79 PB in total. Faced with such a large-scale, multi-source, heterogeneous data asset, how to effectively solve the problem of massive small files, and further build a refined management system based on data records and provide diversified data services to support researchers in conducting efficient data queries and free-exploratory in-depth data analysis, is the core technical challenge facing the Meridian Engineering Data Center.

[0006] The current technical solutions face the following main problems: 1. The Challenge of Massive Small Files: The problem of massive small files has a significant impact on various storage systems, including traditional file systems, distributed file systems, and tape libraries. In traditional file system architectures, massive small files drastically increase the burden of metadata management. Each small file requires a separate inode entry to store metadata information, which not only occupies a large amount of disk storage space but also requires maintaining a large inode cache table in system memory, leading to rapid consumption of memory resources. Simultaneously, small files exacerbate disk fragmentation, requiring frequent seek operations by the read / write head, significantly reducing random read / write performance and making it difficult to utilize the high bandwidth characteristics of sequential disk access. Consequently, the complexity of backup, retrieval, and management tasks increases exponentially, resulting in a significant decline in overall system performance.

[0007] In distributed storage systems, the problem of massive small files mainly manifests in two aspects: metadata and data paths. Firstly, at the metadata level: each file / object introduces fixed metadata overhead (name, permissions, timestamps, location mappings, verification information, etc.). Regardless of whether metadata is stored in memory, disk, or an external database / distributed key-value store, a surge in the number of files significantly increases the capacity and throughput pressure of the metadata service, specifically manifested as increased memory or storage usage and higher access latency, thus limiting the system's scalability and stability. Secondly, at the data level: small files weaken the system's optimization assumptions for large-block sequential read / write operations, leading to inefficient use of disk and network resources, increased proportion of random I / O and protocol overhead, amplified fixed costs per file such as block / object placement and replication, reduced cache hit rates, and increased burden on scheduling and background maintenance tasks. In summary, massive small files amplify overhead on both the control plane and data plane, becoming a key bottleneck for the expansion and performance of large-scale storage systems. In upper-layer systems like HBase that rely on HDFS, the compaction and region splitting processes further generate a large number of small HFile files, further amplifying the systemic impact of the small file problem and severely restricting overall performance.

[0008] Compared to disk-based file systems, tape libraries experience more significant performance degradation when dealing with massive amounts of small files. Tape media, primarily designed for sequential read / write operations, is suitable for storing large, continuous data streams but not for frequent random access. When a tape library is used to store a large number of small files, the drive needs to frequently perform volume loading, addressing, and starting / stopping, resulting in a high proportion of mechanical movements and significantly reducing throughput. Compared to the hundreds of MB / s bandwidth achievable with sequential writing of large files, the actual transfer rate in small file mode is often less than one-tenth, while access latency increases dramatically. Because tape organizes data in fixed-size record blocks, independent writing of small files leads to space fragmentation and padding overhead, reducing media utilization and causing external directories and indexing systems to expand rapidly, resulting in decreased retrieval efficiency. High-frequency loading operations also accelerate wear on the read / write heads and tape media, reducing system reliability and lifespan. In data recovery scenarios, the scattered layout of small files complicates the location process, significantly extending recovery time and hindering batch archiving and long-term storage tasks.

[0009] In general, while the problem of massive small files manifests differently across various storage architectures, they share commonalities: an increase in the number of small files amplifies the metadata burden, weakens sequential access performance, and significantly increases the management complexity of the storage system. This problem is particularly pronounced in sequential access storage systems such as tape libraries. Therefore, in practical applications such as space weather scientific data, strategies such as file aggregation, hierarchical indexing, and tiered storage should be employed to logically or physically integrate large numbers of small files, thereby improving system storage and access efficiency and fully leveraging the performance advantages of various storage media.

[0010] 2. Limited Data Service Model: The Meridian Project currently employs a core data service model that only provides basic services based on raw data files. Under this model, users ultimately receive only a single file, lacking the ability to directly access the data content at a fine-grained level. Users cannot directly obtain the filtered and preprocessed target dataset; instead, they must download the complete raw file and process it themselves.

[0011] 3. Operational challenges arising from system architecture complexity: For example, HBase heavily relies on several core components of the Hadoop ecosystem, primarily including: HDFS: Responsible for underlying data storage and replica management. ZooKeeper: Responsible for cluster coordination, metadata management, and fault detection. HBase Master: Responsible for Region allocation and load balancing scheduling tasks. RegionServer: Responsible for the actual data service processing. This multi-layered distributed architecture makes system deployment and configuration extremely cumbersome, requiring a deep understanding of the configuration parameters and interaction mechanisms of each component. Monitoring and maintenance work needs to cover multiple component levels, and fault diagnosis and troubleshooting are highly complex. Scaling up and down involves multiple technical aspects such as Region reallocation and data migration, placing high demands on the technical capabilities of the operations and maintenance team and significantly increasing system stability risks and maintenance costs.

[0012] In the field of space environment monitoring and scientific observation, all raw data files recording physical parameters are highly valuable observational data assets that must be securely, reliably, and efficiently preserved. In shared file systems employing hierarchical archiving, the underlying layer typically consists of disk arrays and tape libraries, with the physical location of files determined by the archiving strategy. Even so, a massive number of small files still puts systemic pressure on the control and data planes, becoming a key bottleneck for end-to-end performance and scalability. Specifically, namespace amplification and request amplification caused by small files significantly increase the memory / storage usage of metadata services, log and snapshot maintenance, fault recovery, and scanning operation costs. Simultaneously, small-granularity I / O on the data path cannot leverage sequential bandwidth and batch processing advantages, leading to decreased throughput, increased latency, and degraded resource utilization. These effects are not eliminated by whether the files reside on disk or tape, but rather manifest in different forms and propagate among themselves.

[0013] At the disk level, small files increase the proportion of random I / O and amplify seek time, making it difficult to effectively utilize queue depth and sequential prefetching, exacerbating cache pollution, and consequently leading to read / write amplification. At the tape level, although small files are not usually directly written to tape, when callbacks or policy adjustments trigger migrations, small files cause frequent loading, positioning, and start / stop operations, significantly lengthening queuing and loading / positioning latency, worsening long-tail latency distribution, and accelerating wear and tear on media and mechanical components. In summary, massive amounts of small files add overhead to multiple stages, including namespace management, I / O scheduling, media access, and lifecycle orchestration, thereby reducing overall storage efficiency and reliability.

[0014] Existing small file merging strategies can alleviate the problems of metadata scale and media utilization to some extent, but their interfaces are usually still at the file level, making it difficult to support record-level predicate filtering, index access, and fine-grained lifecycle management. This is particularly limiting in scenarios requiring highly selective retrieval, incremental updates, or transparent access across layers. Therefore, in a layered archiving environment, if fine-grained data accessibility is to be achieved while maintaining physical aggregation to suppress the small file effect, a self-describing data container with statistical and indexing capabilities, along with its upper-layer metadata / directory services, should be introduced to achieve a better performance-functionality balance between the control plane and the data plane. One of the main purposes of this approach is to simplify the system architecture, but this simplification also limits the fine-grained capabilities of data services. Summary of the Invention

[0015] The purpose of this invention is to overcome the defects of the prior art and to propose a method for integrating storage management of massive small files and data records. This invention also discloses an apparatus for integrating storage management of massive small files and data records.

[0016] In view of this, the present invention proposes a method for integrating storage management of massive small files and data records, comprising: Step 1: Establish an external metadata database to manage the hierarchical relationships of scientific observation data sources and the mapping relationship between aggregated files and the original files of scientific observation data; Step 2: According to the preset aggregation rules based on data product category and timeline, multiple original files are aggregated and encapsulated in a target file container with columnar storage format. The target file container stores the content of the original files and the structured data records parsed from the original files. Step 3: Based on the external metadata database and the target file container, provide file-level access to the original file content and record-level retrieval services for structured data records.

[0017] As an improvement to the above method, the external metadata database established in step 1 is a relational database, including: The site table is used to record basic information about the observation sites; The instrument table is used to record instrument information at each observation station. The data product table is used to list the data products generated by the instrument, forming a hierarchical mapping relationship of "instrument → data product". The file type table is used to define the file type structure of data products and describe the content and data column structure of each file type. A data list is used to store data columns and data types. File type-column mapping table, used to define the set of columns corresponding to each file type; and The file container table is used to establish the mapping relationship between data products and packaged files in the object storage system, which is compatible with the S3 support range request.

[0018] As an improvement to the above method, the aggregation rules in step 2 based on data product category and timeline include: Package multiple raw files belonging to the same data product category into the same Parquet file in chronological order; The Parquet file includes a Header, a Block, and a Footer, wherein... Header: Used to contain file identification information; Block: Used to store specific column data. Each Block contains multiple row groups. During the packaging process, the original files to be processed are assigned to the corresponding row groups. If the file is unparsable, the fileName, content, and time fields of the file are recorded. If the file is parsable, the parsed structured content is appended to the same row group after the basic information of the file is written in the first line. Footer: Used to record the row groups, column blocks, and offset information of the Parquet file.

[0019] As an improvement to the above method, step 2 further includes: establishing corresponding metadata records in an external metadata database and maintaining association with the footer inside the Parquet file. The external metadata is responsible for global indexing and data location.

[0020] As an improvement to the above method, step 3, which involves file-level access to the original file content, includes: The target file container is located based on the external metadata index, and the corresponding line group of the file content is read to return the complete original data file.

[0021] As an improvement to the above method, step 3, which is a record-level retrieval service for structured data records, includes: In response to a data record-level query request, the target file container is located based on the external metadata index. The predicate pushdown query is then performed on the data record columns in conjunction with the query conditions. Only the blocks of data record columns that are hit are read and returned.

[0022] As an improvement to the above method, the method further includes: a step of adding original files in real time using an asynchronous access mechanism based on a message queue, specifically including: Receive new data file events via message queue; Based on the information of the newly added data file, dynamically match the corresponding parsing and encapsulation strategy; According to the strategy, after processing the newly added data files, they are written to the target file container and the external metadata index is updated.

[0023] On the other hand, the present invention also provides an apparatus for integrating storage management of massive amounts of small files and data records, comprising: The metadata management module is used to establish an external metadata database, manage the hierarchical relationship of scientific observation data sources, and the mapping relationship between aggregated files and the original files of scientific observation data; The file aggregation and encapsulation module is used to aggregate and encapsulate multiple original files into a target file container in a columnar storage format according to preset aggregation rules based on data product categories and timelines. The target file container simultaneously stores the contents of the original files and structured data records parsed from the original files. The data service module provides file-level access to the original file content and record-level retrieval services for structured data records, based on an external metadata database and target file containers.

[0024] As an improvement to the above-mentioned device, the device further includes a real-time file processing module for adding original files in real time using an asynchronous access mechanism based on a message queue.

[0025] Compared with the prior art, the advantages of the present invention are: The technical solution proposed in this application has significant technical advantages and comprehensive improvement effects in terms of system dependency, storage efficiency, space utilization, and data service capabilities, specifically reflected in the following aspects: 1) Lower system dependency and significantly reduced operation and maintenance costs: For example, the HBase (distributed database) technology system heavily relies on Hadoop ecosystem components, including HDFS (file storage layer), ZooKeeper (coordination service), and multiple layers of distributed components such as HBase Master and RegionServer. This results in complex system deployment, difficult monitoring and maintenance, cumbersome scaling processes, high overall operation and maintenance costs, and poor stability. In contrast, the solution proposed in this application relies only on an object storage system and a relational database, achieving a lightweight architecture design that significantly reduces system deployment complexity and operation and maintenance costs, while improving system stability and scalability.

[0026] 2) Effectively solves the problem of storing massive amounts of small files: To address the metadata pressure and inefficient access issues caused by a large number of small files in the file system in the original solution, this method uses an "aggregation and packaging mechanism based on data products and timelines" to aggregate and package small files from the same source into a unified Parquet container file in chronological order. This achieves structured integration and efficient management of small files, fundamentally eliminating the impact of a large number of small files on system performance and stability.

[0027] 3) Higher space utilization efficiency: This method employs the Parquet columnar storage format, which features data compression and encoding optimization, significantly improving space utilization compared to directly storing the original small files. Experimental results show that, after aggregating and packaging the original small files, an average of approximately 50% of storage space can be saved, thereby reducing storage overhead and improving I / O efficiency.

[0028] 4) Supports dual data services at both the file level and the data record level: The original solution only supported data access at the file level, providing coarse-grained services that failed to meet the needs of research users for record-level data analysis. This application's solution, by synchronously storing the original file and parsed data within a Parquet file and combining this with external relational database metadata indexing, achieves integrated support for both file-level access and record-level query services. This ensures both the integrity and usability of the original data and meets the high-granularity data access requirements of research analysis.

[0029] 5) This method is designed for the field of space weather science data, aiming to explore new models for space weather science data services and to build fundamental data storage and management methods for the concept of digital space (digital space). By optimizing data organization and model design, it improves the system's disk space utilization and data management efficiency, providing support for the efficient service of space weather science data.

[0030] In summary, the technical solution of this application, by introducing object storage, columnar aggregation and metadata collaborative indexing mechanism, has achieved significant improvements over the original solution in terms of system simplification, performance optimization, space saving and service capabilities, providing a highly scalable new approach for efficient storage and multi-level services of space weather scientific data. Attached Figure Description

[0031] Figure 1 This is the ER diagram of the present invention; Figure 2 This is a flowchart of the data processing and storage process of the present invention; Figure 3 This is a deployment diagram of the present invention. Detailed Implementation

[0032] This invention proposes a two-tiered data service method based on Parquet (column-oriented storage format) columnar storage. This method not only effectively solves the storage and access performance bottleneck caused by small files in traditional file systems, but also simultaneously supports file-level and data record-level data service capabilities. It addresses the issues of system load and low management efficiency resulting from the storage of massive amounts of small files, while providing upper-layer applications with both raw file-level access and data record-level query services.

[0033] In the field of space environment monitoring and scientific observation, all raw data files recording physical parameters are valuable scientific data assets that must be securely, reliably, and efficiently preserved. However, storing and accessing massive amounts of small files directly as is becomes a performance killer on almost any storage medium and architecture. This is because each file carries a significant fixed overhead: metadata maintenance, opening / closing and permission verification, network and protocol round trips, index and log updates, caching and queue management, etc. Simultaneously, small I / O cannot fully utilize the underlying sequential bandwidth and batch processing capabilities, leading to decreased throughput, increased latency, and poorer resource utilization, while also amplifying the risks of background maintenance and media corruption. This significantly reduces overall storage efficiency and reliability. While existing small file merging schemes can alleviate file storage pressure, they only provide file-level access interfaces and cannot achieve flexible querying and fine-grained management at the data record level. One of the main purposes of this approach is to simplify the system architecture, but this simplification also limits the fine-grained capabilities of data services. Therefore, a two-tier data service method based on Parquet columnar storage is proposed. This method not only effectively solves the storage and access performance bottleneck caused by small files in traditional file systems, but also supports both file-level and data record-level data service capabilities. It addresses the issues of system load and low management efficiency resulting from the storage of massive amounts of small files, while simultaneously providing upper-layer applications with both raw file-level access and data record-level query services.

[0034] The technical solution proposed in this invention adopts the following core design concept: Parquet columnar storage format, as a data file container, is used to aggregate small source files of the same type while retaining the original file content and the parsed structured records, providing unified data storage and multi-level services, and achieving efficient data compression and columnar reading. As an underlying distributed storage medium, the object storage system provides highly scalable and fault-tolerant data persistence support; the object storage system in this invention is compatible with the S3 support range request.

[0035] Relational databases, as external metadata management systems, are responsible for managing the organizational structure of sites, instruments, and data products, storing external metadata for files, and providing index support for file storage and querying.

[0036] In the system implementation process, firstly, based on the business structure of the observation data, a multi-level metadata model of observation stations → observation instruments → data products is constructed in a relational database, forming a complete data hierarchy and lineage. Then, taking the Meridian Project as an example, the original small files from the same source in the Meridian Project are parsed and their content deconstructed, and then aggregated and packaged into Parquet files according to logical relationships. Simultaneously, corresponding external metadata information is established in the relational database. Finally, by combining the metadata in the external relational database with the embedded metadata in the footer area of ​​the Parquet files, efficient joint retrieval and data location are achieved. This design not only solves the system load problem caused by storing massive amounts of small files, but also provides upper-layer applications with the service capabilities of accessing raw files and querying data records.

[0037] This invention provides a method for integrating storage management of massive small files and data records. It is a method for managing massive small files and providing two-tiered data services based on Parquet and object storage systems. In one embodiment, taking Meridian Project as an example, it includes: Step 1: Predefining and modeling the external metadata database During the system initialization phase, an external relational database metadata management system is established. A multi-level hierarchical data structure is constructed within the relational database, from observation sites to observation instruments to data products, realizing the mapping relationship from upper-level observation sites to lower-level data products. This hierarchical model supports subsequent data aggregation, indexing, and retrieval operations, ensuring logical consistency and scalability.

[0038] Step 2: Small file parsing and packaging rule definition Small file packaging and aggregation rules are established based on the Parquet columnar storage structure. A Parquet file consists of three parts: Header, Block, and Footer. Header: Contains file identification information; Block: Stores specific column data; Footer: Automatically calculates and records the file's global metadata.

[0039] The customized Parquet schema is: [fileName, content, time, <parsed data column>]. Each block contains multiple row groups. During the packaging process, small files to be processed are assigned to the corresponding row groups: if a file is unparseable, the fileName, content, and time fields are recorded; if a file is parsable, after writing the basic file information on the first line, the parsed structured content is appended to the same row group, ensuring the structured access capability of the data records.

[0040] Step 3: Aggregating Historical Small Files and Associating Metadata For historical observation data from the Meridian Project, grouping and time-series aggregation are performed according to different data product categories. The system packages source files in chronological order into corresponding Parquet file containers and creates corresponding metadata records in MySQL. The external metadata in MySQL is linked to the internal footer metadata of the Parquet files. External metadata is responsible for global indexing and data location; The internal metadata (Footer) records the row groups, column blocks, and offset information of the Parquet file for quick access.

[0041] For parsable files, their content and the parsed time-series data are stored in the same Row Group; for unparsable files, only the original file data is retained. This mechanism solves the load problem caused by a large number of small files and enables dual data access capabilities at both the file level and the data record level.

[0042] Step 4: Access and Strategic Processing of Real-Time Files For subsequent new observation data from the Meridian Project, an asynchronous access mechanism based on message queues is adopted. When consuming new file messages, the system uses a design scheme combining template pattern, strategy pattern, and factory pattern to dynamically match the corresponding parsing and packaging strategy for each type of data product.

[0043] The consumer end automatically extracts site, instrument, and data product information according to the naming rules of small files, and instantiates the corresponding processing strategy class through the factory class to realize a flexible and scalable online file processing workflow.

[0044] Step 5: Data Access and External Services External data access follows a process of relational database metadata indexing → Parquet file location → precise content retrieval. When a user submits a query request, the system first performs an index retrieval in the relational database metadata to determine the target Parquet file and its object path in the object storage system; then, it performs secondary location based on the metadata information in the footer to determine the data segment and execute a partial download request, thereby achieving efficient partial reading.

[0045] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0046] Example 1 Embodiment 1 of the present invention proposes a method for integrating storage management of massive small files and data records. Taking Meridian Project as an example, it specifically includes: Step 1: Establish an external metadata database During the system initialization phase, an independent relational database metadata management system is first created to manage the mapping relationships between observation stations, instruments, data products, and files in object storage, thereby providing structured metadata support for upper-level data indexing and retrieval.

[0047] The established database mainly includes the following seven core data tables: such as Figure 1 As shown: The SITE table records basic information about the observed sites. Fields: id (primary key), name (site name).

[0048] The instrument table records instrument information for each observation site. Fields: id, site_id (foreign key, related to SITE), name.

[0049] The Data Products table (DATA_PRODUCT) records the data products generated by the instrument, forming a hierarchical mapping relationship of "instrument → data product". Fields: id, instrument_id (related to INSTRUMENT), name, filetype_id (related to FILETYPE).

[0050] The File Type table (FILETYPE) defines the file type structure of the data product, describing the content and data column structure of that file type. Fields: id, name.

[0051] The COLUMN_LIST stores all possible data columns in the system and their data types. Fields: id, column_name, data_type.

[0052] The File Type-Column Mapping Table (FILETYPE_COLUMN) establishes a many-to-many mapping relationship between file types and columns, defining the set of columns corresponding to each file type. Fields: filetype_id, column_id.

[0053] The file container table (PRODUCT_FILE) is used to establish the mapping relationship between data products and packaged files (Parquet files) in the object storage system. Fields: id: Primary key identifier; data_product_id: Foreign key, associated with the DATA_PRODUCT table, used to identify the data product to which the file belongs; file_path: VARCHAR(2048), records the access path of the file in the object storage system; year: CHAR(4), indicating the time partition to which the file belongs.

[0054] Function Description: The PRODUCT_FILE table is a key index table in the system that connects MySQL metadata with object files in the object storage system. It mainly implements: maintaining the correspondence between data products and actual packaged files; unified management and fast location of file paths in the object storage system; supporting timeline data retrieval for the annual partition field; and providing an index foundation for file-level and record-level data access.

[0055] After completing the table structure design, each table is initialized and populated to ensure consistency with the existing station, instrument, and product information of the Meridian Project.

[0056] Step 2: Establish rules for parsing and packaging small files To efficiently store massive amounts of small files, this method uses the Parquet columnar storage format as a unified file container. Since Parquet is based on a strict schema, small files from different sources (heterogeneous) require independent schema structures to adapt to various data products.

[0057] To ensure that both the original file and the parsed data can be stored, this method designs a compatibility schema, which includes the following: time (timestamp), fileName (filename), content (file content), col_1, col_2, ..., col_n (parsed content columns) In terms of data layout: each file corresponds to a Row Group; the first row of the Row Group is used to store time, fileName, and content, and the parsed columns are empty; subsequent rows are used to store the parsed structured content, with the parsed columns filled with data, while the file columns are left empty.

[0058] This design allows the system to preserve both the original appearance of the file and its structured information, thus accommodating the access needs of both.

[0059] Step 3: Pack and store historical small files For small files of historical observation data from the Meridian Project, the system aggregates them according to data product type, packaging files from the same source along the time series into corresponding Parquet containers. The generated Parquet files are stored in an object storage system, and their paths and metadata are registered in the PRODUCT_FILE table to ensure consistency between external metadata and stored objects.

[0060] For parsable files, the original content and the parsed structured record are stored in the same Row Group; for unparsable files, only the time, filename, and file content fields are retained.

[0061] Step 4: Process newly added small files in real time For newly added files each day, the system receives file events through a message queue mechanism. Employing a design approach combining template, strategy, and factory patterns, it dynamically assigns individual parsing and packaging strategies to different data products. The message consumer parses the file name and extracts information about the site, instrument, and data product. Factory method instantiates the matching data processing strategy object; Perform parsing and packaging operations, and write the results to the object storage system and external metadata database; Synchronously update the PRODUCT_FILE metadata to achieve real-time data import and metadata management. (See also) Figure 2 (See flowchart.) Step 5: Data Reading and External Services The system provides file-level access and record-level retrieval services. Its implementation mechanism is as follows: Based on the query criteria, retrieve the corresponding PRODUCT_FILE record from the external metadata database to locate the target Parquet file and its object storage system path; Read the Footer section of the target file and parse the metadata of the row groups and column chunks; Calculate the starting position of the footer: Fs = L 8 Lm Where L is the total file length and Lm is the Footer metadata length; Parquet files use a standard footer metadata storage method, including a data area, footer metadata, a 4-byte length field, and a 4-byte magic number. Therefore, in the above formula, 8 bytes represent the 4-byte length field and the 4-byte magic number.

[0062] Extract row group sequence: G={G1,G2,…,Gn} Each row group contains several column blocks, and each column block contains the file offset, size, data type, and schema path; Based on the query conditions, select the set of hit column paths Pc to form the set of hit column blocks C′; Perform predicate pushdown optimization and continuous interval merging to generate the final request range R; Implement the corresponding API service to provide data services to the outside world: internally perform partial downloads and only read the target data area to achieve efficient access.

[0063] Innovation points: This invention proposes a comprehensive method for storing massive amounts of small files and providing efficient data services. It aims to solve the technical challenges of handling massive amounts of small files while enhancing the system's external data service capabilities. This method achieves efficient data management and a two-tiered service capability through the integration of file aggregation and columnar storage technologies. Specific innovations are as follows: File aggregation mechanism based on data products and time dimension: To address the issue of multi-source, heterogeneous small files in the Meridian Project, this method proposes a strategy for file aggregation based on a dual dimension of "data product - timeline". The system aggregates and packages small files from the same source within a given time window according to the data product category, forming a logically unified and physically contiguous large file. This mechanism effectively reduces the number of files in the underlying storage system, significantly reduces metadata load and I / O overhead, thereby improving the overall storage and access performance of the system.

[0064] A two-tier data service architecture based on Parquet columnar storage: This method uses the Parquet columnar storage format as a unified data container, encapsulating the original file and its parsed structured data within the same Row Group. Through this design, the system can simultaneously support: File-level data service: Provides complete access to and download of raw files; Data record-level service: Enables efficient querying and refined retrieval of file content based on a columnar structure.

[0065] This data service system, which integrates file-level and record-level data, balances the integrity of raw data preservation with the flexibility of scientific data analysis, and constructs a unified data management architecture that combines efficient storage with diverse data service capabilities.

[0066] Example 3 Embodiment 3 of the present invention provides an apparatus for integrating storage management of massive small files and data records, implemented based on a method embodiment, comprising: The metadata management module is used to establish an external metadata database, manage the hierarchical relationship of scientific observation data sources, and the mapping relationship between aggregated files and the original files of scientific observation data; The file aggregation and encapsulation module is used to aggregate and encapsulate multiple original files into a target file container in a columnar storage format according to preset aggregation rules based on data product category and timeline. The target file container simultaneously stores the content of the original files and the structured data records parsed from the original files. The data service module is used to provide file-level access to the original file content and record-level retrieval services for structured data records, based on an external metadata database and target file containers. The real-time file processing module is used to add new original files in real time using an asynchronous access mechanism based on message queues.

[0067] The specific deployment diagram is as follows: Figure 3 As shown.

[0068] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for integrating storage management of massive amounts of small files and data records, comprising: Step 1: Establish an external metadata database to manage the hierarchical relationships of scientific observation data sources and the mapping relationship between aggregated files and the original files of scientific observation data; Step 2: According to the preset aggregation rules based on data product category and timeline, multiple original files are aggregated and encapsulated in a target file container with columnar storage format. The target file container stores the content of the original files and the structured data records parsed from the original files. Step 3: Based on the external metadata database and the target file container, provide file-level access to the original file content and record-level retrieval services for structured data records.

2. The method for integrated storage management of massive small files and data records according to claim 1, characterized in that, The external metadata database established in step 1 is a relational database, including: The site table is used to record basic information about the observation sites; The instrument table is used to record instrument information at each observation station. The data product table is used to list the data products generated by the instrument, forming a hierarchical mapping relationship of "instrument → data product"; The file type table is used to define the file type structure of data products and describe the content and data column structure of each file type. A data list is used to store data columns and data types. File type-column mapping table, used to define the set of columns corresponding to each file type; and The file container table is used to establish the mapping relationship between data products and packaged files in the object storage system, which is compatible with the S3 support range request.

3. The method for integrated storage management of massive small files and data records according to claim 1, characterized in that, The aggregation rules based on data product category and timeline in step 2 include: Package multiple raw files belonging to the same data product category into the same Parquet file in chronological order; The Parquet file includes a Header, a Block, and a Footer, wherein... Header: Used to contain file identification information; Block: Used to store specific column data. Each Block contains multiple row groups. During the packaging process, the original files to be processed are assigned to the corresponding row groups. If the file is unparsable, the fileName, content, and time fields of the file are recorded. If the file is parsable, the parsed structured content is appended to the same row group after the basic information of the file is written in the first line. Footer: Used to record the row groups, column blocks, and offset information of the Parquet file.

4. The method for integrated storage management of massive small files and data records according to claim 3, characterized in that, Step 2 further includes: establishing corresponding metadata records in an external metadata database and maintaining association with the footer inside the Parquet file. The external metadata is responsible for global indexing and data location.

5. The method for integrated storage management of massive small files and data records according to claim 3, characterized in that, Step 3 refers to file-level access to the original file content, including: The target file container is located based on the external metadata index, and the corresponding line group of the file content is read to return the complete original data file.

6. The method for integrated storage management of massive small files and data records according to claim 3, characterized in that, Step 3, the record-level retrieval service for structured data records, includes: In response to a data record-level query request, the target file container is located based on the external metadata index. The predicate pushdown query is then performed on the data record columns in conjunction with the query conditions. Only the blocks of data record columns that are hit are read and returned.

7. The method for integrated storage management of massive small files and data records according to claim 1, characterized in that, The method further includes: a step of adding original files in real time using an asynchronous access mechanism based on a message queue, specifically including: Receive new data file events via message queue; Based on the information of the newly added data file, dynamically match the corresponding parsing and encapsulation strategy; According to the strategy, after processing the newly added data files, they are written to the target file container and the external metadata index is updated.

8. A device for integrating storage and management of massive amounts of small files and data records, characterized in that, include: The metadata management module is used to establish an external metadata database, manage the hierarchical relationship of scientific observation data sources, and the mapping relationship between aggregated files and the original files of scientific observation data; The file aggregation and encapsulation module is used to aggregate and encapsulate multiple original files into a target file container in a columnar storage format according to preset aggregation rules based on data product category and timeline. The target file container simultaneously stores the content of the original files and the structured data records parsed from the original files. and The data service module provides file-level access to the original file content and record-level retrieval services for structured data records, based on an external metadata database and target file containers.

9. The apparatus for integrated storage management of massive small files and data records according to claim 8, characterized in that, The device also includes a real-time file processing module, which uses a message queue-based asynchronous access mechanism to add new original files in real time.

Images