Data dynamic compression and transmission system based on multi-level storage model

Abstract

The invention discloses a data dynamic compression and transmission system based on a multi-level storage model, and the system is characterized in that the system comprises a data preparation unit; a multi-level storage model unit; a data feature analysis unit; a dynamic data compression unit; a dynamic data transmission unit; wherein the multi-level storage model unit further comprises a data coding unit, a data structure unit and a data entity unit. According to the method, a multi-level storage model is designed, data is divided into three levels of'code-structure-entity ', and low-frequency synchronization and efficient coupling with a terminal are realized at a center end by taking a'code-structure' part in a data model as a constraint, so that efficient management and storage of the data are realized; in a data entity transmission process, data security, data volume and redundancy evaluation, network fluctuation monitoring and user real-time evaluation are performed on a transmission file, and dynamic switching between a compression tool and a transmission protocol is performed, so that the compression effect and the transmission efficiency are further improved.

Description

Technical Field

[0001] This invention relates to a data dynamic compression and transmission system based on a multi-level storage model, belonging to the field of data compression and transmission technology. Background Technology

[0002] With the rapid development of science and technology, the amount of data in various fields is experiencing explosive growth. In the field of biomedical research, especially in phenomics studies targeting large-scale population cohorts, the complexity and diversity of data have increased dramatically. This data encompasses molecular-level genomics, proteomics, and metabolomics; cellular-level immune cell phenotypes; organ-level imaging data; and functional-level physiological indicators. Furthermore, this multidimensional data is intertwined with individual lifestyles, environmental exposures, and other factors, further exacerbating the complexity of the data.

[0003] Faced with massive and complex biomedical data, existing storage and transmission technologies face severe challenges. On the one hand, data sources are diverse and highly heterogeneous, with different research institutions and projects generating data in inconsistent formats and standards, making data integration difficult. For example, different cohort studies may employ different questionnaire survey methods, sample collection methods, and laboratory testing techniques, making seamless integration of cross-cohort data challenging. On the other hand, the explosive growth in data volume places higher demands on storage and transmission. Taking genomic data as an example, the widespread application of high-throughput sequencing technology has generated massive amounts of gene sequence data, and the storage and rapid transmission of this data have become bottlenecks in research.

[0004] Therefore, the biomedical research field urgently needs a solution to effectively address the challenges of storing and transmitting massive amounts of complex data. Big data compression and high-speed transmission technologies have emerged to address this need, aiming to improve data processing efficiency, reduce storage costs and bandwidth requirements by optimizing data storage and transmission methods, and ensure that data can flow quickly and stably between different systems and nodes. This not only helps improve the efficiency and quality of biomedical research but also promotes the development of precision medicine and provides high-quality data support for population health research. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic data compression and transmission technology based on a multi-level storage model. By designing a multi-level storage model of queued data consisting of "encoding-structure-data entity", the central terminal uses the "encoding-structure" part of the data model as a constraint to achieve low-frequency synchronization and efficient coupling with the terminal. The distributed terminal completes the automatic identification, parsing and adaptation of the aggregated data protocol to meet the needs of high-speed and real-time access of data aggregation in different application scenarios. Ultimately, it significantly reduces file size and transmission time, and realizes effective compression and lightweight circulation of large-scale sample data entities.

[0006] To achieve the above objectives, the technical solution of the present invention discloses a data dynamic compression and transmission system based on a multi-level storage model, characterized in that it includes: The data preparation unit is used to integrate the collected multi-dimensional phenomics data into a cross-scale, multi-dimensional data management cloud platform. A multi-level storage model unit is used to divide data into three levels: "encoding-structure-data entity". Through a layered decoupling design, it achieves efficient data management and storage, further including data encoding sub-units, data structure sub-units, and data entity sub-units. The data coding subunit is used to define the unique identifier and association rules of data, ensure the global traceability of data, and standardize the data coding of special fields; Data structure sub-units are used to describe the organization and constraints of data, and support dynamic version management and structural evolution; The data entity sub-unit is used to store the actual data content, and combines multi-level storage technology to achieve hot and cold data stratification and resource optimization. The data selection unit is used to select multimodal data resources or single-modal data resources that require data compression and transmission; The data feature analysis unit is used to analyze the data features of the data resources selected by the data selection unit, including data size, data security level, data redundancy, and user real-time requirements. The dynamic data compression unit is used to dynamically select more suitable compression algorithms or tools (such as Zstd, Snappy and Gzip) for block compression based on the analysis results of the data feature analysis unit, thereby further improving the compression effect. The dynamic data transmission unit is used to dynamically select the transmission protocol based on the analysis results of the data feature analysis unit, and then quickly transmit the compressed data in the dynamic data compression unit, thereby further improving the transmission efficiency.

[0007] Preferably, the central terminal uses the data encoding subunit to generate a globally unique code using a hybrid encoding of UUIDv5 and timestamp, ensuring uniqueness across terminals and systems, and standardizing data encoding for special fields. The data encoding subunit generates the globally unique code using the following steps: Step 101: Select the UUID namespace; Step 102: Define a dataset name, which must be unique and may contain key business-related information. Step 103: Generate UUIDv5 using the UUID5 algorithm uuid5(namespace,name); Step 104: Get the timestamp. Get the Unix timestamp of the current time (in seconds or milliseconds), which is usually an integer value since January 1, 1970, 00:00:00 UTC. Step 105: Generate mixed encoding by concatenating the file type, UUIDv5 string, and timestamp string to form the final mixed encoding.

[0008] Preferably, the data structure sub-unit uses JSONSchema or Protobuf format to describe the data structure, supports adding, deleting, modifying, and querying fields, defines dynamic structures, records historical versions of the structure, and supports DeltaSync and rollback. The implementation steps are as follows: Step 201, Unified Abstraction: Decouple the "logical structure" from the "physical storage". Logically, any piece of data = structure version number + entity payload; Step 202, Dual Format Description: Supports both JSON Schema (human-readable, web-friendly) and Protobuf dual formats (cross-language, efficient serialization), automatically generating bidirectional mapping code through a code generator; Step 203, Version Lifecycle: Uses a dual label of "semantic version number (SemVer)" + "timestamp", with each version corresponding to an immutable schema snapshot; Furthermore, the data structure sub-unit adopts the following compatibility strategy: Forward compatibility (old reads new): Only optional fields can be added; deletion / renaming of required fields is prohibited. Backward compatibility (new read old): default values ​​are filled for missing fields, and field alias mapping is supported; Difference synchronization: The terminal only pulls the "structure diff", reducing network overhead by more than 90%; Rollback capability: Switching between any historical version within seconds ensures zero service interruption.

[0009] Preferably, the data entity subunit classifies data into levels (hot data, warm data, and cold data) and storage media (SSD or high-speed memory, SATA hard drive or QLCSSD and object storage or tape) according to popularity (access frequency); at the same time, the data entity subunit formulates a data migration strategy to automatically migrate data from high-hot level to low-hot level based on access frequency and storage cost; and performs lifecycle management on data, automatically cleaning or archiving data according to preset rules (such as time period, access number).

[0010] Preferably, the data migration strategy includes: If the number of accesses in 24 hours is less than 100, the system will automatically migrate to a SATA-SSD, and the heat level will change from hot data to warm data. If there is no access for 7 consecutive days, the data will be dumped to object storage, and its popularity level will change from warm data to cold data.

[0011] Preferably, in the multi-level storage model unit, the central end and the terminal set a synchronization period, and within the synchronization period, change detection is performed. The central end only transmits the change information of the "encoding-structure" part in the data model to the terminal, thereby reducing the amount of synchronization data and network bandwidth, and realizing a low-frequency synchronization strategy.

[0012] Preferably, in the multi-level storage model unit, after the terminal receives the "encoding-structure" synchronization data, it quickly locates the corresponding data entity through the encoding information and structure definition; through the indexing mechanism and the mapping relationship of data storage, it realizes the rapid loading of data entities, ensuring the availability and timeliness of data; at the same time, the terminal periodically sends data usage and feedback information to the central end, and the central end dynamically adjusts the "encoding-structure" model and synchronization strategy according to the feedback information to meet the actual needs of the terminal and realize efficient coupling between the central end and the terminal.

[0013] Preferably, the data features obtained by the data feature analysis unit include data size, data security level, data redundancy, network fluctuations, and real-time requirements of end users, etc. The calculation formula for data redundancy is: Data redundancy = Duplicate data volume / Total data volume.

[0014] Preferably, the dynamic data compression unit performs block compression using the following steps: Step 301, Data Blocking: Based on the data size characteristics, small data volumes are skipped directly, while large data volumes are divided into data blocks of equal size. Step 302, Intra-block compression: Based on the data feature analysis results, apply compression algorithms or tools independently to each data block; Step 303: Data merging. The compressed data blocks are merged into a complete data stream or file for easy storage or transmission.

[0015] Preferably, in the dynamic data transmission unit, the selectable transmission protocols include HTTPS, FTPS, and TCP, and the compressed data in the dynamic data compression unit is transmitted quickly using a parallel transmission tool, thereby further improving transmission efficiency.

[0016] Compared with existing technical solutions, the present invention has the following beneficial effects: 1) By designing a multi-level storage model for queue data consisting of “encoding-structure-data entity”, the “encoding-structure” part of the data model is used as a constraint at the central end to achieve low-frequency synchronization and efficient coupling with the terminal.

[0017] 2) Based on the data feature analysis results of the transmitted files, security, network fluctuation monitoring, and users' real-time requirements, dynamic selection of compression tools and transmission protocols is performed to further improve compression effect and transmission efficiency, and achieve rapid loading of data entities. Attached Figure Description

[0018] Figure 1 The data processing flow of the system disclosed in this invention is illustrated. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0020] Taking a medical institution as an example, using the hospital's genomics database as the original data source, this invention discloses a data dynamic compression and transmission system based on a multi-level storage model, which specifically includes the following parts: The data preparation unit integrates the collected multi-dimensional phenomics data into a cross-scale, multi-dimensional data management cloud platform, mainly including FASTQ files, SAM files, and VCF files, which are commonly used file formats for next-generation sequencing (NGS).

[0021] The multi-level storage model unit divides the data on the cross-scale, multi-dimensional data management cloud platform into three levels: "encoding-structure-data entity". Through layered decoupling design, it achieves efficient data management and storage, and further includes data encoding sub-units, data structure sub-units and data entity sub-units.

[0022] The data encoding subunit defines the unique identifier and association rules for data, ensuring global traceability. The central terminal uses a hybrid encoding of UUIDv5 and timestamps to generate a globally unique code, ensuring uniqueness across terminals and systems. The steps are as follows: Step 1. Select a UUID namespace, such as the DNS namespace or the URL namespace; Step 2. Define the dataset name 'name'. This name must be unique and can contain key business-related information, such as table names. Step 3. Use the UUID5 algorithm to generate UUIDv5 from UUID5(namespace, name); Step 4. Get the timestamp. Get the Unix timestamp of the current time (in seconds or milliseconds), which is usually an integer value since January 1, 1970, 00:00:00 UTC. Step 5. Generate mixed encoding by concatenating the file type, UUIDv5 string, and timestamp string to form the final mixed encoding.

[0023] Table 1. Example of data encoding calculation

[0024] Furthermore, the data encoding subunit establishes classification and encoding standards based on file type and purpose to facilitate rapid classification and retrieval. Simultaneously, a mapping table between encodings and data attributes is created, recording the specific meaning and associated information represented by the encodings to facilitate data parsing and related queries.

[0025] Table 2 Examples of Data File Encoding

[0026] The data structure subunit defines the data organization and version evolution rules, supporting dynamic updates and compatibility management. In this embodiment of the invention, the data structure subunit uses JSONSchema or Protobuf format to describe the data structure, supports adding, deleting, modifying, and querying fields, and defines a dynamic structure. Simultaneously, the data structure subunit records historical versions of the structure, supporting DeltaSync and rollback.

[0027] Table 3 Examples of Dynamic Structural Meta-Models

[0028] All versions are written to immutable storage (S3 / HDFS / object storage), and the filename is schemaId@version.

[0029] JSON Schema Dynamic Field Specification: I. Adopt Draft-2020-12, with all fields set to additionalProperties: false by default to prevent pollution from unknown fields; II. New field: "newField": {"type":"string","default":"N / A"}, which will be automatically filled with the default value when read by older clients; III. Delete a field: Mark "deprecated": true, and retain it until the next major version; IV. Modify types: Use oneOf or anyOf for type union to gradually phase out old types.

[0030] Protobuf dynamic field specification: I. Reserve 100 tags for each message as expansion space; use reserved blocks for new fields to avoid tag conflicts; II. Use `option (field_meta).deprecated = true;` to mark a field as deprecated; III. When generating code, the default value validation logic is automatically injected through the protoc-gen-validate plugin.

[0031] Version-based chained storage: I. Each version saves the full schema (not incrementally), facilitating rollback in seconds; II. Incremental diffs are calculated in real time at runtime using the json-patch / proto-diff library, reducing storage redundancy.

[0032] DeltaSync differential synchronization algorithm steps: I. Input the current terminal structure version vOld, and the latest version vNew at the central end. If vOld == vNew, then return an empty packet. II. If there is a major version jump, the full vNew schema will be issued directly; III. If a minor version upgrade is needed, calculate the JSON Patch (add, delete, or modify fields) or Proto FieldMask, and output the minimum Patch package; Rollback process: I. Business gray-scale abnormality or manual instruction triggering rollback; II. Switch the current write stream to the "shadow version" path (i.e. roll back the external service, but retain a temporary buffer for the new version's write data, thereby ensuring forward compatibility, zero data loss, and zero business interruption) to ensure that the newly written data is not lost; III. Set SchemaMeta.parent to the previous compatible version.

[0033] The data entity sub-unit is used to store the actual data content, and combines multi-level storage technology to achieve hot and cold data stratification and resource optimization. The data is divided into levels (hot data, warm data, and cold data) and storage media (SSD or high-speed memory, SATA hard drive or QLCSSD and object storage or tape) according to the popularity (access frequency), as shown in Table 4.

[0034] Table 4. Examples of hot and cold data classification

[0035] Meanwhile, data migration strategies are formulated in the data entity sub-units. Based on access frequency and storage cost, data is automatically migrated from high-hot levels to low-hot levels, and data lifecycle management is performed. Data is automatically cleaned or archived according to preset rules (such as time period and access count).

[0036] Data migration strategy: I. If the number of accesses in 24 hours is less than 100, the system will automatically migrate to a SATA-SSD, and the heat level will change from hot data to warm data. II. If there is no access for 7 consecutive days, the data will be dumped to object storage, and the popularity level will change from warm data to cold data.

[0037] The multi-level storage model unit sets a synchronization period between the central end and the terminal, and performs change detection within the synchronization period. The central end only transmits the change information of the "encoding-structure" part of the data model to the terminal, reducing the amount of synchronization data and network bandwidth, and realizing a low-frequency synchronization strategy.

[0038] After receiving the "encoding-structure" synchronization data, the terminal can quickly locate the corresponding data entity through the encoding information and structure definition. Through an indexing mechanism and data storage mapping, the data entity is loaded quickly, ensuring data availability and timeliness. Simultaneously, the terminal periodically sends data usage and feedback information to the central terminal, such as data access frequency and data quality issues. Based on the feedback, the central terminal dynamically adjusts the "encoding-structure" model and synchronization strategy to meet the actual needs of the terminal, achieving efficient coupling between the central terminal and the terminal.

[0039] The data selection unit is used to select the data file that needs to be compressed and transmitted, that is, the terminal locates the required data entity through "encoding-structure".

[0040] The data feature analysis unit is used to analyze the data characteristics of the data resources selected by the data selection unit, including data size, data security level, data redundancy, and real-time requirements of end users. The formula for calculating data redundancy is: Data redundancy = Duplicate data volume / Total data volume.

[0041] The feature analysis of the sample data is shown in Table 5.

[0042] Table 5. Example of feature analysis for sample data The dynamic data compression unit is used to dynamically select more suitable compression algorithms or tools (such as Zstd, Snappy, and Gzip) for block compression based on the analysis results of the data feature analysis unit, thereby further improving the compression effect. The block compression steps are as follows: Step 1. Data segmentation: Based on the data size characteristics, small data volumes are skipped directly, while large data volumes are divided into data blocks of equal size. Step 2. Intra-block compression: Based on the data feature analysis results, apply compression algorithms or tools independently to each data block. Since the data blocks are small, compression tools can complete the process faster and adapt to local features within the data blocks. Step 3. Data merging: Merge the compressed data blocks into a complete data stream or file for easy storage or transmission.

[0043] Examples of dynamic data compression rules are shown in Table 6.

[0044] Table 6 Examples of Dynamic Data Compression Rules

[0045] The dynamic data transmission unit is used to dynamically select the transmission protocol (such as HTTPS, FTPS, and TCP) based on the analysis results of the data feature analysis unit. The selection rules are shown in Table 7. Furthermore, the compressed data in the dynamic data compression unit is transmitted quickly through a parallel transmission tool, thereby further improving the transmission efficiency.

[0046] Table 7 Examples of Dynamic Transmission Rules

Claims

1. A data dynamic compression and transmission system based on a multi-level storage model, characterized in that, include: The data preparation unit is used to integrate the collected multi-dimensional phenomics data into a cross-scale, multi-dimensional data management cloud platform. The multi-level storage model unit is used to divide data into three levels: "encoding-structure-data entity". Through layered decoupling design, it realizes efficient data management and storage. It further includes data encoding sub-unit, data structure sub-unit and data entity sub-unit: The data encoding sub-unit is used to define the unique identifier and association rules of the data, ensure the global traceability of the data, and standardize the data encoding of special fields. Data structure sub-units are used to describe the organization and constraints of data, and support dynamic version management and structural evolution; The data entity sub-unit is used to store the actual data content, and combines multi-level storage technology to achieve hot and cold data stratification and resource optimization. The data selection unit is used to select multimodal data resources or single-modal data resources that require data compression and transmission; The data feature analysis unit analyzes the data characteristics of the data resources selected by the data selection unit, including data size, data security level, data redundancy, and user real-time requirements. The dynamic data compression unit dynamically selects a more suitable compression algorithm or tool for block compression based on the analysis results of the data feature analysis unit, thereby further improving the compression effect. The dynamic data transmission unit dynamically selects a transmission protocol based on the analysis results of the data feature analysis unit, and then quickly transmits the compressed data from the dynamic data compression unit, thereby further improving transmission efficiency.

2. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, The central terminal uses the data encoding subunit to generate a globally unique code using a hybrid encoding of UUIDv5 and timestamp, ensuring uniqueness across terminals and systems, and standardizing data encoding for special fields. The data encoding subunit generates the globally unique code using the following steps: Step 101: Select a UUID namespace; Step 102: Define a dataset name, which is unique and can contain key business-related information; Step 103: Generate a UUIDv5 using the UUID5 algorithm; Step 104: Obtain the timestamp, specifically the current Unix timestamp; Step 105: Generate the hybrid code by concatenating the file type, the UUIDv5 string, and the timestamp string to form the final hybrid code.

3. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, The data structure sub-unit uses JSON Schema or Protobuf format to describe the data structure, supports adding, deleting, modifying, and querying fields, defines dynamic structures, records historical versions of the structure, and supports difference synchronization and rollback. The implementation steps are as follows: Step 201, Unified Abstraction: Decouple the "logical structure" from the "physical storage". Any piece of data is logically equal to the structure version number + entity payload; Step 202, Dual-Format Description: Supports both JSON Schema and Protobuf formats, and automatically generates bidirectional mapping code through a code generator; Step 203, Version Lifecycle: Uses "semantic version number" + "timestamp" dual tags, with each version corresponding to an immutable schema snapshot; and the data structure sub-unit adopts the following compatibility strategies: Forward compatibility: Only optional fields are allowed to be added, and the deletion / renaming of required fields is prohibited; Backward compatibility: Default values ​​are filled for missing fields, and field alias mapping is supported; Difference Synchronization: The terminal only pulls the "structure diff", reducing network overhead by more than 90%; Rollback Capability: Any historical version can be switched in seconds, ensuring zero business interruption.

4. The data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, The data entity subunit categorizes data into levels and storage media based on popularity; simultaneously, it formulates data migration strategies within the data entity subunit, automatically migrating data from high-popularity levels to low-popularity levels based on access frequency and storage costs; and it performs lifecycle management on the data, automatically cleaning or archiving data according to preset rules.

5. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, The data migration strategy includes: if the number of accesses is less than 100 times in 24 hours, the data will be automatically migrated to SATA-SSD and its popularity level will change from hot data to warm data; if there are no accesses for 7 consecutive days, the data will be dumped to object storage and its popularity level will change from warm data to cold data.

6. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, In the multi-level storage model unit, the central end and the terminal set a synchronization period, and change detection is performed within the synchronization period. The central end only transmits the change information of the "encoding-structure" part of the data model to the terminal, reducing the amount of synchronization data and network bandwidth, and realizing a low-frequency synchronization strategy.

7. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, In the multi-level storage model unit, after receiving the "encoding-structure" synchronization data, the terminal quickly locates the corresponding data entity through the encoding information and structure definition; through the indexing mechanism and the mapping relationship of data storage, the data entity is loaded quickly, ensuring the availability and timeliness of the data; at the same time, the terminal periodically sends data usage and feedback information to the central end, and the central end dynamically adjusts the "encoding-structure" model and synchronization strategy according to the feedback information to meet the actual needs of the terminal and achieve efficient coupling between the central end and the terminal.

8. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, In the data feature analysis unit, the formula for calculating data redundancy is: Data redundancy = Amount of duplicate data / Total amount of data.

9. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, The dynamic data compression unit performs block compression using the following steps: Step 301, Data Blocking: Based on the data size characteristics, small data volumes are skipped directly, while large data volumes are divided into data blocks of equal size; Step 302, Intra-Block Compression: Based on the data feature analysis results, a compression algorithm or tool is applied independently to each data block; Step 303, Data Merging: The compressed data blocks are merged into a complete data stream or file for easy storage or transmission.

10. A data dynamic compression and transmission system based on a multi-level storage model as described in claim 1, characterized in that, In the dynamic data transmission unit, selectable transmission protocols include HTTPS, FTPS, and TCP, and compressed data in the dynamic data compression unit is transmitted quickly using a parallel transmission tool, thereby further improving transmission efficiency.

Images