Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to a data processing method and device, computer equipment and a storage medium. The method comprises the steps of obtaining original sequencing data; performing multi-stage compression on the original sequencing data to obtain target compressed data; and performing hierarchical storage on the target compressed data to obtain storage data. According to the data processing method provided by the invention, the storage efficiency and the transmission efficiency of the data can be improved, and the computing resources of the data can be reduced.

Description

Technical Field

[0001] This application relates to the field of underwater acoustic communication technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the rapid development and widespread application of genome sequencing technology, the scale of biological data is growing exponentially.

[0003] Traditional techniques typically involve directly storing the raw data generated by whole-genome sequencing (WGS). However, due to the large volume of raw data, storage efficiency is relatively low. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, and storage medium, aiming to solve the above-mentioned technical problems.

[0005] In a first aspect, embodiments of this application provide a data processing method, the data processing method comprising:

[0006] Obtain raw sequencing data;

[0007] The raw sequencing data is compressed in multiple stages to obtain the target compressed data;

[0008] The target compressed data is stored in layers to obtain the stored data.

[0009] In some possible implementations, the multi-stage compression includes primary compression, secondary compression, and tertiary compression. The step of performing multi-stage compression on the raw sequencing data to obtain the target compressed data includes:

[0010] The original sequencing data is compressed in one stage to obtain compressed data in one stage.

[0011] The first-level compressed data is then subjected to a second-level compression to obtain second-level compressed data;

[0012] The secondary compressed data is then subjected to tertiary compression to obtain tertiary compressed data, which is used as the target compressed data.

[0013] In some possible implementations, the raw sequencing data contains several sequencing reads, and the step of performing primary compression on the raw sequencing data to obtain primary compressed data includes:

[0014] Each sequencing read in the raw sequencing data is compared with a preset reference genome to obtain an intermediate file containing the comparison difference sites;

[0015] The intermediate file containing the compared differential sites is compressed in one stage.

[0016] In some possible implementations, before aligning each sequencing read in the raw sequencing data with a preset reference genome, the method further includes:

[0017] Multiple sequences to be aligned are selected from the reference genome that fall within the preset target sequence value range;

[0018] A hash table is established for the plurality of sequences to be compared;

[0019] The credibility of the sequence to be compared in the hash table is dynamically calculated to obtain the credibility score of the sequence to be compared.

[0020] The sequence with the highest confidence score was selected as the starting point for comparison.

[0021] In some possible implementations, the step of performing tertiary compression on the secondary compressed data to obtain tertiary compressed data includes:

[0022] The secondary compressed data is divided into blocks to obtain the target data blocks;

[0023] The target data block is compressed in parallel to obtain three-level compressed data.

[0024] In some possible implementations, the stored data includes hot data, warm data, and cold data, and the hierarchical storage of the target compressed data to obtain the stored data includes:

[0025] The frequently accessed variant data and comparison data in the target compressed data are stored in the hot data layer to obtain the stored hot data.

[0026] The intermediate frequency access data of the target compressed data is stored in the warm data layer to obtain the stored warm data;

[0027] The low-frequency access data in the target compressed data is stored in the cold data layer to obtain the stored cold data.

[0028] In some possible implementations, after storing the target compressed data in layers to obtain the stored data, the method further includes:

[0029] The stored data is retrieved to obtain the search results corresponding to the original sequencing data.

[0030] In some possible implementations, retrieving the stored data to obtain the retrieval results corresponding to the original sequencing data includes:

[0031] Receive user query requests;

[0032] Determine the query type of the user's query request;

[0033] If the query type is a mutation site query, the stored data is filtered before the search is performed to obtain the search results corresponding to the original sequencing data;

[0034] If the query type is a sequence similarity query, then the stored data is layered to obtain layered stored data;

[0035] The layered stored data is then subjected to distance-sensitive editing to generate candidate sequences;

[0036] Based on the candidate sequences, a search is performed to obtain the search results corresponding to the original sequencing data;

[0037] If the query type is a genome-wide query, then spatial indexing is performed on the stored data to obtain index data;

[0038] By using coordinate mapping, data block positioning is performed on the index data to obtain the retrieval results corresponding to the original sequencing data.

[0039] Secondly, embodiments of this application also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0040] Thirdly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0041] This application provides a data processing method, apparatus, computer device, and storage medium. The method includes acquiring raw sequencing data; performing multi-level compression on the raw sequencing data to obtain target compressed data; and storing the target compressed data in layers to obtain stored data.

[0042] This application significantly reduces data volume by performing multi-level compression on raw sequencing data. Compared to the traditional method of directly storing raw files, this application reduces data storage space requirements, thereby greatly reducing the storage cost of raw sequencing data and improving storage efficiency. Furthermore, storing compressed data in layers allows for the rational allocation of different data types to corresponding storage layers, avoiding the situation where all data is "queuing" for processing on the same storage layer, thus improving resource utilization and ultimately increasing data storage efficiency. In addition, storing the target compressed data in layers also improves the convenience and efficiency of subsequent retrieval and analysis. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0046] Figure 1 A flowchart illustrating a first embodiment of a data processing method provided in this application;

[0047] Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0050] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0054] With the rapid development and widespread application of genome sequencing technology, the scale of biological data is growing exponentially, which has led to some problems in data storage, processing and transmission.

[0055] 1) Raw data generated by whole-genome sequencing (WGS) is usually stored in FASTQ format, and the data volume of a single 30X coverage sample can reach approximately 200GB. In sequencing projects involving tens of thousands of people, the total amount of raw data can reach the petabyte level. That is, due to the very large amount of raw data, the required storage space is very large and the storage efficiency is relatively low.

[0056] 2) Analyzing raw genome data (such as alignment and variant detection) usually requires a lot of computing resources.

[0057] 3) Traditional hard drive mailing methods cannot meet the needs of exchanging TB-level genetic data, resulting in relatively low data transmission efficiency.

[0058] To address the aforementioned problems, this application provides a data processing method that can improve data storage and transmission efficiency while reducing computational resources required for data processing.

[0059] See Figure 1 , Figure 1 This is a flowchart illustrating a first embodiment of a data processing method provided in this application. The data processing method includes the following steps:

[0060] In a first aspect, embodiments of this application provide a data processing method, the data processing method comprising:

[0061] Step 110: Obtain raw sequencing data.

[0062] Step 120: Perform multi-level compression on the raw sequencing data to obtain the target compressed data.

[0063] Step 130: Store the target compressed data in layers to obtain the stored data.

[0064] This embodiment significantly reduces data volume by performing multi-level compression on the raw sequencing data. Compared to the traditional method of directly storing raw files, this application can reduce data storage space requirements, thereby significantly reducing the storage cost of raw sequencing data and improving storage efficiency. In addition, storing compressed data in layers allows for the rational allocation of different types of data to corresponding storage layers, avoiding the situation where all data is "queuing up" for processing on the same storage layer, thus improving resource utilization and data storage efficiency. Furthermore, storing the target compressed data in layers also improves the convenience and efficiency of subsequent retrieval and analysis.

[0065] In some possible implementations, the multi-stage compression includes primary compression, secondary compression, and tertiary compression. The step of performing multi-stage compression on the raw sequencing data to obtain the target compressed data includes:

[0066] Step 121: Perform primary compression on the raw sequencing data to obtain primary compressed data.

[0067] In some embodiments, primary compression mainly employs lossless compression using Compressed Reference-oriented Alignment Map (CRAM) format, combined with k-mer encoding technology, and secondary compression of the quality value using entropy encoding.

[0068] For example, primary compression may include the following process:

[0069] 1) Convert raw sequencing data in FASTQ or BAM format to CRAM format;

[0070] 1-1) Reference genome alignment: First, the sequencing reads are aligned with the reference genome, recording only the points of change relative to the reference genome, rather than the entire sequence.

[0071] 1-2) Differential site storage: Only those sites that are different from the reference genome are stored, which greatly reduces the amount of data that needs to be stored.

[0072] Thus, efficient data compression can be achieved through reference genome alignment and storage of differential sites.

[0073] 2) Integrating k-mer spectral analysis technology (k=31), which is an effective method for compressing repetitive sequences:

[0074] 2-1) Short code mapping: For each k-mer of length 31, it is mapped to a 4-bit short code. Since many k-mers are highly repetitive in the genome, this mapping can significantly reduce storage requirements.

[0075] 2-2) Dynamic dictionary update mechanism: In order to adapt to the variation characteristics of different populations, a dynamic k-mer dictionary update mechanism is designed, which allows the k-mer dictionary to be adjusted according to the sample characteristics, thereby improving compression efficiency and adaptability.

[0076] In some possible implementations, the raw sequencing data contains several sequencing reads, and the step of performing primary compression on the raw sequencing data to obtain primary compressed data includes:

[0077] Step 11: Align each sequencing read in the raw sequencing data with a preset reference genome to obtain an intermediate file containing the alignment difference sites.

[0078] Step 12: Perform a first-level compression on the intermediate file containing the comparison difference sites.

[0079] In some possible implementations, before aligning each sequencing read in the raw sequencing data with a preset reference genome, the method further includes:

[0080] Step 21: Select multiple sequences to be aligned from the reference genome that fall within the preset target sequence value range.

[0081] Step 22: Build a hash table for the plurality of sequences to be compared.

[0082] Step 23: Perform dynamic confidence calculation on the sequence to be compared in the hash table to obtain the confidence score of the sequence to be compared.

[0083] Step 24: Select the sequence to be compared with the highest confidence score as the starting point for comparison.

[0084] In other words, a hash table is created for all k-mers (k = 10 to 32) in the reference genome, with the key being the k-mer quaternary hash value and the value being a list of genome locations.

[0085] The confidence score is calculated for the k-mer expression of the sequence to be aligned, where $L_r = 0.2 \times

[0086] len(r) is the fault tolerance length, and delta is the indicator function. The k-mer with the highest score is selected as the comparison starting point.

[0087] 3) Entropy encoding of quality values

[0088] Even if the quality value is compressed again using ANS (Asymmetric Numeral Systems) entropy encoding, the file size will be further reduced;

[0089] In addition, by statistically analyzing the quality values ​​and utilizing their distribution characteristics, a suitable encoding method can be selected to achieve a higher compression rate.

[0090] 4) Hardware acceleration

[0091] 4-1) SIMD Parallel Alignment: Hardware-accelerated SIMD (Single Instruction Multiple Data) parallel alignment is implemented using the AVX-512 instruction set, which significantly improves the processing speed.

[0092] Through primary compression, 300GB of raw sequencing data can be compressed to 35GB of primary compressed data. Compared with BAM, the compression rate of primary compression in this application can reach about 88.3%. Furthermore, on an Intel Xeon Gold 6348 processor, the decompression speed exceeds 500MB / s, thereby improving the efficiency of data processing. In addition, the primary compression adopts a dynamic k-mer dictionary update mechanism, which can adapt to different population variation characteristics and improve adaptability.

[0093] In other words, CRAM introduces reference sequence-dependent compression on top of BAM binary, storing only alignment difference sites (such as SNPs and InDels) instead of the complete sequence. Through coordinate positioning and hierarchical encoding of base quality values ​​(such as using arithmetic encoding to compress quality scores), it achieves 30%-50% storage savings.

[0094] Step 122: Perform secondary compression on the primary compressed data to obtain secondary compressed data.

[0095] Two-stage compression can include the following processes:

[0096] 1) Multi-modal Joint Compression

[0097] 1-1) File association recognition: The system automatically recognizes the association between input files, such as the correspondence between FASTQ and BAM, and the dependency between BAM and VCF;

[0098] 1-2) Unified compression context construction: Integrate the metadata of multiple files (such as sample ID, sequencing platform, alignment parameters) into a shared context model;

[0099] 1-3) Cross-file Deduplication:

[0100] By leveraging the recurrence of the same metadata in different files, a dictionary mapping can be established across files;

[0101] Encode duplicate fields (such as read group name and sequencing platform identifier) ​​once, and then reference the rest, saving storage space.

[0102] 2) Deep Context Modeling and Entropy Optimization

[0103] 2-1) Metadata-driven context modeling:

[0104] Specifically, metadata such as sequencer type, read length, GC content, and alignment quality can be extracted;

[0105] Secondly, context-aware coding strategies can be constructed. For example, different sequencers generate FASTQ quality values ​​with significant differences in distribution, so different entropy coding models can be used for adaptation.

[0106] 2-2) Entropy prediction model based on machine learning:

[0107] For example, the XGBoost algorithm can be used to predict the entropy distribution of the input data; compression parameters (such as block size and encoding method) can be dynamically adjusted based on the prediction results to achieve a balance between the optimal compression ratio and decompression speed.

[0108] 3) Adaptive compression and index optimization

[0109] 3-1) Dynamic block size selection mechanism (512B-4MB):

[0110] Among them, data can be divided into blocks according to chromosome regions, each block is compressed independently and attached with metadata index, and the optimal compressed block size is automatically selected for different data types (such as FASTQ quality values, BAM label fields) and access modes;

[0111] For example, small blocks are suitable for frequently accessed data, while large blocks are suitable for archived data that is read continuously, in order to support random access.

[0112] Alternatively, a typical compression ratio can be used, such as 1:6 for SAM→BAM, and then BAM→CRAM is compressed to less than 50% of the original BAM.

[0113] Thus, compared to the BGZIP method, the two-stage compression of this application can improve the compression ratio by 40%, and the multi-file associated compression can save 35% of redundant metadata and improve compression efficiency; in addition, the two-stage compression can improve the subsequent index building speed by 3.8 times.

[0114] Step 123: Perform tertiary compression on the secondary compressed data to obtain tertiary compressed data, wherein the tertiary compressed data is used as the target compressed data.

[0115] In some possible implementations, the step of performing tertiary compression on the secondary compressed data to obtain tertiary compressed data includes:

[0116] Step 1231: Divide the secondary compressed data into blocks to obtain the target data block.

[0117] Step 1232: Perform parallel compression on the target data block to obtain three-level compressed data.

[0118] The three-level compression can be implemented using a distributed framework based on MapReduce to achieve parallel compression of 64KB data blocks, and may specifically include the following process:

[0119] I. Data Sharding and Input Preparation

[0120] Data source access: Receives intermediate compressed data files from the upper compression stage (such as first-level compression, second-level compression), in formats including but not limited to CRAM, Genozip compression format, etc.

[0121] Data sharding: Divide the original data into fixed-size 64KB data blocks to accommodate the efficient execution of compression algorithms and load balancing of distributed processing;

[0122] Metadata extraction: Extract metadata information for each data block, including offset, verification information, compression algorithm identifier, etc., for subsequent decompression and consistency verification.

[0123] II. Map Node Compression Processing

[0124] Parallel compression task allocation: Distribute 64KB data blocks to multiple Map nodes, with each node executing the compression task independently;

[0125] Compression algorithm selection: The compression algorithm is automatically selected based on the data characteristics (LZ4 is used for high-speed compression, and Zstd is used for high compression ratio scenarios);

[0126] Compression process optimization: Memory binding strategy based on NUMA architecture improves single-node compression efficiency; supports SIMD instruction set acceleration (such as AVX2 / AVX-512); Compression result encapsulation: Each compressed block is accompanied by a CRC32 checksum to ensure data integrity during the compression process.

[0127] III. Shuffle Sorting and Data Transmission Optimization

[0128] Data sorting and grouping: Sort and classify the compression results according to the logical order of the compressed blocks or the file ownership relationship;

[0129] RDMA-based network acceleration: Using the RoCEv2 (RDMA over Converged Ethernet) protocol to achieve zero-copy, low-latency data transmission between nodes; reducing CPU overhead and latency caused by the traditional TCP / IP protocol stack;

[0130] Task scheduling optimization: Introduce a data-aware scheduler to intelligently allocate compression tasks based on the node's NUMA topology, load status, and network location;

[0131] This enables local scheduling of computation and data, reducing cross-node communication overhead.

[0132] Therefore, using RDMA networks and SGX encrypted transmission technology can improve efficiency and solve the problem that traditional hard drive mailing methods cannot meet the needs of TB-level gene data exchange, resulting in low data transmission efficiency.

[0133] IV. Reduce Node Merging and Output

[0134] Target compressed block merging: During the Reduce phase, target compressed blocks from multiple Map nodes are merged in an ordered manner according to the original file structure;

[0135] Consistency Guarantee: Uses the Raft consensus protocol to manage metadata and compression task status, ensuring high availability and consistency in the distributed compression process; supports fault recovery and resume interrupted downloads.

[0136] Output index construction: Constructs a logical index table for compressed files, supporting fast location and on-demand decompression; index information is synchronously written to HDFS or object storage.

[0137] V. HDFS Distributed Storage

[0138] Compression result writing: The final compressed file is written to HDFS (Hadoop Distributed File System), supporting multi-replica storage and disaster recovery;

[0139] Storage format optimization: Supports HDFS small file merging mechanisms (HAR, HARFS) to improve read efficiency; supports integration with object storage (such as S3, OSS) to achieve cloud-native deployment;

[0140] Metadata Management: All compressed block metadata is managed in a unified manner, supporting fast retrieval, version control, and permission management.

[0141] Thus, through the above method, the throughput of the three-level compression of this application can reach 24GB / s (100-node cluster), the horizontal scalability can be linearly increased to the PB level, and the compression ratio can be improved by 15% compared to a single machine.

[0142] In some possible implementations, the stored data includes hot data, warm data, and cold data, and the hierarchical storage of the target compressed data to obtain the stored data includes:

[0143] Step 131: Store the frequently accessed variant data and comparison data in the target compressed data in the hot data layer to obtain the stored hot data.

[0144] The hot data layer can adopt an adaptive cache replacement caching strategy and uses an Intel Optane P5800XNVMe SSD as the storage medium, which has ultra-low access latency, with a measured latency of less than 1ms, making it suitable for high-frequency access genomic data storage and fast retrieval.

[0145] The stored data structure includes mutation data and alignment data. The mutation data is the VCF of the RoaringBitmap index, and the alignment data can be BAM slices, which are small files or data blocks of about 1MB each.

[0146] In some embodiments, the performance metrics of the hot data layer can be as follows:

[0147] 1) Latency: 0.8ms (4K random read);

[0148] That is, under a 4K random read load, the average access latency is as low as 0.8 milliseconds (ms), which meets the stringent response requirements of real-time mutation query and interactive analysis.

[0149] 2) Input / output operations per second (IOPS): 1.5M (70 / 30 read / write mixed load);

[0150] Under a mixed load of 70% read and 30% write, the system can provide a throughput of up to 1.5 million IOPS, which is suitable for high-concurrency variability data retrieval and comparison analysis.

[0151] 3) Bandwidth: For large file or continuous data access scenarios, it supports sequential read bandwidth of up to 14GB / s, which significantly improves the efficiency of batch data loading and analysis.

[0152] Thus, in high-frequency access scenarios at the hot data layer, two key technologies were introduced: persistent memory accelerated indexing (PMem+DRAM hybrid architecture) and FPGA-based mutation query filtering (Bloom Filter). An FPGA acceleration module was integrated into the data access path. This module implemented a Bloom Filter-based filtering mechanism to pre-determine the existence of a specific mutation site. When a mutation query request is received, it is first pre-screened using the Bloom Filter on the FPGA to exclude non-existent mutation sites, reducing unnecessary disk I / O operations. This significantly improves not only the access performance and system response speed of the hot data layer but also enhances system reliability and resource utilization.

[0153] Step 132: Store the intermediate frequency access data of the target compressed data in the warm data layer to obtain the stored warm data.

[0154] In some embodiments, an erasure coding (8+3) configuration based on the Ceph distributed storage system can be used, combined with the Bluestore storage engine and the RocksDB metadata management mechanism, to build a warm data layer that balances storage efficiency, high availability, and performance assurance. This layer is mainly used to store genomic data that is accessed less frequently but still requires fast response (such as archived BAM files, historical VCF variant data, etc.).

[0155] The technical parameters of the warm data layer are as follows:

[0156] Erasure coding configuration: EC 8+3 (8 data blocks + 3 parity blocks);

[0157] Storage pool: Bluestore + RocksDB metadata engine;

[0158] Data distribution: CRUSH algorithm (rack awareness).

[0159] The performance indicators are:

[0160] Availability: 99.999% (annual downtime <5 minutes);

[0161] Recovery speed: 1TB / hour (node ​​failure scenario).

[0162] Thus, by introducing the LRC mechanism on the basis of EC 8+3, some check blocks are localized, reducing the number of data nodes involved in the reconstruction process; significantly reducing network and computing overhead during the reconstruction process, improving data recovery efficiency, and shortening the fault recovery time window.

[0163] Furthermore, a QoS strategy based on Cgroups and Ceph RBD is introduced to allocate bandwidth to different service flows; a minimum of 200MB / s of stable bandwidth is reserved for genome analysis tasks to ensure the performance of critical analysis processes under high concurrency access.

[0164] Step 133: Store the low-frequency access data in the target compressed data in the cold data layer to obtain the stored cold data.

[0165] In some embodiments, the cold data layer may use Glacier Deep Archive+ tape library as storage medium, employing bgzip (Blocked GZIP) compression format and tabix (CSI index format) indexing mechanism.

[0166] In some embodiments, the performance metric for the cold data layer is a compression ratio of 5:1 (FASTQ → bgzip).

[0167] For retrieval latency: Deploy archived data that occasionally requires fast access (such as fast backtracking of historical samples) in the S3 standard layer to achieve a fast response time of <5 seconds;

[0168] Cold data that is rarely accessed but needs to be stored for a long time (such as compliance archives) can be stored using Glacier and tape libraries. Data recovery and access can be completed in less than 5 minutes in standard retrieval mode.

[0169] Thus, by adopting a tiered storage strategy, frequently accessed data is kept on SSDs, while infrequently accessed data is automatically migrated to object storage.

[0170] In some possible implementations, after storing the target compressed data in layers to obtain the stored data, the method further includes:

[0171] The stored data is retrieved to obtain the search results corresponding to the original sequencing data.

[0172] In some possible implementations, retrieving the stored data to obtain the retrieval results corresponding to the original sequencing data includes:

[0173] Step 141: Receive the user's query request.

[0174] Step 142: Determine the query type of the user's query request.

[0175] Step 143: If the query type is a mutation site query, the stored data is filtered before the search is performed to obtain the search results corresponding to the original sequencing data.

[0176] For example, a mutation site query could be rs12345.

[0177] Specifically, mutation site queries can be pre-screened using a Bloom filter. The Bloom filter can use a 256-bit hash array, with a false positive rate of <0.1%, a response time of <2ms, and a filtering efficiency of 98% for invalid queries.

[0178] Step 144: If the query type is a sequence similarity query, then the stored data is layered to obtain the layered stored data.

[0179] For example, a sequence similarity query could be "ATGCTAGCT".

[0180] Step 145: Perform distance-sensitive editing on the layered stored data to generate candidate sequences.

[0181] Step 146: Perform a search based on the candidate sequences to obtain the search results corresponding to the original sequencing data.

[0182] For sequence similarity queries, a hierarchical FM-index structure is employed, dividing the index into multiple levels and deploying them on storage media of different performance levels to achieve efficient alignment and rapid retrieval of large-scale genomic sequences. The specific structure is as follows:

[0183] Top layer: 3-mer coarse index (resident in memory), used to quickly locate candidate regions, build a lightweight global index, and reside in memory to support millisecond-level response;

[0184] Middle layer: 7-mer fine index (SSD cache), which provides finer-grained matching capabilities on the basis of coarse index. It is cached in high-performance SSD to accelerate the accurate comparison process;

[0185] The underlying layer is a full-sequence BWT index (distributed storage), which stores the complete Burrows-Wheeler Transform (BWT) index structure distributed across the underlying storage system, supporting precise alignment and extended queries across the entire genome.

[0186] Among them, BWT (Burrows-Wheeler Transform) is the last character of the reference genome after round-rotation sorting, such as "acaacg$" → BWT(T) = "gc$aaac".

[0187] Checkpoint sampling: Occ(c,r) (the cumulative number of times the character 'c' appears before line 'r') is stored every 448 lines, accounting for 14% of the BWT after compression.

[0188] Simplified SA (suffix array): Store one SA value (original genome position) every 32 / 36 rows, and backtrack the remaining positions through LF mapping ($SA[r]=SA[i]+\text{steps}$), accounting for 50% of BWT.

[0189] For edit distance calculation, fuzzy matching based on Levenshtein distance is supported, with a maximum supported edit distance of ≤3, suitable for alignment requirements involving mutation types such as insertion, deletion, and replacement. Response time: With multi-level index collaborative acceleration, typical query response time is <50 milliseconds, meeting the needs of real-time or near real-time gene sequence retrieval and alignment scenarios.

[0190] Step 147: If the query type is a genome-wide query, then spatial index the stored data to obtain index data.

[0191] A genome range query could be, for example, chr1:1000000-2000000.

[0192] Step 148: Using coordinate mapping, locate data blocks in the index data to obtain the retrieval results corresponding to the original sequencing data.

[0193] The spatial index, specifically the Z-order spatial index, maps two-dimensional genomic coordinates (chromosome number + position) to a 64-bit integer Z-order encoded value using the function f(chr,pos) = (chr << 48)|pos. This compresses multidimensional data into a one-dimensional index key, facilitating efficient storage and retrieval. By utilizing space-filling curves to encode genomic regions traversally, spatial locality is maintained, ensuring that physically adjacent index keys also exhibit high local relevance in the genomic space, thereby improving the efficiency of range queries.

[0194] Furthermore, the index data is stored in 64KB blocks to ensure good alignment and cache friendliness between the index and data blocks in the storage system. This is suitable for I / O optimization features of SSD and distributed storage systems. When performing range queries, the system automatically prefetches neighboring data blocks of the target area based on the spatial continuity characteristics of the Z-order index, improving cache hit rate and reducing random I / O, significantly optimizing the performance of large-scale range queries.

[0195] Dynamic block partitioning strategy: Based on device memory and the number of threads, the reference genome is divided into evenly sized blocks. Each thread independently loads the FM-index blocks and executes the FM-search algorithm (iterwise from the end of the read, narrowing the search interval by $sp\leftarrow C[c]+Occ(c,sp)$ and $ep\leftarrow C[c]+Occ(c,ep)-1$). Semaphores can synchronize results and merge matching positions, reducing the human genome (3Gbp) index from 12GB to 1.1GB and supporting multi-threaded concurrent queries.

[0196] Performance advantages: The human genome (3Gbp) index has been reduced from 12GB to 1.1GB, and multi-threaded concurrent queries have been supported.

[0197] This Z-order spatial indexing scheme achieves efficient organization, fast retrieval, and low-latency response of genomic data in large-scale storage systems through multidimensional coordinate encoding, spatial filling curve mapping, block storage alignment, and range query prefetching mechanism.

[0198] Thus, by using a Bloom filter to pre-screen mutation sites, the false positive rate can be less than 0.1%; by using a hierarchical FM-index to support edit distance-sensitive queries, the response time can be less than 50ms; and by using a Z-order curve spatial index to optimize genomic coordinate range queries, the hybrid indexing approach can improve the convenience and efficiency of indexing.

[0199] In some embodiments, when the query type includes mutation site query, sequence similarity query and genome range query, after the retrieval is performed based on the corresponding processing methods above, the processing results corresponding to mutation site query, sequence similarity query and genome range query can be aggregated to obtain the aggregated result, and then the aggregated result is input into the hardware acceleration layer for processing.

[0200] Specifically, after inputting the aggregation result into the hardware acceleration layer, the following processes can also be included:

[0201] 1) Sort and filter the aggregation results;

[0202] 2) Determine if it meets the requirements for big data analytics;

[0203] If so, SparkSQL is used to directly read and compress the data, followed by distributed computation to make the results more persistent.

[0204] Among them, SparkSQL directly queries compressed data, which reduces the time taken for joint VCF analysis of 100,000 samples from 8 hours to 23 minutes.

[0205] If not, return the API result.

[0206] In some embodiments, the hardware acceleration process may include the following:

[0207] 1) FPGA acceleration;

[0208] 2) BWA-MEM pipeline;

[0209] 3) Seed generation;

[0210] 4) Strip alignment;

[0211] 5) Smith-Waterman.

[0212] Specifically, the performance improvements at each stage are as follows: seed generation: 18x; strip alignment: 9x; SW algorithm: 15x; overall speedup: 12x.

[0213] In some embodiments, a heterogeneous accelerated system architecture CPU-FPGA co-design can be adopted:

[0214] CPU program: responsible for I / O (parsing FASTQ / CRAM), seed filtering, and seed chaining.

[0215] FPGA Accelerator: Focus on optimizing two computationally intensive modules:

[0216] SMEM search accelerator: It adopts a dual-module pipeline (forward search + backward search), and immediately switches modules when a path cannot be expanded, so as to process multiple sequences in parallel.

[0217] Smith-Waterman (SW) Extended Accelerator: Uses a systolic array, PE cells are computed in parallel along the anti-diagonal, and support ping-pong buffer for prefetching seed data.

[0218] Key optimization techniques may include the following:

[0219] 1) Batch Reconstruction

[0220] 1-1) SMEM batch processing: In mem_collect_intv_fpga(), the FPGA is started in advance to process the SMEM search of all sequences.

[0221] 1-2) SW Extension Batch Processing: In ksw_extend2_fpga(), all seed chains of a sequence are bidirectionally extended in batches, and only the scores are returned for CPU filtering.

[0222] 2) Hardware-friendly algorithm modification:

[0223] 2-1) Column-based dimensionality reduction and band scoring: Limit the bandwidth of the SW matrix calculation to $b=0.1\times l_{rd}$, thereby reducing FPGA resource consumption.

[0224] 2-2) Data stream compression: The PCIe interface is encapsulated through the Intel CCI-P bus protocol to reduce CPU-FPGA data transmission latency.

[0225] The latency for pathogenic mutation retrieval in the 1000 Genomes Database was controlled within 50ms13. The Spark cluster supports joint queries of 100,000 VCF files, and the PCA computation time was reduced from 8 hours to 23 minutes.

[0226] Thus, the compression storage technology provided in this application can analyze the genomes of large-scale populations. At the same time, IntelSGX encrypted transmission and AES-256 storage encryption meet the requirements for desensitization of clinical genetic data and ensure data security.

[0227] Combining compressed data with FPGA acceleration can increase the BWA-MEM comparison speed by 12 times, thereby solving the problem that direct analysis of raw data requires a large amount of CPU / GPU resources.

[0228] Corresponding to the above data processing methods, this application also provides a data processing apparatus. This data processing apparatus includes a unit for performing the above data processing methods and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal.

[0229] like Figure 2 As shown in the figure, this application provides a computer device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0230] Memory 113 is used to store computer programs;

[0231] In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the data processing method provided in any of the foregoing method embodiments, including:

[0232] Obtain raw sequencing data;

[0233] The raw sequencing data is compressed in multiple stages to obtain the target compressed data;

[0234] The target compressed data is stored in layers to obtain the stored data;

[0235] The stored data is retrieved to obtain the search results corresponding to the original sequencing data.

[0236] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0237] Therefore, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the data processing method provided in any of the foregoing method embodiments, including:

[0238] Obtain raw sequencing data;

[0239] The raw sequencing data is compressed in multiple stages to obtain the target compressed data;

[0240] The target compressed data is stored in layers to obtain the stored data;

[0241] The stored data is retrieved to obtain the search results corresponding to the original sequencing data.

[0242] The storage medium is a physical, non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0243] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0244] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0245] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0246] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0247] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0248] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Since these modifications and variations fall within the scope of the claims and their equivalents, this application also intends to include these modifications and variations.

[0249] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The data processing method includes: Obtain raw sequencing data; The raw sequencing data is compressed in multiple stages to obtain the target compressed data; The target compressed data is stored in layers to obtain the stored data.

2. The method according to claim 1, characterized in that, The multi-stage compression includes primary compression, secondary compression, and tertiary compression. The process of performing multi-stage compression on the original sequencing data to obtain the target compressed data includes: The original sequencing data is compressed in one stage to obtain compressed data in one stage. The first-level compressed data is then subjected to a second-level compression to obtain second-level compressed data; The secondary compressed data is then subjected to tertiary compression to obtain tertiary compressed data, which is used as the target compressed data.

3. The method according to claim 2, characterized in that, The raw sequencing data contains several sequencing reads. The process of performing primary compression on the raw sequencing data to obtain primary compressed data includes: Each sequencing read in the raw sequencing data is compared with a preset reference genome to obtain an intermediate file containing the comparison difference sites; The intermediate file containing the compared differential sites is compressed in one stage.

4. The method according to claim 3, characterized in that, Before comparing each sequencing read in the raw sequencing data with a preset reference genome, the method further includes: Multiple sequences to be aligned are selected from the reference genome that fall within the preset target sequence value range; A hash table is established for the plurality of sequences to be compared; The credibility of the sequence to be compared in the hash table is dynamically calculated to obtain the credibility score of the sequence to be compared. The sequence with the highest confidence score was selected as the starting point for comparison.

5. The method according to claim 2, characterized in that, The step of performing tertiary compression on the secondary compressed data to obtain tertiary compressed data includes: The secondary compressed data is divided into blocks to obtain the target data blocks; The target data block is compressed in parallel to obtain three-level compressed data.

6. The method according to claim 1, characterized in that, The stored data includes hot data, warm data, and cold data. The process of storing the target compressed data in layers to obtain the stored data includes: The frequently accessed variant data and comparison data in the target compressed data are stored in the hot data layer to obtain the stored hot data. The intermediate frequency access data of the target compressed data is stored in the warm data layer to obtain the stored warm data; The low-frequency access data in the target compressed data is stored in the cold data layer to obtain the stored cold data.

7. The method according to claim 1, characterized in that, After storing the target compressed data in layers to obtain the stored data, the method further includes: The stored data is retrieved to obtain the search results corresponding to the original sequencing data.

8. The method according to claim 7, characterized in that, The step of retrieving the stored data to obtain the retrieval results corresponding to the original sequencing data includes: Receive user query requests; Determine the query type of the user's query request; If the query type is a mutation site query, the stored data is filtered before the search is performed to obtain the search results corresponding to the original sequencing data; If the query type is a sequence similarity query, then the stored data is layered to obtain layered stored data; The layered stored data is then subjected to distance-sensitive editing to generate candidate sequences; Based on the candidate sequences, a search is performed to obtain the search results corresponding to the original sequencing data; If the query type is a genome-wide query, then spatial indexing is performed on the stored data to obtain index data; By using coordinate mapping, data block positioning is performed on the index data to obtain the retrieval results corresponding to the original sequencing data.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images