System and method for accelerating gene alignment execution process

Abstract

The invention discloses a system and a method for accelerating a gene alignment execution process. The system comprises a processing thread management module, one or more dynamic caches and a dynamic cache management module. Wherein the processing thread management module is configured to set a gene comparison process as a multi-stage assembly line, each execution step of the gene comparison process corresponds to one assembly line stage on the multi-stage assembly line, and a plurality of processing threads are distributed for each assembly line stage; and dividing gene sequence data to be compared into a plurality of data blocks and providing the data blocks to the multi-stage assembly line for processing. The dynamic caches are used for data transmission between two pipeline stages, and each dynamic cache comprises at least two cache blocks. And the dynamic cache management module is configured to manage distribution and recovery of the dynamic caches between the two pipeline stages and control switching of working states between different cache blocks in each dynamic cache. According to the method, the overlapping degree of process calculation and input and output is higher, the memory efficiency is higher, and the performance of overall execution is improved.

Description

Systems and methods for accelerating gene alignment processes Technical Field

[0001] This application relates to the field of gene data analysis technology, specifically to a system and method for accelerating gene alignment processes. Background Technology

[0002] The statements in this section are merely to provide background information in relation to this application to aid in understanding it, and such background information does not necessarily constitute prior art.

[0003] Whole-genome sequencing technology has developed rapidly in recent years, with its cost decreasing exponentially, enabling it to quickly penetrate from the research field into clinical applications. This technology can provide precise molecular evidence for tumor molecular diagnosis, tumor staging, and disease prognosis assessment, and is gradually becoming an indispensable key technology in clinical diagnosis and treatment. In the data analysis workflow of whole-genome sequencing, gene alignment is the core pivotal step connecting raw sequencing data with clinical interpretation. Its core task is to match and locate the short reads generated by the sequencer with known human reference genomes, clarifying the specific location and variation information of each short read in the genome. The computational complexity of gene alignment is extremely high, causing the gene alignment step to typically account for a large portion of the entire analysis workflow, becoming a major bottleneck restricting the rapid delivery of sequencing results. Therefore, how to improve the computational performance of gene alignment and shorten its processing time has become a pressing issue in the field of gene data analysis. Summary of the Invention

[0004] Through in-depth research and practice, the inventors noticed that current related work mainly focuses on optimizing the core algorithm of gene alignment, such as improving the performance of a single step by improving the BWT (Burrows-Wheeler Transform) index construction strategy. However, the inventors believe that gene alignment is not an isolated algorithm execution process, but a complete chain of processes involving multiple stages, including file reading and parsing, memory data management, data computation, and result generation and output. Even if the efficiency of the core gene alignment algorithm is improved, if other stages of gene alignment encounter problems, such as frequent cache misses due to memory data scheduling, the overall process time will still be difficult to significantly reduce. Therefore, improving the performance of a single algorithm alone cannot provide a complete solution for overall process optimization. How to improve the overall process performance by adjusting the computation process and optimizing memory resources from the perspective of the entire process is a worthwhile approach to consider.

[0005] In this regard, this application aims to provide a new solution to accelerate the gene alignment process, which optimizes the overall execution flow of gene alignment from a global perspective, thereby improving the efficiency and performance of gene alignment.

[0006] The objective of this application is achieved through the following technical solution: According to the first aspect of this application, a system for accelerating the gene alignment execution process is provided, comprising a processing thread management module, one or more dynamic caches, and a dynamic cache management module. The processing thread management module is configured to set the gene alignment process as a multi-stage pipeline, with each execution step of the gene alignment process corresponding to a pipeline stage in the multi-stage pipeline, and to allocate multiple processing threads to each pipeline stage, and to divide the gene sequence data to be aligned into multiple data blocks and provide them to the multi-stage pipeline for processing; the dynamic cache is used for data transfer between two pipeline stages, and each dynamic cache includes at least two cache blocks; the dynamic cache management module is configured to manage the allocation and reclamation of dynamic caches between two pipeline stages, and to control the switching of working states between different cache blocks in each dynamic cache.

[0007] Preferably, the gene alignment process includes at least the following execution steps: Step 1: Decompress and read the sequence file to generate the raw data stream of the sequence file; Step 2: Parse the sequence file to generate structured sequence data; Step 3: Accurately match sequences to generate matching seeds; Step 4: Expand seed sequences and calculate matching scores; Step 5: Adjust candidate matches and calculate global matching scores; Step 6: Filter high-score matching results and generate alignment files; and wherein the one or more dynamic caches include at least: a first dynamic cache, set between the first pipeline level corresponding to Step 1 and the second pipeline level corresponding to Step 2; a second dynamic cache, set between the second pipeline level corresponding to Step 2 and the third pipeline level corresponding to Step 3; and a third dynamic cache, set between the fifth pipeline level corresponding to Step 5 and the sixth pipeline level corresponding to Step 6.

[0008] Preferably, the dynamic cache management module is further configured to control the switching between available and processing states of different cache blocks in the first dynamic cache in the following manner: a cache block in the available state receives and stores the raw data stream provided by the processing thread of the first pipeline, and after storage, the working state of the cache block is switched from the available state to the processing state; the raw data stream in the cache block in the processing state is read by the processing thread executing the second pipeline, and after reading, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state; wherein, at the same time, at least one cache block in the first dynamic cache is in the available state and at least one cache block is in the processing state.

[0009] Preferably, the dynamic cache management module is further configured to control the switching of different cache blocks of the second dynamic cache between a processing state and an available state in the following manner: a cache block in the available state receives and stores sequential data provided by the processing thread of the second pipeline, and after storage, the working state of the cache block is switched from the available state to the processing state; the sequential data in the cache block in the processing state is read by the processing thread executing the third pipeline, and after reading, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state; wherein, at the same time, at least one cache block in the second dynamic cache is in the available state and at least one cache block is in the processing state.

[0010] Preferably, the dynamic cache management module is further configured to control the switching of different cache blocks in the third dynamic cache between the processing state and the available state in the following manner: a cache block in the available state receives and stores a global matching score provided by the processing thread of the fifth pipeline, and after storage, the working state of the cache block is switched from the available state to the processing state; the global matching score in the cache block in the processing state is read by the processing thread executing the sixth pipeline, and after reading, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state; wherein, at the same time, at least one cache block in the third dynamic cache is in the available state and at least one cache block is in the processing state.

[0011] Preferably, the memory address range occupied by the first dynamic cache is contiguous, the memory address range occupied by the second dynamic cache is non-contiguous, and the memory address range occupied by the third dynamic cache is non-contiguous.

[0012] Preferably, the dynamic cache management module is also configured to expand the space of each cache block in the second and / or third dynamic cache according to the amount of data stored.

[0013] Preferably, the processing thread management module is further configured to determine the number of processing threads allocated to the first pipeline stage based on the number of data blocks to be processed; and to determine the number of processing threads allocated to the third and fourth pipeline stages based on preset multi-threading parameters.

[0014] Preferably, the dynamic cache management module is also configured to allocate an independent dynamic cache for each compressed file.

[0015] According to a second aspect of this application, a method for accelerating a gene alignment process is provided, comprising: acquiring a compressed file of a gene sequence to be aligned; and inputting the compressed file into the system of the first aspect to accelerate the execution of the gene alignment process.

[0016] According to a third aspect of this application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in the second aspect.

[0017] Compared with existing technologies, the advantages of this application are mainly as follows: It employs a fine-grained pipeline partitioning and multi-threading strategy, clarifying the sequential relationship between each step in the gene alignment process. Each step is assigned a corresponding independent processing thread, and each thread has a clearly defined dedicated task, achieving asynchronous collaborative execution of each stage. The coordinated cooperation of all threads enhances the parallelism of the entire pipeline. Furthermore, to address the issue of frequent data transfer between different steps, a dynamic cache management mechanism is adopted, designing at least two cache blocks for each data interaction stage, enabling parallel data writing and reading. Through these two collaborative optimizations, the overall process computation and input / output overlap are higher, memory efficiency is improved, ultimately achieving higher computational performance and shorter runtime, resulting in a performance leap for the entire gene alignment process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 is a schematic diagram of a system structure for accelerating gene alignment according to an embodiment of this application; Figure 2 is a schematic diagram of the working state switching of a dynamic cache block according to an embodiment of this application; Figure 3 is a schematic diagram of the dynamic expansion of a second dynamic cache according to an embodiment of this application; Figure 4 is a schematic diagram of the dynamic expansion of a third dynamic cache according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided through specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0020] The inventors discovered in their research that current efforts to improve gene alignment computational performance mainly focus on optimizing the core algorithms used in gene alignment. However, they believe that gene alignment is not an isolated algorithm execution process, but rather a complete chain of processes involving multiple stages. Improving the performance of a single algorithm cannot create a synergistic optimization effect for the entire process. Therefore, a holistic approach, focusing on the performance of each stage in a coordinated manner, is the best solution to overcome the performance bottlenecks in gene alignment.

[0021] Furthermore, the inventors discovered that the traditional gene alignment process pipeline is too coarsely divided, with many steps being executed together. It also suffers from problems such as unreasonable thread allocation and rigid memory management, which leads to frequent waiting between data reading, processing, and calculation, resulting in very low overall process efficiency.

[0022] To address this, this application provides solutions for optimizing the gene alignment process by using a parallel pipeline and by optimizing the caching system to accelerate the entire gene alignment process. Since the input gene sequence data is very large, it is difficult to load it all into memory and process it all at once. Therefore, a data block approach is typically used, processing the data piece by piece. Each data block is usually stored in a compressed sequence file. In the embodiments of this application, the gene alignment process is divided into a more fine-grained pipeline, using a multi-stage pipeline to process the sequence file corresponding to each data block. The gene alignment process is mainly divided into the following steps: Step 1: Decompress and read the sequence file to generate the raw data stream of the sequence file; Step 2: Parse the sequence file to generate structured sequence data; Step 3: Accurately match sequences to generate matching seeds; Step 4: Expand seed sequences and calculate matching scores; Step 5: Adjust candidate matches and calculate global matching scores; Step 6: Filter high-score matching results and generate alignment files.

[0023] The above-described steps further refine the tasks at each stage of the gene alignment process, reducing dependencies and waiting times between tasks. A multi-stage pipeline (six stages in total) can be constructed based on this division of the gene alignment process. Each step in the gene alignment process corresponds to one stage in the multi-stage pipeline. Therefore, the above steps can be executed through this multi-stage parallel pipeline to process the gene sequence data to be aligned, thereby improving overall computational performance. The specific processing details and procedures involved in each of the above steps are existing technologies and will not be described in detail in this application.

[0024] Figure 1 is a schematic diagram of a system for accelerating gene alignment execution according to an embodiment of this application. As shown in Figure 1, the system includes: a decompression and sequence parsing module 101, a sequence alignment calculation module 102, a matching result output module 103, a processing thread management module (not shown in Figure 1), a dynamic cache, and a dynamic cache management module 104. The decompression and parsing module 101, the sequence alignment calculation module 102, and the alignment result output module 103 are connected sequentially and are responsible for implementing a pipelined processing flow for gene alignment. Specifically, the decompression and sequence parsing module 101 is configured to execute steps 1 and 2. The sequence alignment calculation module 102 is configured to execute steps 3 and 4. The matching result output module 103 is configured to execute steps 5 and 6. The processing thread management module is configured to set the gene alignment process as a multi-stage pipeline. Each execution step of the gene alignment process corresponds to a pipeline stage in the multi-stage pipeline, and multiple processing threads are allocated to each pipeline stage. The gene sequence data to be aligned is divided into multiple data blocks and provided to the multi-stage pipeline for processing. Dynamic caches are used for data transfer between two pipeline stages, and each dynamic cache includes at least two cache blocks. The dynamic cache management module 104 is configured to manage the allocation and reclamation of dynamic caches between two pipeline stages and control the switching of working states between different cache blocks in each dynamic cache.

[0025] The gene alignment process involves many steps with clear dependencies between them. For example, data must be parsed before alignment calculations can be performed; the order cannot be reversed. Traditional solutions often use a single thread to handle multiple steps, or the thread allocation and task dependencies may mismatch, leading to uneven workloads. This application's embodiment first clarifies the sequential relationships between each task, then maps each step (task) to a pipeline stage. Each pipeline stage's steps are executed by multiple processing threads. This way, each thread has a specific task, and all threads cooperate to maximize the parallelism of the entire pipeline. For the problem of frequent data transfer between steps, traditional solutions use a single cache, resulting in a situation where writing data cannot be read, and reading data cannot be written. This application's embodiment, however, designs multiple cache blocks for each data interaction stage, enabling parallel writing and reading of data without waiting for each other, significantly improving memory efficiency.

[0026] In some embodiments, a first dynamic cache (dynamic cache 1 in Figure 1) can be configured in the decompression and sequence parsing module 101 (i.e., between the first pipeline corresponding to step 1 and the second pipeline corresponding to step 2) to store the decompressed raw data stream. The first dynamic cache includes at least two cache blocks, each of which can store the raw data stream of one data block. After storage, it can be read by the parsing processing thread (step 2) of the decompression and sequence parsing module 101 to perform parsing operations on the raw data stream and generate the sequence data corresponding to the data block. A second dynamic cache (dynamic cache 2 in Figure 1) can be configured between the decompression and sequence parsing module 101 and the sequence alignment calculation module 102 (i.e., between the second pipeline and the third pipeline corresponding to step 3) to store the parsed sequence data. The second dynamic cache includes at least two cache blocks, each of which can store the sequence data of one data block. After storage, it can be read by the processing thread (the fourth pipeline corresponding to step 4) of the sequence alignment calculation module 102 to execute the alignment algorithm.

[0027] In traditional gene alignment processes, step 5 calculates the result of one data block, then stops all calculations, waits for the result to be completely written to a file (step 6), and only then starts calculating the next data block. However, writing to a file takes time, causing the calculation process to wait indefinitely for file writing, resulting in significant time wasted and extremely low overall efficiency. In some embodiments, a third dynamic cache (dynamic cache 3 in Figure 1) is located in the matching result output module 103 (between the fifth pipeline stage corresponding to step 5 and the sixth pipeline stage corresponding to step 6) to store the calculation results. The calculation results are obtained by the matching result output module 103 after executing step 5 based on the matching score output by the sequence alignment calculation module 102. The stored calculation results can be read by the processing thread corresponding to step 6 of the matching result output module 103 to write to a file and generate an alignment result report. The third dynamic cache includes at least two cache blocks, each of which can store the calculation result corresponding to one data block. In this embodiment, after calculating the gene alignment result of one data block, the result is not directly written to a file, but is temporarily stored in the third dynamic cache before processing the next data block. While the processing thread in step 5 is processing the new data block, the processing thread corresponding to the sixth pipeline stage in step 6, which writes the file, slowly writes the calculation result of the previous data block stored in the third dynamic cache into the file.

[0028] In some embodiments, the dynamic cache management module 104 is further configured to control the switching between available and processing states of different cache blocks in the first dynamic cache in the following manner: A cache block in the available state receives and stores the raw data stream provided by the processing thread of the first pipeline. After storage, the working state of the cache block is switched from the available state to the processing state. The raw data stream in the cache block in the processing state is read by the processing thread of the second pipeline. After reading, the cache block is cleared and its working state is switched from the processing state to the available state. The initial working state of multiple cache blocks in the first dynamic cache is the available state. At any given time (except at the initial moment), at least one cache block in the first dynamic cache is in the available state and at least one cache block is in the processing state, to ensure that data writing and data processing (being read) in the first dynamic cache are executed in parallel, improving the overall throughput and execution speed.

[0029] In some embodiments, the dynamic cache management module 104 is further configured to control the switching of different cache blocks of the second dynamic cache between a processing state and an available state in the following manner: A cache block of the second dynamic cache in the available state receives and stores sequential data provided by a processing thread of the second pipeline stage; after storage, the working state of the cache block is switched from the available state to the processing state. Sequential data in a cache block in the processing state is read by a processing thread executing the third pipeline stage; after reading, the cache block is cleared and its working state is switched from the processing state to the available state. The initial working state of multiple cache blocks in the second dynamic cache is the available state. At any given time (except at the initial moment), at least one cache block in the second dynamic cache is in the available state, and at least one cache block is in the processing state. This embodiment of the application can ensure that data writing and data processing in the second dynamic cache are executed in parallel, improving the overall throughput of the process.

[0030] In some embodiments, the dynamic cache management module 104 is further configured to control the switching of the working states among multiple cache blocks in the third dynamic cache. Specifically: a cache block in the third dynamic cache that is in an available state receives and stores a global matching score provided by the processing thread of the fifth pipeline. After storage, the dynamic cache management module 104 switches the working state of the cache block from the available state to the processing state. The global matching score in the cache block in the processing state is read by the processing thread executing the sixth pipeline. After reading, the dynamic cache management module 104 clears the cache block and switches the working state of the cache block from the processing state to the available state. The initial working state of multiple cache blocks in the third dynamic cache is all available. At any given time (except at the initial moment), at least one cache block in the third dynamic cache is in an available state, and at least one cache block is in a processing state.

[0031] Figure 2 is a schematic diagram illustrating the switching of the working state of a cache block in a dynamic cache according to an embodiment of this application. The following description uses a first dynamic cache comprising two cache blocks (Cache 1-1 and Cache 1-2) as an example to illustrate the switching process between the processing state and the available state of the cache blocks in the first dynamic cache controlled by the dynamic cache management module. As shown in Figure 2, a cache block in the available state of the first dynamic cache receives and stores the original data stream of one data block. After storage, the dynamic cache management module switches the working state of the cache block from the available state to the processing state. Because the first dynamic cache has only two cache blocks, at any given time, it can store the original data streams corresponding to at most two data blocks. To load the original data stream corresponding to a third data block into the first dynamic cache, one of the occupied cache blocks must first be cleared before the new data block is loaded. At this time, the original data stream in the cache block in the processing state can be read by the processing thread executing the second pipeline stage. After reading, the dynamic cache management module can clear the cache block and switch its working state from the processing state to the available state, continuing to receive and store the original data stream of the third data block. Cache 1-1 and Cache 1-2 take turns serving the roles of writing new data and providing data for subsequent steps to read.

[0032] The following example, using a second dynamic cache comprising two cache blocks (Cache 2-1 and Cache 2-2), illustrates the process by which the dynamic cache management module controls the switching between the processing and available states of the cache blocks in the second dynamic cache. Referring again to Figure 2, the cache block in the available state of the second dynamic cache receives and stores sequence data. After storage, the dynamic cache management module switches the working state of the cache block from the available state to the processing state. Because the second dynamic cache has only two cache blocks, at any given time, only the sequence data corresponding to two data blocks can be stored. To load the sequence data of a third data block into the second dynamic cache, one of the occupied cache blocks must first be cleared, and then the sequence data of the new data block must be loaded. At this time, the sequence data in the cache block in the processing state can be read by the processing thread executing the third pipeline stage. After reading, the dynamic cache management module clears the cache block and switches its working state from the processing state to the available state, continuing to receive and store the sequence data of the third data block. Cache 2-1 and Cache 2-2 alternately assume the roles of writing new data and providing data for subsequent steps to read.

[0033] The following example, using a third dynamic cache comprising two cache blocks (Cache 3-1 and Cache 3-2), illustrates the process by which the dynamic cache management module controls the switching between the processing and available states of the cache blocks in the third dynamic cache. Referring again to Figure 2, the processing thread corresponding to step 5 in the matching result output module reads the matching score (calculation result) output by the sequence alignment calculation module 102 and stores it in the third dynamic cache. Because the third dynamic cache has only two cache blocks (Cache 3-1 and Cache 3-2), it can only store the calculation results corresponding to two data blocks at any given time. To load the calculation result corresponding to the third data block into the third dynamic cache, one of the occupied cache blocks must first be cleared before the new data is loaded. At this point, the processing thread corresponding to step 6 in the matching result output module reads the calculation result from the third dynamic cache. After reading, the dynamic cache management module can clear the cache block and switch its working state from the processing state to the available state, continuing to receive and store the calculation result corresponding to the third data block.

[0034] Because the first dynamic cache stores decompressed data, the size of the data blocks is fixed during block decompression. Therefore, in some embodiments, the size of the cache blocks in the first dynamic cache can be fixed, and the memory address range it occupies is contiguous. However, when parsing data blocks, the length, name (different number of characters), and structure (whether it contains inserted / missing segments) of different sequence data have no fixed standard. If a fixed-size contiguous memory is allocated for each sequence data, two extreme cases will occur: if the allocated cache space is too large, memory is wasted (short sequences occupy a large space); if the allocated cache space is too small, when encountering a long sequence, the current memory must be released first, and then a larger memory must be allocated to copy the data. Frequent allocation and release will generate a large number of system calls, which is not only time-consuming but also leads to memory fragmentation, severely reducing memory efficiency. Therefore, in some embodiments, the memory address range occupied by the second dynamic cache is non-contiguous. As shown in Figure 3, a sequence of data is split into multiple independent parts (such as sequence name, sequence bases, sequence annotations, quality score, etc.), each occupying its own independent memory space (for example, the name occupies 16 bytes and is stored in memory at addresses 0x10000020-0x1000002F, and the sequence bases occupy 110 bytes and are stored in memory at addresses 0x20000100-0x2000016D), with their physical addresses being non-contiguous. When expansion is needed, only the part that needs to be expanded needs to be dynamically adjusted (e.g., if the sequence bases need to occupy 150 bytes, then expand to 0x20000100-0x20000195), while other parts (such as the name) remain unaffected, thus achieving dynamic expansion of a single part of the second dynamic cache. Figure 3 also shows the expansion of the cache size occupied by the quality score; since the principle is the same, it will not be elaborated here. Each part of each sequence in this embodiment occupies a separate memory block, which facilitates the dynamic expansion of a single part. When a larger sequence is parsed, the dynamic memory is expanded. If the sequence is less than or equal to the current cache space, the current cache is used directly. This dynamic expansion strategy can reduce frequent memory allocation and release, thereby improving memory efficiency.

[0035] Because the size of the sequence data is not fixed, the size of the alignment result is also not fixed. In some embodiments, the memory address range occupied by the third dynamic cache is non-contiguous. As shown in Figure 4, the alignment result occupies 200B in the third dynamic cache. Because the memory address occupied by the cache block is non-contiguous, the size of the cache block occupied by the alignment result can be dynamically expanded as needed. Figure 4 shows the cache block being expanded to 300B.

[0036] To further improve the parallel processing capability of the system, in some embodiments, the processing thread management module is also configured to allocate a decompression processing thread for each input file based on the number of input files; determine the number of processing threads allocated to the first pipeline stage based on the number of data blocks to be processed; and determine the number of processing threads allocated to the third and fourth pipeline stages based on the processor capability (or user-preset multithreading parameters).

[0037] In some embodiments, the processing thread management module is also configured to allocate an independent dynamic cache for each compressed file.

[0038] In some embodiments, the processing thread management module assigns a number to each data block. When the last data block is parsed, the steps handled by the decompression and sequence parsing modules are set to end, and the steps handled by the subsequent modules are processed accordingly. After the system completes the last data block, the pipeline stops running, the dynamic cache management module releases all cache blocks, and the operation ends.

[0039] According to another embodiment of this application, a method for accelerating gene alignment is provided, comprising: obtaining a compressed file of gene sequences to be aligned; and inputting the compressed file into the system of the aforementioned embodiment to accelerate the execution of the gene alignment process.

[0040] This application's embodiment decomposes the pipeline with finer granularity. The traditional input module is broken down into independent small steps: a dedicated step for file decompression and a dedicated step for parsing the decompressed data. This transforms the messy raw data into structured gene sequence data that the computer can recognize. This decomposition eliminates the need to wait for one step to complete before starting the next; for example, while decompressing, the previously decompressed data block can be parsed simultaneously. Secondly, the multi-threading strategy refines the dependencies between tasks, assigning a dedicated processing thread to each step. This application's embodiment first clarifies the sequential relationship between each step, then allocates corresponding processing threads based on the workload and computational complexity of each step. For example, if the data parsing step has a large workload, more threads are allocated. Each thread has a clearly defined task, and all threads cooperate to improve the parallelism of the entire pipeline. Finally, a dynamically expanding multi-caching strategy is adopted for data interaction between different steps. Multiple cache blocks are designed for each data interaction stage, and dynamic cache expansion is supported. Multiple cache blocks can work alternately; some are responsible for receiving new data from the previous step, while others are responsible for passing existing data to the next step, enabling parallel data writing and reading. Meanwhile, the cache size is not fixed, and the memory addresses are not contiguous. When encountering short sequence data, the current cache space is used, avoiding memory waste; when encountering long sequence data, the cache automatically expands, eliminating the need for frequent memory reallocation. This avoids memory waste and reduces the time spent on frequent memory operations, significantly improving memory efficiency. Compared to traditional solutions, this embodiment can process more sequencing data and output alignment results faster, perfectly meeting the core needs of massive data and rapid analysis in whole-genome sequencing technology.

[0041] References to "various embodiments," "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a specific feature, structure, or property described in connection with an embodiment that is included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that such combination is not illogical or inoperable.

[0042] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments or implementation methods. Similar or identical parts between the various embodiments of the present invention can be referred to mutually. The implementation principles and technical effects of the inventive concept can be referred to mutually, and will not be repeated here. Where there is no conflict, the various embodiments or implementation methods in this invention can be combined with each other.

[0043] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0044] Embodiments of this application may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0045] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0046] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used in the embodiments of the present invention is chosen to best explain the principles, practical applications, or technological improvements in the market, or to enable other those skilled in the art to understand the disclosed embodiments of the present invention.

Claims

1. A system for accelerating the gene alignment process, wherein, This includes a thread management module, one or more dynamic caches, and a dynamic cache management module; The processing thread management module is configured to set the gene alignment process as a multi-level pipeline, each execution step of the gene alignment process corresponds to a pipeline level in the multi-level pipeline, and multiple processing threads are allocated to each pipeline level. The gene sequence data to be aligned is divided into multiple data blocks and provided to the multi-level pipeline for processing. Dynamic caching is used for data transfer between two pipeline stages. Each dynamic cache includes at least two cache blocks. The dynamic cache management module is configured to manage the allocation and reclamation of dynamic caches between two pipeline stages and control the switching of working states between different cache blocks in each dynamic cache.

2. The system according to claim 1, wherein, The gene alignment process includes at least the following steps: Step 1: Decompress and read the sequence file to generate the raw data stream of the sequence file; Step 2: Parse the sequence file to generate structured sequence data; Step 3: Accurately match sequences to generate matching seeds; Step 4: Expand seed sequences and calculate matching scores; Step 5: Adjust candidate matches and calculate global matching scores. Step 6: Filter high-score matching results and generate a comparison file; And wherein, the one or more dynamic caches include at least: a first dynamic cache, set between the first pipeline corresponding to step 1 and the second pipeline corresponding to step 2; a second dynamic cache, set between the second pipeline corresponding to step 2 and the third pipeline corresponding to step 3; and a third dynamic cache, set between the fifth pipeline corresponding to step 5 and the sixth pipeline corresponding to step 6.

3. The system according to claim 2, wherein, The dynamic cache management module is also configured to control the switching of different cache blocks of the first dynamic cache between the available state and the processing state in the following manner: the cache block in the available state receives and stores the raw data stream provided by the processing thread of the first pipeline, and after the storage is completed, the working state of the cache block is switched from the available state to the processing state. The raw data stream in the cache block in the processing state is read by the processing thread executing the second pipeline stage. After reading is completed, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state. At the same time, at least one cache block in the first dynamic cache is in the available state and at least one cache block is in the processing state.

4. The system according to claim 2, wherein, The dynamic cache management module is also configured to control the switching of different cache blocks in the second dynamic cache between a processing state and an available state in the following manner: a cache block in the available state receives and stores sequential data provided by the processing thread of the second pipeline, and after storage, the working state of the cache block is switched from the available state to the processing state; the sequential data in the cache block in the processing state is read by the processing thread executing the third pipeline, and after reading, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state; wherein, at the same time, at least one cache block in the second dynamic cache is in the available state and at least one cache block is in the processing state.

5. The system according to claim 2, wherein, The dynamic cache management module is also configured to control the switching of different cache blocks of the third dynamic cache between the processing state and the available state in the following manner: the cache block in the available state receives and stores the global matching score provided by the processing thread of the fifth pipeline, and after the storage is completed, the working state of the cache block is switched from the available state to the processing state. The global matching score in the cache block that is in the processing state is read by the processing thread executing the sixth pipeline stage. After reading, the cache block is cleared and the working state of the cache block is switched from the processing state to the available state. At the same time, at least one cache block in the third dynamic cache is in the available state and at least one cache block is in the processing state.

6. The system according to claim 2, wherein, The memory address range occupied by the first dynamic cache is contiguous, the memory address range occupied by the second dynamic cache is non-contiguous, and the memory address range occupied by the third dynamic cache is non-contiguous.

7. The system according to claim 6, wherein, The dynamic cache management module is also configured to expand the space of each cache block in the second and / or third dynamic caches based on the amount of data stored.

8. The system according to claim 2, wherein, The processing thread management module is also configured to determine the number of processing threads allocated to the first pipeline stage based on the number of data blocks to be processed; And the number of processing threads allocated to the third and fourth pipeline stages is determined based on preset multi-threading parameters.

9. The system according to claim 1, wherein, The dynamic cache management module is also configured to allocate an independent dynamic cache for each compressed file.

10. A method for accelerating gene alignment processes, wherein, This includes acquiring a compressed file of the gene sequence to be compared; and inputting the compressed file into the system according to any one of claims 1-9 to accelerate the execution of the gene comparison process.

Images