Distributed sequence alignment method and system based on Spark

Abstract

The invention discloses a distributed sequence alignment method and system based on Spark, and relates to the technical field of bioinformatics, and the method comprises the steps: receiving a Minimap2 reference sequence, and generating a multi-part index of the reference sequence based on a multi-part index generation function of the Minimap2; partitioning the multi-part index of the reference sequence based on a character string in the multi-part index of the reference sequence as an identifier to obtain a partitioned copy, and distributing the partitioned copy to each node of a Spark cluster; receiving a query sequence, partitioning the query sequence through a file format of FASTA or FASTQ, and generating a multi-node shared task queue based on the number of partitions of the reference sequence, the number of partitions of the query sequence and partition copies on each node; and based on the multi-node shared task queue, each node selects the task in the task queue which needs to process all the remaining tasks and takes the longest time in the task queues corresponding to all the index file partition copies of the node to process, and a task processing result is obtained. According to the method, strategies of query sequence partitioning and reference sequence database partitioning are combined, dynamic load balancing is achieved by establishing a multi-node shared task queue, and the efficiency of comparison between Minimap2 and a large-scale data set is optimized. The method is suitable for long sequence data generated by a third-generation sequencing technology, and is especially suitable for application scenes of on-site rapid sequencing and analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics, and specifically to a Spark-based distributed sequence alignment method and system. Background Art

[0002] In recent decades, gene sequencing technology has been continuously developing and improving. Although the first generation sequencing technology represented by the dideoxy termination method and chemical degradation method has good accuracy and sequencing length, it is expensive, slow, and requires a large amount of DNA templates, so it cannot be widely used. The next generation sequencing (NGS) is a revolutionary gene sequencing technology, also known as high-throughput sequencing. Its main advantage is that it can sequence a large number of DNA molecules at the same time. Therefore, compared with the first generation sequencing technology, the second generation sequencing technology can perform gene sequencing at a lower cost, faster speed and larger scale. The second generation sequencing technology is widely used, but the shorter read segments make it difficult to solve the assembly problems of highly repetitive sequences and complex genomic regions, and although its sequencing cost is much lower than the first generation, it is still high. Currently, sequencing technology has developed to the third generation, also known as single-molecule sequencing technology. Representative companies of the third-generation sequencing technology include Oxford Nanopore Technologies (ONT) and Pacbio. Long read length is also a significant feature of the third-generation sequencing technology. The third-generation sequencing technology can generate read segments of up to 10Kbp or even up to 1Mbp. The nanopore sequencing technology developed by ONT has the characteristics of portable equipment and fast sequencing speed, and has a longer read length than Pacbio's Single Molecule, Real-Time (SMRT) technology.

[0003] Metagenomics is a field that studies all genomes in complex microbial communities. Unlike traditional microbiological research, metagenomics does not rely on culturing microorganisms, but directly extracts DNA from environmental samples (such as soil, seawater, human intestines, etc.), and uses sequencing technology to perform gene sequencing to analyze the diversity, function, and dynamic changes of the microorganisms. The portable and fast speed of nanopore sequencing makes it possible to conduct real-time gene detection in the field, which greatly facilitates the study of metagenomics. However, as the cost of sequencing becomes lower and lower, the amount of genetic data generated by data collection is also exploding. Metagenomic sequencing data can often reach tens or even hundreds of GB. Therefore, how to effectively and quickly store and analyze these huge data has become an important challenge for research and application.

[0004] Sequence alignment is a basic data analysis technique in bioinformatics, which is mainly used to discover the similarities between biological sequences (DNA, RNA or protein sequences). By comparing the similarities and differences between sequences, the associations between these sequences in terms of function, structure and evolutionary relationship can be inferred. Sequence alignment is widely used in taxonomic assignment, gene function annotation, evolutionary relationship analysis, drug design and target identification, identification of disease-related variants, and so on. Taxonomic assignment often requires the comparison between the query sequence and a large reference sequence database, while traditional gene sequence alignment tools often affect the speed of data processing due to the limitations of computing power and storage resources when processing large-scale data, thus further affecting the efficiency of the entire data analysis process. Nowadays, the third-generation sequencing can achieve on-site real-time sequencing and is widely used in various aspects of biological research, especially metagenomic research that can collect samples on-site without laboratory cultivation. Accelerating the speed of sequence alignment can help researchers collect data on-site while also allowing them to quickly analyze data on-site.

[0005] Minimap2 is a general sequence alignment tool that can be used for mapping reads to reference genomes, aligning query sequences with large reference sequence databases, aligning whole genomes, and so on. It is fast and highly accurate. Minimap2 is particularly suitable for processing long reads generated by third-generation sequencing, and chain extension enables Minimap2 to better cope with the high error rate of third-generation sequencing. Although the alignment speed has been greatly improved compared to other sequence alignment tools, the traditional stand-alone serial Minimap2 still leads to inefficiency in the entire data analysis process due to insufficient computing power when processing large-scale data. Summary of the invention

[0006] In order to solve the deficiencies mentioned in the above background technology, the object of the present invention is to provide a distributed sequence alignment method and system based on Spark.

[0007] In a first aspect, the purpose of the present invention can be achieved by the following technical solution: a distributed sequence alignment method based on Spark, the method comprising the following steps:

[0008] Receive a Minimap2 reference sequence, generate a multi-part index of the reference sequence based on the multi-part index generation function of Minimap2; partition the multi-part index of the reference sequence based on the string in the multi-part index of the reference sequence as an identifier, obtain partition copies, and distribute the partition copies to each node of the Spark cluster;

[0009] Receive the query sequence, partition the query sequence using the FASTA or FASTQ file format, and generate a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node;

[0010] Based on the multi-node shared task queue, each node selects the task in the task queue corresponding to all its own index file partition copies that takes the longest time to process all remaining tasks, and obtains the task processing result.

[0011] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of generating the multi-part index of the reference sequence based on the multi-part index generation function of Minimap2 is as follows:

[0012] Split the reference sequence into parts according to pre-set rules, and each part will be assigned to a processing unit for processing;

[0013] Generate a local index for each sequence block by using a specific hash algorithm;

[0014] Merge local indexes. The merging process is performed on the central node.

[0015] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of partitioning the multi-part index of the reference sequence based on the character string in the multi-part index of the reference sequence as an identifier:

[0016] Specify the number of partitions, generate partition files and number each file starting from 0;

[0017] Read the reference sequence index file in binary mode, and initialize the file pointer to point to the partition file numbered 0;

[0018] Read a byte. If the byte is not character M, write the character to the partition file pointed to by the current file pointer and judge the byte; otherwise, read three bytes 5 in succession.

[0019] Determine whether the read byte is the last byte in the index file. If so, end the partition; otherwise, read the byte again.

[0020] Read three bytes continuously and determine whether the three bytes are M, I, and \x02 respectively. If so, move the file pointer to the partition file corresponding to the next number. If it has already pointed to the partition file with the largest number, re-point the current file pointer to the partition file with number 0; otherwise, move the file pointer to the partition file with the next number, write the read byte into the partition file pointed to by the current file pointer, and then determine whether the read byte is the last byte in the index file.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: storing copies of multiple different index file partitions on each node, and establishing a multi-node shared task queue, and each node selects the task to be processed based on the time required to process all remaining tasks in the task queue.

[0022] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the node selection formula is:

[0023]

[0024] db is the task queue finally selected, P is the task queue corresponding to the node, and all the tasks in the priority queue have not been fully executed. x The number of tasks remaining in the task queue that have not been executed, N x is the number of tasks being executed in the task queue. argmax() returns The input value x (x∈P) at which the function reaches its maximum value.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a process of generating the multi-node shared task queue:

[0026] Specify the number of replicas per database partition, the number of database partitions, and the number of nodes in the cluster;

[0027] Generates a partition copy of the index file stored on each node based on the number of replicas, the number of database partitions, and the number of nodes;

[0028] Specify the number of query sequence partitions;

[0029] Generate query sequence partitions and upload partition files to HDFS;

[0030] A multi-node shared task queue is generated for each index file partition, and the path of each query sequence file partition in HDFS is stored in the task queue corresponding to each index partition.

[0031] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the task processing process of each node based on the multi-node shared task queue is as follows:

[0032] Each node traverses the task queues corresponding to all its index partition replicas and finds the task queue with the largest number of remaining tasks divided by the number of tasks being executed, and obtains the path of the query sequence partition file in HDFS from it; if there are no tasks in all task queues, the task is processed;

[0033] If there are tasks in the task queue, the query sequence partition in HDFS is pulled to the local node and compared with the index partition file corresponding to the task queue;

[0034] Continue to obtain the query sequence partition path from the task queue. If there are no remaining tasks in the task queue, re-determine whether there are tasks in the task queue; otherwise, re-perform the comparison process;

[0035] Collect the output results of all comparison tasks and merge them into task processing results.

[0036] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a distributed sequence alignment system based on Spark, comprising:

[0037] A sequence partitioning module is used to receive a Minimap2 reference sequence, generate a multi-part index of the reference sequence based on the multi-part index generation function of Minimap2, partition the multi-part index of the reference sequence based on the string in the multi-part index of the reference sequence as an identifier, obtain partition copies, and distribute the partition copies to each node of the Spark cluster;

[0038] A task queue generation module is used to receive a query sequence, partition the query sequence in a FASTA or FASTQ file format, and generate a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node;

[0039] The task processing module is used for each node to select the task in the task queue corresponding to all its own index file partition copies based on the multi-node shared task queue, and to process the task that takes the longest time to process all remaining tasks to obtain the task processing result.

[0040] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, a distributed sequence alignment method based on Spark as described above is adopted.

[0041] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a computer-readable storage medium is disclosed, in which a computer program is stored. When the computer program is loaded and executed by a processor, a distributed sequence alignment method based on Spark as described above is adopted.

[0042] Beneficial effects of the present invention:

[0043] The present invention realizes dynamic load balancing by sharing task queues among multiple nodes, so that the resource utilization of the cluster is further improved, and the running speed of the program is improved. The present invention is suitable for comparing a query sequence with a large reference sequence database, and improves the CPU utilization of the program when comparing the query sequence with the large reference sequence database by performing reference sequence index partitioning. The present invention combines two partitioning strategies, query sequence partitioning and reference sequence index partitioning, so that the program reaches a balance between load balancing capability and CPU utilization, so that the program has better load balancing capability while not causing a significant decrease in CPU utilization due to too many query sequence partitions, thereby causing the running speed of the program to decrease. The present invention directly uses Minimap2 to perform sequence alignment during the alignment process, and can make the result of distributed alignment completely consistent with the result of stand-alone alignment by specifying the -f parameter of Minimap2, and will not reduce the alignment accuracy of Minimap2 due to distributed parallelization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a schematic flow chart of the method of the present invention;

[0046] Figure 2 This is a schematic diagram of multi-node shared task queue;

[0047] Figure 3 This is a comparison chart of the running time in Example 1 of the present invention;

[0048] Figure 4 This is a comparison diagram of load standard deviation in Example 1 of the present invention;

[0049] Figure 5 This is a comparison chart of the running time in Example 2 of the present invention;

[0050] Figure 6 This is a comparison diagram of load standard deviation in Example 2 of the present invention;

[0051] Figure 7 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1:

[0054] like Figure 1 As shown, a distributed sequence alignment method based on Spark includes the following steps:

[0055] S101: receiving a Minimap2 reference sequence, generating a multi-part index of the reference sequence based on a multi-part index generation function of Minimap2; partitioning the multi-part index of the reference sequence based on a string in the multi-part index of the reference sequence as an identifier, obtaining partition copies, and distributing the partition copies to each node of the Spark cluster;

[0056] The process of generating a multi-part index for a reference sequence based on the multi-part index generation function of Minimap2 is as follows:

[0057] Split the reference sequence into parts according to pre-set rules, and each part will be assigned to a processing unit for processing;

[0058] Generate a local index for each sequence block by using a specific hash algorithm;

[0059] Merge local indexes. The merging process is usually performed on a central node to facilitate global comparison operations.

[0060] The process of partitioning a multi-part index of a reference sequence based on a string within the multi-part index of the reference sequence as an identifier:

[0061] Specify the number of partitions, generate partition files and number each file starting from 0;

[0062] Read the reference sequence index file in binary mode, and initialize the file pointer to point to the partition file numbered 0;

[0063] Read a byte. If the byte is not character M, write the character to the partition file pointed to by the current file pointer and judge the byte; otherwise, read three bytes 5 in succession.

[0064] Determine whether the read byte is the last byte in the index file. If so, end the partition; otherwise, read the byte again.

[0065] Read three bytes continuously and determine whether the three bytes are M, I, and \x02 respectively. If so, move the file pointer to the partition file corresponding to the next number. If it has already pointed to the partition file with the largest number, re-point the current file pointer to the partition file with number 0; otherwise, move the file pointer to the partition file with the next number, write the read byte into the partition file pointed to by the current file pointer, and then determine whether the read byte is the last byte in the index file.

[0066] S102: receiving a query sequence, partitioning the query sequence in a FASTA or FASTQ file format, and generating a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node;

[0067] Multiple copies of different index file partitions are stored on each node, and a multi-node shared task queue is established. Each node selects the task to be processed based on the time required to process all remaining tasks in the task queue.

[0068] The node selection formula is:

[0069]

[0070] db is the task queue finally selected, P is the task queue corresponding to the node, and all the tasks in the priority queue have not been fully executed. x The number of tasks remaining in the task queue that have not been executed, N x is the number of tasks being executed in the task queue. argmax() returns The input value x (x∈P) at which the function reaches its maximum value.

[0071] The multi-node shared task queue is as follows: Figure 2 As shown, the generation process is as follows:

[0072] Specify the number of replicas per database partition, the number of database partitions, and the number of nodes in the cluster;

[0073] Generates a partition copy of the index file stored on each node based on the number of replicas, the number of database partitions, and the number of nodes;

[0074] Specify the number of query sequence partitions;

[0075] Generate query sequence partitions and upload partition files to HDFS;

[0076] A multi-node shared task queue is generated for each index file partition, and the path of each query sequence file partition in HDFS is stored in the task queue corresponding to each index partition.

[0077] S103: Based on the multi-node shared task queue, each node selects and processes the task in the task queue corresponding to all its own index file partition copies that takes the longest time to process all remaining tasks, and obtains the task processing result.

[0078] The task processing process of each node based on the multi-node shared task queue is as follows:

[0079] Each node traverses the task queues corresponding to all its index partition replicas and finds the task queue with the largest number of remaining tasks divided by the number of tasks being executed, and obtains the path of the query sequence partition file in HDFS from it; if there are no tasks in all task queues, the task is processed;

[0080] If there are tasks in the task queue, the query sequence partition in HDFS is pulled to the local node and compared with the index partition file corresponding to the task queue;

[0081] Continue to obtain the query sequence partition path from the task queue. If there are no remaining tasks in the task queue, re-determine whether there are tasks in the task queue; otherwise, re-perform the comparison process;

[0082] Collect the output results of all comparison tasks and merge them into task processing results.

[0083] Specifically, the scheme of the present invention is further elaborated through an embodiment as follows: The present invention realizes multi-node shared task queues for each database partition through the list data structure of Redis. The key of each list is set to the corresponding database partition name, and after the partitioning of the query sequence file is completed, lpush is used to put the files of the query file partition into the list. The execution node only needs to call rpop to the corresponding task queue according to the priority of its own task when acquiring the task to obtain the file name of the query file partition, and then download the corresponding partition file from HDFS to the local to call the Minimap2 program through the Runtime class, and obtain the output result of Minimap2 through the standard output stream of the process. The present invention also stores the number of tasks running in each task queue through Redis. Whenever a node selects a task in a task queue to run, the number of executed tasks corresponding to the corresponding task queue in Redis will increase by one.

[0084] The running speed and load balancing capability of the distributed Minimap2 (i.e., IMOS) based on query sequence partitioning and the present invention were tested and compared. The number of reference sequence database partitions used in the test is 29, and the number of query sequence partitions is 2. The test uses docker to establish a container cluster on the server, wherein the hardware configuration of the server physical machine is shown in Table 1, and the performance configuration of the container is shown in Table 2. The query sequence used in the test is a real metagenomic assembly data set in the research project PRJEB48021, which contains 11,346 sequences. The reference sequence database used is the bacteria, archaea, and viral genome sequences with an assembly level of "Complete Genome" in the Refseq sequence database of NCBI, which contains 35,938 sequences. The parameters used to generate the index file are instructions such as "minimap2-x asm20-I 1536m-k 15-w 10-d". The running time test results are shown in the figure. Figure 3 As shown, the load standard deviation test results are as follows Figure 4 shown.

[0085] The running time and load balancing capabilities of IMOS and the present invention were tested and compared when there were differences in node performance in the cluster. The number of reference sequence database partitions used in the test was 29, and the number of query sequence partitions was 2. The test used Docker to establish a container cluster on the server, where the hardware configuration of the server physical machine is shown in Table 1 and the performance configuration of the container is shown in Table 3. The data set and parameters used in the test are the same as above. The running time test results are shown in Figure 5 As shown, the load standard deviation test results are as follows Figure 6 shown.

[0086] Table 1 shows the server hardware configuration

[0087]

[0088] Table 2 shows the cluster configuration with the same node performance.

[0089]

[0090] Table 3 Cluster configurations with different node performance

[0091]

[0092] Embodiment 2: In the second aspect, as Figure 7 As shown, in order to achieve the above purpose, the present invention discloses a distributed sequence alignment system based on Spark, comprising:

[0093] The sequence partitioning module 11 is used to receive the Minimap2 reference sequence, generate a multi-part index of the reference sequence based on the multi-part index generation function of Minimap2; partition the multi-part index of the reference sequence based on the string in the multi-part index of the reference sequence as an identifier, obtain partition copies, and distribute the partition copies to each node of the Spark cluster;

[0094] A task queue generation module 12 is used to receive a query sequence, partition the query sequence in a FASTA or FASTQ file format, and generate a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node;

[0095] The task processing module 13 is used for each node to select the task in the task queue corresponding to all its own index file partition copies based on the multi-node shared task queue, and to process the task that takes the longest time to process all remaining tasks to obtain the task processing result.

[0096] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0097] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0099] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A distributed sequence alignment method based on Spark, characterized in that: The method comprises the following steps: Receive a Minimap2 reference sequence, and generate a multi-part index of the reference sequence based on a multi-part index generation function of Minimap2; Partition the multi-part index of the reference sequence based on the string in the multi-part index of the reference sequence as an identifier, obtain partition copies, and distribute the partition copies to each node of the Spark cluster; Receive the query sequence, partition the query sequence using the FASTA or FASTQ file format, and generate a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node; Based on the multi-node shared task queue, each node selects the task in the task queue corresponding to all its own index file partition copies that takes the longest time to process all remaining tasks, and obtains the task processing result.

2. A distributed sequence alignment method based on Spark according to claim 1, characterized in that: The process of generating a multi-part index of a reference sequence based on the multi-part index generation function of Minimap2 is as follows: Split the reference sequence into parts according to pre-set rules, and each part will be assigned to a processing unit for processing; Generate a local index for each sequence block by using a specific hashing algorithm, k-mer matching, and graph methods; Merge local indexes. The merging process is performed on the central node.

3. The Spark-based distributed sequence alignment method according to claim 1, characterized in that: The process of partitioning a multi-part index of a reference sequence based on a string within the multi-part index of the reference sequence as an identifier: Specify the number of partitions, generate partition files and number each file starting from 0; Read the reference sequence index file in binary mode, and initialize the file pointer to point to the partition file numbered 0; Read a byte. If the byte is not character M, write the character to the partition file pointed to by the current file pointer and judge the byte; otherwise, read three bytes 5 in succession. Determine whether the read byte is the last byte in the index file. If so, end the partition; otherwise, read the byte again. Read three bytes continuously to determine whether the three bytes are M, I, and \x02 respectively. If so, move the file pointer to the partition file corresponding to the next number. If it has already pointed to the partition file with the largest number, re-point the current file pointer to the partition file with number 0. Otherwise, the file pointer is moved to the partition file with the next number, and the read byte is written to the partition file pointed to by the current file pointer, and then it is determined whether the read byte is the last byte in the index file.

4. The Spark-based distributed sequence alignment method according to claim 1, characterized in that: The nodes store multiple copies of different index file partitions, and establish a multi-node shared task queue. Each node selects the task to be processed based on the time required to process all remaining tasks in the task queue.

5. A Spark-based distributed sequence alignment method according to claim 4, characterized in that: The node selection formula is: db is the task queue finally selected, P is the task queue corresponding to the node, and all the tasks in the priority queue have not been fully executed. x The number of tasks remaining in the task queue that have not been executed, N x is the number of tasks being executed in the task queue. argmax() returns The input value x at which the function reaches its maximum value, where x∈P.

6. The Spark-based distributed sequence alignment method according to claim 1, characterized in that: The generation process of the multi-node shared task queue: Specify the number of replicas per database partition, the number of database partitions, and the number of nodes in the cluster; Generates a partition copy of the index file stored on each node based on the number of replicas, the number of database partitions, and the number of nodes; Specify the number of query sequence partitions; Generate query sequence partitions and upload partition files to HDFS; A multi-node shared task queue is generated for each index file partition, and the path of each query sequence file partition in HDFS is stored in the task queue corresponding to each index partition.

7. The Spark-based distributed sequence alignment method according to claim 1, characterized in that: The task processing process of each node based on the multi-node shared task queue is as follows: Each node traverses the task queues corresponding to all its index partition replicas and finds the task queue with the largest number of remaining tasks divided by the number of tasks being executed, and obtains the path of the query sequence partition file in HDFS from it; if there are no tasks in all task queues, the task is processed; If there are tasks in the task queue, the query sequence partition in HDFS is pulled to the local node and compared with the index partition file corresponding to the task queue; Continue to obtain the query sequence partition path from the task queue. If there are no remaining tasks in the task queue, re-determine whether there are tasks in the task queue; otherwise, re-perform the comparison process; Collect the output results of all comparison tasks and merge them into task processing results.

8. A distributed sequence alignment system based on Spark, characterized in that: include: A sequence partitioning module, which is used to receive a Minimap2 reference sequence and generate a multi-part index of the reference sequence based on the multi-part index generation function of Minimap2; Partition the multi-part index of the reference sequence based on the string in the multi-part index of the reference sequence as an identifier, obtain partition copies, and distribute the partition copies to each node of the Spark cluster; A task queue generation module is used to receive a query sequence, partition the query sequence in a FASTA or FASTQ file format, and generate a multi-node shared task queue based on the number of partitions of the reference sequence and the number of partitions of the query sequence and the partition copies on each node; The task processing module is used for each node to select the task in the task queue corresponding to all its own index file partition copies based on the multi-node shared task queue, and to process the task that takes the longest time to process all remaining tasks to obtain the task processing result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on a processor. When the processor loads and executes the computer program, a distributed sequence alignment method based on Spark described in any one of claims 1 to 7 is adopted.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by the processor, a Spark-based distributed sequence alignment method according to any one of claims 1 to 7 is adopted.

Images