Target pathogen detection method based on feature vector hash table, software and application

Abstract

The invention discloses a target pathogen detection method based on a feature vector hash table, software and application. Based on the combination of a deep learning neural network model and a K-mer sequence comparison technology, a K-mer sequence is converted into a feature vector considering local features and global features so as to predict possible species classification information. In the actual operation process, only the belonging-level reference hash table corresponding to the model prediction result needs to be loaded, and the whole genome reference database does not need to be loaded. Compared with the traditional method, the method provided by the invention has the advantages that the memory is reduced by 90% when the 136GB data is processed, the analysis time is shortened from more than 3 minutes to less than 10 seconds, and huge potential and practical application value are shown in large-scale sequencing application.

Description

Technical Field

[0001] The present invention relates to bioinformatics, and in particular to a gene sequence comparison method, software and application based on a feature vector hash table. Background Art

[0002] With the continuous development of bioinformatics, pathogen data has been increasingly used in epidemic monitoring, tracing, and evolutionary research. In-depth mining of these data has also provided important support for clinical diagnosis. As one of the top ten causes of death worldwide, infectious diseases have made pathogen detection increasingly important in the fields of medicine and scientific research. However, traditional morphological detection and laboratory culture methods are inefficient, time-consuming, and highly dependent on manual judgment, which limits their widespread application in clinical practice. In contrast, second-generation metagenomic sequencing can identify pathogens more comprehensively and accurately by extracting genetic information from samples and comparing them with databases, significantly improving detection efficiency and accuracy.

[0003] Currently, pathogen detection using second-generation metagenomic data includes steps such as sample collection, nucleic acid extraction, library construction and sequencing, and data analysis. The FASTQ files generated by sequencing must be used for pathogen species alignment or annotation classification using software such as BWA, Centrifuge, MetaPhlAn, and Kraken2. However, these methods have the following limitations when dealing with large-scale data:

[0004] High computational cost: For example, BWA relies on a global comparison algorithm, which has low analysis efficiency in large-scale databases and may take several hours to process samples, making it difficult to meet clinical real-time detection needs.

[0005] Insufficient short sequence matching capability: For example, Kraken2 uses K-mer and lowest common ancestor (LCA) alignment strategies, but its recognition capability for short sequences (e.g., <100bp) is limited, with an accuracy rate of only about 54%, making it difficult to effectively detect low-abundance pathogens or mixed infection samples.

[0006] Limited feature expression capabilities: Existing K-mer-based methods mainly rely on hash indexes, but do not fully utilize the contextual information and deep features of the sequence, affecting the accuracy and generalization ability of classification.

[0007] In recent years, the rapid development of deep learning technology has provided new insights for metagenomic data analysis, demonstrating significant potential in feature extraction and generalization. However, no publicly available technology utilizes K-mer feature vectors combined with deep learning models for efficient, real-time pathogen identification and detection to meet the needs of clinical diagnosis and epidemic monitoring. Summary of the Invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a target pathogen detection method based on K-mer feature vectors and hash table indexes combined with deep learning technology, which can significantly improve the comparison efficiency and detection accuracy of metagenomic sequencing data.

[0009] Technical solution: To solve the above technical problems, the present invention provides a target pathogen detection method based on a feature vector hash table, comprising:

[0010] Constructing a reference K-mer feature vector hash table, wherein the key of the reference K-mer feature vector hash table is a reference hash value obtained by hashing the reference K-mer feature vector, and the value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer;

[0011] Get the target K-mer feature vector and calculate the target hash value corresponding to the target K-mer;

[0012] Query the target hash value in the reference K-mer feature vector hash table to obtain a matching K-mer;

[0013] Calculate the similarity between the target sequence and each corresponding matching K-mer, and output the species classification result of the target pathogen.

[0014] The target sequence described in this invention refers to a DNA or RNA sequence of 50-10,000 bp. Unlike existing techniques, this method can still achieve high identification efficiency and accuracy even for short target sequences of 50-300 bp. The reference sequence described in this invention is a dataset containing the complete genome sequence of pathogens, including but not limited to bacteria, fungi, and viruses; this dataset can be downloaded and accessed through NCBI.

[0015] As a further optimization of the present invention, the present invention uses a deep learning neural network model to extract and classify K-mer features to make up for the limitations of traditional methods in identifying short sequences and low-abundance species. Specifically, by traversing the reference sequence or target sequence, the split K-mer is input into the deep learning neural network model to obtain the reference K-mer feature vector and the target K-mer feature vector. Therefore, the reference K-mer feature vector is a feature vector obtained after the equal-length split K-mer fragment data set is input into the model, and the target K-mer feature vector is also the same. Assuming that the length of the reference sequence or target sequence is L, the K-mer split length is K, and the step size is λ, ⌈(LK) / λ⌉+1 split K-mers can be traversed to obtain for input into the deep learning neural network model.

[0016] Furthermore, the segmentation length K of the K-mer segmentation is 15-30 bp, preferably 15-25 bp.

[0017] Furthermore, the K-mer segmentation step length of the reference sequence is 15-30 bp, which is conducive to building a compact hash index and saving computing resources. As a further optimization of the present invention, the K-mer segmentation step length of the reference sequence is 15-20 bp. As a preferred embodiment of the present invention, the K-mer segmentation step length of the reference sequence is 15 bp.

[0018] Furthermore, the K-mer segmentation step length of the target sequence is 1-5 bp, which is beneficial for improving the accuracy of sequence matching and ensuring the sensitivity of pathogen detection. As a further optimization of the present invention, the K-mer segmentation step length of the target sequence is 1-3 bp. As a preferred embodiment of the present invention, the K-mer segmentation step length of the target sequence is 1 bp.

[0019] The selection of the K-mer segmentation step size affects the system performance. A smaller step size can improve the ability to accurately locate the target sequence, but increases the amount of calculation and storage requirements. A larger step size can reduce the computational cost and memory usage, but may affect the recognition accuracy.

[0020] Furthermore, the position information of the reference K-mer in the reference sequence includes the reference sequence file ID, the reference sequence index number, and the starting position of the K-mer in the reference sequence. Specifically, the position information of the reference K-mer in the reference sequence consists of fileid, sequence_index, and offset, which are used to accurately locate the position of a K-mer segment in the reference sequence dataset.

[0021] More specifically, the fileid is the file identification ID, which is obtained by parsing the Fasta file path or database index. It can be stored as a string or an unsigned integer (such as a hash code) to uniquely identify the reference file to which the split K-mer belongs. The sequence_index is the reference sequence index number, that is, the sequential number of the sequence in the Fasta file, indicating the storage position of the sequence in the file. This value can be obtained by a loop counting method when reading the Fasta file. Each sequence is numbered incrementally according to the order of its appearance for index query. The offset represents the starting position of the split K-mer in the sequence, which is usually calculated by traversing the variable i, that is, offset=i, where i is the index position of the current K-mer when it is split in the sequence. This value is used to quickly locate the specific position of the K-mer in the original sequence and support efficient retrieval. These three sets of data are all unsigned integer types, which can ensure the compactness of data storage and the efficiency of calculation, and are particularly suitable for indexing and alignment of large-scale gene sequences. Through the combination of fileid, sequence_index and offset, the position of a K-mer in the reference database can be uniquely determined, providing a reliable indexing mechanism for pathogen detection, alignment acceleration and traceability analysis.

[0022] Furthermore, the taxonomic information corresponding to the reference K-mer at least includes a genus-level TaxID.

[0023] As a preferred embodiment, the taxonomic information corresponding to the reference K-mer is species TaxID, and the classification file is read through the species TaxID to obtain taxonomic information containing at least file name, species name, species TaxID, genus name, and genus-level TaxID.

[0024] Using the species TaxID as a classification index ensures the accuracy of K-mer attribution to specific pathogens and provides an efficient classification query mechanism for comparison tasks. Specifically, the classification file name is used to identify the fasta file to which the K-mer belongs. The species name describes the specific species to which the K-mer belongs, such as Escherichia coli. The species TaxID is its unique classification number in the NCBI or GTDB (Genome Taxonomy Database) database, such as 562. The genus name indicates the genus-level classification to which the K-mer belongs, such as Escherichia. The genus-level TaxID is the unique identification number of the genus-level classification, such as 561.

[0025] As another possible embodiment of the present invention, the taxonomic information of the reference K-mer in the reference sequence does not need to read the classification file, and directly includes the taxonomic information of the genus-level TaxID.

[0026] Furthermore, the detection method described herein also includes the step of inputting the target sequence into a deep learning neural network model to obtain a genus-level prediction classification file for the target sequence. Specifically, the genus-level prediction classification file includes at least one genus-level TaxID and its probability distribution information. As a preferred embodiment of the present invention, the genus-level prediction classification file includes at least three genus-level TaxIDs with the highest distribution probabilities (arranged in descending order of probability) and their respective confidence levels. This method significantly saves memory and shortens computation time compared to loading a hash table of all genus-level classifications.

[0027] Furthermore, the deep learning neural network model described in the present invention includes at least one convolution operation module for capturing local features and at least one fusion module for fusing local features into global features. The convolution operation model is selected from a convolutional neural network (CNN) model based on any one or more combinations of multi-scale convolution kernels, multi-channel convolution, dynamic convolution, or residual convolution, which facilitates efficient extraction of local features of K-mers and generation of feature vectors. The fusion module can fuse local features into global features based on feature concatenation, feature pooling, and an attention mechanism to characterize the characteristics of the entire sequence.

[0028] As a preferred embodiment of the present invention, the deep learning neural network model is a pre-trained and fine-tuned model based on Hyena. This model utilizes long convolution techniques and bitwise gating. The former uses larger convolution kernels to adapt to 15-30 bp K-mers, while the latter uses an attention mechanism to dynamically weight feature vectors to improve global feature quality. While optimizing computational resources and reducing time complexity, this model extends the pre-trained segmentation length to 1 million tokens. While maintaining prediction accuracy, it effectively captures long-range dependencies and contextual information in gene sequences, thereby deeply exploring the semantic features of gene sequences. After fine-tuning on a genus-level sequence classification task, the model achieved 95% accuracy (ranking in the top three in predictions), effectively generating embeddings that characterize different pathogen sequences and accurately predicting genus-level information for target sequences.

[0029] Furthermore, the algorithm for converting the target K-mer feature vector or the reference K-mer feature vector into a hash value is independently selected from any one of xxHash, MurmurHash, SHA-256, CityHash, SpookyHash, FarmHash, SipHash, and FNV-1a.

[0030] As a further optimization of the present invention, the xxHash and MurmurHash algorithms are preferred, which are more suitable for the K-mer feature vectors obtained by the present invention. Their main advantages are that these two algorithms have fast calculation speed and low conflict rate, and can efficiently generate hash values in large-scale K-mer feature vector data processing, thereby improving the real-time performance and computational efficiency of pathogen detection.

[0031] Furthermore, when calculating the similarity between the target sequence and each corresponding K-mer, an extension window is formed by bidirectionally extending the matching K-mer. If the extension is insufficient, it is extended forward and backward, with a total extension length of 2 × K. Ultimately, the extension range is confined to the valid range of the sequence to ensure its legitimacy. Based on the adjusted start and end positions, the extended K-mer is extracted from the original sequence and used for sequence alignment. This method enhances the robustness of the alignment, captures genomic variants (such as insertions and deletions), and reduces the probability of alignment failure, thereby improving detection sensitivity and accuracy.

[0032] Furthermore, the similarity calculation method described in the present invention includes but is not limited to any one or more combinations of the Smith-Waterman algorithm, the Hamming distance algorithm, the Needleman-Wunsch algorithm, the Levenshtein distance algorithm (also known as the edit distance algorithm), and the Jaccard similarity algorithm.

[0033] As a preferred approach, the Smith-Waterman algorithm achieves the highest matching score between two sequences and is particularly effective at analyzing sequence similarity between homologous genes. Its core steps include: initialization, which constructs a two-dimensional score matrix with rows and columns corresponding to the base positions of the two sequences, and all values initialized to zero; recursive filling, which uses dynamic programming techniques to gradually fill the matrix and calculate the score at each step; and backtracking, which starts from the highest-scoring point in the matrix and backtracks along the path to ultimately determine the optimal local alignment. Ultimately, it returns the start and end positions of the aligned segment, along with the corresponding score. Through these steps, the Smith-Waterman algorithm not only reveals subtle differences between sequences but also provides the specific location and alignment score of matching regions, providing powerful support for in-depth bioinformatics research.

[0034] The Hamming distance algorithm is a preferred method for efficiently evaluating the number of differences between two sequences of the same length. This algorithm can be quickly calculated using a linear scan of the sequence, achieving a time complexity of O(N). The Hamming distance algorithm takes two sequences of equal length (such as DNA or RNA) as input and compares each base (or character) at each position in the sequence. If a difference is found, a counter is incremented by 1, and the total number of positions that differ between the two sequences is output as the Hamming distance.

[0035] The present invention also provides a target pathogen detection software based on a feature vector hash table, which can be used in high-throughput pathogen genome sequencing and / or species identification, specifically including:

[0036] A hash table generation module is used to obtain a reference K-mer feature vector hash table, wherein the key of the reference K-mer feature vector hash table is a reference hash value obtained by converting the reference K-mer feature vector, and the value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer;

[0037] Target sequence hash value generation module, used to calculate the target hash value corresponding to the target K-mer;

[0038] A retrieval module, configured to search for a target hash value in the reference K-mer feature vector hash table to obtain a matching K-mer;

[0039] The calculation module is used to calculate the similarity between the target sequence and each matching K-mer;

[0040] The output module is used to output the species classification results of the target pathogen.

[0041] The present invention also provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the target pathogen detection method of the present invention is implemented.

[0042] The target pathogen species classification results described in this invention include the most likely genus-level classification information calculated based on the K-mer features of the target sequence. This information can also be further included, including alignment consistency scores and the start and end positions of the target sequence on the reference sequence, to enhance the accuracy and interpretability of the detection results.

[0043] This invention achieves efficient and accurate pathogen detection and classification by combining deep learning models with k-mer sequence alignment technology. During the pre-training phase, the model uses the Hyena algorithm to capture global sequence features through an efficient long-distance dependency modeling mechanism. For downstream tasks, fine-tuning training is performed by combining the Transformer architecture with a convolutional neural network (CNN). Multi-scale convolutional modules and attention mechanism modules are introduced to achieve deep extraction and fusion of k-mer-level sequence features, enhancing the model's ability to express complex biological sequence features and the accuracy of downstream prediction tasks. Pre-trained on a large-scale genomic database, the model can capture evolutionary and taxonomic differences in k-mer segments, thereby enhancing detection sensitivity and specificity.

[0044] The K-mer feature vectors output by the model are hashed to construct an index table, supporting efficient alignment and fast retrieval. High-performance hashing algorithms (such as xxHash or MurmurHash) are used to map high-dimensional vectors into fixed-length hash values. These values, along with the K-mer's reference positional information and taxonomic information (such as species or genus-level TaxID), form hash table entries. This index structure eliminates the need to load the entire reference database, significantly reducing memory consumption and improving query efficiency.

[0045] In addition, the genus-level classification results can be used to filter the target reference hash table, loading only the index information of the relevant genus level, further narrowing the comparison range and improving processing speed and accuracy. Compared with traditional methods, the present invention shows significant advantages in processing large-scale pathogen sequencing data. For example, when processing 136GB of data, memory usage is reduced by about 90%, and the analysis time is shortened from 3 minutes to less than 10 seconds. Therefore, this method is particularly suitable for high-throughput sequencing, low-abundance pathogen identification and mixed infection sample analysis, and has broad application prospects.

[0046] Compared with the existing technology, the present invention has the following significant innovations in K-mer feature representation and index structure design: (1) A deep learning model is used to extract K-mer semantic features, replacing the traditional K-mer representation; (2) The index table not only records position information, but also integrates taxonomic information to enhance classification performance and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The target pathogen detection process of the present invention;

[0048] Figure 2 It is a structural block diagram of the gene sequence comparison device of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a target pathogen detection method based on a feature vector hash table, which specifically includes the following steps:

[0052] S1: Construct a reference K-mer feature vector hash table. The key of the reference K-mer feature vector hash table is the reference hash value obtained by hashing the reference K-mer feature vector. The value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer.

[0053] S2: Obtain the target K-mer feature vector and calculate the target hash value corresponding to the target K-mer;

[0054] S3: Look up the target hash value in the reference K-mer feature vector hash table to obtain the matching K-mer;

[0055] S4: Calculate the similarity between the target sequence and each matching K-mer;

[0056] S5: Output the species classification results of the target pathogen.

[0057] As a preferred embodiment, the reference sequence or the target sequence is traversed, and the segmented K-mer is input into a deep learning neural network model to obtain a reference K-mer feature vector and a target K-mer feature vector.

[0058] As a preferred embodiment, the K-mer segmentation length is 15-30 bp. The K-mer segmentation step length of the target sequence is 1-5 bp; the K-mer segmentation step length of the reference sequence is 15-30 bp.

[0059] In a preferred embodiment, the position information of the reference K-mer in the reference sequence includes the reference sequence file ID, the reference sequence index number, and the starting position of the K-mer in the reference sequence. The taxonomic information corresponding to the reference K-mer is the species TaxID, and the taxonomic information is read from the classification file using the species TaxID to obtain taxonomic information including at least the file name, species name, species TaxID, genus name, and genus-level TaxID.

[0060] As a preferred embodiment, the target sequence is input into a deep learning neural network model to obtain a target sequence genus-level prediction classification file, and the genus-level prediction classification file includes at least one genus-level TaxID and its probability distribution information.

[0061] As a preferred embodiment, the deep learning neural network model includes at least one convolution operation model for capturing local features and at least one fusion model for fusing local features into global features. Specifically, the deep learning neural network model includes a pre-trained fine-tuning model based on Hyena.

[0062] As a preferred embodiment, the algorithm for converting the target K-mer feature vector or the reference K-mer feature vector into a hash value is independently selected from any one of xxHash, MurmurHash, SHA-256, CityHash, SpookyHash, FarmHash, SipHash, and FNV-1a.

[0063] As a preferred embodiment, when calculating the similarity between a target sequence and each corresponding matching K-mer, the target K-mer is first expanded before the calculation is performed. Optionally, the target K-mer can be expanded in any of the following ways: context expansion, multi-scale expansion, and frequency expansion. The expansion range is initially determined by expanding the length of the sequence forward and backward by 2×K, respectively. If the expansion length of the starting position is less than 2×K, the expansion is further expanded backward to the ending position; if the expansion length of the ending position is less than 2×K, the expansion is further expanded forward to the starting position.

[0064] As a preferred implementation, the similarity calculation method includes: any one or more combinations of the Smith-Waterman algorithm, the Hamming distance algorithm, the Needleman-Wunsch algorithm, the Levenshtein distance algorithm, and the Jaccard similarity algorithm.

[0065] Example 2

[0066] like Figure 2 As shown, this embodiment provides a target pathogen detection software based on a feature vector hash table, the software including:

[0067] A hash table generation module is used to obtain a reference K-mer feature vector hash table. The key of the reference K-mer feature vector hash table is the reference hash value obtained by converting the reference K-mer feature vector. The value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer.

[0068] Target sequence hash value generation module, used to calculate the target hash value corresponding to the target K-mer;

[0069] The retrieval module is used to find the target hash value in the reference K-mer feature vector hash table to obtain the matching K-mer;

[0070] The calculation module is used to calculate the similarity between the target sequence and each matching K-mer;

[0071] The output module is used to output the species classification results of the target pathogen.

[0072] In addition, this embodiment further provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the target pathogen detection method of Example 1 is implemented.

[0073] Example 3

[0074] Step 1: Build a sequence dataset

[0075] In this example, the seqio (Version 0.0.1) library was used for sequence reading, the reference sequence was read from the FASTA file, and the target sequence was read from the FASTQ file.

[0076] Reference sequence source: The reference sequences for this study are from the NCBI database (as of December 2024). The entire All Bacteria Genomes reference dataset was used to construct this dataset, which contains complete genome sequences of nearly 2,000 genera and over 14,200 different bacteria. This dataset is continuously expanded as the NCBI database is updated. The dataset can be downloaded from the NCBI FTP server or as an NCBI Assembly Summary file, and can be downloaded and processed in batches using the wget or ncbi-genome-download tools. This large dataset contains reliable genomic taxonomic information, ensuring accurate classification and validation of the test set.

[0077] Target sequence generation: To simulate a real-world sequencing environment, we used ART (Version 2.5.8) software to generate 50bp Illumina sequencing reads and randomly selected 100 sequence reads as target sequences. These sequences were used for classification testing of the deep learning model to verify the model's accuracy in detecting short pathogen sequences.

[0078] Step 2: Output the predicted classification file

[0079] For the target sequence, we run the trained deep learning model (based on the Hyena algorithm), input the fastq file of the target sequence, and output the Top 3 genus-level prediction classification file for each target sequence.

[0080] This embodiment classifies the target sequence based on a deep learning model and combines it with the NCBI reference sequence classification information to generate a predicted classification file for the target sequence and a genus-level classification file for the reference sequence to achieve efficient pathogen detection.

[0081] (1) Obtain target sequence classification file

[0082] The target sequence is classified using a deep learning model based on the Hyena algorithm. This model can extract single-base features of the fragment and predict the genus-level classification information to which the target sequence belongs. The specific process is as follows:

[0083] Input data: target_sequences.fastq, containing 100 50bp DNA sequences (FASTQ format).

[0084] Model architecture: A model based on the Hyena algorithm (hyena-dna Version 1.0.0), which inputs a DNA sequence and outputs a genus-level classification probability distribution.

[0085] Output data: top3_genera_predictions.csv, CSV format, contains the top 3 genus-level classification information of the target sequence. The fields are described as follows:

[0086] SequenceID: unique identifier of the target sequence;

[0087] PredictedGenus1, Probability1: the genus-level classification with the highest probability and its confidence level;

[0088] PredictedGenus2, Probability2: the second most probable genus classification and its confidence level;

[0089] PredictedGenus3, Probability3: The third most probable genus-level classification and its confidence.

[0090] [Target sequence FASTQ] → Deep learning model → Top3 genus prediction file (CSV).

[0091] (2) Obtain reference sequence classification file

[0092] The NCBI reference sequences were used to construct a genus-level taxonomic index to optimize the efficiency of hash table queries. The specific process is as follows:

[0093] Data source: The reference sequence is based on the NCBI Assembly Summary file, and the species TaxID corresponding to the genome is extracted.

[0094] TaxID conversion: The species TaxID was converted to the genus TaxID using taxonkit software (Version 0.15.0) to generate reference_genera_mapping.txt.

[0095] Take Escherichia coli K-12 as an example:

[0096] Species-level TaxID: 562 → corresponds to the species Escherichia coli.;

[0097] Genus TaxID: 561 → corresponds to the genus Escherichia.

[0098] [Reference sequence FASTA] → TaxID extraction → Taxonkit conversion → Genus-level classification file (txt).

[0099] The file for reading the genus-level classification information of the reference sequence includes the file name, species name, species TaxID, genus name, and genus-level TaxID of the reference sequence. Specifically:

[0100] File name: file identification ID, the file name of the reference sequence;

[0101] Species name: species name of the reference sequence;

[0102] Species TaxID: the species-level TaxID corresponding to the reference sequence;

[0103] Genus name: the genus name of the reference sequence;

[0104] Attribute TaxID: The attribute-level TaxID corresponding to the reference sequence.

[0105] Step 3: Get the reference K-mer feature vector hash table

[0106] (1) Traverse the reference sequence and generate K-mer fragments

[0107] Traverse all K-mer segments in the reference sequence, extract K-mers, and store their position information. This example uses a K-mer segmentation length of K = 20 and a step size of 10. Users can adjust these parameters based on actual needs. Assuming the length of the reference sequence or target sequence is L, the K-mer segmentation length is K, and the step size is λ, ⌈(LK) / λ⌉+1 K-mer segments can be traversed. The specific operation is as follows:

[0108] Starting from the beginning of the reference sequence, move backwards by 10 steps each time, extract a K-mer segment, and if the last segment is less than K, fill it forward;

[0109] For example, if the input sequence is "TTCGGAAATCCGTAAATTACATGCGT", its length L=26, K=20, then the traversed K-mers are as follows:

[0110] The first K-mer: TTCGGAAATCCGTAAATTAC;

[0111] The second K-mer: AATCCGTAAATTACATGCGT.

[0112] This process ensures that all reference K-mers are indexed, forming a K-mer fragment dataset.

[0113] (2) Extract K-mer feature vectors using a deep learning neural network model

[0114] A deep learning neural network model is used to embed each K-mer into a feature vector, thereby capturing the characteristic information of the sequence. This embodiment selects a pre-trained fine-tuning model based on Hyena. The model is based on long convolution technology and bit-by-bit gating technology. The former is based on a larger convolution kernel to adapt to 15-30 bp K-mers, and the latter is to dynamically weight the feature vectors in combination with the attention mechanism to enhance the global feature representation capability of K-mer.

[0115] After each K-mer is processed by the model, a 256-dimensional feature vector is generated.

[0116] (3) K-mer feature vector is converted into hash value and the location information is recorded

[0117] To improve query efficiency, this paper uses the xxHash hash function library (Version 0.8.3) to convert high-dimensional feature vectors into 32-bit hash values. Assuming the input DNA sequence is "TTCGGAAATCCGTAAATTACATGCGT", the sequence ID is "seq1", and K is set to 20, then:

[0118] The K-mer is first embedded in the deep learning model to obtain a feature vector: [10, 20, 30, ..., 2560] (256 dimensions). After that, it is hashed using xxHash32 to generate a hash value of "12345". This hash value will be associated with the K-mer's position in the reference sequence:

[0119]

[0120] The key values are represented as follows:

[0121] Key: hash value of the K-mer feature vector (32-bit hash value generated by xxHash);

[0122] Value: Contains the position information of the K-mer in the reference sequence:

[0123] FileID (file identification ID): generated by the file path of the reference sequence;

[0124] SequenceIndex (sequence index number): the sequence number of the sequence in the file;

[0125] Offset (the starting position of the K-mer in the sequence);

[0126] Species TaxID (for subsequent classification).

[0127] This design avoids the extra overhead of storing high-dimensional vectors and optimizes the retrieval efficiency of hash indexes.

[0128] (4) Read the genus-level classification information of the reference sequence

[0129] Read the genus-level classification file of the reference sequence, including the file name (filename), species scientific name (scientific_name), species TaxID (s_tax_id), genus name (genus), genus-level TaxID (g_tax_id), and other information of the reference sequence.

[0130]

[0131] (5) Constructing K-mer feature vector hash table

[0132] The hash value of each reference K-mer feature vector is used as the key, and the corresponding value consists of the position information of the reference K-mer in the reference sequence and the species TaxID corresponding to the reference K-mer. This embodiment uses the following data storage format:

[0133]

[0134] Ultimately, the K-mer feature vector hash table can be used for rapid alignment of target sequences, enabling efficient pathogen detection. By storing only K-mer positional and classification information, rather than loading the entire reference genome database, this method optimizes memory usage and query speed.

[0135] Step 4: Get the target hash value of the target K-mer feature vector

[0136] The target sequence is segmented into k-mers according to the method in step 3, and feature vectors are embedded for each k-mer using the same deep learning neural network model (based on the Hyena algorithm). The target k-mer feature vectors are then converted to target hash values using the same xxHash hash function as in step 3 for subsequent hash table lookups. This method ensures that target and reference k-mers are processed consistently, thus ensuring alignment consistency and accuracy.

[0137] Step 5: Hash value lookup

[0138] Load the reference K-mer feature vector hash table and compare the target hash value of the target K-mer feature vector with the reference hash value in the hash table to find / locate whether there is a match.

[0139] If there is a match: proceed to step 6 for K-mer expansion to further analyze the context information of the matching K-mer.

[0140] If there is no match: it is considered that the target sequence has no significant alignment with the known sequences in the database, and the following message is printed: "No significant alignment for sequence: sequence name"

[0141] Because the hash table in this embodiment uses a key-value uniqueness design, the time complexity of hash value lookups is O(1). This means that query time is virtually unaffected by the size of the data, enabling efficient computational performance on large datasets. This method avoids the overhead of global alignment computations, improves alignment speed, and ensures that the target sequence is quickly matched to the most likely pathogen category.

[0142] Step 6: Expand K-mer

[0143] After matching the K-mer, it is first context-expanded to capture its surrounding sequence information before subsequent alignment analysis, thereby enhancing the robustness of the sequence alignment.

[0144] In this embodiment, the design of the extended K-mer is as follows:

[0145] Extend 2×K forward from the starting position; extend 2×K backward from the ending position; if the extended length is less than 2×K, adjust the starting position or the ending position to make up for the extended range.

[0146] (1) Input the following parameters:

[0147] original_sequence: original DNA sequence string;

[0148] kmer_start: the starting position of K-mer in the original sequence;

[0149] kmer_end: the end position of K-mer in the original sequence;

[0150] target_seq: target sequence;

[0151] k: length of K-mer;

[0152] kmer_positions_ref: the position of K-mer in the reference sequence;

[0153] kmer_positions_query: query the positions of K-mers in the sequence.

[0154] (2) Determine the initial extension range based on kmer_start and kmer_end. The initial extension length is 2×K:

[0155] Extend forward: extended_start = kmer_start - 2×k;

[0156] Backward extension: extended_end = kmer_end + 2×k.

[0157] (3) Boundary adjustment rules:

[0158] If the extension of the starting position is less than 2×K, it is supplemented by extending extended_end backward until the requirement is met;

[0159] If the extension of the end position is less than 2×K, it is supplemented by extending extended_start forward until the requirement is met;

[0160] Make sure the extension does not exceed the original sequence boundaries.

[0161] (4) Output

[0162] The expanded K-mer segment contains the context information of 2×K length before and after it;

[0163] The expanded target sequence is used for further alignment and classification.

[0164] Step 7: Similarity Judgment

[0165] After K-mer alignment, the Smith-Waterman local alignment algorithm is used to calculate the similarity between the target sequence and the expanded reference K-mer. This method uses the Smith-Waterman algorithm (aligner library, Version 1.2.6) for local alignment. This algorithm is suitable for sequences of unequal lengths and can be used in bioinformatics to find the best local matching region.

[0166] (1) Input

[0167] masked_extended_kmer: extended K-mer fragment (masked);

[0168] masked_target_seq: target sequence (after masking).

[0169] (2) Calculation

[0170] The Smith-Waterman algorithm uses dynamic programming to calculate the local alignment score between the target sequence and the reference sequence. The matching rules are as follows:

[0171] Match score: +2

[0172] Mismatch penalty: -2;

[0173] Gap Penalty: -3;

[0174] Gap Extension Penalty: -1;

[0175] The final score is in the range of [0, 2 × length(target_seq)], and a larger value indicates a higher similarity between the target sequence and the reference sequence.

[0176] (3) Output

[0177] Similarity score: used to measure the similarity between the target sequence and the extended K-mer;

[0178] Target pathogen detection results: Possible target pathogen species classification results are screened out based on the comparison scores.

[0179] The output results of this example are shown in the following table, including the target sequence ID, consistency score, species taxonomy ID (TaxID), the starting position (Start) and ending position (End) of the target sequence on the reference sequence.

[0180]

[0181] Example 4

[0182] 1. Purpose

[0183] In order to verify the advantages of the method of the present invention in terms of memory usage, comparison speed, and detection accuracy in pathogen detection, the following experiment was designed and compared with two commonly used traditional methods (BWA2 and Kraken2).

[0184] 2. Experimental Methods

[0185] Use the 136GB All bacteria genomes reference sequence database (as of December 2024), which contains thousands of complete bacterial genome sequences, to ensure experimental coverage of a wide range of pathogens.

[0186] This experiment compares the performance of three methods under the same computing environment:

[0187] The detection method described in this invention (experimental group, see Example 3) employs a K-mer feature vector hash table approach, optimizing the alignment process through feature vector hash indexing to reduce memory usage and accelerate alignment. Only the genus-level reference hash table to which the target sequence belongs is loaded, avoiding the need to load the entire database and improving computational efficiency.

[0188] Traditional alignment method BWA2 (Comparison Example 1): Traditional genome alignment tools use an indexing algorithm based on the Burrows-Wheeler transform (BWT). This requires indexing before whole-genome alignment, resulting in high computational overhead. Specifically, bwa index is used to pre-build an index of the entire 136 GB reference genome. After the target sequence is input, bwa mem is executed to perform the alignment and calculate the match score.

[0189] Traditional classification method Kraken2 (Comparison Example 2): uses the lowest common ancestor (LCA) method to perform rapid species classification through K-mer statistical analysis. It requires pre-building a database and loading the entire database into memory during comparison.

[0190] 3. Experimental Procedure

[0191] (1) Data preprocessing

[0192] Experimental group: Please refer to the contents described in Example 3.

[0193] The reference sequence is preprocessed and a genus-level hash table index is generated based on the K-mer feature vector. After the target sequence is input, only the hash table of the corresponding genus-level TaxID is loaded to reduce computing resource requirements.

[0194] Comparative Example 1: Use bwa index to pre-build an index for the entire 136 GB reference genome. After the target sequence is input, bwa mem is executed for alignment and the matching score is calculated.

[0195] Comparative Example 2: Using the kraken2-build pre-built database and loading the complete 136 GB reference sequence, after inputting the target sequence, kraken2 was used to perform K-mer statistical classification.

[0196] (2) Database loading and comparison

[0197] Experimental group: Please refer to the contents described in Example 3.

[0198] Comparative Example 1: First load the entire reference genome index. When aligning the target sequence, it is necessary to traverse the entire database for global alignment, which is computationally intensive.

[0199] Comparative Example 2: The target sequence needs to be aligned with all K-mers in the database, and the classification information is calculated using the LCA algorithm.

[0200] (3) Performance evaluation

[0201] Record memory usage, comparison time, and classification accuracy (based on species / genus matching criteria).

[0202] All methods were run using the same computing resources (128-core CPU, 512GB RAM) and parallel experiments were performed to ensure comparability.

[0203] (4) Experimental results

[0204] A. Memory consumption

[0205] Comparative Example 1: When loading the entire 136 GB reference genome, the memory usage was 321.2 GB, and the index construction time exceeded two weeks.

[0206] Comparative Example 2: When loading the database, the memory occupied 72 GB.

[0207] Experimental group: Through the optimization of the attribute-level hash table, only 0.9 GB of memory is required, which is nearly 80 times lower than Kraken2 and more than 350 times lower than BWA2.

[0208] B. Calculation speed

[0209] Comparative Example 1: It takes 3 minutes and 14 seconds to align 100 50bp short sequences.

[0210] Comparative Example 2: It takes 3 minutes to align 100 50bp short sequences.

[0211] Experimental group: Aligning 100 50bp short sequences took only 6 seconds, 30 times faster than BWA2 and 30 times faster than Kraken2.

[0212] C. Classification Accuracy

[0213] BWA2: 92% accuracy (some short sequences could not be aligned to the complete genome).

[0214] Kraken2: The accuracy is 88%, of which 34% of sequences can only be identified to the genus or family level, and there is a high misclassification rate at the species level.

[0215] Experimental group: The accuracy reached 95%, and it could provide classification results accurate to the species level.

[0216] (5) Results analysis

[0217]

[0218] First, the hash table provided by this invention optimizes indexing, significantly reducing memory usage. This invention employs a tax ID aggregation strategy at the attribute level, reducing memory requirements by 80 times compared to Kraken2 and 350 times compared to BWA2. By loading only the reference hash table for the target attribute level, it avoids large-scale database loading and improves computational efficiency.

[0219] Secondly, the present invention optimizes computational complexity and increases alignment speed. BWA2 uses whole-genome alignment, which has high computational complexity (O(N) level). Kraken2 uses the lowest common ancestor (LCA) strategy, which requires traversing the entire database for statistical analysis. The present invention uses O(1) hash queries, which is 30 times faster than Kraken2 and 30 times faster than BWA2.

[0220] Thirdly, classification accuracy has been improved. Traditional Kraken2 achieved only 34% species-level classification accuracy, with some sequences misclassified as genus or family. This new approach uses deep learning K-mer feature vector matching combined with Smith-Waterman local alignment to increase species-level classification accuracy to 95%.

[0221] In summary, the method of the present invention shows great application value in high-throughput pathogen detection applications. It can significantly optimize comparison efficiency, improve detection accuracy, and reduce computing resource consumption, making it suitable for clinical diagnosis and large-scale epidemic monitoring.

[0222] Based on the best matching K-mer, the present invention can accurately identify the pathogen type corresponding to the target sequence and output the final test result, including the matching species name and TaxID. In addition, by comparing the annotation information of the target sequence with the test results, the detection accuracy of the present invention can be calculated. Experimental results show that the present invention is superior to traditional methods in database loading, comparison speed and detection accuracy. In terms of database loading, when BWA2 processes the 136GB Allbacteria genomes reference database, the memory usage is approximately 321.2GB, and the index construction time exceeds two weeks, while Kraken2 requires 72GB of memory to complete database loading and comparison. The present invention optimizes the storage structure through a hash table at the sub-level, so that the memory consumption during database loading is reduced to 0.9GB, which is 80 times lower than Kraken2 and 350 times lower than BWA2, greatly reducing the computing resource requirements and improving the lightweightness of the system operation.

[0223] In terms of alignment speed, the method of the present invention can complete the detection of 100 50bp short sequences within 6 seconds, while Kraken2 takes 3 minutes and the BWA2 method takes more than 3 minutes and 14 seconds, and has higher computing resource requirements. After adopting the hash index method based on K-mer feature vectors, the present invention can achieve O(1) complexity alignment query, avoiding the high computational overhead of BWA2's reliance on whole genome alignment, and also reducing the large-scale search space when Kraken2 constructs the LCA (lowest common ancestor) index, so that the alignment efficiency is significantly improved. In addition, in terms of detection accuracy, the present invention reaches 95%, which is further improved compared to BWA2's 92%, while Kraken2's accuracy is only 88%, and 34% of the sequences can only be identified to the genus or family level, and cannot give an accurate species-level classification. The present invention combines deep learning K-mer feature vector matching and Smith-Waterman local alignment to improve the accuracy of species-level classification, while optimizing the matching ability of variant sequences and reducing the misclassification rate.

[0224] The method of the present invention demonstrates significant advantages in computational efficiency, storage optimization, and detection accuracy. An O(1)-level query method based on K-mer feature vector hash table indexing makes the comparison 30 times faster than BWA2 and Kraken2, and requires only 1 / 80 of the memory compared to Kraken2 and 1 / 350 of the memory compared to BWA2. Furthermore, by adopting a genus-level TaxID aggregation strategy, the storage and computational burden of loading a complete database is avoided. While maintaining high classification accuracy, the pathogen detection process is optimized, improving detection accuracy and computational efficiency.

[0225] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A target pathogen detection method based on a feature vector hash table, characterized by: Constructing a reference K-mer feature vector hash table, wherein the key of the reference K-mer feature vector hash table is a reference hash value obtained by hashing the reference K-mer feature vector, and the value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer; Get the target K-mer feature vector and calculate the target hash value corresponding to the target K-mer; Searching for a target hash value in the reference K-mer feature vector hash table to obtain a matching K-mer; Calculate the similarity between the target sequence and each corresponding matching K-mer, and output the species classification result of the target pathogen.

2. A target pathogen detection method based on feature vector hash table according to claim 1, characterized in that: The reference sequence or the target sequence is traversed, and the segmented K-mer is input into a deep learning neural network model to obtain the reference K-mer feature vector and the target K-mer feature vector.

3. A target pathogen detection method based on a feature vector hash table according to claim 1 or 2, characterized in that: The position information of the reference K-mer in the reference sequence includes the reference sequence file identification ID, the reference sequence index number and the starting position of the K-mer in the reference sequence.

4. A target pathogen detection method based on feature vector hash table according to claim 3, characterized in that: The method further includes: inputting the target sequence into the deep learning neural network model to obtain a target sequence genus-level prediction classification file, wherein the genus-level prediction classification file includes at least one genus-level TaxID and its probability distribution information; The deep learning neural network model includes a pre-trained model based on Hyena, at least one convolution operation module for capturing local features, and at least one fusion module for fusing local features into global features.

5. The target pathogen detection method based on feature vector hash table according to claim 4 is characterized in that: The algorithm for converting the target K-mer feature vector or the reference K-mer feature vector into a hash value is independently selected from any one of xxHash, MurmurHash, SHA-256, CityHash, SpookyHash, FarmHash, SipHash, and FNV-1a.

6. A target pathogen detection method based on a feature vector hash table according to any one of claims 1-2, 4-5, characterized in that: When calculating the similarity between the target sequence and each matching K-mer, an extension window is formed by bidirectionally extending the matching K-mer. If the extension length is insufficient, it is extended forward and backward, and the total extension length is 2×K.

7. The target pathogen detection method based on feature vector hash table according to claim 6 is characterized in that: The similarity calculation method includes: any one or more combinations of the Smith-Waterman algorithm, the Hamming distance algorithm, the Needleman-Wunsch algorithm, the Levenshtein distance algorithm, and the Jaccard similarity algorithm.

8. A target pathogen detection software based on feature vector hash table, characterized in that: The software includes: A hash table generation module is used to obtain a reference K-mer feature vector hash table, wherein the key of the reference K-mer feature vector hash table is a reference hash value obtained by converting the reference K-mer feature vector, and the value of the reference K-mer feature vector hash table includes the position information of the reference K-mer in the reference sequence and the taxonomic information corresponding to the reference K-mer; Target sequence hash value generation module, used to calculate the target hash value corresponding to the target K-mer; A retrieval module, configured to search for a target hash value in the reference K-mer feature vector hash table to obtain a matching K-mer; The calculation module is used to calculate the similarity between the target sequence and each matching K-mer; The output module is used to output the species classification results of the target pathogen.

9. A storage medium, characterized in that: The storage medium stores computer-executable instructions, which, when executed by a processor, implement a target pathogen detection method based on a feature vector hash table according to any one of claims 1 to 7.

10. Use of the software according to claim 8 in high-throughput pathogen genome sequencing and / or species identification.

Images