Dynamic multi-modal biological sequence staged pre-training system and method for precise breeding

Abstract

The invention provides a dynamic multi-modal biological sequence staged pre-training system for precise breeding, which comprises a multi-modal data organization unit used for collecting a single-modal sequence, generating a two-modal pairing sequence, constructing a three-modal interpenetrating sequence and simulating biological information transmission; the unified sequence representation unit is used for performing unified word segmentation on each modal sequence and adding modal marks; the progressive pre-training strategy unit is used for training in three stages and dynamically adjusting the modal mixing proportion in combination with a simulated annealing strategy; the cross-modal autoregression unit realizes prediction conversion between modals; the sequence feature extraction and prediction unit is used for obtaining a quantitative prediction value. The invention further provides a dynamic multi-mode biological sequence staged pre-training method for precise breeding. Therefore, the biological sequence feature prediction precision can be remarkably improved, the training convergence speed is increased, the performance equivalent to that of a large model is achieved, the cost is low, flexible deployment in various environments can be achieved, and efficient, accurate and low-cost technical support is provided for biological sequence analysis.

Description

Technical Field

[0001] This invention relates to the field of intelligent breeding technology, and in particular to a dynamic multimodal biological sequence staged pre-training system and method for precision breeding. Background Technology

[0002] In the interdisciplinary field of bioinformatics and intelligent breeding, the development of biological sequence pre-training models has gradually evolved from single-modal modeling to multi-modal fusion.

[0003] In existing technologies, such as ESM-2, ProGen2 and DNABERT, the focus is usually on modeling single-type biological sequences (such as DNA only or protein only). Since the central dogma of biology states that there is an information flow relationship between DNA, RNA and protein, single-modality modeling cannot capture this essential relationship. Moreover, the model architecture is mostly Transformer encoder, but most of them are not specifically optimized for the characteristics of biological sequences.

[0004] Existing input representation methods employ single-nucleotide segmentation, K-mer segmentation, non-overlapping segmentation, and sequence concatenation. Models like HyenaDNA use single-nucleotide segmentation with a stride of 1, which accurately models individual nucleotide variations but results in excessively long sequences. DNABERT (a deep learning model based on the Transformer architecture) uses overlapping K-mer segmentation (stride = 1), preserving single-nucleotide accuracy but with low training efficiency. Nucleotide Transformer (a large-scale language model designed specifically for nucleic acid sequences (DNA, RNA)) uses non-overlapping K-mer segmentation (stride = K), reducing sequence length but sacrificing single-nucleotide accuracy. Models like LucaOne simply concatenate data from different modalities, lacking effective interaction and fusion mechanisms between modalities. Existing pre-training methods employ traditional masked language modeling (MLM) and multi-task learning. However, traditional masked language modeling (MLM) only predicts on single-modal sequences and cannot learn cross-modal relationships. Multi-task learning performs different biological domain pre-training tasks on different modalities, but lacks interaction between tasks.

[0005] The existing data processing workflow involves collecting single-modality biological sequence data (DNA, RNA, or protein) and applying K-mer or other segmentation strategies to convert the sequences into model input; however, it does not consider the biological relationships between different modalities. Model training and application involve self-supervised pre-training on large-scale but single-modality data; using traditional masked language modeling (MLM) or autoregressive prediction tasks; however, it uses fixed mask ratios and training strategies, lacking optimization for the characteristics of biological sequences; and it adapts to downstream tasks through simple fine-tuning, ignoring the unique requirements of different biological sequence tasks.

[0006] Existing biological sequence pre-training models have the following problems:

[0007] 1. Insufficient modal fusion leads to one-sided information in breeding decisions. Existing models either focus only on a single biological sequence type (such as DNA only) or simply splice together different types of data, failing to fully utilize the intrinsic connections between DNA, RNA, and proteins under the central dogma to comprehensively assess the genetic potential of breeding individuals. In breeding practice, complex traits (such as yield and stress resistance) are the result of multi-gene, multi-level molecular interactions. The lack of effective cross-modal learning mechanisms makes it difficult for models to capture these complex interaction networks, thus limiting the accuracy of predictions for these key breeding traits.

[0008] 2. The challenge of balancing single nucleotide precision and regional pattern recognition in breeding applications. DNA sequences have a dual nature: single nucleotide variations (SNPs) can lead to significant phenotypic changes, which are crucial for improving specific traits; at the same time, gene functional elements (such as promoters and enhancers) depend on the identification of regional patterns, and variations in these elements also affect breeding value. Previous models have struggled to balance these two aspects: for example, models such as HyenaDNA can accurately capture single nucleotide variations but are inefficient at processing long sequences (such as whole genomes), making them unsuitable for large-scale breeding population analysis; while non-overlapping K-mer models are efficient but may lose crucial single nucleotide mutation information, affecting the accuracy of breeding decisions. There is a lack of a unified solution that can simultaneously and efficiently process large-scale genomic data and accurately identify single nucleotide variations and their regional functional backgrounds in breeding applications.

[0009] 3. Constraints of Pre-training Efficiency and Generalization Ability on Breeding. Existing high-performance biological sequence models typically rely on large-scale parameters and computational resources (such as Evo and NT), which are too costly for many breeding institutions and difficult to widely apply. Pre-training strategies are singular and not optimized for complex plant and animal genomes and diverse breeding objectives, resulting in slow model convergence or insufficient generalization ability on specific breeding populations or specific traits. Furthermore, the lack of consideration for pre-training tasks of varying difficulty levels prevents the gradual improvement of the model's understanding of complex genetic mechanisms, making it difficult to adapt to the analytical needs of breeding, ranging from simple traits to highly complex traits.

[0010] 4. Limitations in application restrict breeding practices. Most models perform well in specific modalities or academic tasks, but their performance is limited in real-world breeding scenarios, such as cross-modal and cross-task requirements for simultaneous improvement of multiple traits and assessment of environmental interaction effects; the models are too large to be deployed in resource-constrained breeding farms or local computing environments, hindering the rapid application of advanced technologies; and there is a lack of flexible adaptation mechanisms and efficient transfer learning capabilities for different breeding objectives (such as different species and different trait combinations).

[0011] In summary, the main problems with existing technologies in breeding applications are:

[0012] 1. Insufficient modality fusion makes it impossible to accurately assess breeding value: Existing models either only analyze DNA or simply combine data, making it difficult to capture the comprehensive impact of complex interactions between DNA, RNA, and proteins on key breeding traits (such as yield, resistance, and quality), resulting in an incomplete assessment of the genetic potential of breeding individuals.

[0013] 2. The contradiction between single nucleotide precision and regional pattern modeling constrains breeding decisions: Breeding must focus on both individual gene locus variations affecting traits and understanding the functional patterns of genomic regions. Existing models struggle to achieve both, impacting the accuracy of breeding decisions such as gene editing target selection and superior allele discovery.

[0014] 3. Challenges in pre-training efficiency and model size hinder the widespread adoption of breeding technologies: Existing high-performance models have large parameters and high computational requirements, making them unsuitable for the actual conditions of most breeding units. Their pre-training strategies are also not optimized for the complex genomes and small sample sizes commonly encountered in breeding, resulting in high application costs and low efficiency.

[0015] 4. Limited cross-task generalization ability, making it difficult to meet diverse breeding needs: Breeding work involves complex scenarios such as prediction of multiple traits, adaptation of different species, and environmental interactions. Existing models lack a unified framework, making it difficult to effectively transfer knowledge between different breeding tasks and objectives. Summary of the Invention

[0016] To address the aforementioned shortcomings, the present invention aims to provide a dynamic multimodal biological sequence phased pre-training system and method for precision breeding, offering breeders more accurate genetic assessment tools, more efficient selection strategies, and lower-cost technical solutions, thereby accelerating the breeding of superior varieties and ensuring food security and sustainable agricultural development.

[0017] To achieve the above-mentioned technical effects, this invention provides a dynamic multimodal biological sequence staged pre-training system for precision breeding, comprising:

[0018] The multimodal data organization unit is used to collect single-modal sequences of DNA, RNA and protein from biological databases, generate two-modal paired sequences through homology alignment and transcription-translation relationship analysis, and interleave DNA, RNA and protein sequences according to predetermined rules to form a three-modal interpenetrating sequence that contains three modal information and simulates the complete information transmission path in the body.

[0019] A unified sequence representation unit is used to perform unified single nucleotide or single amino acid segmentation on biological sequences of various modalities and to add corresponding modality embedding tags to each modality.

[0020] A progressive pre-training strategy unit is used to sequentially perform a next-label prediction task for first-order training on the single-modal sequence; perform a cross-modal next-label prediction task for second-order training on a mixture of the single-modal sequence and the two-modal pairing sequence, and dynamically adjust the mixing ratio of the single-modal sequence and the two-modal pairing sequence according to a simulated annealing strategy; and perform a next-label prediction task for third-order training on a mixture of the single-modal sequence, the two-modal pairing sequence, and the three-modal interpenetrating sequence, and dynamically adjust the mixing ratio of each modality according to a simulated annealing strategy.

[0021] Cross-modal autoregressive units are used to achieve context-based prediction transformation and sequence generation between different biological sequence modalities;

[0022] The sequence feature extraction and prediction unit is used to input the biological sequence to be tested into the trained model in order to obtain the quantitative prediction value or subsequent sequence output by the model.

[0023] Optionally, it also includes a breeding-specific fine-tuning unit, which includes:

[0024] The SNP data integration module is used to receive and process SNP (Single Nucleotide Polymorphism) information from breeding populations.

[0025] The model dictionary and embedding layer extension module is used to modify the dictionary and / or embedding layer of the pre-trained model by creating new SNP-specific tokens (markers) to achieve variation-aware embedding or extended conditional embedding based on the input SNP information, so that the model can specifically identify and respond to key SNP sites.

[0026] The supervised fine-tuning module is used to supervise the learning of the extended and modified model for a specific breeding task using individual sequence data containing the SNP information and their corresponding breeding phenotypic data.

[0027] Optionally, the progressive pre-training strategy unit or the breeding-specific fine-tuning unit is specifically used for: dynamically adjusting the data mixing ratio.

[0028] Set the initial mixing ratio range;

[0029] Define one or more evaluation functions based on the model's performance on the validation set;

[0030] During the iterative process of pre-training or fine-tuning, the model is allowed to explore diverse mixing ratios in the early stages of training, accepting temporary performance fluctuations to escape local optima.

[0031] As the training process progresses, the randomness of the blending ratio is gradually reduced, and a blending ratio that continuously optimizes the evaluation function is selected, eventually causing the blending ratio to converge to one or a set of optimal or suboptimal configurations at the current stage.

[0032] Optionally, the multimodal data organization unit includes:

[0033] The single-modal basic sequence subunit is used to collect single-modal sequences of DNA, RNA and protein from biological databases as the initial characterization of each independent information molecule corresponding to the central dogma, and to perform preprocessing.

[0034] The bimodal pairing sequence subunit is used to generate bimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data through homology alignment and transcription-translation relationship analysis methods.

[0035] Trimodal interpenetrating sequence subunits are used to interleave DNA, RNA and protein sequences according to predetermined rules to form trimodal complex sequences that contain information of three modalities and simulate the complete information transmission pathway in a living organism.

[0036] Optionally, the bimodal pairing sequence subunit is specifically based on the central dogma of biology, pairing DNA coding regions with corresponding RNA transcripts or ultimately expressed protein products to form contextual target pairs for the next marker prediction.

[0037] On the other hand, based on the same inventive structure, this invention also provides a method for phased pre-training of dynamic multimodal biological sequences for precision breeding, comprising the following steps:

[0038] The process involves collecting single-modal sequences of DNA, RNA, and protein from biological databases, generating bimodal paired sequences through homology alignment and transcription-translation relationship analysis, and interleaving DNA, RNA, and protein sequences according to predetermined rules to form a trimodal interpenetrating sequence that contains information of three modalities and simulates the complete information transmission pathway in the body.

[0039] Perform unified single nucleotide or single amino acid segmentation on biological sequences of each modality, and add corresponding modality embedding tags for each modality;

[0040] The training process involves sequentially performing a next-label prediction task on the single-modal sequence for first-order training; performing a cross-modal next-label prediction task on a mixture of the single-modal sequence and the two-modal pairing sequence for second-order training, with the mixing ratio of the single-modal sequence and the two-modal pairing sequence dynamically adjusted according to a simulated annealing strategy; and performing a next-label prediction task on a mixture of the single-modal sequence, the two-modal pairing sequence, and the three-modal interpenetrating sequence for third-order training, with the mixing ratio of each modality dynamically adjusted according to a simulated annealing strategy.

[0041] Context-based prediction transformation and sequence generation between different biological sequence modalities;

[0042] The biological sequence to be tested is input into the trained model to obtain the quantitative prediction value or subsequent sequence output by the model.

[0043] Optional, also includes:

[0044] Receive and process SNP information of breeding populations;

[0045] Based on the input SNP information, the dictionary and / or embedding layer of the pre-trained model are modified by creating new SNP-specific tokens to achieve variation-aware embedding or extended conditional embedding, so that the model can specifically identify and respond to key SNP sites.

[0046] Supervised learning of the extended and modified model for a specific breeding task is performed using individual sequence data containing the SNP information and their corresponding breeding phenotypic data.

[0047] Optionally, it can be used specifically when dynamically adjusting the data mixing ratio:

[0048] Set the initial mixing ratio range;

[0049] Define one or more evaluation functions based on the model's performance on the validation set;

[0050] During the iterative process of pre-training or fine-tuning, the model is allowed to explore diverse mixing ratios in the early stages of training, accepting temporary performance fluctuations to escape local optima.

[0051] As the training process progresses, the randomness of the blending ratio is gradually reduced, and a blending ratio that continuously optimizes the evaluation function is selected, eventually causing the blending ratio to converge to one or a set of optimal or suboptimal configurations at the current stage.

[0052] Optionally, the step of collecting single-modal sequences of DNA, RNA, and protein from a biological database, generating bimodal paired sequences through homology alignment and transcription-translation relationship analysis, and interleaving DNA, RNA, and protein sequences according to predetermined rules to form a trimodal interpenetrating sequence containing three modalities and simulating a complete information transmission pathway in vivo includes:

[0053] Single-modal sequences of DNA, RNA, and protein were collected from biological databases as initial characterizations of each independent information molecule corresponding to the central dogma, and then preprocessed.

[0054] Dimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data are generated using homology alignment and transcription-translation relationship analysis methods.

[0055] DNA, RNA, and protein sequences are interleaved according to predetermined rules to form a trimodal complex sequence that simulates a complete information transmission pathway within a living organism, containing information in three modalities.

[0056] Optionally, the generation of bimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data through homology alignment and transcription-translation relationship analysis methods specifically includes:

[0057] Based on the central dogma of biology, DNA coding regions are paired with corresponding RNA transcripts or the final expressed protein products to form context-target pairs for predicting the next marker.

[0058] In terms of pre-training data design, this invention draws profound inspiration from the information processing flow of the general central dogma, constructing a progressively layered multimodal data system: First, high-quality DNA, RNA, and protein single-modal sequences from biological databases are used as basic representations, corresponding to each independent stage of the information flow; on this basis, genome-protein two-modal pairing sequences are innovatively constructed, and paired inputs are generated through homology alignment to simulate the direct association of information flow at specific stages, and the complementary characteristics of cross-modal contexts are used to strengthen the linkage between sequences; furthermore, DNA, RNA, and proteins are cross-inserted according to predetermined rules (such as the order of simulating the central dogma) to form interconnected three-modal sequences, providing the model with richer intermodal dependency signals that simulate the complete information transmission path.

[0059] In terms of model structure, this invention employs a universal single nucleotide segmenter to perform unified segmentation on three sequence types while preserving single nucleotide precision, and adds specialized modality embedding markers for different modalities. This design achieves an efficient cross-modal attention mechanism, enabling the model to simultaneously understand minute changes at the single nucleotide level and functional patterns at the long-range region level.

[0060] Regarding the pre-training strategy, this invention designs a three-stage training process corresponding to the data organization hierarchy, which also embodies the progressive idea of ​​the central dogma: the first stage is to perform pre-training on single-modal sequences, so that the model can master the basic rules of each single information carrier; the second stage is to carry out cross-modal mask prediction on bimodal genome-protein pairing sequences, so as to enhance the model's ability to capture the rules of specific information transformation (such as DNA-protein mapping); the third stage is mainly based on trimodal interpenetrating sequences, and uses the principle of simulated annealing to dynamically adjust the mixing ratio of different modalities, deeply explore the multi-level complex associations between sequences, and obtain a unified expression that takes into account both single-modal and multimodal information, just like understanding a complete biological process.

[0061] In the application phase, the genome sequence of the sample to be tested (which may contain variation information such as SNPs) is input into the optimized pre-trained model, which can output quantitative prediction scores of various biological characteristics and support diverse downstream tasks, including specialized breeding fine-tuning.

[0062] Compared with existing single-modal or simple splicing models, this invention significantly improves the prediction accuracy of biological sequence features, accelerates training convergence speed, achieves performance comparable to large models with only a small number of parameters, greatly reduces computing power and storage requirements, and can be flexibly deployed in various environments, providing efficient, accurate and low-cost technical support for biological sequence analysis. Attached Figure Description

[0063] Figure 1 This is a schematic block diagram of the structure of the dynamic multimodal biological sequence staged pre-training system for precision breeding provided in an embodiment of the present invention;

[0064] Figure 2 A three-stage pre-training flowchart of the dynamic multimodal biological sequence staged pre-training system for precision breeding provided in an embodiment of the present invention;

[0065] Figure 3 The dynamic multimodal biological sequence phased pre-training system for precision breeding provided in one embodiment of the present invention is a schematic diagram of multimodal data construction;

[0066] Figure 4 This is a schematic diagram illustrating the application of the dynamic multimodal biological sequence staged pre-training system for precision breeding provided in an embodiment of the present invention in a downstream task.

[0067] Figure 5 The flowchart of the step-by-step pre-training method for dynamic multimodal biological sequences for precision breeding provided in another embodiment of the present invention is shown. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0069] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0070] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0071] Before describing the embodiments of this application in detail, the technical concept of this application is briefly described first: This application addresses the problems of insufficient modality fusion and low efficiency in existing biological sequence pre-training models. Based on the central dogma of biology, it constructs a multi-level data system of single-modality, two-modality pairing, and three-modality interpenetrating sequences. It adopts unified word segmentation and modality embedding to achieve cross-modal representation. Through simulated annealing strategy, it dynamically adjusts the modality mixing ratio in three stages for pre-training, enabling the model to capture the deep associations between DNA, RNA, and proteins. Ultimately, it realizes the prediction of biological sequence features in precision breeding, improves accuracy, and reduces computing power requirements.

[0072] This invention aims to construct a pre-trained basic model with broad applicability in the field of plant and animal breeding. The specific principles of the dynamic multimodal biological sequence staged pre-training system and method for precision breeding described below are illustrated with specific embodiments.

[0073] Example 1

[0074] Figure 1This invention illustrates a dynamic multimodal biological sequence phased pre-training system 100 for precision breeding, as provided in Embodiment 1 of the present invention. The system includes a multimodal data organization unit 10, a unified sequence representation unit 20, a progressive pre-training strategy unit 30, a cross-modal autoregressive unit 40, and a sequence feature extraction and prediction unit 50, wherein:

[0075] The multimodal data organization unit 10 is used to collect single-modal sequences of DNA, RNA, and protein from biological databases, generate bimodal pairing sequences through homology alignment and transcription-translation relationship analysis, and interleave DNA, RNA, and protein sequences according to predetermined rules to form a trimodal interpenetrating sequence containing information of all three modalities, simulating a complete information transmission path in the organism; the unified sequence representation unit 20 is used to perform unified single-nucleotide or single-amino acid segmentation on biological sequences of each modality, and add corresponding modality embedding tags to each modality; the progressive pre-training strategy unit 30 is used to perform first-order training by sequentially executing the next tag prediction task on the single-modal sequences; and in the mixed single-modal sequences... The system performs second-order training on a cross-modal next-label prediction task on single-modal sequences and bimodal paired sequences, and dynamically adjusts the mixing ratio of the single-modal sequences and bimodal paired sequences according to a simulated annealing strategy; and performs third-order training on a mixed single-modal sequences, bimodal paired sequences, and trimodal interpenetrating sequences, and dynamically adjusts the mixing ratio of each modality according to a simulated annealing strategy; the cross-modal autoregressive unit 40 is used to realize context-based prediction transformation and sequence generation between different biological sequence modalities; the sequence feature extraction and prediction unit 50 is used to input the biological sequence to be tested into the trained model to obtain the quantitative prediction value or subsequent sequence output by the model.

[0076] The design concept of the multimodal data organization unit 10 draws inspiration from the hierarchical information transmission and integration principle in the central dogma of biology. In specific implementation, the multimodal data organization unit 10 includes a unimodal basic sequence subunit, a bimodal paired sequence subunit, and a trimodal interpenetrating sequence subunit, wherein:

[0077] The single-modal basic sequence subunit is used to collect single-modal sequences of DNA, RNA, and protein from biological databases as the initial characterization of each independent information molecule corresponding to the central dogma, and to perform preprocessing. The two-modal pairing sequence subunit is used to generate two-modal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data through homology alignment and transcription-translation relationship analysis methods. The three-modal interpenetrating sequence subunit is used to interleave DNA, RNA, and protein sequences according to predetermined rules (such as simulating the central dogma) to form a three-modal complex sequence containing three modal information and simulating the complete information transmission path in the organism.

[0078] The central dogma of biology reveals the pathways of genetic information transmission within organisms. Its core content is: genetic information is transferred from DNA to DNA (DNA replication), or from DNA to RNA (transcription), and then from RNA to protein (translation), i.e., the information flow hierarchy of "DNA→RNA→protein." This dogma elucidates the processes of genetic information storage, transmission, and expression, and is the molecular basis for understanding biological heredity, development, and trait formation. This invention draws upon this dogma to construct a multimodal data system and design a phased pre-training strategy to uncover deep correlations between biological sequences.

[0079] Furthermore, the bimodal pairing sequence subunit is specifically based on the biological central dogma, which pairs DNA coding regions with corresponding RNA transcripts or the final expressed protein products to form contextual target pairs for predicting the next marker.

[0080] Trimodal interpenetrating sequence subunits are used to interleave DNA, RNA and protein sequences according to predetermined rules (e.g., simulating the complete information flow and regulatory feedback of the central dogma) to form a complex sequence containing information of three modalities that simulates the complete information transmission pathway in a living organism.

[0081] The core of the multimodal data design in this embodiment lies in profoundly simulating the hierarchical transmission and integration process of information in the central dogma of biology. By constructing sequence combinations of different levels and complexities, it provides structured learning materials that precisely correspond to the information flow of the central dogma for subsequent advanced pre-training specifically for precision breeding.

[0082] In practice, the single-modal basic sequence subunit screens high-quality DNA, RNA, and protein sequences from the NCBI RefSeq (Reference Sequence Database). These basic sequences not only correspond to the initial states of each independent information molecule in the central dogma, but more importantly, they enable the model to learn the universal characterization of various biological macromolecules. This forms a solid foundation for the model to understand subsequent, more complex molecular interaction mechanisms that are crucial to the formation of breeding target traits.

[0083] The bimodal pairing sequence subunit constructs a gene-protein pairing dataset through intelligent alignment of homologous genomes and protein sequences. This design directly simulates the key information transformation steps in the central dogma from the genetic blueprint (possibly breeding markers or QTL (Quantitative Trait Locus) regions) to the functional execution unit (protein). This allows the model to learn how DNA sequence variations (such as SNPs within QTLs) affect protein structure and function, ultimately relating to core breeding traits such as yield and disease resistance. Innovatively, some pairings can include intermediate RNA transcript information, thus more completely depicting the regulatory pathways of the central dogma and enhancing the model's understanding of how gene expression regulation—a key link in trait manifestation—leads to breeding-related phenotypic variations.

[0084] Trimodal interpenetrating sequence subunits interleave DNA, RNA, and protein sequence fragments according to pre-defined rules (precisely simulating the complete information flow of the central dogma and its possible regulatory feedback). This design forces the model to learn the dynamic synergistic regulation and information transmission patterns between different modalities in an environment that more closely resembles the complex interactions in a living organism, much like understanding multiple interconnected links in a complete biological process. This is crucial for understanding and predicting quantitative traits (such as growth rate and environmental adaptability) determined by complex multi-gene, multi-pathway regulatory networks, and is key to unlocking the ability to make deep predictions about advanced and complex breeding goals.

[0085] The multimodal data organization unit 10 and / or the unified sequence representation unit 20 are also used to preprocess the data, specifically including sequence length unification, special character processing, and quality filtering.

[0086] The difficulty and data complexity of the prediction tasks in each stage of the progressive pre-training strategy unit 30 increase progressively. The first stage is autoregressive prediction within a single modality, the second stage is cross-modal prediction between two modalities, and the third stage is complex cross-modal and joint prediction under the coexistence of three modalities. This design aims to gradually deepen the model's understanding of information flow guided by the biological central dogma.

[0087] Preferably, in the model architecture design of this embodiment, a unified single nucleotide segmenter is constructed, suitable for processing three types of sequences: DNA, RNA, and protein. A specific modality embedding layer is designed to provide unique modality identifiers for different types of sequences such as DNA, RNA, and protein. A multi-layer Transformer encoder is constructed, including a standard self-attention mechanism and a dedicated cross-modal attention layer to promote the effective fusion of information from different modalities. Optimizations such as residual connections and layer normalization are added to ensure model training stability. Thus, in terms of sequence representation and cross-modal architecture, the system uses a single nucleotide / single amino acid level segmenter to uniformly process DNA, RNA, and protein sequences, supplemented by dedicated modality embedding vectors to accurately capture key information. Through specific modality identifiers, the model can distinguish and integrate information at different molecular levels. The core cross-modal attention mechanism promotes the dynamic exchange and fusion of features between different modalities. The overall architecture achieves lightweight parameter design through efficient modality fusion and structural optimization, aiming to lower the deployment threshold for breeding applications while ensuring or even improving prediction performance.

[0088] This embodiment utilizes a progressive pre-training strategy unit 30 to simulate an annealing-based three-stage pre-training strategy to enhance breeding generalization capabilities. The core of this strategy lies in progressively increasing task difficulty and data complexity, combined with simulated annealing to guide dynamic optimization of the mixing ratios of different modalities. The aim is to efficiently learn robust feature representations that can broadly adapt to different breeding scenarios (such as different species, different traits, and different genetic backgrounds). This phased training is precisely to simulate the gradual decoding and integration of central dogma information by biological systems.

[0089] Specifically, in the single-modal warm-up phase: initial training is performed on single-modal sequences using a next-label prediction task. This phase aims to allow the model to master the intrinsic structure and sequence patterns of various basic biological sequences (DNA, RNA, proteins), much like learning the basic "alphabet" and "grammar" of molecules in the central dogma, laying the foundation for subsequent understanding of more complex cross-modal breeding-related information. In the two-modal enhancement phase: gene-protein pairing sequences (which may include RNA bridges) are introduced to perform cross-modal NTP tasks. This phase dynamically adjusts the mixing ratio of single-modal and two-modal sequences using the principle of simulated annealing: the model's performance or loss function value on the validation set for breeding-related tasks (such as known gene function prediction or simple trait association) is used as the "energy function". In the early stages of training (high-temperature zone), more diverse mixing ratios are allowed to explore in order to escape local optima; as training progresses (cooling process), the mixing ratio gradually stabilizes at a ratio that optimizes the "energy function". This design aims to enhance the model's ability to capture key transformation steps in the central dogma (such as how DNA sequence variations directly or indirectly affect proteins through RNA) and thus relate them to breeding traits. In the trimodal deep fusion stage: interpenetrating trimodal sequences are mainly used for training, and the NTP task continues. Similarly, the mixing ratio of single-modal, two-modal, and trimodal data is dynamically adjusted according to the simulated annealing strategy. The goal of this stage is to deeply explore the complex synergistic and regulatory relationships among the three modalities in determining complex breeding traits, much like understanding the complete biological process guided by the central dogma. The aim is to find the modal combination that optimizes the model's overall breeding prediction performance (such as the joint prediction accuracy of multiple key traits), ultimately forming a unified and efficient multimodal representation capable of adapting to diverse and challenging breeding prediction needs.

[0090] Modal balancing during pre-training can dynamically adjust the data ratio based on the objectives of different pre-training stages and the guidance of simulated annealing strategies, ensuring that the model learns different model information in a balanced way and their contribution to breeding traits.

[0091] This embodiment can also be fine-tuned for downstream tasks. Specifically, it involves: designing appropriate model output layers and loss functions based on the specific biological task requirements; using a small amount of labeled data to supervise and fine-tune the pre-trained model to adapt it to the specific task; and utilizing specially designed data input formats to achieve efficient inference in cross-modal tasks such as RNA-protein interaction prediction (ncRPI). Specifically, a flexible fine-tuning framework can be designed to support various downstream applications, ranging from single-sequence trait prediction (such as functional prediction of specific genes or estimation of the effect size of a QTL region) to complex cross-modal breeding decision-making tasks (such as genotype x-environment interaction effect prediction, multi-trait heritability analysis, and optimal hybridization recommendation). A unified interface handles different biological problems, making it convenient for breeders to use.

[0092] To address the challenges of high labeling costs and relatively small sample sizes in breeding data (especially phenotypic data), this study optimizes the model's fine-tuning performance under small or even zero-sample conditions. This enhances the model's ability to transfer learning to new breeding populations or new traits, reducing reliance on large-scale phenotypic assays.

[0093] This embodiment provides a deep multimodal data fusion framework that organically combines DNA, RNA, and protein sequence information to construct a unified representation space that comprehensively reflects the genetic basis of complex traits. This significantly improves the prediction accuracy of key breeding traits such as yield, quality, disease resistance, and stress resistance, as well as the reliability of breeding value assessment. A sequence representation method combining single-nucleotide accuracy and regional pattern recognition capabilities is designed, enabling the model to simultaneously capture subtle variations at key gene loci and their roles in genomic functional elements, providing precise guidance for accurate selection, gene editing, and the discovery of superior gene resources. A simulated annealing three-stage pre-training strategy is developed. By progressively increasing task difficulty and optimizing modality mixing ratios, it not only significantly improves training efficiency and enhances the model's generalization ability to different breeding populations and environments but also greatly reduces the model parameter size and computational requirements, enabling advanced sequence analysis techniques to be widely applied in various breeding practices. A lightweight yet high-performance pre-trained model is constructed, allowing it to run cost-effectively on standard computing resources. This facilitates rapid deployment by breeders in actual breeding processes such as early selection, parental mating, and hybrid combination prediction, thereby accelerating the breeding process.

[0094] This embodiment achieves efficient pre-training and modeling of multimodal biological sequences, which can accurately capture variation information at the single nucleotide level, understand regional functional patterns and complex interactions between modalities, and achieve performance comparable to large models with a relatively small parameter scale, providing an efficient and practical technical path for biological sequence analysis.

[0095] Example 2

[0096] See Figure 3 The following describes a method for organizing multimodal biological sequence data in an optional implementation based on Embodiment 1. This method profoundly reflects the idea of ​​information hierarchical transmission and integration in the central dogma of biology, and is the foundation for achieving efficient NTP pre-training, closely corresponding to the subsequent three-stage pre-training strategy. This method is implemented through the multimodal data organization unit 10 and the unified sequence representation unit 20.

[0097] Data organization and implementation steps include:

[0098] S1.1, Construction of Single-Modal Base Sequences (i.e., constructing a single-modal base dataset of 400, corresponding to independent information units of the central dogma, used for the first stage of pre-training):

[0099] High-quality DNA sequences 410, RNA sequences 420, and protein sequences 430 were collected from the NCBI RefSeq database. All sequences underwent quality control to remove low-complexity regions and ambiguous bases / amino acids.

[0100] The sequence is divided into subsequences of variable length, including context and target prediction parts;

[0101] The specific implementation results are as follows: a single-modal sequence of approximately 220B tokens is constructed as the basic pre-training corpus, which enables the model to establish a basic understanding of the three modalities of DNA, RNA and protein, and how to learn the basic characteristics of each independent information molecule in the central dogma.

[0102] S1.2, Construction of bimodal pairing sequences (i.e., constructing a bimodal pairing dataset of 500, simulating the direct information transformation in the central dogma, used for the second stage of pre-training):

[0103] Using RefSeq gene annotation information, DNA coding regions and corresponding protein products were extracted;

[0104] By analyzing homologous genes and aligning sequences, genome-protein pairing relationships can be constructed, which in some cases may include RNA as a bridge to simulate the DNA→(RNA)→Protein conversion.

[0105] Cross-organizing paired sequences enables the model to predict the next label of one modality from another, learning the direct mapping relationship between modalities.

[0106] The specific implementation effect is as follows: constructing approximately 36B bimodal pairing sequences, establishing a predictive mapping relationship between DNA and protein (or through RNA), enabling the model to begin to understand how information in the central dogma is transmitted and transformed between different molecules.

[0107] S1.3, Construction of Trimodal Interpenetrating Sequences (i.e., constructing a trimodal interpenetrating dataset of 600 to simulate the complete information flow and complex interactions of the central dogma, for use in the third stage of pre-training):

[0108] DNA fragment 610, its corresponding transcribed RNA fragment 620, and the final translated protein fragment 630 were selected from the validated genomic region.

[0109] Following the natural flow of the central dogma (DNA→RNA→protein) or specific research needs (such as regulatory feedback), we design staggered insertion points and organization methods for sequence fragments to construct highly integrated multimodal inputs.

[0110] Add special markers at modal crossover points and design next marker prediction tasks that cross modalities (e.g., predict RNA from DNA, predict protein from RNA, or even cross multiple levels, such as predicting protein directly from DNA).

[0111] Specific implementation results: Constructing a trimodal interpenetrating sequence of approximately 13.3B tokens enables the model to learn and understand the complete information flow and complex interdependencies between the three modalities, much like understanding a complete biological process regulated by the central dogma.

[0112] S1.4 Data Preprocessing and Sequence Preparation:

[0113] A uniform single nucleotide / single amino acid segmentation strategy is applied to all sequences (DNA, RNA, protein);

[0114] Add dedicated modality identifiers (such as [DNA_MODALITY]) for different modalities.

[0115] [RNA_MODALITY], [PROTEIN_MODALITY]), and add positional embeddings and fragment embeddings as necessary;

[0116] Construct context-target pairs for training autoregressive next label prediction (NTP).

[0117] Specific implementation effects: It forms standardized NTP pre-training inputs, supports the model to efficiently learn the next label of the predicted sequence, and can distinguish and integrate information from different modalities.

[0118] Through the aforementioned multi-level data organization strategies (unimodal, bimodal pairing, and trimodal interleaving), a sequence prediction task with progressively increasing complexity was constructed, corresponding to the information flow of the biological central dogma. This not only provides the model with rich and structured biological sequence learning materials, enabling it to master powerful autoregressive prediction capabilities, but more importantly, it allows it to deeply understand the internal rules of a single modality and the complex transformation and synergistic relationships among DNA, RNA, and proteins based on the central dogma, laying a solid foundation for the subsequent three-stage pre-training.

[0119] Example 3

[0120] See Figure 2 The following describes a simulated annealing three-stage NTP pre-training strategy in an optional implementation based on Embodiment 1. This strategy is closely coupled with the multimodal data organization method embodying the central dogma idea in the above-mentioned multimodal biological sequence data organization method. By performing staged training on multimodal data of different complexities, efficient learning of biological sequence representations is achieved. This strategy is executed by the progressive pre-training strategy unit 30.

[0121] The steps for implementing a pre-training strategy include:

[0122] S2.1, First Stage: Single-Mode Autoregressive Prediction (Learning the fundamental laws of each molecule of the central dogma):

[0123] Data usage: Single-modal RefSeq sequence data (DNA, RNA, and protein independent sequences) constructed using the above-mentioned multimodal biological sequence data organization method;

[0124] Training task: Perform the next-to-n tag prediction (NTP) task for each modality. Given the first n nucleotides / amino acids in the sequence, predict the (n+1)th tag. Specifically, perform about 100K steps of next-to-n tag prediction (NTP) pre-training on a single-modality RefSeq sequence (containing DNA, RNA, and protein) of about 220B tokens.

[0125] Preliminary objectives: To establish the model's ability to understand and generate the three basic biological sequences of DNA, RNA, and protein through autoregression, and to gain a preliminary grasp of the "vocabulary" and "grammar" of each information carrier in the central dogma.

[0126] Specific implementation results: The model gains preliminary predictive ability for the subsequent development trend of each single modality sequence.

[0127] S2.2, Second Stage: Bimodal Reinforcement (Understanding Direct Information Transformation in the Central Dogma)

[0128] Data usage: The bimodal paired sequences constructed in the above-mentioned multimodal biological sequence data organization method are mixed with a portion of the unimodal sequences in a specific ratio.

[0129] Training task: Perform cross-modal label prediction (e.g., predict the next amino acid in a corresponding protein sequence given a DNA sequence context) and learn the rules of direct intermodal transitions guided by the central dogma.

[0130] Hybrid Strategy and Simulated Annealing Application: Simulated annealing is used to guide the dynamic adjustment of the mixing ratio of unimodal and bimodal data. Specifically, an evaluation function is set (e.g., a performance metric for a specific cross-modal task on the validation set). In the early stages of training (high-temperature zone), a more diverse mixing ratio is allowed for exploration, even if some ratios may cause short-term performance fluctuations, with the aim of escaping local optima. As training progresses (cooling down), the randomness of exploration is gradually reduced, favoring mixing ratios that can continuously improve the performance of the evaluation function, eventually converging to a relatively optimized ratio. This involves combining approximately 120B tokens of unimodal sequences and approximately 36B tokens of bimodal pairing sequences (DNA-protein, potentially containing RNA bridges) for approximately 80K steps of cross-modal NTP pre-training. In this stage, the mixing ratio of unimodal and bimodal data is dynamically adjusted based on the simulated annealing strategy: an evaluation function (such as cross-modal prediction loss on the validation set) is set, allowing a larger proportion of the exploration range in the early stage of training, and gradually converging to a better mixing ratio based on the feedback of the evaluation function, so as to enhance the model's capture of cross-modal mapping in the DNA-RNA-protein chain.

[0131] Specific implementation effects: Significantly enhances the model's ability to capture interconversions between DNA, RNA, and proteins (especially DNA to protein, or through indirect mapping via RNA, i.e., the key steps of the central dogma) and improves cross-modal prediction accuracy.

[0132] S2.3, Third Stage: Trimodal Deep Integration (Mastering a Complete Understanding of Biological Processes Guided by the Central Dogma):

[0133] It primarily uses approximately 13.3B tokens of trimodal interpenetrating sequences (DNA-RNA-protein), and can mix in a small amount of bimodal and unimodal data for approximately 120K steps of deep fusion NTP training.

[0134] Similarly, the simulated annealing strategy is applied to dynamically adjust the mixing ratio of different types of modal data. The goal is to find the optimal ratio combination that enables the model to achieve the best overall performance in multimodal understanding and cross-modal prediction tasks, thereby forming a unified and powerful multimodal biological sequence representation.

[0135] Within each stage, the application of simulated annealing strategies aims to ensure the efficiency of the model learning path and the superiority of the final performance by systematically exploring and optimizing the data mixing ratio.

[0136] Example 4

[0137] See Figure 4The following describes the application of NTP-based multimodal fusion biological sequence prediction in precision breeding, based on any of the above embodiments. This embodiment demonstrates the application of a pre-trained model in the actual process of precision breeding of plants and animals, aiming to accelerate the selection of superior varieties and improve breeding efficiency. Specifically, it includes:

[0138] S3.1 Model fine-tuning and deployment for specific breeding tasks:

[0139] The three-stage NTP pre-trained multimodal biological sequence model has been loaded.

[0140] For specific downstream breeding tasks (such as Figure 4 The example shown is: Estimation of genomic breeding value for lean meat percentage in pigs. Design or select appropriate output layers (such as regression heads for predicting breeding values ​​or trait performance, and classification heads for resistance grading).

[0141] Supervised fine-tuning of the pre-trained model is performed using a small amount of relevant marker data (genotype, transcriptome, proteome, and corresponding phenotypic data for a breeding population, perhaps only a few hundred to a few thousand individuals with phenotypic records) of relevant data.

[0142] The finely tuned model is deployed to the breeding unit's servers, cloud computing platforms, or integrated into the breeding decision support system for breeders to use in practice.

[0143] S3.2, Breeding trait prediction and genetic evaluation (e.g.) Figure 4 (Flowchart shown)

[0144] Input (input module): DNA sequence (e.g., whole genome SNP chip data, resequencing data) of the breeding material to be tested (e.g., new hybrid offspring, candidate parents, materials in the germplasm resource nursery). If conditions permit, RNA transcript data or key protein expression profiles at specific developmental stages or environmental conditions can be combined.

[0145] Processing (Model Processing Module): The input sequence is processed using a unified word segmenter consistent with pre-training, adding corresponding modality tags and possible conditional embeddings (such as environmental factors and management measures); the model executes the following based on task requirements:

[0146] 1. Breeding value / phenotype prediction model: The input sequence is transformed into a high-dimensional feature vector through a pre-trained model (or a specific layer thereof), and then these feature vectors are used to directly predict the numerical value (e.g., predicting yield kg / hectare, predicting daily weight gain g / day in beef cattle) or grade (e.g., disease resistance grade 1-5) of the target breeding trait through the output layer added during fine-tuning.

[0147] 2. Candidate gene / marker screening mode: Based on the model's sensitivity to sequence variations, identify gene regions, SNP sites, or RNA expression patterns that have a significant impact on target traits, providing candidate targets for marker-assisted selection or gene editing.

[0148] Output (output module): Predicted scores of target breeding traits (such as individual genetic potential ranking), carrier status of superior alleles, and adaptability rating to specific environmental conditions, to assist breeders in making selection decisions.

[0149] Specific implementation effects: For example, in rice yield trait prediction, after fine-tuning the pre-trained model of this invention, the accuracy of genomic prediction for complex yield components (such as number of grains per panicle and thousand-grain weight) (usually measured by the correlation between predicted breeding values ​​and actual breeding values) can be improved by 15-30% compared to the traditional GBLUP method. In the prediction of prolific meat yield in domestic pigs, the Spearman correlation coefficient can be improved by an average of 0.1-0.2. This improvement in accuracy means that low-yielding individuals can be eliminated more accurately and superior individuals can be selected in the early stages of breeding, thereby greatly accelerating the breeding generation process and improving selection efficiency.

[0150] S3.3 Case Studies of Practical Breeding Application Scenarios

[0151] Crop genome selection and accelerated breeding: Early genome prediction of large-scale hybrid offspring populations allows for rapid screening of individuals with comprehensive superior traits such as high yield, high quality, and multiple resistances to enter the next round of breeding. This replaces some time-consuming and labor-intensive field phenotyping and can shorten the breeding cycle by 1-3 years.

[0152] Livestock genetic assessment and germplasm innovation: Conduct more precise genetic assessments of core economic traits (such as litter size, feed conversion ratio, and meat quality indicators) of breeding males (such as breeding pigs and cattle) to guide the introduction, utilization, and breeding of superior breeds and new strains.

[0153] Stress-resistant breeding and climate-smart agriculture: By inputting genomic and multi-omics data under different environmental stresses (such as drought, high temperature, and salinity), we can predict the stress resistance performance of varieties and screen and breed new "climate-smart" varieties that are better adapted to future climate change.

[0154] Hybrid vigor prediction and parent selection: By analyzing the multimodal sequence information of candidate parents, the level of hybrid vigor that may appear in their hybrid offspring can be predicted, guiding the selection of the optimal hybrid combination and improving the success rate of hybridization breeding.

[0155] Specific implementation effects: Compared with traditional breeding methods or predictive models that rely solely on single-genome information, the pre-trained model provided by this invention can more quickly and accurately mine genetic information related to breeding objectives from large-scale, multi-dimensional sequence data. This will significantly accelerate the breeding process of superior varieties, reduce breeding costs, and is expected to achieve breakthroughs in increasing crop yields, improving livestock production performance, and enhancing disease and stress resistance, providing a powerful technological engine for ensuring national food security, enhancing the international competitiveness of agricultural products, and promoting sustainable agricultural development.

[0156] This embodiment demonstrates that NTP-based multimodal (DNA, RNA, protein) fusion pre-trained models have significant application potential in actual plant and animal breeding, through accurate trait prediction, genetic evaluation, and candidate gene mining, thereby accelerating genetic improvement, increasing breeding efficiency, and reducing costs. Its application process (such as...) Figure 4 As shown in the diagram, the results are clear and can provide more accurate predictions and innovative analytical methods for various breeding projects, providing core technical support for modern precision agriculture and intelligent breeding.

[0157] Example 5

[0158] This embodiment, based on the first embodiment described above, also includes a breeding-specific fine-tuning unit 60, which comprises an SNP data integration module, a model dictionary and embedding layer extension module, and a supervised fine-tuning module, wherein:

[0159] The SNP data integration module is used to receive and process SNP information from the breeding population; the model dictionary and embedding layer extension module is used to modify the dictionary and / or embedding layer of the pre-trained model by creating new SNP-specific tokens to achieve variation-aware embedding or extended conditional embedding, based on the input SNP information, so that the model can specifically identify and respond to key SNP sites; the supervised fine-tuning module is used to supervise the learning of the extended and modified model for specific breeding tasks using individual sequence data containing the SNP information and their corresponding breeding phenotypic data.

[0160] Furthermore, the progressive pre-training strategy unit 30 or the breeding-specific fine-tuning unit 60 is specifically used for the following when dynamically adjusting the data mixing ratio: setting an initial mixing ratio range; defining one or more evaluation functions based on the model's performance on the validation set (as a representation of "energy state"); during the iterative process of pre-training or fine-tuning (considered a "cooling process"), allowing the model to explore diverse mixing ratios in the early training stage (i.e., the "high-temperature zone"), accepting temporary performance fluctuations to escape local optima; gradually reducing the randomness of mixing ratio exploration as the training progresses (temperature decreases), selecting mixing ratios that continuously optimize the evaluation function, and ultimately converging the mixing ratios to one or a set of optimal or suboptimal configurations at the current stage.

[0161] The following describes the fine-tuning module for a specific breeding application and its extension to the model dictionary in this embodiment.

[0162] This embodiment focuses on the fine-tuning process of the pre-trained model of the present invention in a specific breeding scenario, particularly the handling of single nucleotide polymorphism (SNP) information that is commonly encountered in breeding practice and not fully covered during the pre-training stage. This demonstrates the flexibility and professionalism of the present invention in breeding applications, and breeding fine-tuning has its unique characteristics compared to fine-tuning for general downstream tasks. Specifically, it includes:

[0163] S4.1 The Special Characteristics of Breeding Fine-tuning: Processing SNP Data;

[0164] Unlike many general bioinformatics tasks, precision breeding relies heavily on fine-grained analysis of genetic variation among individuals, with SNPs being one of the most important and common types of variation. These SNP sites are directly associated with key breeding traits (such as yield, disease resistance, and quality). However, standard pre-training sequence data (such as RefSeq) typically represent a reference genome or typical sequences and may not include or highlight specific SNP information present in the population. Therefore, during the breeding fine-tuning phase, the model needs to effectively integrate this SNP information.

[0165] S4.2 The necessity and methods for modifying the LLM (Large Language Model) dictionary for SNP are as follows:

[0166] 1. Necessity: The tokenizer and corresponding vocabulary used in the pre-trained model are built based on the pre-training data. If a specific SNP base substitution pattern is uncommon in the pre-training data or is not explicitly distinguished as an independent token, the model may not fully understand its biological significance or may treat it as ordinary sequence noise. In order for the model to be able to "recognize" and specifically respond to these SNPs that are crucial for breeding, the model's input representation layer needs to be tuned.

[0167] 2. Modification method:

[0168] a. SNP Information Integration: First, collect SNP data from the target breeding population. This data is usually in VCF (Variant Call Format) files or other similar formats, clearly indicating the variant bases at specific locations in the genome relative to the reference genome.

[0169] b. Extended dictionary / embedding layer:

[0170] i. Option 1 (Special SNP Token): For key SNPs known to be functionally important or frequently used within a group, a new, specific token can be created and added to the model's segmenter dictionary. For example, for an SNP "chr1:12345A>G" that plays a key role in corn drought resistance, "G_at_chr1:12345" can be used as a new token. Subsequently, new embedding vectors are initialized for these new tokens in the model's embedding layer (this can be done by random initialization followed by fine-tuning, or by referencing embeddings of similar nucleotides).

[0171] ii. Option Two (Variation-Aware Embedding): During input sequence construction, SNP information can be explicitly encoded into the input sequence. For example, a special marker can be used at the corresponding position in the original DNA sequence to indicate the presence of an SNP, along with variant type information. Alternatively, more precisely, the variant base can be directly replaced in the input sequence, ensuring that the tokenizer can process it with single-nucleotide precision.

[0172] iii. Option 3 (Conditional Embedding / Modal Embedding Extension): The individual's SNP profile (e.g., a combination of a set of key SNPs) can be used as an additional conditional input or a new "SNP modality" information. A special embedding vector can be designed for this modality and combined with the original DNA, RNA, and protein modality embeddings, and fused through a cross-modal attention mechanism.

[0173] c. Update model input processing: Adjust the model's data loading and preprocessing pipeline accordingly to ensure that, during fine-tuning, new format sequences containing SNP information can be correctly segmented and converted into input tensors acceptable to the model.

[0174] S4.3 Breeding Fine-tuning Process

[0175] Data preparation: Prepare a fine-tuned dataset containing individual genome sequences (integrated with SNP information), optional other modal data (RNA, protein), and corresponding breeding phenotypic data.

[0176] Model Loading and Modification: Load the pre-trained multimodal model. Modify the segmenter dictionary and / or embedding layer according to the strategy selected in S4.2 to adapt to the SNP data.

[0177] Fine-tuning the task design: Design the output layer and loss function according to the specific breeding objectives. For example, a regression head can be used for breeding value prediction; a classification head can be used for disease resistance grading.

[0178] Supervised fine-tuning: The modified model is subjected to supervised learning using breeding phenotypic data. During fine-tuning, the model learns how to use the basic biological knowledge it acquired from the pre-training phase, combined with newly added SNP information, to predict the target trait.

[0179] S4.4 Expected Results

[0180] Through this fine-tuning method specifically optimized for breeding SNP data, the model can: more accurately capture phenotypic variations caused by specific SNPs or SNP combinations; improve the understanding and utilization efficiency of genetic diversity within breeding populations; enhance the accuracy of genomic selection, thereby more effectively guiding breeding decisions, such as parent selection and hybridization prediction; and significantly improve the prediction accuracy of complex quantitative traits compared to general fine-tuning that does not consider the specificity of SNPs.

[0181] The breeding-specific fine-tuning unit 60 provided in this embodiment, especially its adaptive modification of the LLM dictionary for SNP data, enables the pre-trained model of this invention to better serve the actual needs of precision breeding, fully explore the impact of genomic variation information on key traits, and thus play a greater role in breeding practice.

[0182] Example 6

[0183] Figure 5 This invention illustrates a phased pre-training method for dynamic multimodal biological sequences for precision breeding, as provided in Embodiment Six of the present invention, comprising the following steps:

[0184] S101: Collect single-modal sequences of DNA, RNA and protein from biological databases, generate two-modal paired sequences through homology alignment and transcription-translation relationship analysis, and interleave DNA, RNA and protein sequences according to predetermined rules to form a three-modal interpenetrating sequence that contains three modal information and simulates the complete information transmission path in the body.

[0185] S102: Perform unified single nucleotide or single amino acid segmentation on biological sequences of each modality, and add corresponding modality embedding tags for each modality.

[0186] S103: Perform the next label prediction task sequentially on the single-modal sequence for first-order training; perform the cross-modal next label prediction task on the mixed single-modal sequence and the two-modal pairing sequence for second-order training, and dynamically adjust the mixing ratio of the single-modal sequence and the two-modal pairing sequence according to the simulated annealing strategy; and perform the next label prediction task on the mixed single-modal sequence, the two-modal pairing sequence and the three-modal interpenetrating sequence for third-order training, and dynamically adjust the mixing ratio of each modality according to the simulated annealing strategy.

[0187] S104: Context-based prediction transformation and sequence generation between different biological sequence modalities.

[0188] S105: Input the biological sequence to be tested into the trained model to obtain the quantitative prediction value or subsequent sequence output by the model.

[0189] In an optional implementation, it further includes:

[0190] Receive and process SNP information of the breeding population; based on the input SNP information, modify the dictionary and / or embedding layer of the pre-trained model by creating new SNP-specific tokens to achieve variation-aware embedding or extended conditional embedding, so that the model can specifically identify and respond to key SNP sites; use individual sequence data containing the SNP information and their corresponding breeding phenotypic data to perform supervised learning of the extended and modified model for specific breeding tasks.

[0191] Furthermore, it is specifically used when dynamically adjusting the data mixing ratio:

[0192] Set an initial range for the blending ratio; define one or more evaluation functions based on the model's performance on the validation set; during the iterative process of pre-training or fine-tuning, allow the model to explore diverse blending ratios in the early stages of training, accepting temporary performance fluctuations to escape local optima; gradually reduce the randomness of blending ratio exploration as the training progresses, select blending ratios that continuously optimize the evaluation functions, and ultimately converge the blending ratios to one or a set of optimal or suboptimal configurations at the current stage.

[0193] In an optional implementation, step S101 specifically includes:

[0194] Single-modal sequences of DNA, RNA, and protein are collected from biological databases as initial characterizations of each independent information molecule corresponding to the central dogma, and are preprocessed. Bimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data are generated through homology alignment and transcription-translation relationship analysis. DNA, RNA, and protein sequences are interleaved according to predetermined rules to form a trimodal complex sequence containing three modal information that simulates the complete information transmission pathway in the organism.

[0195] The method of generating bimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data through homology alignment and transcription-translation relationship analysis specifically includes: pairing DNA coding regions with corresponding RNA transcripts or finally expressed protein products based on the central dogma of biology, and constructing them as contextual target pairs for the next marker prediction.

[0196] This sixth embodiment and the first embodiment belong to the same inventive structure, so for specific implementation details, please refer to the content of the above embodiments, which will not be repeated here.

[0197] The following describes optional modifications to the above embodiments during implementation.

[0198] To prevent others from circumventing patent protection by making only minor modifications to certain aspects of this invention, several feasible variations are listed below, along with their correspondence to the core technical solution of this invention, so that they can be covered in the claims:

[0199] I. Variations in word segmentation methods

[0200] Modification Scheme 1: Replace the character-level tokenizer with k-mer segmentation (such as 3-mer, 6-mer) or Byte-Pair Encoding to segment and encode DNA / RNA / protein sequences.

[0201] The corresponding technical points are: the three modalities are still segmented uniformly and modal embedding is added, but the segmentation granularity is different and the data input method is changed.

[0202] II. Number and Sequence Variations of Pre-training Stages

[0203] Modification scheme 2: Simplify the three-stage pre-training into two stages (such as single-modal warm-up followed by three-modal fusion), or change it to four stages (adding an environmental factor fusion stage in the middle of the two stages).

[0204] Corresponding technical points: Each stage still expands the representational ability through NTP or contrastive learning, but the stage division and order are slightly different.

[0205] III. Variations of Pre-training Tasks

[0206] Variation 3: In addition to the next token prediction, add one or a combination of the following tasks: contrastive learning, masked-token prediction, auto-encoding, or knowledge distillation.

[0207] Corresponding technical point: The goal remains to enhance sequence representation, but the form of pre-training loss can be changed.

[0208] IV. Model Structural Deformation

[0209] Modification scheme 4: Insert a graph neural network (GNN) module in the middle of the standard Transformer backbone to combine gene-protein interaction networks, or replace part of the self-attention layer with a lightweight convolutional residual block (Conv-ResBlock).

[0210] Corresponding technical points: The overall multi-layer network architecture still includes self-attention and cross-modal fusion, with only the implementation of sub-modules being adjusted.

[0211] V. Modal extension deformation

[0212] Modification scheme 5: Based on the three modes (DNA / RNA / protein), a fourth mode, such as environmental factors (temperature, humidity) or phenotypic time series, is introduced, and cross-modal fusion is achieved by adding modal markers and corresponding attention heads.

[0213] Corresponding technical points: It still adopts the modal embedding and fusion mechanism, but supports more modal inputs.

[0214] VI. Parameter Scale and Depth Deformation

[0215] Modification scheme 6: The model parameters are adjusted to different scales such as 50M, 200M, and 500M, and the number of network layers, attention heads, and hidden dimensions can be linearly scaled according to actual computing power requirements.

[0216] The corresponding technical points are as follows: the core fusion and pre-training strategies remain unchanged, with only the macroscopic size of the architecture differing.

[0217] VII. Data Augmentation and Sampling Variation

[0218] Variation 7: In the pre-training dataset, apply data augmentation strategies such as random spanning, noise injection, and token shuffling to the sequences to enhance the robustness of the model.

[0219] Corresponding technical points: The pre-training task and data modality design are consistent, only the data augmentation methods are different.

[0220] VIII. Fine-tuning and Multi-objective Optimization Deformation

[0221] Modified Solution 8: In the fine-tuning stage, adopt multi-task learning or multi-objective loss (such as simultaneously optimizing three indicators: yield, disease resistance, and quality), or improve the generalization ability with few samples through meta-learning.

[0222] Corresponding technical points: Trait prediction and decision-making processes are preserved, and the loss function and training strategy can be flexibly expanded.

[0223] IX. Deployment Variations in Reasoning

[0224] Variant Solution 9: Deploy the inference process on edge device microservices, WebAssembly, or mobile apps, or deploy the model after pruning, quantization, or distillation to meet the needs of offline or low-computing-power environments.

[0225] The corresponding technical points are as follows: the functions of the prediction and decision-making modules remain unchanged, only the hardware deployment and model compression methods are different.

[0226] All the above-mentioned variations are based on the core ideas of multimodal data design, cross-modal fusion architecture and phased pre-training strategy disclosed in this invention. Although there are differences in implementation details, they are essentially still within the protection scope of this invention to prevent others from circumventing the claims by fine-tuning word segmentation granularity, modifying fusion operators, adding or removing pre-training stages, adjusting model size or changing deployment methods.

[0227] This invention, through a multimodal deep fusion-based enhanced biological sequence model and a simulated annealing three-stage pre-training strategy, has demonstrated the following advantages compared to existing technologies, based on benchmark tests for multiple crop breeding trait predictions and protein function predictions:

[0228] I. Significantly Improved Prediction Accuracy. Experiments show that in the prediction task of backfat thickness in pigs weighing 100 kg, the Spearman correlation coefficient of the Decoder-only Transformer model using the method of this invention can reach 0.82, which is about 17% higher than that of the traditional single DNA sequence Transformer model (approximately 0.70). In small-sample, low-shot learning scenarios, the model in this paper improves the average prediction accuracy of multiple quantitative traits such as lean meat percentage in pigs weighing 100 kg by 12% to 20%.

[0229] II. Optimization of Convergence Speed ​​and Training Efficiency. By adopting a simulated annealing three-stage pre-training strategy, the model of this invention only requires 50% of the training steps in the single-modal preheating stage to reach baseline accuracy; in the two-modal and three-modal fusion stages, the overall training time is shortened by about 30% to 40% compared with the existing single-round multimodal stitching model, significantly reducing the consumption of computing resources.

[0230] III. High-throughput deployment and operational efficiency. The system built based on the method of this invention can predict thousands of whole genome sequences per minute in batches and in parallel through API interfaces or front-end interfaces. Compared with traditional LSTM / CNN models, the overall processing accuracy is greatly improved, while the inference time meets practical needs and satisfies the requirements of large-scale breeding projects.

[0231] IV. Estimated Beneficial Socioeconomic Benefits. Significantly Shortened Breeding Cycle: In the screening of major grain crops such as maize and rice, the iterative cycle from genotype to trait prediction can be shortened; Significantly Reduced Breeding Costs: Compared with traditional field trials and hotspot gene candidate methods, breeding experiments and labor costs can be reduced; Improved Agricultural Yield and Stress Resistance: Combining prediction and decision optimization schemes can increase the yield of target crop lines, and the improvement effects on traits such as disease resistance and drought resistance are stable and reliable; Promotion of Sustainable Development: The method of this invention supports flexible deployment on cloud, edge devices, and local servers, providing efficient and low-cost technical support for modern precision breeding and intelligent agriculture, promoting food security and ecological environmental protection.

[0232] In summary, this invention has significant advantages such as high prediction accuracy, fast convergence, low cost, high throughput, and good economic and social benefits. It is significantly superior to existing breeding models with single sequences or simple splicing of multimodal models, fully demonstrating the innovation and practical value of this invention in the field of plant and animal breeding.

[0233] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0234] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A dynamic multimodal biological sequence staged pre-training system for precision breeding, characterized in that, Including: The multimodal data organization unit is used to collect single-modal sequences of DNA, RNA and protein from biological databases, generate two-modal paired sequences through homology alignment and transcription-translation relationship analysis, and interleave DNA, RNA and protein sequences according to predetermined rules to form a three-modal interpenetrating sequence that contains three modal information and simulates the complete information transmission path in the body. A unified sequence representation unit is used to perform unified single nucleotide or single amino acid segmentation on biological sequences of various modalities and to add corresponding modality embedding tags to each modality. A progressive pre-training strategy unit is used to sequentially perform a next-label prediction task for first-order training on the single-modal sequence; perform a cross-modal next-label prediction task for second-order training on a mixture of the single-modal sequence and the two-modal pairing sequence, and dynamically adjust the mixing ratio of the single-modal sequence and the two-modal pairing sequence according to a simulated annealing strategy; and perform a next-label prediction task for third-order training on a mixture of the single-modal sequence, the two-modal pairing sequence, and the three-modal interpenetrating sequence, and dynamically adjust the mixing ratio of each modality according to a simulated annealing strategy. Cross-modal autoregressive units are used to achieve context-based prediction transformation and sequence generation between different biological sequence modalities; The sequence feature extraction and prediction unit is used to input the biological sequence to be tested into the trained model in order to obtain the quantitative prediction value or subsequent sequence output by the model.

2. The dynamic multimodal biological sequence staged pre-training system for precision breeding according to claim 1, characterized in that, It also includes a breeding-specific fine-tuning unit, which includes: The SNP data integration module is used to receive and process SNP information of breeding populations; The model dictionary and embedding layer extension module is used to modify the dictionary and / or embedding layer of the pre-trained model by creating new SNP-specific tokens to achieve variation-aware embedding or extended conditional embedding based on the input SNP information, so that the model can specifically identify and respond to key SNP sites. The supervised fine-tuning module is used to supervise the learning of the extended and modified model for a specific breeding task using individual sequence data containing the SNP information and their corresponding breeding phenotypic data.

3. The dynamic multimodal biological sequence staged pre-training system for precision breeding according to claim 1 or 2, characterized in that, The progressive pre-training strategy unit or the breeding-specific fine-tuning unit is specifically used for dynamically adjusting the data mixing ratio in the following ways: Set the initial mixing ratio range; Define one or more evaluation functions based on the model's performance on the validation set; During the iterative process of pre-training or fine-tuning, the model is allowed to explore diverse mixing ratios in the early stages of training, accepting temporary performance fluctuations to escape local optima. As the training process progresses, the randomness of the blending ratio is gradually reduced, and a blending ratio that continuously optimizes the evaluation function is selected, eventually causing the blending ratio to converge to one or a set of optimal or suboptimal configurations at the current stage.

4. The dynamic multimodal biological sequence staged pre-training system for precision breeding according to claim 1, characterized in that, The multimodal data organization unit includes: The single-modal basic sequence subunit is used to collect single-modal sequences of DNA, RNA and protein from biological databases as the initial characterization of each independent information molecule corresponding to the central dogma, and to perform preprocessing. The bimodal pairing sequence subunit is used to generate bimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data through homology alignment and transcription-translation relationship analysis methods. Trimodal interpenetrating sequence subunits are used to interleave DNA, RNA and protein sequences according to predetermined rules to form trimodal complex sequences that contain information of three modalities and simulate the complete information transmission pathway in a living organism.

5. The dynamic multimodal biological sequence staged pre-training system for precision breeding according to claim 4, characterized in that, The bimodal pairing sequence subunit is specifically based on the central dogma of biology, which pairs DNA coding regions with corresponding RNA transcripts or the final expressed protein products to form contextual target pairs for the next marker prediction.

6. A phased pre-training method for dynamic multimodal biological sequences for precision breeding, characterized in that, Including the following steps: The process involves collecting single-modal sequences of DNA, RNA, and protein from biological databases, generating bimodal paired sequences through homology alignment and transcription-translation relationship analysis, and interleaving DNA, RNA, and protein sequences according to predetermined rules to form a trimodal interpenetrating sequence that contains information of three modalities and simulates the complete information transmission pathway in the body. Perform unified single nucleotide or single amino acid segmentation on biological sequences of each modality, and add corresponding modality embedding tags for each modality; The training process involves sequentially performing a next-label prediction task on the single-modal sequence for first-order training; performing a cross-modal next-label prediction task on a mixture of the single-modal sequence and the two-modal pairing sequence for second-order training, with the mixing ratio of the single-modal sequence and the two-modal pairing sequence dynamically adjusted according to a simulated annealing strategy; and performing a next-label prediction task on a mixture of the single-modal sequence, the two-modal pairing sequence, and the three-modal interpenetrating sequence for third-order training, with the mixing ratio of each modality dynamically adjusted according to a simulated annealing strategy. Context-based prediction transformation and sequence generation between different biological sequence modalities; The biological sequence to be tested is input into the trained model to obtain the quantitative prediction value or subsequent sequence output by the model.

7. The method for phased pre-training of dynamic multimodal biological sequences for precision breeding according to claim 6, characterized in that, Also includes: Receive and process SNP information of breeding populations; Based on the input SNP information, the dictionary and / or embedding layer of the pre-trained model are modified by creating new SNP-specific tokens to achieve variation-aware embedding or extended conditional embedding, so that the model can specifically identify and respond to key SNP sites. Supervised learning of the extended and modified model for a specific breeding task is performed using individual sequence data containing the SNP information and their corresponding breeding phenotypic data.

8. The method for phased pre-training of dynamic multimodal biological sequences for precision breeding according to claim 6 or 7, characterized in that, Specifically used when dynamically adjusting the data mixing ratio: Set the initial mixing ratio range; Define one or more evaluation functions based on the model's performance on the validation set; During the iterative process of pre-training or fine-tuning, the model is allowed to explore diverse mixing ratios in the early stages of training, accepting temporary performance fluctuations to escape local optima. As the training process progresses, the randomness of the blending ratio is gradually reduced, and a blending ratio that continuously optimizes the evaluation function is selected, eventually causing the blending ratio to converge to one or a set of optimal or suboptimal configurations at the current stage.

9. The method for phased pre-training of dynamic multimodal biological sequences for precision breeding according to claim 6, characterized in that, The process involves collecting single-modal sequences of DNA, RNA, and proteins from biological databases, generating bimodal paired sequences through homology alignment and transcription-translation relationship analysis, and interleaving DNA, RNA, and protein sequences according to predetermined rules to form a trimodal interpenetrating sequence containing three modalities and simulating a complete information transmission pathway within a living organism. This includes: Single-modal sequences of DNA, RNA, and protein were collected from biological databases as initial characterizations of each independent information molecule corresponding to the central dogma, and then preprocessed. Dimodal pairing sequences of DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data are generated using homology alignment and transcription-translation relationship analysis methods. DNA, RNA, and protein sequences are interleaved according to predetermined rules to form a trimodal complex sequence that simulates a complete information transmission pathway within a living organism, containing information in three modalities.

10. The method for phased pre-training of dynamic multimodal biological sequences for precision breeding according to claim 9, characterized in that, The bimodal pairing sequences generated through homology alignment and transcription-translation relationship analysis methods for DNA-protein pairing data, DNA-RNA pairing data, or RNA-protein pairing data specifically include: Based on the central dogma of biology, DNA coding regions are paired with corresponding RNA transcripts or the final expressed protein products to form context-target pairs for predicting the next marker.

Images