A method, system, and inference method for multi-task training of neural networks
By introducing a covariant loss branch and segmented adaptive scheduling of multi-sequence alignment (MSA) during the neural network training phase, the performance degradation problem of neural networks in shallow/missing MSA scenarios is solved, and the effective utilization of co-evolutionary signals and the convenience of single-sequence inference are realized.
Patent Information
- Application Number
- CN202611586947.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Existing neural network training schemes exhibit significant performance degradation in shallow/missing multiple sequence alignment (MSA) scenarios, have high inference costs and limited applicability, and fail to effectively address the issue of adaptive injection of co-evolutionary signals during the pre-training stage.
By introducing a covariant loss branch of multi-sequence alignment (MSA) during the multi-task training phase of the neural network, and performing segmented adaptive scheduling based on the effective depth D, the covariant loss weights are calculated. Combined with the masked language model loss, the co-evolutionary signal is fully utilized and training stability is achieved, requiring only a single sequence input during the inference phase.
While maintaining inference performance, it reduces training instability and cross-depth level metric dispersion, improves co-evolutionary modeling capabilities and inference deployment convenience, and reduces computational costs.
Smart Images

Figure CN123072513A_ABST
Abstract
Description
Technical Field
[0001] This application mainly relates to the fields of artificial intelligence and machine learning, specifically to a neural network multi-task training method, system, and inference method. Background Technology
[0002] In existing technologies, neural network training schemes have attempted to incorporate co-evolutionary information from multiple sequence alignments (MSA) to enhance sequence representation capabilities. For example, the MSA Transformer directly uses MSA as input and performs masked language modeling on alignments through row-column interleaved attention, demonstrating outstanding performance in tasks such as contact prediction. In the structure prediction pipeline, MSA generation / completeness schemes, as well as co-evolutionary assisted mutation screening and sequence-structure co-generation schemes, utilize co-evolutionary information for downstream task scenarios.
[0003] However, the MSA Transformer inference stage relies on MSA input, which significantly reduces performance in shallow / missing MSA scenarios, resulting in high inference costs and limited applicability. Furthermore, downstream application solutions such as MSA generation and completion, and co-evolution-assisted mutation screening do not involve loss scheduling design in the neural network training stage, and cannot directly solve the problem of adaptive injection of co-evolutionary signals in the pre-training stage. Summary of the Invention
[0004] The core objective of this invention is to overcome the aforementioned deficiencies of the prior art and provide a neural network multi-task training method, system, and inference method. This method fully utilizes the co-evolutionary signals of deep MSA during the neural network multi-task training stage, while avoiding training instability and excessive dispersion of indicators across depth levels caused by fixed covariant weights on shallow MSA / orphan sequences. Furthermore, it maintains the effect of inference based solely on a single query sequence after pre-training.
[0005] In a first aspect, this application provides a method for multi-task training of neural networks, comprising the following steps: S1: Obtain the protein sample query sequence and its corresponding multiple sequence alignment (MSA), perform quality control filtering preprocessing on the MSA to obtain the quality control MSA, and determine the effective depth D of multiple sequence alignment based on the quality control MSA; S2: Calculate the covariance scores of all locus pairs in the MSA after quality control, sort them in descending order of covariance scores, and select the top [percentages]. Each locus pair is considered as a set of covariant locus pairs, where Based on the length of the query sequence Sure; S3: Perform random masking on the query sequence to obtain the masked sequence. Input the masked sequence into the sequence encoder to obtain the hidden representation of each bit of the query sequence. Calculate the masked language model loss and covariance loss respectively. S4: Calculate the covariance loss weights based on the effective depth D using a piecewise function. ; S5: Weight the masked language model loss and the covariance loss according to the covariance loss weight. The total training loss is obtained by weighted summation and backpropagation. The parameters of the sequence encoder are updated by backpropagating the total training loss. Multiple protein samples in the training corpus are traversed, and steps S1 to S5 are repeated until the maximum number of pre-training rounds is reached or the total training loss does not decrease within the specified number of rounds.
[0006] In one embodiment, in S4, the covariant loss weight is calculated based on the effective depth D according to a piecewise function. Including: when the effective depth D is lower than the first threshold =0, when the effective depth D is higher than the second threshold. A preset saturation value is taken, and when the effective depth D is between the first threshold and the second threshold... As D becomes monotonically non-decreasing.
[0007] In one embodiment, the covariant loss weight in S4 is: , in The first threshold, The second threshold, and < .
[0008] In one embodiment, the piecewise function in S4 can also be a superposition of a smooth monotonic function and a hard-gated function, i.e., when D is below the first threshold... =0, when D is between the first threshold and the second threshold. Determined by a smooth monotonic function, the smooth monotonic function satisfies the condition that when D approaches the first threshold... Approaching zero, when D approaches the second threshold Smooth monotonic functions that approach 1 and are monotonically increasing in the intermediate region include the sigmoid function or the exponentially saturated function.
[0009] In one embodiment, the covariant loss in S5 is a loss branch that is independent of the masked language model loss. The covariant loss is calculated by minimizing the square of the cosine distance between the latent representations of the site pairs within the covariant site pair set, so that the latent representations of the site pairs within the covariant site pair set are close to each other in the vector space.
[0010] In one embodiment, the covariance loss in S3 is calculated as follows: The number of site pairs participating in the calculation is determined based on the set of covariant site pairs. Calculate the loss term for each site pair (i,j). ,in and These are the hidden representations at sites i and j in the last layer of the sequence encoder, respectively. The covariance fraction of that locus pair; The covariant loss is obtained by averaging the loss terms for all sites involved in the calculation and multiplying them by a scaling factor.
[0011] In one embodiment, the number of loci involved in the calculation The first half of the set of covariant site pairs, arranged in descending order of covariance fraction, is... ,in, ,in This is the preset scaling factor.
[0012] In one embodiment, the covariant loss in S3 is calculated via paired scoring heads or attention coupling. The calculation via the paired scoring head includes receiving the concatenation of the hidden representations of the site pairs and the sum of the positions from the sequence encoder as input, and outputting the covariance prediction score of the site pair, wherein the covariance loss is the difference between the covariance prediction score and the covariance score of the site pair; or The attention coupling calculation includes using the covariance score of the site pairs in the covariance site pair set as the target value at the corresponding position in the attention matrix after averaging multiple attention heads in the last layer of the sequence encoder, and the covariance loss is the difference between the attention weight at the corresponding position in the attention matrix and the covariance score.
[0013] In one embodiment, the covariance fraction in S2 is calculated using mutual information MI or mutual information MIp corrected for average product; for each site pair (i,j) in the quality-controlled MSA, the joint occurrence frequency matrix of the standard amino acids is statistically analyzed. Where a and b represent the types of amino acids appearing at site i and site j, and the frequency of marginal amino acids at site i, respectively. and the edge amino acid frequency of site j , Mutual Information MI is: ; The mutual information MIp corrected for average product is: .
[0014] In one embodiment, if a site in the set of covariant site pairs is masked in S3, then the site pair does not participate in the covariant loss calculation; or a context-based inference-based implicit representation is used to participate in the covariant loss calculation.
[0015] In one embodiment, the function of the total training loss is: , in, For masked language model loss, For covariance loss, The weights are for covariant loss.
[0016] Secondly, this application provides a neural network multi-task inference method, including pre-training using the aforementioned neural network multi-task training method. After pre-training, the sequence encoder only receives the target protein query sequence as input during the inference phase, and outputs the hidden representation of each position in the target protein query sequence as the protein representation.
[0017] Thirdly, this application provides a neural network multi-task training system, including: The MSA preprocessing module is used to obtain the protein sample query sequence and its corresponding multiple sequence alignment MSA, perform quality control filtering preprocessing on the multiple sequence alignment MSA to obtain the quality control MSA, determine the effective depth D of multiple sequence alignment based on the quality control MSA, and calculate the covariance score of all site pairs in the quality control MSA, sort them in descending order of covariance score, and select the top ones. Site pairs are considered as a set of covariant site pairs, where Based on the length of the query sequence Sure; The sequence encoder module is used to receive the sequence after performing a random mask and output the implicit representation of each bit. The dual-task loss module is used to calculate the masked language model loss and the covariant loss separately. The depth-adaptive weighting module is used to calculate the covariant loss weights based on the effective depth D according to a piecewise function. ; The masked language model loss and covariant loss are arranged according to The total training loss is obtained by weighted summation, and then backpropagated to update the parameters of the sequence encoder.
[0018] Fourthly, this application provides an electronic device, including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the above-described neural network multi-task training method and the above-described protein language model inference method.
[0019] Fifthly, this application provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the steps of the above-described neural network multi-task training and protein language model inference methods.
[0020] Sixthly, this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-described neural network multi-task training method and the above-described protein language model inference method.
[0021] By adopting a dual-task architecture that decouples training and inference, an independent covariant loss branch is introduced during the training phase using multi-sequence alignment (MSA) as the supervision source. Based on the effective depth of MSA, sample-level segmented adaptive scheduling of covariant loss weights is achieved. This fully utilizes the long-range co-evolutionary signals in deep MSA while avoiding training instability caused by noise covariant supervision interfering with the main task in shallow MSA / orphan sequence scenarios. Furthermore, after pre-training, only a single sequence input is required during the inference phase, without the need for MSA support. This balances co-evolutionary modeling capabilities with convenient inference deployment, effectively reducing the variance of training metrics between samples of different MSA depths. Attached Figure Description
[0022] To more clearly illustrate the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a flowchart of one embodiment of the neural network multi-task training method provided in this application.
[0024] Figure 2 This is a structural block diagram of one embodiment of the neural network multi-task training system provided in this application. Detailed Implementation
[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this application. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.
[0026] As indicated in this application, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.
[0027] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.
[0028] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.
[0029] Furthermore, although the terminology used in this application is selected from commonly known and used terms, some terms mentioned in this application's specification may have been chosen by the applicant according to his or her judgment, and their detailed meanings are explained in the relevant sections of the description herein. Moreover, this application is to be understood not only by the actual terms used, but also by the meaning implied by each term.
[0030] This application uses flowcharts to illustrate the operations performed by an apparatus or device according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.
[0031] The protein language model (PLM) involved in this application refers to an artificial intelligence model that draws on language modeling techniques from the field of natural language processing, uses protein amino acid sequences as modeling objects, learns the contextual rules of amino acid sequences, evolutionary constraints, and structure-function relationships. Its core function is to output a distributed representation of the amino acid sequence to support downstream protein-related tasks. This model uses a sequence encoder as its core structure, and in this application, it achieves the fusion of single-sequence modeling capabilities and co-evolutionary signals through task design during the pre-training phase. In one exemplary embodiment, the protein language model uses a sequence encoder with a single protein sequence as input as its backbone. During the training phase, it maintains basic single-sequence contextual modeling capabilities through a masked language model task, while simultaneously injecting co-evolutionary signals through an independent covariant loss branch. After training, only a single protein sequence needs to be input to output an enhanced sequence representation. In other words, the core technical concept of this application is not necessarily bound to the specific scale of the protein language model, the selection of the backbone network, or the source of the pre-training data. Those skilled in the art can adaptively adjust the above configuration according to actual application scenarios and performance requirements, and such conventional adjustments do not depart from the protection scope of this invention.
[0032] This application provides a neural network multi-task training method and system, which can realize explicit and controllable injection of long-range co-evolutionary signals, ensure the training stability of samples with different multi-sequence alignment depths, improve the characterization consistency of proteins with different homologous sequence abundances, and reduce the computational overhead and latency in the inference stage.
[0033] According to one embodiment of this application, a multi-task training method based on neural networks is proposed, such as... Figure 1 This includes the following steps: Step S1: Obtain the protein sample query sequence and its corresponding multiple sequence alignment (MSA). Perform quality control filtering preprocessing on the MSA to obtain the quality control MSA, and determine the effective depth D of the multiple sequence alignment based on the quality control MSA.
[0034] Specifically, the protein sample query sequences are derived from each protein sequence in a pre-trained dataset (such as the UniRef50 protein sequence database). For each protein sample query sequence, a multiple sequence alignment (MSA) is constructed by retrieving its homologous sequences from the sequence database. Quality control processing includes two steps: sequence identity redundancy reduction and gap filtering. Sequence identity redundancy reduction removes excessively similar redundant sequences (such as multiple closely related variants of the same species) to avoid bias in joint frequency statistics. Gap filtering removes low-quality sequences with a large number of gaps. The quality-controlled MSA is used simultaneously for the statistics of the effective depth D of multiple sequence alignment and the extraction of covariant site pairs. Quality control processing also includes an optional column coverage filtering step. After the first two steps, columns with a non-gap residue ratio below a preset coverage threshold are removed. The sites corresponding to the removed columns do not participate in the subsequent covariance score calculation.
[0035] The effective depth D of multiple sequence alignment (MSA) reflects the abundance of independent evolutionary information. This application provides two methods for calculating the effective depth D: one is the sequence counting method, which counts the number of non-gap sequences retained after redundancy removal; the other is the weighted effective sequence count method, which assigns a weight based on sequence identity to each sequence and sums all weights.
[0036] For example, the effective depth D is determined using sequence counting: When performing quality control filtering preprocessing on the multiple sequence alignment MSA, column gap ratio filtering, column coverage filtering, and sequence identity redundancy reduction are performed sequentially. The number of sequences with non-full gaps in the corresponding column of the query sequence in the quality-controlled MSA is counted, and this number is used as the effective depth D, and as input to the piecewise function to calculate the covariance loss weight. Gap filtering on the multiple sequence alignment MSA filters columns where the gap ratio in a single column exceeds a preset ratio threshold. This preset ratio threshold is set by the technician as needed, for example, preferably 0.5. Column coverage filtering on the multiple sequence alignment MSA filters columns where the column coverage is lower than a preset coverage threshold. This preset coverage threshold is set by the technician according to actual needs, for example, 0.5. Sequence identity redundancy reduction processing on the multiple sequence alignment MSA filters redundant sequences where the sequence identity exceeds a preset identity threshold. This preset identity threshold is set by the technician according to actual needs, for example, 0.9. For example, performing quality control filtering on an MSA containing 120 homologous sequences, after removing columns with a gap ratio exceeding 0.5 by gap filtering, filtering columns with a coverage ratio below 0.5, and performing sequence identity redundancy reduction filtering to remove redundant sequences with an identity ratio exceeding 0.9, 40 valid sequences remain, resulting in an effective depth D=40. By performing multi-dimensional quality control preprocessing on multi-sequence alignment MSA, including gap filtering, column coverage filtering, and sequence identity redundancy reduction, invalid columns, redundant sequences, and low-coverage sites in the multi-sequence alignment MSA can be effectively eliminated, improving the reliability of covariance score calculation results and reducing the interference of noise signals on subsequent covariance loss calculations.
[0037] For example, the weighted effective sequence number is calculated as follows: each sequence in the quality-controlled multiple sequence alignment (MSA) is assigned a weight based on sequence identity, all weights are summed to obtain N_eff, and N_eff is used to replace the sequence number in the same piecewise function. One example of the weight is the reciprocal of the average identity of the sequence with the remaining sequences; another example is to distribute the weight equally within the cluster after clustering according to an identity threshold.
[0038] The two methods for calculating the effective depth D complement each other in terms of accuracy and computational efficiency: the sequence counting method is simple to calculate and uses the number of redundant sequences as a lower bound estimate of the independent evolutionary information content, making it suitable for fast pre-training scenarios with limited computational resources; the weighted effective sequence count method uses soft weighting based on sequence identity to more accurately quantify the impact of partial redundancy, making it suitable for scenarios with high accuracy requirements for depth scheduling. Both methods use the quality-controlled MSA as input, reflecting the abundance of independent evolutionary information in the samples at different accuracy levels, providing reliable data support for the subsequent adaptive calculation of covariant loss weights.
[0039] Step S2: Calculate the covariance scores of all locus pairs in the MSA after quality control, sort them in descending order of covariance scores, and select the top [percentages]. Site pairs are considered as a set of covariant site pairs, where Based on the length of the query sequence Sure.
[0040] The covariance score is a statistical measure of the degree of co-evolutionary association between two loci. The covariance score can be calculated using mutual information (MI) of the locus pair, mutual information (MIp) corrected for APC, etc.
[0041] Mutual information (MI) calculation includes: for each pair of sites (i,j) in the quality-controlled MSA, statistically analyzing the co-occurrence frequency matrix of standard amino acids. Where a and b represent the types of amino acids appearing at site i and site j, respectively, with 20 standard amino acids selected for each site, and the frequency of edge amino acids at site i. and the edge amino acid frequency of site j Mutual information MI is: A higher mutual information (MI) value indicates a stronger co-evolutionary association between the two sites, and they are more likely to be spatially adjacent residue pairs in the three-dimensional structure of the protein.
[0042] The APC-corrected mutual information calculation includes: applying average product correction (APC) to the mutual information MI to obtain MIp, in order to eliminate the systematic bias introduced by the phylogenetic tree structure. The APC-corrected mutual information MIp is: .
[0043] In some embodiments, the set of covariant sites The set of triplets is arranged in descending order of covariance strength. Each triplet includes the first point identifier i, the second point identifier j, and the corresponding covariance strength s, i.e. The number of site pairs in the set of covariant site pairs ,in The preset proportional coefficient is set by technicians according to actual needs, and is preferably selected. The value range is 0.5 to 5, for example, , To determine the length of the query sequence, for a query sequence of length 100, calculate the MIp values of all locus pairs as covariance scores. Select the top 50 locus pairs in descending order of scores to form a set of covariance locus pairs. Each locus pair is stored as a triple in the form of (i,j,s), where s is the mutual information MI value of the corresponding locus pair or the mutual information MIp value corrected by APC.
[0044] By supporting multiple covariance score calculation methods such as mutual information MI and APC-corrected mutual information MIp, the covariance signal extraction scheme can be flexibly selected according to different pre-training data scales and downstream task requirements, thereby improving the flexibility and adaptability of covariance sites for screening.
[0045] Step S3: Perform random masking on the query sequence to obtain the masked sequence, input the masked sequence into the sequence encoder to obtain the hidden representation of each bit of the query sequence; calculate the masked language model loss and covariance loss respectively.
[0046] Sequence encoders can employ a Transformer architecture. For example, a small-scale sequence encoder reproducible on a CPU could be used: a 6-layer Transformer encoder with 320 hidden dimensions, 8 attention heads, a vocabulary size of 25 (20 standard amino acids + special tokens: mask markers, start / end markers, padding markers, etc.), and a maximum sequence length of 256. For larger-scale pre-training, a larger-scale Transformer (e.g., 30 layers, 1280 hidden dimensions, 12 attention heads) can be used, trained on the full UniRef50 dataset.
[0047] Sequence encoders can also employ a structure combining small-scale embedding layers with a feedforward network. During both the training and inference phases, the input to a sequence encoder is a single protein sequence, and the output, the query sequence, is a vector-based residue-level hidden representation of each bit. .
[0048] The masked language model loss adopts the standard MLM strategy. The masked language model loss is the cross-entropy loss calculated at the mask position after performing random masking on the query sequence according to a preset mask ratio. The preset mask ratio is preferably 0.15.
[0049] In some embodiments, the covariant loss is a loss branch independent of the masked language model loss. The covariant loss is calculated based on the relationship between the latent representations of point pairs in the covariant point pair set, so that the latent representations of point pairs within the covariant point pair set are close to each other in the vector space.
[0050] In some embodiments, the covariant loss is calculated by minimizing the square of the cosine distance between the latent representations of site pairs within the covariant site pair set, such that the latent representations of site pairs within the covariant site pair set are close to each other in the vector space. Specifically, the top covariant site pairs are selected from the covariant site pair set in descending order of covariance score. Each locus pair is used as a computational locus pair for the calculation. Corresponding subset , subset For each site pair (i,j,s), the loss term is calculated. ,in and These are the hidden representations of the sequence encoder at sites i and j, respectively. This represents the covariance fraction of the locus pair. The hidden representation of site i The hidden representation of site j The cosine similarity.
[0051] The covariant loss is obtained by averaging the loss terms for all sites involved in the calculation and multiplying the result by a scaling factor. The scaling factor is determined by the technicians based on actual needs; for example, a scaling factor of 4 may be used.
[0052] It should be noted that the scale factor of 4 is merely an example value aligned with the numerical values in the embodiment and is not intended to limit the scope of the claims. Those skilled in the art can scale the scale factor proportionally based on training stability, exemplarily using 2 or 8. As an optional implementation, the covariant site pair set can be pre-trunculated, and all pairs can be used to calculate the covariant loss. The above substitution only changes the numerical scale or summation range of the covariant loss; it does not change the structural position of the covariant loss as an independent loss branch, multiplied by the covariant loss weights, and synthesized with the masked language model loss to form the total training loss.
[0053] The covariance loss is calculated as follows: , like If empty, then =0. The scaling factor 4 is an example of an implementation. The above covariant loss can be expressed in words as follows: Let the subset of site pairs prefixes involved in the calculation be... Its size is ; , These are the hidden representations of the sequence encoder at positions i and j, respectively. Let cosine similarity be the two values. This is the covariance score for that locus pair. If the prefix subset is empty, then the covariance loss is... Set to 0; otherwise, set each item in the prefix subset to 0. squared multiplied by Then sum them up, and then divide by . And multiply by a scaling factor. In this example, the scaling factor is 4, therefore Multiply by the sum of the above terms. The scale factor of 4 is used as an example; it can be replaced with other positive numbers as mentioned above, or it can be changed to the mean of all paired sets.
[0054] In some embodiments, the covariance loss is calculated via a paired scoring head that receives the implicit representation of the sequence encoder at point pairs i and j. and splicing ; As input, the output is the covariance prediction score of the locus pair, and the covariance loss is the difference between the covariance prediction score and the covariance score of the locus pair.
[0055] In some embodiments, through attention coupling computation, the covariance score of the site pairs in the covariance site pair set is used as the target value at the corresponding position in the attention matrix after averaging multiple attention heads in the last layer of the sequence encoder, and the covariance loss is the difference between the attention weight and the covariance score at the corresponding position in the attention matrix.
[0056] The covariance loss is calculated only for the first half of the high-confidence site pairs in the set of covariance site pairs arranged in descending order of covariance strength. This can further filter out noise interference from low-confidence covariance pairings, while reducing the computational cost of the covariance loss and shortening the training iteration time per step.
[0057] If a site in the set of covariant site pairs is masked, that site pair is not included in the covariant loss calculation; alternatively, a context-inference-based implicit representation is used to participate in the covariant loss calculation. This avoids interference from masked site implicit representation noise in the covariant loss calculation and improves the reliability of the covariant monitoring signal.
[0058] In some embodiments, the loss calculation of the masked language model employs a covariance-aware masking strategy. This can involve masking two sites simultaneously for site pairs (i,j) in the covariance site pair set, retaining at least one site as a context anchor; or masking both sites simultaneously for site pairs (i,j) in the covariance site pair set, forcing the sequence encoder to predict the masked amino acid based on long-range covariance relationships rather than local context. This allows for further enhancement of the model's ability to model long-range co-evolutionary relationships through adjustments to the masking strategy, adapting to different pre-training objectives.
[0059] In some embodiments, the loss calculation for the masked language model does not employ covariant-aware masking. Instead, it performs conventional random masking on the query sequence according to a preset mask ratio, and co-evolutionary injection is completed by the independent covariant loss branch and depth-adaptive external multiplication weighting. In other optional embodiments, the aforementioned mask constraints of not simultaneously masking a pair or simultaneously masking a pair can be used as an additional implementation method for the masked language model task, without changing the structure of the independent covariant loss branch and the effective depth-based external multiplication weighting.
[0060] The covariant supervision signal is independent of the masked encoding path, avoiding the limitation of the masked strategy on the injection range of co-evolutionary signals and improving the injection accuracy of long-range co-evolutionary constraints; the independent loss branch can schedule weights separately without changing the original training logic of the masked language model, reducing the parameter tuning complexity and loss coupling interference of dual-task training.
[0061] Step S4: Calculate the covariant loss weights based on the effective depth D using a piecewise function. .
[0062] Where the effective depth D is lower than the first threshold =0, when the effective depth D is higher than the second threshold. Take a preset saturation value, when the effective depth D is between the first threshold and the second threshold. As D monotonically decreases. The preset saturation value is set by technicians according to actual needs; for example, the preset saturation value is 1.
[0063] Specifically, the covariance loss weight is: , in, The first threshold, The second threshold, and < . and The value is set by technical personnel according to actual needs. For example, Take 20, If i is 100, then the transition region λ_ It equals (D-20) / 80.
[0064] The core design basis of this piecewise function is the "D-noise coupling" in the statistical properties of multiple sequence alignment (MSA): when When the number of sequences in multiple sequence alignment (MSA) is too small, it is a shallow MSA, the risk of false positives in mutual information MI ordination is high, and the noise in covariance estimation is large. In this case, hard shutting down the covariance branch is necessary. =0), causing the model to degenerate into pure MLM training, completely avoiding noise interference; when When the reliability of covariant estimation increases with D, it is a medium MSA, and the weights are linearly increased to gradually introduce covariant supervision; when When the covariance is deep MSA, the covariance estimation tends to be saturated and reliable, so the weight is set to 1 to make full use of the covariance signal.
[0065] In some embodiments, the piecewise function can also be a superposition of a smooth monotonic function and a hard-gated function, i.e., when D is below a first threshold. =0, when D is between the first threshold and the second threshold. Determined by a smooth monotonic function, the smooth monotonic function satisfies the condition that when D approaches the first threshold... Approaching zero, when D approaches the second threshold Smooth monotonic functions that approach 1 and are monotonically increasing in the intermediate region include the sigmoid function or the exponentially saturated function.
[0066] In some embodiments, the piecewise function may also be a piecewise constant function that keeps the shallow MSA closed, the medium MSA with intermediate constants, and the deep MSA with saturated weights.
[0067] In some embodiments, covariant loss weights For sample-level weights, each training sample has... The effective depth D of its corresponding multiple sequence alignment is calculated separately.
[0068] This application is approved by the When the coefficient of variation (COP) is 0, the gradient contribution of the covariant loss branch to the sequence encoder parameters is zero, the gradient generated by the covariant branch is 0, and the sequence encoder parameters are updated entirely by the MLM gradient. The backpropagation path of the noise gradient is completely cut off through the hard gating design.
[0069] In shallow MSA / orphan sequence scenarios, gradient propagation of high false positive covariant pairings is completely cut off to avoid interference from erroneous covariant signals with the main task of single-sequence modeling, thereby improving the numerical stability of the training process. Shallow-depth samples are degenerated into pure mask language model training, which aligns with the optimization objective of single-sequence pre-training, reducing the training convergence difficulty of shallow homologous sequence samples and reducing the number of training iterations.
[0070] The sample-level adaptive weights match the reliability of covariant signals at different MSA depths, keeping the total loss magnitude of samples at different depths within a consistent range and reducing the dispersion of training metrics across different depths. The weights in the medium MSA depth range monotonically increase with D, achieving a smooth transition in covariant supervision intensity, avoiding training oscillations caused by step weight switching, and improving the stability of the optimization process. The weights in the deep MSA range take saturation values, fully releasing the supervisory value of high-confidence co-evolutionary signals, without needing to further increase the weights with depth to cause loss imbalance, thus ensuring the training convergence efficiency of deep MSA samples.
[0071] Step S5: Weight the masked language model loss and the covariant loss according to the covariant loss weights. The weighted summation yields the total training loss, which is then backpropagated to update the sequence encoder parameters. Steps S1 to S5 are repeated across multiple protein samples in the training corpus until the maximum number of pre-training epochs is reached or the total training loss fails to decrease within a specified number of epochs. The total training loss function is: , in, For masked language model loss, For covariance loss, These are the weights for the covariant loss, i.e., the total training loss. Equal to masked language model loss Add covariance loss weights Covariance loss The product of.
[0072] During training, the effective depth D of multiple sequence alignment (MSA) varies for different protein samples, and each has its own... The results are also different. For multi-sequence alignment MSA samples with a large effective depth D, the covariance loss weight is high, while for multi-sequence alignment MSA samples with a small effective depth D, the covariance loss weight is low or zero. This achieves sample-level adaptive covariance supervision intensity control, avoiding training instability and excessive dispersion of indicators across depth levels caused by fixed covariance weights on shallow MSA / orphan sequences.
[0073] This application also provides a neural network multi-task inference method, which uses steps S1 to S5 of the aforementioned neural network multi-task training method for pre-training. After the pre-training is completed, the sequence encoder only receives the target protein query sequence as input during the inference stage, and outputs the hidden representation of each position in the target protein query sequence as the protein representation.
[0074] During the inference phase, steps S1 (multiple sequence alignment (MSA) construction and quality control), S2 (site pair covariance score calculation), S3 (covariance loss calculation), S4 (weight calculation), and S5 (parameter update) are omitted. Only a single target protein query sequence is input into the sequence encoder, and the hidden representations for each position are obtained through a single forward propagation. These hidden representations, having incorporated co-evolutionary information through covariance loss constraints during training, can be directly used for downstream tasks such as contact map prediction, mutation effect prediction, and functional annotation, without requiring preprocessing steps like MSA retrieval and mutual information calculation.
[0075] During the training phase (steps S1 to S5), Multiple Sequence Alignment (MSA) is only used to calculate the set of covariant site pairs and the covariant loss weights; it is not used as input to the sequence encoder, which always only receives the masked query sequence. Therefore, after removing the MSA preprocessing branch and the covariant loss branch during the inference phase, the encoder's forward propagation path remains completely unchanged, and the output latent representation still retains the co-evolutionary information learned during the training phase.
[0076] The inference phase eliminates the need for MSA retrieval, quality control, and covariant calculations, significantly reducing computational overhead and end-to-end latency for single-sample inference and improving inference throughput in high-concurrency scenarios. Model inference performance is unaffected by MSA quality and abundance, maintaining stable performance even in scenarios with shallow MSA, orphan sequences, or even no homologous sequences, thus expanding the model's applicability. With all training-specific modules removed, the number of model parameters and the size of the computation graph on the inference side are reduced, lowering memory usage and hardware compatibility barriers during deployment.
[0077] An example is the end-to-end pre-training process (S1 to S5) and experimental verification.
[0078] I. Experimental Dataset and Model Configuration
[0079] Pre-training dataset: Data is derived from Pfam-A seed multiple sequence alignment (downloaded in Stockholm format via EBI InterPro API), frozen on 2026-07-22. The subset contains N=30 real MSAs (all public seeds, no synthetic alignments). For each seed, the first L_max non-gap columns of the query sequence are taken. Seeds are mutually exclusive based on their natural seed size—deep (D>=100), medium (20<=D<100), and shallow (D<20), with 10 families in each category. During training, the number of homologous sequences retained is shallow / medium / deep = 10 / 40 / 120 (documented subset protocol).
[0080] Backbone model: A small-scale sequence encoder reproducible on CPU (non-large-scale pre-trained configuration). This model employs a sequence encoder architecture, learning residue-level hidden representations using only the query sequence as input.
[0081] Baseline scheme: fixed weight λ=1 (i.e., the covariant loss always participates in the total loss with full intensity).
[0082] This application proposes a segmented scheduling approach. (default =20, =100, saturation value=1).
[0083] II. S1: MSA Preprocessing and Depth Statistics
[0084] Step S1: Construct or read the query sequence and MSA of the target protein, and calculate the effective depth D according to the preset quality control criteria (gap filtering, column coverage filtering, identity redundancy reduction).
[0085] The specific process is as follows: Step 1.1: Obtain the query sequence and MSA.
[0086] Step 1.2: Quality control filtration pretreatment.
[0087] Step 1.3: Determine the effective depth D.
[0088] Step 1.4: Example of D-values for three sample tiers. In this N=30 subset, after mutually exclusive tiering by natural seed size: (1) Deep-file (10 families): D>=100, 120 homologous families are retained during training. The D value of all deep-file samples exceeds the second threshold D2=100, and subsequent... =1.
[0089] (2) Mid-range (10 families): 20 <= D < 100, 40 homologous families are retained during training. The D value of mid-range samples is between the first threshold D1 = 20 and the second threshold D2 = 100, and subsequently... It rises linearly with D.
[0090] (3) Shallow tier (10 families): D < 20, retain 10 homologous families during training. The D value of shallow tier samples is lower than the first threshold D1 = 20, and subsequent... =0, the covariant pairing set is set to empty (K=0), and the covariant loss is automatically zero.
[0091] This subset's measured average values for light / medium / dark levels The values are approximately 0 / 0.2475 / 0.9988, consistent with the segmented pattern.
[0092] III. S2: Covariant Paired Extraction
[0093] Step S2: Calculate the covariance scores of all site pairs in the MSA after quality control, and obtain the supervision set by taking the Top-L pairs in descending order of covariance scores.
[0094] The specific process is as follows: Step 2.1: Calculate mutual information (MI). The higher the MI value, the stronger the co-evolutionary association between the two sites, and the more likely they are to be spatially adjacent residue pairs in the three-dimensional structure of the protein.
[0095] Step 2.2: APC Correction (Optional). Perform average product correction (APC) on the mutual information MI to obtain MIp: .
[0096] Step 2.3: Select Top-L pairs. Sort by covariance score (MI or MIp) in descending order, and select the top K pairs as the covariance pair set. .
[0097] Step 2.4: Special handling for shallow depths. When the effective depth D is lower than the first threshold (D < 0.05), When K=20, the number of sequences in the MSA is too small, and the risk of false positives in MI / MIp sorting is extremely high—all paired covariance scores may be statistical noise rather than true covariance signals. In this case, setting the covariance pairing set to empty (K=0) automatically reduces the covariance loss to zero, and no covariance correlation calculations are performed. This design not only avoids noise interference but also saves computational overhead on shallow MSA samples.
[0098] IV. S3-S4: Dual-Task Loss Calculation
[0099] Steps S3-S4: Parallel computation of masked language model loss (reconstruction after random masking of query sequence) and covariant loss (representation similarity alignment of site pairs within the covariant pair set).
[0100] The specific process is as follows: Step 3.1: Masking and encoder forward propagation.
[0101] Step 3.2: Calculate the loss of the masked language model.
[0102] Step 3.3: Calculate the covariant loss. The covariant loss is a loss branch independent of the masked language model loss. Let P be the set of pairs ordered in descending order of covariance strength s (MI in this example), |P|=K. Take the size of the prefix subset. Let P' be the first K' terms of P. The covariance loss is calculated as follows: If P' is empty, then L_cov = 0; otherwise: The factor 4 and prefix truncation are scale choices consistent with the publicly available reproducible embodiments. Factor 4 is the reciprocal of the theoretical maximum value of cosine similarity alignment loss (1-(-1))^2=4, used to normalize the loss value to the [0,1] interval. Prefix truncation (taking K' to be approximately equal to K / 2) can further filter out noise interference from low-confidence covariant pairings, while reducing the computational cost of covariant loss.
[0103] Step 3.4: Processing of masked sites. If a site in the covariant pair set is masked, the pair is not included in the covariant loss calculation; or a context-inference-based implicit representation is used to participate in the covariant loss calculation.
[0104] Step 3.5: Independence of Covariant Supervision. The covariant supervision signal is independent of the masked encoding path, avoiding the limitation of the co-evolutionary signal injection range by the masking strategy. Independent loss branches can schedule weights independently without modifying the original training logic of the masked language model, reducing the parameter tuning complexity and loss coupling interference of dual-task training.
[0105] V. S4: Depth-Adaptive Weighted
[0106] Step S4: Calculate the covariant loss weights based on the effective depth D using a piecewise function. .
[0107] Default piecewise function (default) =20, =100): .
[0108] Example of lambda(D) calculation for three samples in this embodiment: (1) Deep sample (D>=100): =1, all covariant branches are open, total loss ; (2) Medium-sized sample (20<=D<100): The linear increase with D. The measured average λ for this subset is approximately 0.2475, with a total loss... Covariant supervision is introduced at approximately 25% intensity; (3) Shallow sample (D<20): =0, covariant branch closed, total loss It completely degenerates into pure MLM training to avoid noise covariance interference.
[0109] VI. S5-S6: Depth Weighting and Parameter Update
[0110] Steps S5-S6: Use External multiplication by the covariant loss term and then with The total loss is obtained by summing the results, and the encoder parameters are updated by backpropagation.
[0111] Total loss synthesis formula: .
[0112] VII. Experimental Results
[0113] Main metric definition: The cross-gear variance of the total loss of the three gears (shallow / medium / deep) and its relative reduction relative to the fixed arm on the N=30 subset (a surrogate metric, used only to measure the suppression of inter-gear dispersion by depth gating; it must not be rewritten as a claim of general protein language model accuracy / downstream task improvement).
[0114] Cross-range variance calculation method: Range mean total loss = average of the total loss of all samples within each depth range in the last epoch; Cross-range variance = sample variance of the three range mean total losses [shallow range total loss, medium range total loss, deep range total loss]; Relative reduction = (Var_fixed - Var_adaptive) / Var_fixed 100%, where Var_fixed is the cross-range variance of the fixed-weight scheme, and Var_adaptive is the cross-range variance of the depth-adaptive scheme of this invention.
[0115] Experiment 1: Fixed λ vs. adaptive scheduling.
[0116] Under the same N=30 subset and random seed protocol, comparing the fixed arm λ with {0, 0.5, 1} with the depth adaptive scheme of this application:
[0117] Results analysis: Even compared to "completely turning off covariance" (λ=0), the depth-adaptive scheme still reduces cross-level variance by approximately 80.92%—because moderate covariance supervision is introduced in the medium / deep levels without sacrificing stability in the shallow levels. This confirms that the gain comes from "depth-specific weighting" rather than accidentally turning off λ.
[0118] Experiment 2: Depth-level sensitivity.
[0119] Maintaining the "shallow closure - mid-layer ascent - deep saturation" pattern, scan the three-group thresholds:
[0120] Results analysis: All three groups outperformed the fixed lambda=1, with relative reductions of >=80%. The threshold can be replaced, but the core pattern of "shallow closure - mid-layer ascent - deep layer saturation" is indispensable.
[0121] Experiment 3: Consistency between average lambda and segmented pattern for each level.
[0122] The measured average λ for the shallow / medium / deep ranges of this subset is approximately 0 / 0.2475 / 0.9988, which is consistent with the default piecewise function ( =20, The shape of (=100) is consistent: (1) The average λ of the shallow range is about 0: the D values of the 10 families in the shallow range are all below 20. If the precision is 0, the covariant branch is completely closed.
[0123] (2) The average λ for the mid-range category is approximately 0.2475: The D values of the 10 mid-range families are distributed between 20 and 100. As D increases linearly, the intensity averages about 25%.
[0124] (3) The average λ for deep-grade data is approximately 0.9988: the D values of all 10 families in the deep-grade data exceed 100. =1, covariant branches are fully open. The average λ for deeper ranges is slightly lower than 1.0 because λ is not fully saturated when the D value of some families is close to 100.
[0125] This application also provides a neural network multi-task training system, such as Figure 2 ,include: The MSA preprocessing module is used to obtain the protein sample query sequence and its corresponding multiple sequence alignment MSA, perform quality control filtering preprocessing on the multiple sequence alignment MSA to obtain the quality control MSA, determine the effective depth D of multiple sequence alignment based on the quality control MSA, and calculate the covariance score of all site pairs in the quality control MSA, sort them in descending order of covariance score, and select the top ones. Each locus pair is considered as a set of covariant locus pairs, where Based on the length of the query sequence Sure; The sequence encoder module is used to receive the sequence after performing a random mask and output the implicit representation of each bit. The dual-task loss module is used to calculate the masked language model loss and the covariant loss separately. The depth-adaptive weighting module is used to calculate the covariant loss weights based on the effective depth D according to a piecewise function. ; The masked language model loss and covariant loss are arranged according to The total training loss is obtained by weighted summation, and then backpropagated to update the parameters of the sequence encoder.
[0126] In some embodiments, when the effective depth D is below a first threshold =0, when the effective depth D is higher than the second threshold. Take a preset saturation value, when the effective depth D is between the first threshold and the second threshold. As D becomes monotonically non-decreasing.
[0127] In some embodiments, the covariant loss is a loss branch independent of the masked language model loss. The covariant loss is calculated based on the relationship between the latent representations of point pairs in the covariant point pair set, so that the latent representations of point pairs within the covariant point pair set are close to each other in the vector space.
[0128] In some embodiments, the sequence encoder module is shared in both the training and inference phases, and in the inference phase, it receives only a single target protein query sequence and outputs the hidden representation of each position in the target protein query sequence as the protein representation.
[0129] In some embodiments, the neural network multi-task training system is used to implement Figure 1 Multi-task training methods for neural networks.
[0130] Based on the above-described neural network multi-task training method, this application embodiment also provides a neural network multi-task training electronic device, which can operate... Figure 1 The neural network multi-task training method shown includes a processor and a memory, wherein the processor and the memory can be connected via a bus or other means.
[0131] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.
[0132] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the neural network multi-task training method in the embodiments of this application. The processor executes various functional applications and data classification by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the neural network multi-task training method in the above method embodiments.
[0133] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0134] One or more modules are stored in memory and, when executed by a processor, perform the neural network multi-task training method in the embodiment.
[0135] This specification also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the steps of the protein language model pre-training method described above.
[0136] This specification also provides a computer program product comprising a computer program or instructions that, when executed by a processor, implement the steps of the aforementioned neural network multi-task training method.
[0137] The aforementioned neural network multi-task training method can be implemented as a computer program, stored on a hard disk, and loaded into a processor for execution to implement the neural network multi-task training method of this application.
[0138] This application also includes a computer-readable medium storing computer program code that, when executed by a processor, implements the aforementioned neural network multi-task training method.
[0139] When a neural network multi-task training method is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.
[0140] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.
[0141] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).
[0142] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.
[0143] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.
Claims
1. A method for multi-task training of neural networks, characterized in that, Includes the following steps: S1: Obtain the protein sample query sequence and its corresponding multiple sequence alignment (MSA), perform quality control filtering preprocessing on the MSA to obtain a quality-controlled MSA, and determine the effective depth D of multiple sequence alignment based on the quality-controlled MSA; S2: Calculate the covariance scores of all locus pairs in the post-quality control MSA, sort them in descending order of covariance scores, and select the top [percentages]. Site pairs are considered as a set of covariant site pairs, where Based on the length of the query sequence Sure; S3: Perform random masking on the query sequence to obtain the masked sequence, input the masked sequence into the sequence encoder to obtain the hidden representation of each bit of the query sequence, and calculate the masked language model loss and covariance loss respectively; S4: Calculate the covariance loss weight based on the effective depth D using a piecewise function. ; S5: Combine the masked language model loss and the covariance loss according to the weight of the covariance loss. The total training loss is obtained by weighted summation and backpropagation of the total training loss to update the parameters of the sequence encoder. The multiple protein samples in the training corpus are traversed, and steps S1 to S5 are repeated until the maximum number of pre-training rounds is reached or the total training loss does not decrease within the specified number of rounds.
2. The neural network multi-task training method according to claim 1, characterized in that, In step S4, the covariance loss weight is calculated based on the effective depth D using a piecewise function. Including: when the effective depth D is lower than the first threshold =0, when the effective depth D is higher than the second threshold. A preset saturation value is taken, and when the effective depth D is between the first threshold and the second threshold... As D becomes monotonically non-decreasing.
3. The neural network multi-task training method according to claim 2, characterized in that, The covariance loss weight in S4 is: , in The first threshold, The second threshold, and < .
4. The neural network multi-task training method according to claim 2, characterized in that, The piecewise function in S4 can also be a superposition of a smooth monotonic function and a hard-gated function, that is, when D is lower than the first threshold. =0, when D is between the first threshold and the second threshold. Determined by a smooth monotonic function, the smooth monotonic function satisfies the condition that when D approaches the first threshold... Approaching zero, when D approaches the second threshold The smooth monotonic function approaches 1 and monotonically increases in the intermediate region, including the sigmoid function or the exponentially saturated function.
5. The neural network multi-task training method according to claim 1, characterized in that, In S5, the covariant loss is a loss branch that is independent of the masked language model loss. The covariant loss is calculated by minimizing the square of the cosine distance between the latent representations of the site pairs within the covariant site pair set, so that the latent representations of the site pairs within the covariant site pair set are close to each other in the vector space.
6. The neural network multi-task training method according to claim 5, characterized in that, The covariance loss in S3 is calculated as follows: Based on the set of covariant site pairs, determine the number of site pairs K^' participating in the calculation, and calculate the loss term for each site pair (i,j). ,in and These are the hidden representations of the last layer of the sequence encoder at sites i and j, respectively. The covariance fraction of that locus pair; The covariant loss is obtained by averaging the loss terms for all sites involved in the calculation and multiplying them by a scaling factor.
7. The neural network multi-task training method according to claim 6, characterized in that, The number of loci involved in the calculation The first half of the set of covariant site pairs, arranged in descending order of covariance fraction, i.e. ,in, ,in This is the preset scaling factor.
8. The neural network multi-task training method according to claim 1, characterized in that, In S3, the covariant loss is calculated using paired scoring heads or attention coupling. The calculation via the pairing scoring head includes receiving the concatenation of the latent representation of the site pair by the sequence encoder as input, and outputting the covariance prediction score of the site pair, wherein the covariance loss is the difference between the covariance prediction score and the covariance score of the site pair. The attention coupling calculation includes using the covariance score of the site pairs in the covariance site pair set as the target value at the corresponding position in the attention matrix after averaging multiple attention heads in the last layer of the sequence encoder, and the covariance loss is the difference between the attention weight at the corresponding position in the attention matrix and the covariance score.
9. The neural network multi-task training method according to claim 1, characterized in that, The covariance fraction in S2 is calculated using mutual information MI or mutual information MIp corrected for average product; for each site pair (i,j) in the quality control MSA, the joint occurrence frequency matrix of standard amino acids is statistically analyzed. Where a and b represent the types of amino acids appearing at site i and site j, and the frequency of marginal amino acids at site i, respectively. and the edge amino acid frequency of site j , The mutual information MI is: ; The mutual information MIp corrected by average product is: 。 10. The neural network multi-task training method according to claim 1, characterized in that, In step S3, if a site in the set of covariant site pairs is masked, then the site pair does not participate in the calculation of the covariant loss; or a context-based implicit representation is used to participate in the calculation of the covariant loss.
11. The neural network multi-task training method according to claim 1, characterized in that, The function of the total training loss is: , in, For masked language model loss, For covariance loss, The weights are for covariant loss.
12. A neural network multi-task inference method, characterized in that, This includes pre-training using the neural network multi-task training method described in any one of claims 1 to 8. After pre-training, the sequence encoder only receives the target protein query sequence as input during the inference phase, and outputs the hidden representation of each position in the target protein query sequence as the protein representation.
13. A neural network multi-task training system, characterized in that, include: The MSA preprocessing module is used to obtain the protein sample query sequence and its corresponding multiple sequence alignment (MSA), perform quality control filtering preprocessing on the MSA to obtain a quality-controlled MSA, determine the effective depth D of multiple sequence alignment based on the quality-controlled MSA, and calculate the covariance score of all site pairs in the quality-controlled MSA, sorting them in descending order of covariance score and selecting the top ones. Site pairs are considered as a set of covariant site pairs, where Based on the length of the query sequence Sure; The sequence encoder module is used to receive the sequence after performing a random mask and output the implicit representation of each bit. The dual-task loss module is used to calculate the masked language model loss and the covariant loss separately. The depth-adaptive weighting module is used to calculate the covariant loss weights based on the effective depth D according to a piecewise function. The masked language model loss and the covariant loss are then compared... The total training loss is obtained by weighted summation, and the total training loss is backpropagated to update the parameters of the sequence encoder.
14. An electronic device, characterized in that, It includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.
16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.
