A KASP primer intelligent typing evaluation method and system TAL-SRX based on multi-model fusion

Abstract

The present invention relates to the field of artificial intelligence technology, and specifically provides a KASP primer intelligent typing evaluation method and system TAL-SRX based on multi-model fusion. First, the RS, PSO and BO algorithms are used to optimize the performance of the basic model and meta-model, and a Stacking integrated learning framework is constructed to integrate the advantages of the XGBoost learner, and a five-fold cross validation is used to enhance the stability of the model. Then, ANN and LSTM are combined to construct a hybrid neural network to capture nonlinear relationships and extract complex features, and the Transformer algorithm is introduced to capture global dependencies in high-dimensional feature space. Finally, the two machine learning algorithms are fused through a soft voting integration strategy to output the KASP marker typing effect score. The present invention can accurately screen KASP markers with excellent typing effects, has high robustness and good and reliable evaluation performance, and provides technical support for molecular marker-assisted breeding and other work.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence and relates to a KASP primer intelligent typing evaluation method and system based on multi-model fusion. Background Art

[0002] Competitive allele-specific PCR (KASP) technology can accurately identify the double allele genotypes of SNPs (single nucleotide polymorphism) at specific sites in different species genome sample types. Due to its unique advantages of flexibility, high efficiency and low cost, it is widely used in molecular marker-assisted selection breeding, quality testing, variety identification and stress resistance assessment. However, the amplification and separation of population genotypes are complex and changeable, and the evaluation of typing results directly affects the efficiency of KASP marker development. Therefore, it is necessary to achieve intelligent and accurate evaluation of the relative independence of population genotyping results, so as to screen excellent KASP markers on a large scale and improve the efficiency of marker development.

[0003] So far, there are three main methods for evaluating typing results in studies using competitive allele-specific PCR technology. The most commonly used method is manual visual inspection of genotyping conditions, which is mainly observed and identified by agricultural experts or technicians. Due to the flexible and changeable performance of typing result graphs, applying this method to breeding practice requires professionals to have long-term and rich experience in typing graph interpretation, and it takes a lot of time to select KASP markers with good typing. Therefore, the evaluation process is accompanied by problems such as long time consumption, strong subjectivity, and large-scale material verification. For example, Yu Shizhou et al. directly observed the fluorescence typing status of different color origins in the KASP fluorescence detector in the screening and verification test of candidate core markers for tobacco variety genotyping, and used the SNP sites with good subjective typing directly as molecular markers in subsequent studies; Schoonmaker et al., in the verification of cotton leaf roll virus resistance gene-associated markers, combined with observation results, proposed that the separation of homozygous and heterozygous populations and the tight morphology within the cluster can prove that the marker is good; Zhao Chuanchao et al., in the rice blast resistance gene Pi2 KASP marker development test, because the visual typing results were not clear enough, further set up negative and positive controls, and preliminarily judged that the marker was feasible after observation. However, because the typing results were relatively scattered, 4 cycles were added to the original detection procedure, and finally a more intuitive and clear KASP genotyping result was obtained; Ruslan et al., in the test of allele identification of barley varieties with known genotypes, used grid lines parallel to the horizontal and vertical axes to segment the KASP genotyping map, and observed whether the homozygous and heterozygous genotypes were located in different segmented areas in the map to evaluate the accuracy and effectiveness of the marker. The second category is the method of quantitative evaluation of index values. With the development of new SNP genotyping technology, some scholars have proposed the use of variance analysis to quantitatively evaluate the difference in index values ​​in comparative tests of KASP and TaqMan technologies. However, the standard of using the relative high and low index values ​​to judge the typing effect proposed by this method is relatively vague, and its application scope in KASP tests is limited. The third category is the traditional machine learning method. Usually, the sample size of SNP genotyping result data is large, the data dimension is high, and it has the complexity of nonlinear relationships. Compared with statistical analysis using a small number of indicators, machine learning, as a powerful data-driven framework, is more suitable for providing accurate solutions for the complex relationship between a large number of variables in KASP test results. For example, Chen et al. proposed a KASP marker primer intelligent typing evaluation model. The model is based on the typing effect grade evaluation standard, introduces the K-Means clustering algorithm in the design module to fit the gene population aggregation and classification effect, and finally realizes the intelligent typing result evaluation through the logical decision tree algorithm.

[0004] However, there are relatively few studies on KASP intelligent typing evaluation, and the existing typing effect evaluation standards lack a more detailed classification level, which may lead to a low recognition rate of good markers in large-scale screening, resulting in a large amount of financial and material resources. These factors have jointly limited the application of KASP technology in assisted breeding. Secondly, shallow learning models still have bottlenecks in processing big data. Given that deep learning is one of the most popular data-driven methods, applying it to automatically extract and learn the intrinsic characteristics of KASP test result data may be a beneficial exploration. Summary of the invention

[0005] Based on the above reasons, the present invention proposes a KASP primer intelligent typing evaluation method and system based on multi-model fusion. An integrated learning framework based on the Stacking algorithm is used to comprehensively apply multiple heterogeneous base learners, and a two-layer structure is constructed to combine and train with the eXtreme Gradient Boosting (XGBoost) model to drive data while improving the prediction accuracy of the model. In addition, two deep learning models are selected and weighted and introduced into the integrated learning framework to further enhance the system's ability to master multidimensional features under complex task conditions. In a specific embodiment, the present invention performs a performance test on the system through the KASP marker typing test results of 3399 groups of cotton variety resource materials to verify the effectiveness of the proposed method.

[0006] The multi-model fusion evaluation method provided by the present invention makes full use of the respective advantages of traditional machine learning algorithms and deep learning algorithms. First, three traditional machine learning algorithms are connected using the "basic model-meta-model" method. In order to accelerate the parameter search process and maximize the performance of each learner, different optimization algorithms are used for the unique tuning requirements of different learners. In the Stacking integrated learning framework based on the hyperparameter optimization algorithm, the heterogeneous learner parses the features of different dimensions in the data set, generates new feature variables with fewer dimensions, and the meta-learner integrates the data to predict the probability. Then the deep learning algorithm is introduced, and the data output after ANN processing is passed to the LSTM layer, which flexibly captures complex features while enhancing the model's adaptability to heterogeneous samples. Then, the Transformer model based on the self-attention mechanism is added to capture the global dependency between input variables and enhance the model robustness. Finally, the voting integration model extracts the shared features of the output data of each basic algorithm to obtain the evaluation results of the KASP primer typing effect. Figure 2 The process of the KASP primer typing effect evaluation method is described.

[0007] 1. Stacking ensemble learning evaluation model based on hyperparameter optimization algorithm

[0008] Stacking ensemble is a powerful and widely used ensemble learning technology. The basic principle of the stacking model of the present invention is to make predictions through a two-layer nested model structure, train two base learner models, and use their prediction results as the input of the meta-learner model to fully mine the original data set and comprehensively play the learning advantages of each base learner, thereby improving the prediction accuracy. For each sub-model, the present invention uses an optimization algorithm to improve the model structure to find the optimal hyperparameter combination, and constructs three improved models, RS-RF, PSO-SVM and BO-XGBoost, respectively. The hyperparameter combination with the best performance is selected to construct a Stacking ensemble learning evaluation model ( Figure 2 ).

[0009] 2. RF model based on RS optimization

[0010] Random Search (RS) randomly samples multiple parameter combinations in a predefined parameter space and evaluates the performance of each combination to find the best model parameters. It does not require gradient information and can explore globally to avoid local optimality.

[0011] RF is an algorithm that integrates multiple decision trees based on the idea of ​​bagging. In RF, many trees are built using randomly selected training data sets and random subsets of predictor variables. The results of each tree are aggregated using the absolute majority voting method to obtain the predicted category. This paper optimizes the modeling process of RF based on RS, extracts sample subsets through the Bootstrap method, randomly selects a subset from the sample features at each node split to find the best split point, and uses these sample subsets and feature subsets to build multiple unpruned decision trees. The model performance is evaluated in each iteration to select the best hyperparameter configuration. The formula for the absolute majority voting strategy is:

[0012] (1)

[0013] Where: h i is the base learner, i.e., the decision tree, c j For category label, T is the total number of base learners, yes h i In category tag c j The output on N for h i In the sample x The predicted output on total.

[0014] The segmentation features and segmentation points of each tree in RF are different. Compared with a single decision tree model, it has stronger nonlinear data processing capabilities and anti-overfitting capabilities. At the same time, random search can help the RF model better adapt to the data set and improve its robustness and generalization capabilities.

[0015] 3. SVM model based on PSO optimization

[0016] The Particle Swarm Optimization (PSO) algorithm originates from the phenomenon of bird flock foraging. Its essence is to simulate the intelligent behavior of the group. Through information sharing and mutual learning between individuals and groups, each particle adjusts its position and speed in the search space and gradually finds the optimal solution to the problem.

[0017] The core idea of ​​SVM is to achieve data classification by solving the maximum margin hyperplane in the feature space. This method is called maximum margin classification. When faced with linearly inseparable data, SVM maps the data to a higher-dimensional space through the kernel method, making it linearly separable in this new space. This paper uses the PSO algorithm to optimize the parameters of the SVM modeling process. By evaluating the fitness of each particle, the optimal position of the individual and the group is updated in each iteration to determine the optimal parameter model. The decision function of SVM can be expressed as:

[0018] (2)

[0019] Where: x is the input feature vector, ϕ(x) The input feature vector x feature maps mapped to a high-dimensional space (defined by a kernel function), w is the normal vector (weight vector) of the hyperplane found in the feature space, b is the bias term, the inner product w⋅ϕ(x) Represents the projection in the feature space.

[0020] SVM has excellent adaptability to class imbalance problems with large feature differences. The PSO algorithm has excellent performance with simple implementation, high accuracy and fast convergence, which can help SVM improve its processing capabilities for high-dimensional complex data sets.

[0021] 4. XGBoost model based on BO optimization

[0022] Bayesian Optimization (BO) has emerged at the forefront of black-box optimization methods due to its high efficiency in finding the global optimal solution in fewer times. Its advantage is that it can infer the posterior distribution of the objective function based on prior information and observed results, and achieve a good balance between exploration and development in the search process.

[0023] XGBoost is a scalable end-to-end tree boosting technology in the Boosting ensemble learning model. Its objective function consists of two parts: loss function and regularization term. The core idea of ​​the model is to measure the deviation through the loss function, and use the regularization term to control the complexity of the model while reducing the deviation to avoid overfitting. This paper uses BO to optimize the modeling process of XGBoost. By evaluating the performance of the sampling points, the Gaussian process proxy model is updated in each iteration to select the optimal parameter combination, thereby gradually approaching the optimal solution of the unknown objective function. The objective function is defined as follows:

[0024] (3)

[0025] In the formula, l is the loss function, Ω is the regularization term, y i is the true value, is the predicted value, f k Represents each tree.

[0026] XGBoost integrates multiple weak learners into a strong learner, which has higher computing speed and better model performance. BO optimization can improve the stability of the XGBoost model on the data set and reduce the risk of overfitting or underfitting.

[0027] 5. Voting integration with deep learning algorithms

[0028] Since the statistical data of KASP marker typing reports are produced from different batches, in order to avoid the threat of genetic data heterogeneity to model robustness, an integration method that introduces deep learning models ANN-LSTM and Transformer on the basis of traditional machine learning algorithm integration is proposed.

[0029] 1) ANN-LSTM: ANN is a type of computational model inspired by biological neural networks and has been widely used in tasks such as data classification. In combination with LSTM, we constructed a hybrid neural network with ANN as the feedforward neural network layer, which comprehensively utilizes the nonlinear relationship capture ability of ANN and the complex data modeling ability of LSTM. In the forward propagation process, the input data first passes through the ANN layer, which uses ReLU as the activation function for nonlinear transformation to increase the expressiveness of the model. The data is initially extracted by 128 hidden units in this layer and then the shape output is adjusted. The LSTM layer also has 128 units. Through the memory unit and forget gate mechanism, it can effectively capture the inherent complex structure of the data. Finally, the output of LSTM is mapped to the target category space. In the back-propagation process, the cross entropy loss function is calculated, and the model parameters are updated in the training rounds (epochs) through the Adam optimizer. After training, the model is predicted on the test set and the probability distribution of each score is calculated.

[0030] 2) Transformer: Transformer is a neural network based mainly on the self-attention mechanism, which is used to capture the global dependencies between input features and focus on the key details of the data. In this architecture, an encoder with feature extraction capabilities is first defined, and the input data is processed through three encoder layers to increase the depth of the model. During the forward propagation process, the self-attention mechanism and the feedforward neural network process the data to capture higher-level relationships and extract more complex features. In order to prevent the model from overfitting, the Dropout layer is added to randomly discard some neurons, and finally the processed features are mapped to the target category space through the linear layer. During the training process, the model calculates the cross entropy loss through backpropagation and updates the model parameters using the Adam optimizer. In the test phase, the model calculates the score category probability of each sample through the softmax function and outputs it.

[0031] The prediction probabilities of the three models, ANN-LSTM, Transformer and Stacking, were weighted averaged to obtain the evaluation results of the KASP primer typing effect. Through this method, we obtained a powerful and stable system that includes multiple heterogeneous base learners. Compared with a single prediction model, TAL-SRX has the ability to adapt to different scenarios and can therefore obtain better prediction results.

[0032] The technical solution provided by the present invention is as follows:

[0033] The first aspect of the present invention provides a KASP primer intelligent typing evaluation method based on multi-model fusion, comprising the following steps:

[0034] (1) Obtain the original data set to construct a data set, and divide the data set into a training set and a test set;

[0035] (2) Constructing a Stacking ensemble learning evaluation model: The training set described in step (1) is subjected to five-fold cross validation for training base models, wherein the base models are RF and SVM models. During the training process, RS and PSO are used to optimize the hyperparameters of the RF and SVM model training processes, respectively. The five-fold cross validation results of the RF and SVM models are combined and spliced ​​into new meta-model feature variables and their original labels for further training of the meta-model XGBoost, and BO is used to optimize the hyperparameters of the training process. After the training is completed, the RF and SVM models are used to predict the test set, and the prediction results are input into the meta-model and then output to obtain the final KASP marker typing effect score probability prediction value;

[0036] (3) Constructing ANN-LSTM model and Transformer model respectively: Using hybrid neural network technology, the advantages of ANN in capturing nonlinear relationships and LSTM in extracting complex features are integrated. At the same time, the Transformer algorithm is introduced to capture global dependencies in high-dimensional feature space, alleviating the impact of data noise and local anomalies in different batches.

[0037] After the ANN-LSTM model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided;

[0038] After the Transformer model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided;

[0039] (4) Through the soft voting integration strategy, based on the weighted weights, the probability prediction average of stacking, ANN-LSTM, and Transformer is calculated, and the underlying integration and deep learning algorithms are integrated to output the final effect score of KASP marker typing.

[0040] Furthermore, in step (1), the acquisition of the original data set to construct the data set is specifically to select the statistical data of the KASP marker typing report of cotton variety resource materials produced by the LGC high-throughput genotyping detection platform SNPline, extract the statistical data of a group of 94 DNA samples and 2 NTC (negative control reaction without adding DNA samples in PCR detection) detection data to establish the original data set, take the SNP site number, HEX fluorescence signal relative value X, FAM fluorescence signal relative value Y and sample number in the original data set to apply the KASP-IEva model for typing, and use the expert's typing effect score result as the standard. Finally, the SNP site number is used as the number of each group of data, and the HEX fluorescence signal relative value X and FAM fluorescence signal relative value Y of each group are combined and reconstructed into 192 high-dimensional feature variables, and the expert score is used as the label category to form a model data set, of which 84% is used as a training set and 16% is used as a test set.

[0041] Furthermore, in step (2), the implementation steps of the Stacking ensemble learning evaluation model are as follows:

[0042] a. The training set S = {(yn, xn), n = 1, ..., N} is preprocessed using the five-fold cross-validation method, and the data set S is randomly divided into five equal-sized and non-intersecting subsets S1, S2, ..., S5;

[0043] b. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the PSO-SVM model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z1.

[0044] c. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the RS-RF model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z2.

[0045] d. Combine the prediction results of the base learners in the five-fold cross validation and horizontally splice them into a new feature variable Z={Z1, Z2}, thereby realizing the feature transformation from the base learner to the meta-model. Use these new features and the original label yn to train the meta-model;

[0046] e. After the model training is completed, the base model is used to predict the test set, and the prediction results are input into the meta-model to obtain the final KASP primer typing effect evaluation results.

[0047] Furthermore, in step (3), the implementation steps of the ANN-LSTM model are as follows:

[0048] In the forward propagation process, the input data first passes through the ANN layer, which uses ReLU as the activation function to perform nonlinear transformation to increase the expressive power of the model. 128 hidden units are set. After the data undergoes preliminary feature extraction in this layer, the shape is adjusted and then passed through the LSTM layer. The LSTM layer also has 128 units. Through the memory unit and forget gate mechanism, it can effectively capture the inherent complex structure of the data. Finally, the output of the LSTM is mapped to the target category space. In the back propagation process, the cross entropy loss function is calculated, and the model parameters are updated in the training rounds through the Adam optimizer. After the training is completed, the ANN-LSTM model makes predictions on the test set and calculates the probability distribution of each score.

[0049] Furthermore, in step (3), the implementation steps of the Transformer model are as follows:

[0050] In the Transformer model, an encoder with feature extraction capability is first defined, and the input data is processed through three encoder layers to increase the depth of the model; in the forward propagation process, the self-attention mechanism and the feedforward neural network process the data to capture higher-level relationships and extract more complex features; in order to prevent the model from overfitting, the Dropout layer is added to randomly discard some neurons, and finally the processed features are mapped to the target category space through the linear layer; during the training process, the model calculates the cross entropy loss through backpropagation and uses the Adam optimizer to update the model parameters. In the test phase, the Transformer model calculates the score category probability of each sample through the softmax function and outputs it.

[0051] The second aspect of the present invention provides a KASP primer intelligent typing evaluation system TAL-SRX based on multi-model fusion, the TAL-SRX system comprising: a memory, a processor, the memory comprising a KASP primer intelligent typing evaluation method program based on multi-model fusion, the KASP primer intelligent typing evaluation method program based on multi-model fusion being executed by the processor to implement the following steps:

[0052] (1) Obtain the original data set to construct a data set, and divide the data set into a training set and a test set;

[0053] (2) Constructing a Stacking ensemble learning evaluation model: The training set described in step (1) is subjected to five-fold cross validation for training base models, wherein the base models are RF and SVM models. During the training process, RS and PSO are used to optimize the hyperparameters of the RF and SVM model training processes, respectively. The five-fold cross validation results of the RF and SVM models are combined and spliced ​​into new meta-model feature variables and their original labels for further training of the meta-model XGBoost, and BO is used to optimize the hyperparameters of the training process. After the training is completed, the RF and SVM models are used to predict the test set, and the prediction results are input into the meta-model and then output to obtain the final KASP marker typing effect score probability prediction value;

[0054] (3) Constructing ANN-LSTM model and Transformer model respectively: Using hybrid neural network technology, the advantages of ANN in capturing nonlinear relationships and LSTM in extracting complex features are integrated. At the same time, the Transformer algorithm is introduced to capture global dependencies in high-dimensional feature space, alleviating the impact of data noise and local anomalies in different batches.

[0055] After the ANN-LSTM model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided;

[0056] After the Transformer model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided;

[0057] (4) Through the soft voting integration strategy, based on the weighted weights, the probability prediction average of stacking, ANN-LSTM, and Transformer is calculated, and the underlying integration and deep learning algorithms are integrated to output the final effect score of KASP marker typing.

[0058] Furthermore, in step (2), the implementation steps of the Stacking ensemble learning evaluation model are as follows:

[0059] a. The training set S = {(yn, xn), n = 1, ..., N} is preprocessed using the five-fold cross-validation method, and the data set S is randomly divided into five equal-sized and non-intersecting subsets S1, S2, ..., S5;

[0060] b. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the PSO-SVM model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z1.

[0061] c. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the RS-RF model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z2.

[0062] d. Combine the prediction results of the base learners in the five-fold cross validation and horizontally splice them into a new feature variable Z={Z1, Z2}, thereby realizing the feature transformation from the base learner to the meta-model. Use these new features and the original label yn to train the meta-model;

[0063] e. After the model training is completed, the base model is used to predict the test set, and the prediction results are input into the meta-model to obtain the final KASP primer typing effect evaluation results.

[0064] Furthermore, in step (3), the implementation steps of the ANN-LSTM model are as follows:

[0065] In the forward propagation process, the input data first passes through the ANN layer, which uses ReLU as the activation function to perform nonlinear transformation to increase the expressive power of the model. 128 hidden units are set. After the data undergoes preliminary feature extraction in this layer, the shape is adjusted and then passed through the LSTM layer. The LSTM layer also has 128 units. Through the memory unit and forget gate mechanism, it can effectively capture the inherent complex structure of the data. Finally, the output of the LSTM is mapped to the target category space. In the back propagation process, the cross entropy loss function is calculated, and the model parameters are updated in the training rounds through the Adam optimizer. After the training is completed, the ANN-LSTM model makes predictions on the test set and calculates the probability distribution of each score.

[0066] Furthermore, in step (3), the implementation steps of the Transformer model are as follows:

[0067] In the Transformer model, an encoder with feature extraction capability is first defined, and the input data is processed through three encoder layers to increase the depth of the model; in the forward propagation process, the self-attention mechanism and the feedforward neural network process the data to capture higher-level relationships and extract more complex features; in order to prevent the model from overfitting, the Dropout layer is added to randomly discard some neurons, and finally the processed features are mapped to the target category space through the linear layer; during the training process, the model calculates the cross entropy loss through backpropagation and uses the Adam optimizer to update the model parameters. In the test phase, the Transformer model calculates the score category probability of each sample through the softmax function and outputs it.

[0068] The third aspect of the present invention provides a refined KASP primer typing effect evaluation standard, as shown in Table 1, setting a score range from 100 to 0, indicating that the primer combination morphology ranges from the best to the worst. Specifically, when the competitive primer combination morphology is manifested as independent aggregation of homozygous and heterozygous genotypes, and the homozygous genotype is located at the maximum value of the two coordinate axes, and the heterozygous genotype is located in the center of the line connecting the two homozygous genotypes, the score is 100, and as the competitive primer combination morphology shows that the degree of independent aggregation of homozygous and heterozygous genotypes decreases, the separation of genotypes decreases, and the score gradually decreases until it is completely relatively diffuse and cannot be accurately typed, and the score is 0. This standard provides a quantitative method for evaluating the competitive primer combination morphology, and provides a powerful tool for analyzing the amplification efficiency, specificity and combination competitiveness of genotypes.

[0069] Table 1 Evaluation criteria for KASP primer typing effect

[0070]

[0071] Beneficial effects of the present invention:

[0072] In order to improve the accuracy of large-scale screening and identification of good markers, the present invention adopts six machine learning algorithms, namely support vector machine (SVM), random forest (RF), extreme gradient boosting (XGBoost), artificial neural network (ANN), long short-term memory network (LSTM) and Transformer, and combines the above algorithms through two integration strategies, stacking and soft voting, to construct a hierarchical hybrid learning method driven by statistical data of KASP marker typing report. This method overcomes the defects of a single learning model, makes full use of the respective advantages of different deep learning models and traditional machine learning algorithms, enhances the functional properties of high-dimensional data feature analysis, and has a powerful, stable and comprehensive learning ability.

[0073] (1) A KASP primer intelligent typing evaluation method and system TAL-SRX based on multi-model fusion are provided, the main goal of which is to intelligently and accurately evaluate the typing effect of KASP markers. The input of the system TAL-SRX is the relative value of the fluorescence signal magnitude, and the output is the typing effect score. The system adopts two integrated strategies, stacking and soft voting, to overcome the defects of a single model and improve the robustness and accuracy of model evaluation. At the same time, the evaluation criteria for the typing effect of KASP primers are refined, and a competitive allele distribution morphology evaluation system with a score of 0-100 is constructed. This evaluation standard provides a powerful quantitative tool for analyzing the amplification efficiency, specificity and combination competitiveness of genotypes. It is not only applicable to cotton KASP test results, but also can provide reference for other variety resources, and can be used to screen excellent KASP markers that meet the characteristics of different crops.

[0074] (2) In the multi-model fusion method proposed in the present invention, the Stacking ensemble learning model is connected using the "basic model-metamodel" method to give full play to the respective advantages of heterogeneous learners. Different optimization algorithms are used for different learners to accelerate the parameter search process and maximize the performance of each model. The training process uses a five-fold cross-validation to enhance the model stability. The ANN-LSTM hybrid neural network model combines the ability to capture nonlinear relationships and the ability to extract complex features, enhancing the model's adaptability to heterogeneous samples. The Transformer model is based on the self-attention mechanism to capture global dependencies in high-dimensional feature space, thereby improving the robustness of TAL-SRX.

[0075] (3) The TAL-SRX system provided was trained and tested using the statistical data of KASP marker typing reports of 3399 groups of cotton variety resource materials. First, the performance of the TAL-SRX system of the present invention was compared with the basic model. The evaluation error rate and error value of the present invention were lower than those of the basic model, with an accuracy rate of 92.83% and an AUC value of 0.9905. It has high accuracy and stability, and its performance is significantly better than that of a single model. Secondly, the influence of each algorithm on the prediction performance of the model was studied by ablation comparison in an integrated combination manner. The experimental results show that the two integrated strategies can enhance the performance of the model, and different sub-algorithms have a positive effect on the model. Finally, the TAL-SRX system was compared with the expert scoring. The number of samples in each score segment of the evaluation results of the present invention is highly consistent with the expert evaluation. Therefore, the present invention has good evaluation performance.

[0076] (4) This paper explores an intelligent KASP primer typing effect evaluation method using deep learning, stacking integration and secondary integration, and verifies its effectiveness through experiments, providing the possibility for the practical application of this algorithm in the KASP typing data of various crop variety resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Construct a flowchart for the dataset.

[0078] Figure 2 This is the overall architecture diagram of the TAL-SRX multi-model fusion method.

[0079] Figure 3 Evaluation model for Stacking ensemble learning.

[0080] Figure 4 Error heatmap of the base model and the TAL-SRX system.

[0081] Figure 5 ROC curves for the base model and the TAL-SRX system.

[0082] Figure 6 Kappa coefficient for the base model and the TAL-SRX system.

[0083] Figure 7 Performance comparison of different integrated combinations.

[0084] Figure 8 Comparison of the TAL-SRX system and expert scoring results. DETAILED DESCRIPTION

[0085] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0086] Example 1 KASP primer typing effect evaluation test

[0087] 1 Materials and methods

[0088] 1.1 Test data

[0089] 1.1.1 Raw Data

[0090] The statistical data of KASP marker typing report of cotton variety resource materials produced by LGC high-throughput genotyping detection platform SNPline were selected. The resource materials include resource varieties, line materials, approved varieties and genetic isolation population materials. The statistical data of genotyping report came from Cotton Research Institute of Chinese Academy of Agricultural Sciences and Cotton Quality Supervision, Inspection and Testing Center of Ministry of Agriculture and Rural Affairs. It included 319 test results of different SNPs and different DNA samples from 2019 to 2023. Due to the different arrangement of samples on the motherboard of each test, we extracted statistical data in the format of 94 DNA samples and 2 NTC (negative control reaction without adding DNA samples in PCR detection) test data to establish the original data set, and a total of 3399 sets of statistical data were obtained.

[0091] 1.1.2 Evaluation criteria for KASP primer typing effect

[0092] Based on a large amount of experimental data, the evaluation criteria for KASP primer typing effect were divided in detail, as shown in Table 1.

[0093] Table 1 Evaluation criteria for KASP primer typing effect

[0094]

[0095] 1.1.3 Dataset Construction

[0096] The SNP site number, HEX fluorescence signal relative value X, FAM fluorescence signal relative value Y and sample number in the original data set were used to perform typing using the KASP-IEva model ( Figure 1 c in the figure), and the expert's scoring result of typing effect was used as the standard. Finally, the SNP site number was used as the number of each group of data, and the relative value X of the HEX fluorescence signal level and the relative value Y of the FAM fluorescence signal level of each group were combined and reconstructed into 192 high-dimensional feature variables, and the expert score was used as the label category to form the model data set ( Figure 1 d, e), of which 84% is used as training set and 16% is used as test set to evaluate the effectiveness of the model. The Stacking model in the fusion method will use five-fold cross-validation of the training set and further divide it into validation subsets.

[0097] 2 Experimental results and analysis

[0098] 2.1 Experimental environment

[0099] The operating system environment is Windows 11, the CPU is 12th Gen Intel(R) Core(TM) i5-125003.00 GHz, the machine has 32.0 GB RAM, and the GPU is NVDIA GeForce RTX 3080 with 10 GB memory. The training environment is created by Anaconda3, and the environment is configured with Python 3.10.13 and PyTorch2.0.0 deep learning framework.

[0100] 2.2 Model Evaluation Metrics

[0101] In order to verify the effectiveness of the proposed model method, the model performance is evaluated using accuracy, precision, recall, F1 score and Cohen's Kappa coefficient. The value range of the first four indicators is 0-1, each scoring category is calculated separately, and then the macro-averaging method is used to obtain the average indicator value. The evaluation formula for each category is as follows:

[0102] (4)

[0103] (5)

[0104] (6)

[0105] (7)

[0106] Where TP, FP and FN represent the number of samples correctly predicted as the label, the number of samples incorrectly predicted as the label and the number of samples incorrectly predicted as other labels, respectively.

[0107] The Kappa coefficient provides a more reliable measure of consistency than simple accuracy by taking into account the accidental factors of classification. Its value range is between -1 and 1, and the scoring criteria are as follows: 0.81 - 1.00 indicates almost complete consistency; 0.61 - 0.80 indicates significant consistency; 0.41 - 0.60 indicates moderate consistency; 0.21 - 0.40 indicates general consistency; 0.00 - 0.20 indicates extremely low consistency; less than 0 indicates no consistency or very poor consistency. The Kappa coefficient formula is as follows:

[0108] (8)

[0109] In the formula, p o Indicates the proportion of predicted labels that are consistent with actual labels, p e It represents the consistency ratio under the assumption that the predictions and actual labels are independent by chance.

[0110] 2.3 Model Hyperparameter Selection and Model Performance Analysis

[0111] The selection of hyperparameter combination of basic learners is crucial to the prediction performance of the fusion model. To ensure the optimal performance of TAL-SRX, RS is used to optimize the RF model, PSO is used to optimize the SVM model, BO is used to optimize the XGBoost model, and the trial and error method is used to optimize the hyperparameters of ANN-LSTM and Transformer. The grid search method is used for voting weights. The optimization and determination of each model parameter are shown in Table 2.

[0112] Table 2 Core hyperparameters of the basic learner

[0113]

[0114] After optimization, the prediction performance of the basic model has been improved to a certain extent. The TAL-SRX integrated strategy integrates a variety of basic algorithms and observes the data space and structure from different angles, obtaining a more comprehensive learning prediction result for the data set. The prediction results of the test set are compared and analyzed between the basic model and the TAL-SRX integrated algorithm. Figure 4 The error between the predicted values ​​and the actual values ​​of each model is displayed, which intuitively shows which samples are incorrectly evaluated and classified during the prediction process of the model. The color bar on the right side of the heat map indicates the corresponding relationship between color and error size, with the color of the maximum error at the top and the color of the minimum error at the bottom. As can be seen from the figure, the number of blue strips of the TAL-SRX algorithm is small and the color is generally lighter, which indicates that the evaluation error rate of the TAL-SRX algorithm is lower than that of other basic models, and the error value is generally smaller than that of a single model.

[0115] Table 3 compares the prediction performance of the basic model and the TAL-SRX algorithm. The accuracy of the five single models ranges from 84.93% to 89.15%, verifying the feasibility of the five algorithms, RS-RF, SO-SVM, and BO-XGBoost, as base learners. The accuracy of the TAL-SRX integrated algorithm we proposed is 92.83%, which is 3.68% higher than the PSO-SVM model with the highest accuracy among the single models. The precision, recall, and F1 score of this method are 93.82%, 87.65%, and 89.88%, respectively, which are all higher than the five single models, indicating that the TAL-SRX integrated method has significantly improved the comprehensive performance of the basic model and has higher prediction accuracy.

[0116] Table 3 Performance comparison between the basic model and the TAL-SRX test set

[0117]

[0118] In order to reflect the ability of the present invention to distinguish different labels, the ROC curve is drawn and the average AUC value of each model under multiple categories is calculated. Figure 5 As shown, the dotted line is the baseline, and AUC represents the area under the ROC curve. The farther the ROC curve is from the baseline, the larger the AUC is, indicating that the ability of the present invention to distinguish samples is stronger. Among the six evaluation algorithms, the AUC value of the basic model BO-XGBoost is 0.9891, ranking second, and the AUC value of the TAL-SRX strategy is the highest, which is 0.0014-0.0276 higher than that of a single learner. Overall, TAL-SRX can effectively identify and evaluate KASP typing samples with different labels.

[0119] The consistency analysis of different label classifications was further performed on each model. The box violin plots of the Kappa coefficients of each model are shown in the following figure. Figure 6 As shown in the figure, it can be seen that the median Kappa coefficient of each single model is distributed between 0.7 and 0.9, and is densely distributed around 0.8. After applying the integrated method, the median Kappa coefficient of TAL-SRX is higher than 0.9, and is densely distributed around 0.9, and the overall distribution is tight, which is a very significant improvement, indicating that the consistency between the predicted labels and the actual labels of the TAL-SRX method is relatively strong, and the model is more stable. Among them, the abnormal Kappa coefficient value of the 90-point label is affected by the uneven distribution of the data set samples.

[0120] In summary, the experimental results show that compared with the basic model, the TAL-SRX method has higher evaluation accuracy, consistency and stability. From a theoretical perspective, on the one hand, TAL-SRX makes full use of the differences of heterogeneous models and gives full play to the powerful nonlinear modeling capabilities of the basic model, so that it can fully capture the detailed characteristics of the data. On the other hand, the basic model has different advantages. For example, BO-XGBoost is based on the gradient boosting framework. By controlling the depth of the tree, it enhances its generalization ability and robustness to noisy data. The characteristics of each internal model enhance the integrated expression ability of TAL-SRX. Therefore, TAL-SRX has obvious advantages over a single model and can achieve effective prediction of the effect of KASP primer typing.

[0121] 2.4 Comparative analysis of integrated combination ablation

[0122] In order to prove the effectiveness of the two integration strategies in TAL-SRX, ablation comparison tests were performed using different integration combinations. First, the Stacking integration and deep learning model integration of traditional machine learning algorithms were used for prediction. Among them, the integration combination of ANN-LSTM and Transformer used grid search to determine the best weight combination, which was 0.45 and 0.4 respectively. Secondly, ANN-LSTM integration was added on the basis of Stacking, and the grid search determined the best weight combination to be 0.25 and 0.05. At the same time, Transformer integration was added on the basis of Stacking, and the grid search determined the best weight combination to be 0.35 and 0.1.

[0123] The test results of each ensemble combination method are shown in Figure 7. It can be seen that compared with the Stacking ensemble that uses traditional machine learning algorithms alone, the accuracy, precision, recall and F1 score of TAL-SRX are 2.02%, 1.00%, 1.86% and 1.58% higher, respectively, indicating that deep learning algorithms are more capable of extracting high-level features and key details when processing complex data. Compared with the ensemble combination that uses deep learning alone, the accuracy, precision, recall and F1 score of TAL-SRX are 4.78%, 8.07%, 3.60% and 5.23% higher, respectively. This is due to the fact that Stacking combines the advantages of more learners and reduces the deviation caused by data segmentation through five-fold cross validation during the model training stage, thereby improving the prediction accuracy. In addition, compared with the simple Stacking model, the accuracy of adding two deep learning algorithms separately is improved, and the TAL-SRX method that introduces both of them together has the highest accuracy, which shows that Stacking, ANN-LSTM and Transformer algorithms have their own advantages and positive effects on specific types of data in the data set, and the samples scored correctly by the three algorithms do not completely cover each other, thereby effectively improving the model scoring performance.

[0124] 2.5 Comparative analysis of expert scoring and TAL-SRX prediction results

[0125] Figure 8The relationship curves between the expert scoring and TAL-SRX methods and the sample size are shown. It can be seen that the trajectories of the orange and blue curves are basically the same, and the sample size distribution curves of the expert evaluation and algorithm evaluation have similar trends in each score segment, indicating that the TAL-SRX evaluation is highly consistent with the expert evaluation. The evaluation results of the multi-model fusion method can well reflect the development success rate of KASP markers, and thus effectively screen out markers with good typing. However, it is worth noting that there is a significant deviation in the curve trajectories of the two score regions of 0 and 10. This is because the Transformer with the strongest global feature capture ability has the smallest weight in the vote and contributes less to the final prediction result, making the integrated model performance weaker on samples with dispersed allele types.

[0126] 3 Conclusion

[0127] The present invention proposes a more accurate KASP primer intelligent typing evaluation method and system TAL-SRX, which can be used to provide more accurate data support for improving the development success rate of KASP markers. The method first introduces an optimization algorithm, and constructs an underlying Stacking ensemble learning model through three improved models of RS-RF, PSO-SVM and BO-XGBoost, which improves the model prediction performance to a certain extent, and improves the model stability and prediction accuracy. Then, ANN and LSTM are used to construct a hybrid neural network to capture nonlinear relationships and extract complex features, enhance the model's adaptability to differential samples, and introduce a Transformer algorithm with a multi-head attention mechanism to capture global dependencies in high-dimensional feature space, effectively enhancing the robustness of the model. A comparative test was carried out on the test set. The TAL-SRX method and the expert evaluation results have a high consistency in the distribution of the number of samples in each score segment. The evaluation accuracy is 92.83%, the AUC value is 0.9905, the error rate and error value are lower than the basic model, and the overall performance is significantly better than each single model. In the integrated combination ablation comparison test, the TAL-SRX integrated method showed better accuracy and was suitable for the evaluation of KASP primer intelligent typing.

[0128] However, it is worth noting that the model proposed in this invention is mainly aimed at the data evaluation problem of medium-scale experiments and is only applicable to experimental data sets with 96-well plates as the mother plate. In the actual operation of KASP, researchers will also obtain experimental data of different well plates at larger and large scales. Therefore, in subsequent experiments, it is necessary to further adjust the model structure and enhance the generalization ability of the model to more data sets.

[0129] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A KASP primer intelligent typing evaluation method based on multi-model fusion, characterized in that: The following steps are involved: (1) Obtain the original data set to construct a data set, and divide the data set into a training set and a test set; (2) Constructing a Stacking ensemble learning evaluation model: The training set described in step (1) is subjected to five-fold cross validation for training base models, wherein the base models are RF and SVM models. During the training process, RS and PSO are used to optimize the hyperparameters of the RF and SVM model training processes, respectively. The five-fold cross validation results of the RF and SVM models are combined and spliced ​​into new meta-model feature variables and their original labels for further training of the meta-model XGBoost, and BO is used to optimize the hyperparameters of the training process. After the training is completed, the RF and SVM models are used to predict the test set, and the prediction results are input into the meta-model and then output to obtain the final KASP marker typing effect score probability prediction value; (3) Constructing ANN-LSTM model and Transformer model respectively: Using hybrid neural network technology, the advantages of ANN in capturing nonlinear relationships and LSTM in extracting complex features are integrated. At the same time, the Transformer algorithm is introduced to capture global dependencies in high-dimensional feature space, alleviating the impact of data noise and local anomalies in different batches. After the ANN-LSTM model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided; After the Transformer model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided; (4) Through the soft voting integration strategy, based on the weighted weights, the probability prediction average of stacking, ANN-LSTM, and Transformer is calculated, the underlying integration and deep learning algorithms are integrated, and the final effect score of KASP marker typing is output; Among them, in step (2), the implementation steps of the Stacking ensemble learning evaluation model are as follows: a. The training set S is preprocessed using the five-fold cross-validation method, and the data set S is randomly divided into five equal-sized and non-intersecting subsets S1, S2, …, S5; b. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the PSO-SVM model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z1. c. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the RS-RF model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z2. d. Combine the prediction results of the base learners in the five-fold cross validation and horizontally splice them into a new feature variable Z={Z1, Z2}, thereby realizing the feature transformation from the base learner to the meta-model. Use these new features and the original label yn to train the meta-model; e. After the model training is completed, the base model is used to predict the test set, and the prediction results are input into the meta-model to obtain the final KASP primer typing effect evaluation results; In step (3), the implementation steps of the ANN-LSTM model are as follows: In the forward propagation process, the input data first passes through the ANN layer, which uses ReLU as the activation function to perform nonlinear transformation to increase the expressive power of the model. 128 hidden units are set. After the data undergoes preliminary feature extraction in this layer, the shape is adjusted and then passed through the LSTM layer. The LSTM layer also has 128 units. Through the memory unit and forget gate mechanism, it can effectively capture the inherent complex structure of the data. Finally, the output of the LSTM is mapped to the target category space. In the back propagation process, the cross entropy loss function is calculated, and the model parameters are updated in the training rounds through the Adam optimizer. After the training is completed, the ANN-LSTM model makes predictions on the test set and calculates the probability distribution of each score.

2. The KASP primer intelligent typing evaluation method based on multi-model fusion according to claim 1, characterized in that: In step (1), the step of obtaining the original data set and constructing the data set is specifically, The statistical data of KASP marker typing report of cotton variety resource materials produced by SNPline, a high-throughput genotyping detection platform of LGC, were selected, and the statistical data in the format of 94 DNA samples and 2 NTC detection data were extracted to establish the original data set. The SNP site number, HEX fluorescence signal level relative value X, FAM fluorescence signal level relative value Y and sample number in the original data set were taken to apply KASP-IEva model for typing, and the results of expert scoring on the typing effect were used as the standard. Finally, the SNP site number was used as the number of each group of data, and the relative value X of the HEX fluorescence signal level and the relative value Y of the FAM fluorescence signal level of each group were combined and reconstructed into 192 high-dimensional feature variables. The expert score was used as the label category to form a model data set. Among the model data sets, 84% was used as a training set and 16% was used as a test set.

3. The KASP primer intelligent typing evaluation method based on multi-model fusion according to claim 1, characterized in that: In step (3), the implementation steps of the Transformer model are as follows: In the Transformer model, an encoder with feature extraction capability is first defined, and the input data is processed through three encoder layers to increase the depth of the model; in the forward propagation process, the self-attention mechanism and the feedforward neural network process the data to capture higher-level relationships and extract more complex features; in order to prevent the model from overfitting, the Dropout layer is added to randomly discard some neurons, and finally the processed features are mapped to the target category space through the linear layer; during the training process, the model calculates the cross entropy loss through backpropagation and uses the Adam optimizer to update the model parameters. In the test phase, the Transformer model calculates the score category probability of each sample through the softmax function and outputs it.

4. A KASP primer intelligent typing evaluation system TAL-SRX based on multi-model fusion, characterized in that: The system includes: a memory and a processor, wherein the memory includes a KASP primer intelligent typing evaluation method program based on multi-model fusion, and when the KASP primer intelligent typing evaluation method program based on multi-model fusion is executed by the processor, the following steps are implemented: (1) Obtain the original data set to construct a data set, and divide the data set into a training set and a test set; (2) Constructing a Stacking ensemble learning evaluation model: The training set described in step (1) is subjected to five-fold cross validation for training base models, wherein the base models are RF and SVM models. During the training process, RS and PSO are used to optimize the hyperparameters of the RF and SVM model training processes, respectively. The five-fold cross validation results of the RF and SVM models are combined and spliced ​​into new meta-model feature variables and their original labels for further training of the meta-model XGBoost, and BO is used to optimize the hyperparameters of the training process. After the training is completed, the RF and SVM models are used to predict the test set, and the prediction results are input into the meta-model and then output to obtain the final KASP marker typing effect score probability prediction value; (3) Constructing ANN-LSTM model and Transformer model respectively: Using hybrid neural network technology, the advantages of ANN in capturing nonlinear relationships and LSTM in extracting complex features are integrated. At the same time, the Transformer algorithm is introduced to capture global dependencies in high-dimensional feature space, alleviating the impact of data noise and local anomalies in different batches. After the ANN-LSTM model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided; After the Transformer model is trained based on the training set described in step (1), a probability prediction value based on the test set described in step (1) is provided; (4) Through the soft voting integration strategy, based on the weighted weights, the probability prediction average of stacking, ANN-LSTM, and Transformer is calculated, the underlying integration and deep learning algorithms are integrated, and the final effect score of KASP marker typing is output; Among them, in step (2), the implementation steps of the Stacking ensemble learning evaluation model are as follows: a. The training set S is preprocessed using the five-fold cross-validation method, and the data set S is randomly divided into five equal-sized and non-intersecting subsets S1, S2, …, S5; b. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the PSO-SVM model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z1. c. Select a subset from step a as the test set and the remaining four subsets as the training set. Train the RS-RF model on the training set, make predictions on the test set and save the prediction results. Then select the remaining four subsets as the test set in turn. Finally, 5 prediction results are obtained and combined into set Z2. d. Combine the prediction results of the base learners in the five-fold cross validation and horizontally splice them into a new feature variable Z={Z1, Z2}, thereby realizing the feature transformation from the base learner to the meta-model. Use these new features and the original label yn to train the meta-model; e. After the model training is completed, the base model is used to predict the test set, and the prediction results are input into the meta-model to obtain the final KASP primer typing effect evaluation results; In step (3), the implementation steps of the ANN-LSTM model are as follows: In the forward propagation process, the input data first passes through the ANN layer, which uses ReLU as the activation function to perform nonlinear transformation to increase the expressive power of the model. 128 hidden units are set. After the data undergoes preliminary feature extraction in this layer, the shape is adjusted and then passed through the LSTM layer. The LSTM layer also has 128 units. Through the memory unit and forget gate mechanism, it can effectively capture the inherent complex structure of the data. Finally, the output of the LSTM is mapped to the target category space. In the back propagation process, the cross entropy loss function is calculated, and the model parameters are updated in the training rounds through the Adam optimizer. After the training is completed, the ANN-LSTM model makes predictions on the test set and calculates the probability distribution of each score.

5. According to claim 4, a KASP primer intelligent typing evaluation system TAL-SRX based on multi-model fusion, characterized in that: In step (3), the implementation steps of the Transformer model are as follows: In the Transformer model, an encoder with feature extraction capability is first defined, and the input data is processed through three encoder layers to increase the depth of the model; in the forward propagation process, the self-attention mechanism and the feedforward neural network process the data to capture higher-level relationships and extract more complex features; in order to prevent the model from overfitting, the Dropout layer is added to randomly discard some neurons, and finally the processed features are mapped to the target category space through the linear layer; during the training process, the model calculates the cross entropy loss through backpropagation and uses the Adam optimizer to update the model parameters. In the test phase, the Transformer model calculates the score category probability of each sample through the softmax function and outputs it.

Images