Text correction method and device based on multi-expert hybrid mechanism, equipment and medium

Abstract

This invention relates to the field of natural language processing. Specifically, it relates to a text correction method, apparatus, device, and medium based on a multi-expert hybrid mechanism. The method includes constructing a feedforward layer based on the multi-expert hybrid mechanism to obtain a gating probability distribution for a set of expert sub-networks; activating expert sub-networks that meet preset conditions in the set of expert sub-networks according to the gating probability distribution to participate in the calculation, obtaining the corresponding expert sub-network output results; performing weighted fusion and normalization processing on the expert sub-network output results according to the gating probability distribution to obtain a fused feature representation; inputting the fused feature representation into a multi-task output header structure to obtain jointly trained model parameters; and performing text correction inference based on the jointly trained model parameters to output the corrected text result. This invention improves the multi-task collaborative performance and correction accuracy of text correction tasks.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to text correction methods, apparatus, devices and media based on a multi-expert hybrid mechanism. Background Technology

[0002] In the traditional field of text correction, common methods for solving text correction problems include rule-based correction methods, which check and correct text according to pre-defined rules such as grammar and spelling. These rules can be formulated by linguists or other professionals. Statistical correction methods also exist, which learn word probabilities and collocation patterns through statistical analysis of large amounts of text data to identify errors. Additionally, machine learning-based correction methods are also frequently used, employing machine learning algorithms such as support vector machines and decision trees to classify and correct text.

[0003] However, existing text correction methods have significant drawbacks. Rule-based methods require manually writing a large number of rules, making it difficult to cover all linguistic phenomena and handle complex contexts and newly emerging vocabulary. Statistical methods rely on large-scale corpora and are ineffective at correcting low-frequency words and rare errors. Machine learning-based methods have limited capabilities in handling complex semantics and contextual information, are prone to overfitting during training, and cannot achieve joint processing of multiple tasks or load balancing constraints on expert subnetworks. Summary of the Invention

[0004] To improve the multi-task collaborative performance and error correction accuracy of text correction tasks, this application provides a text correction method, apparatus, device, and medium based on a multi-expert hybrid mechanism.

[0005] The above-mentioned objective of this application is achieved through the following technical solution:

[0006] A text correction method based on a multi-expert hybrid mechanism, comprising:

[0007] Obtain the input text information, perform word embedding and position embedding operations on the input text information to obtain the embedded representation containing semantic vectors and position information;

[0008] The embedded representation is input into a Transformer encoding network containing a multi-layer encoding structure. In the Transformer encoding network, multi-head self-attention computation based on query vector, key vector and value vector is performed to obtain context-dependent representation features.

[0009] In the feedforward stage of the Transformer coding network, a feedforward layer based on a multi-expert hybrid mechanism is constructed. Two-layer nonlinear gating computation is performed on each context-related representation feature to obtain the gating probability distribution for the expert sub-network set.

[0010] Based on the gating probability distribution, expert subnetworks that meet the preset conditions are activated in the expert subnetwork set to participate in the calculation, and the corresponding expert subnetwork output results are obtained. The expert subnetwork output results are then subjected to weighted fusion and normalization processing according to the gating probability distribution to obtain the fusion feature representation.

[0011] The fused feature representation is input into the multi-task output head structure, and joint training operations of language modeling, text correction and named entity recognition tasks are performed. In the joint training operation, an expert balance constraint loss function is constructed based on the activation frequency of each expert sub-network to perform load balancing constraint on the activation process of the expert sub-network, and the model parameters after joint training are obtained.

[0012] Based on the jointly trained model parameters, text error correction inference is performed, and the corrected text result is output.

[0013] By adopting the above technical solutions, multi-task collaborative optimization and dynamic expert selection mechanisms can be realized in the text correction process, enabling the model to have adaptive feature extraction capabilities when facing different semantic scenarios. By regulating the activation state of expert sub-networks through gating probability distribution, the bias problem caused by excessive participation of a single expert can be effectively avoided. At the same time, based on the introduction of the expert balance constraint loss function, the load balance of each expert sub-network is maintained during training, thereby improving the stability and convergence efficiency of the overall network. Finally, under the joint training of language modeling, text correction and named entity recognition, the model can more accurately capture contextual information and significantly improve the error correction accuracy and semantic consistency in complex contexts.

[0014] In a preferred embodiment, this application can be further configured as follows: performing a two-layer nonlinear gating computation on each context-dependent representation feature to obtain a gating probability distribution for the expert subnetwork set includes:

[0015] Each context-dependent representation feature is input into the first linear transformation layer, and a linear mapping operation is performed on the context-dependent representation features to obtain the first transformation result;

[0016] The first transformation result is input into the activation function layer, and nonlinear activation processing is performed on the first transformation result to obtain the activation output;

[0017] The activation output is input to the second linear transformation layer to obtain the second transformation result;

[0018] The softmax operation is performed on the second transformation result. A learnable temperature parameter is introduced into the softmax operation to generate a gated probability distribution.

[0019] By adopting the above technical solutions, efficient mapping from context-related representation features to gating probability distributions can be achieved in the feedforward stage. This allows the model to balance linear transformation and nonlinear expression capabilities during feature transfer. A two-layer linear transformation structure captures semantic relationships and feature patterns at different levels, and activation function layers enhance the nonlinear discriminativeness of features. A learnable temperature parameter is introduced into the softmax operation, allowing the gating probability distribution to adaptively adjust its smoothness and concentration during training. This makes the selection process of expert subnetworks more flexible and stable, avoiding training instability caused by excessively large gradients or overly sharp distributions. Dynamic expert activation and probability control under a multi-expert hybrid mechanism are achieved, improving the model's ability to model complex semantic features.

[0020] In a preferred embodiment, this application can be further configured as follows: Based on the gating probability distribution, expert subnetworks meeting preset conditions are activated in the expert subnetwork set to participate in the calculation, obtaining the corresponding expert subnetwork output results. Weighted fusion and normalization processing is then performed on the expert subnetwork output results according to the gating probability distribution to obtain a fused feature representation, including:

[0021] Based on the numerical values ​​of the gating probability distribution, the expert subnetworks in the expert subnetwork set are sorted to obtain the sorting results;

[0022] Based on the ranking results, determine the set of target expert subnetworks whose gating probability distributions satisfy preset conditions;

[0023] The input data, consisting of context-dependent representation features, is sequentially fed into each target expert subnetwork in the target expert subnetwork set to obtain the output results of each target expert subnetwork.

[0024] The output of the target expert subnetwork is matched with the weight information corresponding to the gating probability distribution to obtain the matching result. A weighted operation is performed based on the matching result to obtain the weighted result. The weighted result is then normalized to obtain the fused feature representation.

[0025] By adopting the above technical solution, a dynamic expert selection and output fusion process based on gating probability distribution can be realized. This enables the model to adaptively activate the most representative expert sub-networks to participate in feature calculation in each forward computation, thereby improving the model's selective expression of input semantics while ensuring computational efficiency. By sorting the expert sub-network set according to the numerical value of the gating probability distribution and selecting target expert sub-networks that meet preset conditions, it is ensured that only the experts most relevant to the current semantics are activated to participate in the computation, avoiding invalid computation and resource waste. In the fusion stage, by accurately matching the output results of each target expert sub-network with the corresponding weight information and performing weighting and normalization processing, the fused feature representation can integrate the differentiated information of multiple expert outputs, achieving coordinated expression of global and local semantics, thereby enhancing the model's ability to perceive complex contexts and output stability.

[0026] In a preferred embodiment, this application can be further configured as follows: During the joint training operation, an expert equilibrium constraint loss function is constructed based on the activation frequencies of each expert sub-network to apply load balancing constraints to the activation process of the expert sub-networks, thereby obtaining the model parameters after joint training, including:

[0027] In the joint training operation, the activation frequency information of each expert subnetwork is obtained by counting the number of activations of each expert subnetwork in multiple training samples.

[0028] The average activation frequency is calculated based on the activation frequency information to obtain the target equilibrium benchmark.

[0029] The frequency deviation is obtained by calculating the difference between the activation frequency of each expert subnetwork and the target equilibrium benchmark.

[0030] Based on the frequency deviation, an expert balance constraint loss function is constructed, and a load balancing constraint is applied to the activation process of the expert sub-network. Under the condition of applying the load balancing constraint, the joint training parameters are updated to obtain the joint trained model parameters.

[0031] By adopting the above technical solution, load balancing among expert subnetworks can be effectively achieved during joint training of multi-expert hybrid structures. By statistically analyzing the activation counts of each expert subnetwork in multiple training samples and calculating the average activation frequency, a unified target balancing benchmark is established, enabling dynamic monitoring of the distribution of expert subnetwork usage during training. Based on this, the frequency deviation between each expert subnetwork and the target balancing benchmark is calculated, and an expert balancing constraint loss function is constructed accordingly. This allows the model to explicitly constrain activation imbalance during optimization. Combining the joint optimization mechanism of load balancing constraints and task loss, the parameter update direction simultaneously considers the performance of the main task and the activation balance among experts, thereby avoiding overfitting caused by excessive activation of individual expert subnetworks, improving the model's stability and generalization ability, and achieving global optimization of resource allocation during the training phase.

[0032] In a preferred embodiment, this application can be further configured as follows: the step of constructing an expert balancing constraint loss function based on the frequency deviation, applying load balancing constraints to the activation process of the expert sub-network, and updating the joint training parameters under the applied load balancing constraints to obtain the jointly trained model parameters includes:

[0033] Calculate the load deviation value of each expert subnetwork based on the frequency deviation;

[0034] Based on the load deviation value, a balance coefficient is generated. The balance coefficient is then introduced into the expert balance constraint loss function. The expert balance constraint loss function is then weighted and combined to obtain a loss function expression that includes load balance constraint terms.

[0035] The gradient information during the joint training process is calculated based on the loss function expression. The activation process of the expert sub-network is then updated with parameters, and the joint-trained model parameters are output.

[0036] By adopting the above technical solution, adaptive constraint adjustment of the activation state of expert sub-networks can be achieved during joint training. First, by calculating the load deviation value of each expert sub-network, the degree of deviation from the ideal activation state is quantified, so that expert sub-networks with different behavioral characteristics have measurable load differences. Then, an equilibrium coefficient is generated based on the load deviation value, and this equilibrium coefficient is introduced into the expert equilibrium constraint loss function. The main task loss and equilibrium constraint term are combined in a weighted manner, so that task performance and expert distribution balance are considered in the optimization objective. Finally, gradient information is calculated based on the combined loss function expression, and the parameters are updated using optimization algorithms such as gradient descent. This allows the activation weights and frequency distribution of each expert sub-network to be dynamically adjusted during the training process, gradually achieving load balance and overall performance synergistic optimization, thereby obtaining the model parameters after joint training and maintaining a stable expert activation structure.

[0037] In a preferred embodiment, this application can be further configured as follows: the text correction inference is performed based on the jointly trained model parameters, and the corrected text result is output, including:

[0038] Obtain the input text information to be corrected, perform word embedding and position embedding operations on the input text information to obtain the embedded representation of the text to be corrected;

[0039] The embedded representation is input into a Transformer encoding network loaded with the parameters of the jointly trained model, and forward computation is performed to obtain the context-related representation features of the text to be corrected.

[0040] The context-related representation features are input into the feedforward layer based on the multi-expert hybrid mechanism. The gated probability distribution is generated based on the jointly trained model parameters, and the corresponding expert sub-network set is activated to obtain the fused feature representation.

[0041] The fused feature representation is input into the output header of the text correction task, inference calculation is performed to obtain the error correction prediction result, and the corrected text result is generated based on the error correction prediction result.

[0042] By adopting the above technical solution, an end-to-end adaptive processing flow from embedding generation to error correction output can be realized in the text error correction inference stage. First, word embedding and position embedding operations are performed on the input text information to be corrected, so that the text is quantified in both semantic and structural dimensions, thereby preserving word order and context-dependent features. Then, the embedded representation is input into the Transformer encoding network loaded with jointly trained model parameters. Context-related representation features are obtained through forward computation, enabling the model to capture syntactic associations and semantic deviations based on context information. Then, a gated probability distribution is generated through a feedforward layer based on a multi-expert hybrid mechanism, and the corresponding expert sub-network set is activated to achieve differentiated modeling and correction of errors of different semantic types. Finally, the fused feature representation is input into the output head corresponding to the text error correction task, inference computation is performed, and error correction prediction results are generated, so that the corrected text results achieve optimal matching in terms of semantic consistency and language fluency, thereby realizing refined and context-adaptive text error correction output.

[0043] In a preferred embodiment, this application can be further configured such that generating the corrected text result based on the error correction prediction result includes:

[0044] Based on the prediction score information in the error correction prediction results, candidate error correction word sequences are determined;

[0045] Perform a probability screening operation on the candidate error correction word sequences and select the target error correction word sequences that meet the preset confidence conditions;

[0046] Based on the correspondence between the target error correction word sequence and the input text information, error correction mapping information is generated;

[0047] Based on the error correction mapping information, the corresponding positions in the input text information are replaced to obtain the corrected text result.

[0048] By adopting the above technical solution, a precise mapping and replacement process from the predicted result to the final corrected text can be achieved in the text correction result generation stage. First, by reading the prediction score information in the error correction prediction result, candidate error correction word sequences are determined, enabling the model to form a candidate set among multiple potential error correction options. Then, a probability screening operation is performed on the candidate error correction word sequences, filtering out low-confidence results based on pre-set confidence conditions, and retaining only the target error correction word sequences that are semantically reasonable and conform to the contextual logic. Next, error correction mapping information is generated based on the correspondence between the target error correction word sequences and the input text information, establishing a one-to-one correspondence between the predicted output and the original text position, thereby clarifying the replacement position of each error correction content in the original text. Finally, a replacement operation is performed on the corresponding position in the input text information according to the error correction mapping information to obtain the corrected text result, ensuring that the output text maintains consistency in semantic integrity and language fluency, and realizing a controllable and verifiable text correction output process.

[0049] The second objective of this invention is achieved through the following technical solution:

[0050] A text correction device based on a multi-expert hybrid mechanism, comprising:

[0051] The text embedding processing module is used to acquire input text information, perform word embedding and position embedding operations on the input text information, and obtain an embedded representation containing semantic vectors and position information.

[0052] The multi-head attention encoding module is used to input the embedded representation into the Transformer encoding network, which contains a multi-layer encoding structure. In the Transformer encoding network, multi-head self-attention computation based on query vector, key vector and value vector is performed to obtain context-related representation features.

[0053] The multi-expert gated feedforward module is used to construct a feedforward layer based on a multi-expert hybrid mechanism in the feedforward stage of the Transformer coding network. It performs two-layer nonlinear gated computation on each context-related representation feature to obtain the gated probability distribution for the set of expert sub-networks.

[0054] The expert fusion computing module is used to activate expert subnetworks that meet preset conditions in the expert subnetwork set to participate in the calculation according to the gating probability distribution, obtain the corresponding expert subnetwork output results, and perform weighted fusion and normalization processing on the expert subnetwork output results according to the gating probability distribution to obtain the fusion feature representation;

[0055] The multi-task joint training module is used to input the fused feature representation into the multi-task output head structure and perform joint training operations for language modeling, text correction and named entity recognition tasks. In the joint training operation, an expert balance constraint loss function is constructed based on the activation frequency of each expert sub-network to perform load balancing constraints on the activation process of the expert sub-networks and obtain the model parameters after joint training.

[0056] The text correction inference module is used to perform text correction inference based on the jointly trained model parameters and output the corrected text results.

[0057] By adopting the above technical solutions, multi-task collaborative optimization and dynamic expert selection mechanisms can be realized in the text correction process, enabling the model to have adaptive feature extraction capabilities when facing different semantic scenarios. By regulating the activation state of expert sub-networks through gating probability distribution, the bias problem caused by excessive participation of a single expert can be effectively avoided. At the same time, based on the introduction of the expert balance constraint loss function, the load balance of each expert sub-network is maintained during training, thereby improving the stability and convergence efficiency of the overall network. Finally, under the joint training of language modeling, text correction and named entity recognition, the model can more accurately capture contextual information and significantly improve the error correction accuracy and semantic consistency in complex contexts.

[0058] The above-mentioned objective three of this application is achieved through the following technical solution:

[0059] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the steps of the above-described method.

[0060] The fourth objective of this application is achieved through the following technical solution:

[0061] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the text correction method based on a multi-expert hybrid mechanism described above.

[0062] In summary, this application includes at least one of the following beneficial technical effects:

[0063] 1. It can achieve multi-task collaborative optimization and dynamic expert selection mechanism in the text correction process, enabling the model to have adaptive feature extraction capabilities when facing different semantic scenarios. By regulating the activation state of expert sub-networks through gating probability distribution, it effectively avoids the bias problem caused by excessive participation of a single expert. At the same time, based on the introduction of the expert balance constraint loss function, the load balance of each expert sub-network is maintained during training, thereby improving the stability and convergence efficiency of the overall network. Finally, under the joint training of language modeling, text correction and named entity recognition, the model can more accurately capture contextual information and significantly improve the error correction accuracy and semantic consistency in complex contexts. Attached Figure Description

[0064] Figure 1 This is a flowchart of a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application;

[0065] Figure 2 This is a flowchart illustrating the implementation of step S30 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application.

[0066] Figure 3 This is a flowchart illustrating the implementation of step S40 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application.

[0067] Figure 4 This is a flowchart of step S50 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application;

[0068] Figure 5 This is a flowchart illustrating the implementation of step S504 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application.

[0069] Figure 6 This is a flowchart illustrating the implementation of step S60 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application.

[0070] Figure 7 This is a flowchart illustrating the implementation of step S604 in a text correction method based on a multi-expert hybrid mechanism in one embodiment of this application.

[0071] Figure 8 This is a schematic diagram of a text correction device based on a multi-expert hybrid mechanism in one embodiment of this application; Detailed Implementation

[0072] The present application will be further described in detail below with reference to the accompanying drawings.

[0073] In one embodiment, such as Figure 1 As shown, this application discloses a text correction method based on a multi-expert hybrid mechanism, which specifically includes the following steps:

[0074] S10: Obtain the input text information, perform word embedding and position embedding operations on the input text information, and obtain the embedded representation containing semantic vectors and position information.

[0075] In this embodiment, the input text information refers to the original text data that needs to be corrected. It can be a natural language input sequence, a document paragraph, or a text sample from a corpus dataset. Word embedding refers to converting each word or sub-word unit in the input text information into a fixed-dimensional vector representation to characterize the semantic relationship between words in the text. Position embedding refers to encoding the position information of each word in the sequence to preserve the sequential features of the text structure. Embedding representation refers to the overall feature representation formed by combining word embedding and position embedding in the vector dimension, which is used as the input to the subsequent Transformer encoding network.

[0076] Specifically, the input text information is first processed by word segmentation. Word segmentation can be performed on the text by dividing it into sub-words based on the byte-pair encoding algorithm. Each word or sub-word is matched with the word index and the corresponding word vector is read from the pre-trained embedding matrix. The embedding matrix is ​​a set of fixed-dimensional vectors learned in the training phase. Each row of the vector corresponds to the semantic representation of a word or sub-word. When performing position embedding, the sine and cosine function values ​​on different dimensions are calculated according to the index number of each word in the text sequence. The periodic encoding of the sequence position information is realized through different frequency parameters. The generated position information vector and the word embedding vector are added on the corresponding dimension to obtain the embedding representation containing semantic vector and position information.

[0077] S20: Input the embedded representation into a Transformer encoding network containing a multi-layer encoding structure. In the Transformer encoding network, perform multi-head self-attention computation based on query vector, key vector and value vector to obtain context-dependent representation features.

[0078] In this embodiment, the Transformer coding network is a deep neural structure composed of multiple layers of coding units stacked sequentially. Each coding unit includes a multi-head self-attention sublayer and a feedforward sublayer. Multi-head self-attention computation is a weighted feature aggregation method based on the correlation between vectors. The query vector, key vector, and value vector are different representations obtained by linear mapping from the embedding representation. The query vector is used to represent the semantic requirements of the current word or subword, the key vector is used to represent the semantic features that can be paid attention to in the context, and the value vector is used to carry the corresponding semantic content. The context-related representation features refer to the comprehensive semantic representation obtained after processing by the multi-head self-attention mechanism, which is used to reflect the dependency relationship between various positions in the input text.

[0079] Specifically, when the embedded representation is input into the Transformer encoding network, firstly, in each encoding unit, a query vector, key vector, and value vector are generated through a linear transformation operation. The linear transformation is calculated by performing matrix multiplication on the input vector with three different weight matrices to obtain the corresponding vector set. Then, a dot product operation is performed on the query vector and the key vector to calculate the relevance score. During the calculation, the inner product of each query vector and all key vectors is scaled according to the dimension and input into the softmax function to obtain a normalized attention weight distribution. Then, the value vector is weighted and summed using this weight distribution to achieve the association and aggregation of semantic information at different positions. The "multi-head" in multi-head self-attention refers to the parallel execution of multiple independent attention computation heads in the same encoding unit. Each attention head uses a different weight matrix to capture the dependencies of different semantic dimensions. The outputs of all attention heads are concatenated in the dimension and passed to the feedforward sub-layer after a linear mapping operation. Finally, they are propagated and fused sequentially in the multi-layer stacked encoding structure, and the output is the context-dependent representation feature.

[0080] S30: In the feedforward stage of the Transformer coding network, a feedforward layer based on a multi-expert hybrid mechanism is constructed. Two-layer nonlinear gating computation is performed on each context-related representation feature to obtain the gating probability distribution for the expert sub-network set.

[0081] In this embodiment, the multi-expert hybrid mechanism refers to setting up multiple parallel expert subnetworks in the feedforward layer. Each expert subnetwork has the same structural form but an independent set of parameters. The feedforward layer is a network structure used to perform nonlinear mapping and dimensional transformation on the input features during the encoding stage. The two-layer nonlinear gating calculation refers to the joint calculation of gating coefficients through two layers of linear transformation and nonlinear activation functions, which are used to determine the activation strength of the expert subnetworks. The gating probability distribution is the weight distribution obtained after normalizing the gating scores of all expert subnetworks, which is used to represent the selection probability of each expert subnetwork. The expert subnetwork set refers to the total number of all expert subnetworks defined as candidates participating in the calculation in the feedforward layer.

[0082] Specifically, context-dependent representation features are input into the gating network structure. First, the first-layer gating mapping result is calculated through a linear transformation operation. The linear transformation is calculated by performing matrix multiplication between the input feature vector and a preset first weight matrix and adding the corresponding bias vector to obtain the first transformation result. Then, nonlinear activation processing is performed on the result. The activation function can be a Gaussian error linear unit function. Smooth nonlinear mapping is achieved by performing piecewise exponential calculation on the input value. The output result is then input into the second linear transformation layer. In the second linear transformation layer, matrix multiplication with the second weight matrix is ​​performed and a bias term is added to obtain the second transformation result. A softmax operation is performed on each component in the second transformation result to normalize the exponential values ​​of all components proportionally. The calculation formula is to take the exponent of each component and divide it by the sum of the exponential values ​​of all components to form the gating probability distribution.

[0083] S40: Based on the gating probability distribution, activate the expert subnetworks that meet the preset conditions in the expert subnetwork set to participate in the calculation, and obtain the corresponding expert subnetwork output results. Perform weighted fusion and normalization processing on the expert subnetwork output results according to the gating probability distribution to obtain the fused feature representation.

[0084] In this embodiment, activating expert subnetworks that meet preset conditions means selecting expert subnetworks with higher activation probabilities based on the numerical value of the gating probability distribution to participate in subsequent calculations. The preset conditions can be determined based on the gating probability threshold or the activation quantity limit. The output result of the expert subnetwork refers to the output vector obtained by each activated expert subnetwork after receiving input features and performing feedforward calculation. Weighted fusion refers to the weighted summation of the output results of each expert subnetwork and its corresponding gating probability. Normalization processing refers to the proportional adjustment of the feature vector after fusion to maintain the consistency of feature scale. The fused feature representation refers to the comprehensive semantic features obtained after weighting and normalization, which are used as inputs in the subsequent task training and inference stages.

[0085] Specifically, the expert subnetwork set is first sorted in descending order based on the values ​​of each dimension of the gating probability distribution. The sorting process is achieved by comparing the probability values ​​corresponding to each dimension. The sorted results are used to determine the candidate activation order. Then, the top few expert subnetworks are selected as target expert subnetworks according to preset activation quantity parameters or probability thresholds. Input data consisting of context-related representation features is sequentially input into each target expert subnetwork. Within the expert subnetwork, feature feedforward calculation is performed through linear transformation and nonlinear mapping operations to obtain the corresponding expert subnetwork output results. The output results of all target expert subnetworks are matched with the weights at the corresponding positions in the gating probability distribution. The matching process achieves a one-to-one correspondence through an index mapping table. Then, a weighting operation is performed, that is, each output vector is multiplied by its corresponding gating probability and summed to obtain the weighted result. Subsequently, normalization is performed on each dimension of the weighted result. The normalization operation can be performed by mean-variance standardization or vector magnitude normalization. The average and standard deviation of the weighted result in each dimension are calculated to adjust the values, so that the overall distribution of the fused vector remains stable, and finally, the fused feature representation is obtained.

[0086] S50: Input the fused feature representation into the multi-task output head structure, and perform joint training operations for language modeling, text correction, and named entity recognition tasks. In the joint training operation, an expert balance constraint loss function is constructed based on the activation frequency of each expert sub-network to perform load balancing constraints on the activation process of the expert sub-networks, and the model parameters after joint training are obtained.

[0087] In this embodiment, the multi-task output head structure refers to a network output layer composed of multiple parallel task branches. Each task branch corresponds to a specific training objective, including a language modeling task branch, a text correction task branch, and a named entity recognition task branch. Joint training operation refers to a training method that simultaneously optimizes parameters for multiple tasks, enabling the shared encoding network to jointly learn semantic features in different tasks. The expert balance constraint loss function refers to introducing a constraint term on the activation frequency of expert sub-networks to control the balance of activation distribution while calculating task loss. The load balancing constraint refers to limiting the frequency difference of activation of each expert sub-network during the training phase, so that the overall computational load of the network remains evenly distributed. The model parameters after joint training refer to the set of all learnable weights updated during this joint optimization process.

[0088] Specifically, the fused feature representations are input into each task branch of the multi-task output head structure. The language modeling task branch calculates the probability distribution of the next word using autoregressive prediction. The text correction task branch predicts the character or word category at each input position through a classification output layer. The named entity recognition task branch labels entity boundaries and categories in the text sequence through a conditional random field layer. Each task branch calculates its corresponding loss function value, including cross-entropy loss or sequence labeling loss. The losses of each task are weighted and summed according to a preset proportional coefficient to obtain the multi-task comprehensive loss value. During the training iteration, the activation of each expert subnetwork throughout the entire training batch is statistically analyzed. The activation frequency of each expert subnetwork is calculated by counting and summing the activation counts and then dividing the sum by the sample size. The average activation frequency is then used as the target equilibrium benchmark to calculate the frequency deviation of each expert subnetwork. This deviation is then substituted into the expert equilibrium constraint loss function, which expresses the degree of imbalance in expert activations through the form of squared difference. This constraint term is then weighted and superimposed with the multi-task comprehensive loss value to form the total loss function. This total loss function is used to calculate gradient information and perform backpropagation to update the parameters of the jointly trained network. This ensures that the training maintains the balanced distribution of expert subnetwork activations while preserving task performance. After iteration, the parameters of the jointly trained model are obtained.

[0089] S60: Based on the jointly trained model parameters, perform text error correction inference and output the corrected text result.

[0090] In this embodiment, text error correction inference refers to the process of performing forward computation and error identification on the text to be corrected using the jointly trained model parameters after the model training is completed. The corrected text result refers to the character-level or word-level modified text sequence obtained through inference computation. Its content is semantically consistent with the input text information but corrects the misspellings or grammatical components.

[0091] Specifically, in this step, the jointly trained model parameters are first loaded and the structure configuration of the Transformer encoding network is initialized. The input text information to be corrected is then input into the text embedding processing flow. Discrete text symbols are converted into embedding representations containing semantic vectors and positional information through word embedding and positional embedding operations. This embedding representation is then input into the Transformer encoding network loaded with the jointly trained model parameters for forward computation. During the encoding stage, contextual dependencies are parsed and context-related representation features are generated through a multi-head self-attention mechanism. These context-related representation features are then input into a multi-head self-attention-based... In the feedforward layer of the hybrid mechanism, a gating probability distribution is generated based on the jointly trained model parameters. The activation strength of each expert sub-network is calculated, and the corresponding expert sub-network is activated to participate in the calculation. The output results of the expert sub-networks are obtained. The output results are weighted, fused, and normalized to obtain a fused feature representation. The fused feature representation is input into the output header structure corresponding to the text correction task to perform inference calculation and output the error correction prediction result. Finally, the error correction position is determined based on the prediction word sequence information in the error correction prediction result. A new text sequence is formed by replacing the characters or words at the corresponding position and output as the corrected text result, thus completing the text correction inference process.

[0092] In one embodiment, such as Figure 2 As shown, in step S30, a two-layer nonlinear gating calculation is performed on each context-related representation feature to obtain the gating probability distribution for the expert subnetwork set, including:

[0093] S301: Input each context-dependent representation feature into the first linear transformation layer, perform a linear mapping operation on the context-dependent representation features, and obtain the first transformation result.

[0094] In this embodiment, the first linear transformation layer refers to the computational layer used to perform linear projection of the input feature space. By setting a trainable weight matrix and bias vector, the high-dimensional feature vector of the input is transformed into a new representation space. The first transformation result refers to the intermediate feature output obtained after the linear mapping, which is used as the computational input for the subsequent nonlinear activation stage.

[0095] Specifically, in this step, each context-related representation feature is first arranged into a vector according to its dimensional order and input into the first linear transformation layer. In the first linear transformation layer, linear mapping is calculated through matrix multiplication. The calculation method is to multiply the context-related representation feature vector with the pre-initialized weight matrix and superimpose the corresponding bias vector on the product result to adjust the offset of the output space. The matrix multiplication operation is achieved by sequentially calculating the inner product of each dimensional component in the input vector with the corresponding column vector in the weight matrix. Each element of the output represents the linear combination strength of the input features in the new feature space. To ensure numerical stability, floating-point standardization or precision truncation can be used to limit the calculation range during the operation. After the above linear mapping operation, the output result is used as the first transformation result.

[0096] S302: Input the first transformation result into the activation function layer, perform nonlinear activation processing on the first transformation result, and obtain the activation output.

[0097] In this embodiment, the activation function layer refers to the computational layer used to introduce nonlinear mapping relationships in the feedforward computation. The activation function is a mathematical mapping function used to establish a nonlinear correspondence between input features and output response, enabling the model to express complex feature relationships. The activation output refers to the new feature vector generated after the function is calculated, which is used as the input of the second linear transformation layer.

[0098] Specifically, during this step, each numerical element of the first transformation result is sequentially input into the activation function layer. Inside the activation function layer, element-by-element calculations are performed according to a preset function form. When the rectified linear unit function (ReLU) is used, the calculation formula for the activation function is: That is, when the input value x > 0, the output is x; when the input value x > 0, the output is x. When the output is 0, the calculation process uses a comparison unit to determine the threshold and output the condition. When using the Gaussian error linear unit function (GELU), the calculation formula is as follows: In the calculation process, the input value x is first cubed and multiplied by a coefficient of 0.044715, then added to x to obtain a weighted term, which is then multiplied by a constant. And input the hyperbolic tangent function A smooth nonlinear transformation is performed, and finally, the input value is multiplied by a constant of 0.5 to achieve a continuously smooth output mapping. When using the Sigmoid linear unit function (SiLU), the calculation formula is as follows: The exponential function is calculated first during execution. The decay value is then used to calculate the Sigmoid weight term. The output is obtained by multiplying the input value x, forming a continuously differentiable nonlinear mapping. In terms of computation, the activation function layer processes the components of multiple input vectors simultaneously in a matrix-level parallel manner, independently calculating the function value for each component. The resulting array constitutes the activation output, which is consistent with the first transformation result in terms of dimension and is used as the input for the subsequent second linear transformation layer.

[0099] S303: Input the activation output to the second linear transformation layer to obtain the second transformation result.

[0100] The second linear transformation layer refers to the linear mapping layer used in the feedforward computation to further adjust the feature space representation. It performs an affine transformation on the activation output through a trainable weight matrix and bias vector, so that the features achieve spatial reprojection while maintaining the semantic structure. The second transformation result refers to the intermediate feature result after this linear transformation, which is used to generate the input data for the gated probability distribution.

[0101] Specifically, during this step, the multidimensional feature vectors of the activation output are sequentially input into the second linear transformation layer. Inside the second linear transformation layer, each input vector is subjected to linear operations according to the matrix multiplication formula, and the calculation expression is as follows:

[0102] Where a represents the activation output vector, W2 represents the weight matrix of the second linear transformation layer, and b2 represents the bias vector. The matrix multiplication operation involves taking each component of the activated output and multiplying it by the corresponding column element in the weight matrix, summing the results, and then adding the summation to the corresponding bias term to achieve an affine transformation. To prevent excessive fluctuations in feature values ​​in high-dimensional space, numerical truncation and floating-point stabilization operations can be performed during the operation to ensure that the results of each multiplication and addition are within a preset precision range. When the input is a batch of samples, the second linear transformation layer uses a parallel matrix multiplication mechanism to perform the same weight calculation on multiple input vectors simultaneously, thereby ensuring computational consistency and speed. The final result matrix is ​​the result of the second transformation.

[0103] S304: Perform a softmax operation on the second transformation result, and introduce a learnable temperature parameter in the softmax operation to generate a gated probability distribution.

[0104] In this embodiment, the softmax operation refers to the mathematical operation process of converting a set of real values ​​in the second transformation result into a probability distribution through exponential normalization mapping. The learnable temperature parameter refers to the trainable coefficient introduced in the softmax calculation, which is used to adjust the relative difference between each component in the normalization process, thereby affecting the smoothness of the output distribution. The gated probability distribution refers to the probability vector obtained after the softmax operation, and each component represents the relative probability of the expert subnetwork being activated.

[0105] Specifically, in performing this step, the second transformation result matrix is ​​first obtained from the output of the second linear transformation layer. Each vector in the matrix is ​​then input as a set of independent input component sequences to the softmax operation unit. In the softmax operation, an exponential operation is performed on each input component xi. ,in This represents a temperature parameter used to control the steepness of the distribution. When the value is small, the growth rate of the exponential function accelerates, and the probability distribution is more concentrated on the components corresponding to larger values. When the value is large, the distribution tends to be smooth, and the probabilities of all components are nearly uniform. Then, the normalization factor is obtained by summing all the exponential results. The calculation formula is as follows: , where pi represents the normalized probability value corresponding to the i-th component, and the denominator is the sum of all exponential results, used to ensure that the sum of all output components is 1. In the matrix-level implementation, the softmax operation is vectorized by simultaneously performing exponential calculation and normalization operations on each component of the input vector. All output probability values ​​form a gated probability distribution vector, used to represent the relative probability of each subnetwork being selected in the expert subnetwork set. To ensure computational stability, the maximum value of the vector is subtracted from the input value before performing the exponential operation to avoid numerical overflow. Meanwhile, temperature parameters are allowed during training. As a learnable variable, it participates in gradient updates, enabling the model to adaptively adjust the smoothness of the gating probability distribution. The final output probability distribution vector is the gating probability distribution.

[0106] In one embodiment, such as Figure 3 As shown, in step S40, according to the gating probability distribution, expert subnetworks that meet preset conditions are activated in the expert subnetwork set to participate in the calculation, and the corresponding expert subnetwork output results are obtained. Weighted fusion and normalization processing is then performed on the expert subnetwork output results according to the gating probability distribution to obtain the fused feature representation, including:

[0107] S401: Sort the expert subnetworks in the expert subnetwork set according to the numerical value of the gating probability distribution to obtain the sorting result.

[0108] In this embodiment, the sorting result refers to the ordered list obtained by rearranging the index positions of each expert subnetwork in the expert subnetwork set according to the numerical value of each component in the gating probability distribution. This ordered list is used to determine the priority order of subsequent activations. Each component in the gating probability distribution corresponds one-to-one with an expert subnetwork in the expert subnetwork set, and the magnitude of the component value reflects the activation tendency of the expert subnetwork.

[0109] Specifically, firstly, the gated probability distribution vector is read from the output of the softmax operation, and a correspondence is established between the probability components and the expert subnetwork identifiers. The identifier number of each subnetwork in the expert subnetwork set and its corresponding probability value are stored as key-value pairs. Then, a sorting operation is performed on this key-value pair set, using the magnitude of the probability values ​​as the sorting criterion. Comparison operations are used to determine the relative magnitude of the probability values ​​of each expert subnetwork. When the probability value of one expert subnetwork is greater than that of another, its position in the sequence is shifted forward, thus forming an ordered structure arranged in descending order of probability values. In practical implementation, this can be achieved by... This operation is performed using the QuickSort or HeapSort algorithms. If index stability is required under the same probability values, a stable sorting mechanism can be used with the original index order as the order reference. When the sorting operation is implemented at the matrix level, it is achieved by calculating the sorting index vector for each component of the probability vector. That is, the index is rearranged for each element in the vector to generate a sorting vector containing the identifier of the expert subnetwork. The sorting result output after sorting represents the arrangement order of each expert subnetwork in the expert subnetwork set. This result will be used in the next step to determine the target expert subnetwork set that satisfies the preset conditions of the gated probability distribution.

[0110] S402: Based on the ranking results, determine the set of target expert subnetworks whose gating probability distribution meets the preset conditions.

[0111] In this embodiment, the target expert subnetwork set refers to a set of expert subnetworks whose gating probability distribution values ​​meet preset selection conditions selected from the ranking results. The preset conditions are screening criteria used to limit the scope of expert subnetworks participating in the calculation, which may include probability threshold conditions or fixed quantity conditions, to ensure that the expert subnetworks participating in the subsequent calculation have high gating response values.

[0112] Specifically, in performing this step, the sorted sequence of expert subnetwork identifiers is first read from the sorting results, and the gated probability distribution value corresponding to each expert subnetwork is obtained. Then, the probability distribution values ​​are filtered according to preset conditions. When a probability threshold condition is used, the probability value corresponding to each expert subnetwork is compared with the threshold. The formula for calculating the size relationship is: Where Ei represents the i-th expert subnetwork, pi represents the corresponding gating probability value, and S represents the set of target expert subnetworks that meet the conditions. When a fixed number of conditions are used, the top K expert subnetworks with the highest probability values ​​are selected as the target set based on the sorting results. The calculation method can be expressed as follows:

[0113] During the filtering operation, the sorted result list is iterated through in a loop, comparing the probability value of each expert subnetwork to see if it meets the preset conditions. If the conditions are met, the subnetwork is added to the target set; otherwise, it is skipped. This process continues until the filtering is complete. To ensure the stability of the filtering results, the filtering operation can be implemented in a batch matrix. That is, multiple filtering mask matrices of gated probability vectors are calculated simultaneously under the same batch of inputs. Batch selection is achieved through logical operations. The final target expert subnetwork set contains all expert subnetworks that meet the preset conditions.

[0114] S403: Input data consisting of context-dependent representation features is sequentially input into each target expert subnetwork in the target expert subnetwork set to obtain the output results of each target expert subnetwork.

[0115] In this embodiment, the output result of the target expert subnetwork refers to the intermediate result vector obtained by each target expert subnetwork after receiving input data and undergoing a feedforward computation process in the target expert subnetwork set. The intermediate result vector is used to characterize the response features of the expert subnetwork under a specific input. The input data is an input matrix formed by vector concatenation or batch encoding of context-related representation features, which is used to drive each target expert subnetwork to perform parallel computation.

[0116] Specifically, in performing this step, the context-related representation features are first organized into an input matrix X according to the sample dimension and the feature dimension, where each row of the matrix corresponds to the context-related feature vector of an input sample. Then, according to the order in the target expert subnetwork set, the input matrix X is sequentially input into each target expert subnetwork for calculation. Within each target expert subnetwork, the input matrix X is first compared with the corresponding expert weight matrix W. Ei Perform matrix multiplication, the calculation formula is: Where Hi represents the linear transformation output of the i-th objective expert subnetwork, and b Ei This represents the corresponding bias vector. Then, a non-linear activation function is performed on each element in _i. The activation function can be ReLU, GELU, or SiLU. For example, when using the ReLU function, the activation formula is: This operation achieves nonlinear mapping through threshold comparison. When using the GELU function, the calculation formula is: When using the Sigmoid linear unit function (SiLU), the calculation formula is as follows: After the activation operation is completed, the output values ​​are recombine into a feature vector, which is used as the output result Oi of the i-th target expert subnetwork. The calculation formula can be expressed as: The outputs of all target expert subnetworks are stored sequentially as an ordered result set. .

[0117] S404: Match the output of the target expert subnetwork with the weight information corresponding to the gating probability distribution to obtain the matching result. Perform weighting operation based on the matching result to obtain the weighted result. Perform normalization processing on the weighted result to obtain the fused feature representation.

[0118] In this embodiment, the fusion feature representation refers to the feature vector obtained by performing weighting and normalization processing on the output results of multiple target expert sub-networks based on the weight information of the gating probability distribution. It is used to form a unified feature expression among different expert sub-networks. The matching result refers to the one-to-one correspondence established between the gating probability distribution and the output results of the target expert sub-networks, which is used to calibrate the weighting coefficient corresponding to each output result.

[0119] Specifically, when performing this step, the set of output vectors of each target expert subnetwork is first obtained from the output result set of the target expert subnetwork. And extract the corresponding weight information vector from the gating probability distribution. During the matching phase, an index-based mapping operation is used to establish a correspondence between the two, that is, to establish a one-to-one matching mapping between each output result Oi and its corresponding probability weight pi, forming a set of matching results. During the weighting phase, an element-wise weighted operation is performed on each matching pair, calculated using the following formula: Where Ri represents the weighted output of the i-th target expert subnetwork, the multiplication operation is performed at the vector level, that is, each component of the output vector Oi is multiplied by the scalar pi, realizing linear scaling based on probability weights. All weighted outputs Ri are summed sequentially to obtain the overall weighted result R, and the calculation formula is as follows: In the normalization stage, to ensure consistency in the numerical ranges output by different experts, the weighted result R is normalized, often using L2 normalization, which divides each component of R by its L2 norm. The formula is as follows:

[0120] Where F represents the fusion feature representation, This represents the L2 norm of the weighted result vector, used to maintain the overall numerical stability of the vector.

[0121] In one embodiment, such as Figure 4As shown, in step S50, i.e., in the joint training operation, an expert equilibrium constraint loss function is constructed based on the activation frequency of each expert sub-network to apply load balancing constraints to the activation process of the expert sub-networks, thereby obtaining the model parameters after joint training, including:

[0122] S501: In the joint training operation, the activation frequency information of each expert subnetwork is obtained by counting the number of activations of each expert subnetwork in multiple training samples.

[0123] In this embodiment, activation frequency information refers to the ratio between the number of times each expert subnetwork is activated when processing all training samples and the total number of samples during the joint training operation. It is used to reflect the usage ratio of each expert subnetwork during the training process. The number of activations refers to the cumulative number of times an expert subnetwork is selected by the gating probability distribution and participates in the calculation in one forward computation.

[0124] Specifically, when performing this step, the counting matrix is ​​first initialized before the joint training operation begins. Where N represents the number of subnetworks in the expert subnetwork set, and each ci is initially set to 0. Subsequently, in each training batch, based on the output of the gating probability distribution, the set of expert subnetworks activated in the current batch is determined. For each activated expert subnetwork in the set, a count accumulation operation is performed, and its corresponding count value is updated according to the following formula: This operation is performed once during the forward propagation phase of each training sample. By iterating through the activation records of all samples, the activation counts of each expert subnetwork are gradually accumulated. After the joint training operation has traversed all training samples, the count value of each expert subnetwork is divided by the total number of samples T to calculate the activation frequency information. The calculation formula is as follows:

[0125] , where fi represents the activation frequency of the i-th expert subnetwork, ci represents its cumulative activation count, and T represents the total number of training samples.

[0126] S502: Calculate the average activation frequency based on the activation frequency information to obtain the target equilibrium benchmark.

[0127] In this embodiment, the target equilibrium benchmark refers to the average reference value calculated based on the activation frequency information of all expert sub-networks. It is used to measure the overall load level among expert sub-networks and to provide a basic standard for the subsequent calculation of frequency deviation and equilibrium constraints. The average activation frequency refers to the arithmetic mean of the activation frequency information of all expert sub-networks, reflecting the average usage ratio of the entire expert sub-network set.

[0128] Specifically, when performing this step, the activation frequency information set is first... The activation frequency values ​​of each expert subnetwork are read, and the sum of all activation frequencies is calculated using the following formula: Where Sf represents the sum of activation frequencies of all expert subnetworks, and fi represents the activation frequency of the i-th expert subnetwork. Then, the sum is divided by the number of expert subnetworks N to obtain the average activation frequency, calculated using the following formula: ,in, The average activation frequency is represented by a vectorized averaging operation in the batch matrix implementation. That is, summation and division operations are performed on the vector storing the activation frequency information to obtain a scalar output. To prevent tiny numerical deviations caused by floating-point precision errors, fixed-point truncation or rounding operations can be used during the calculation process to ensure calculation stability. The final calculated average activation frequency is used as the target equalization benchmark.

[0129] S503: Calculate the difference between the activation frequency of each expert subnetwork and the target equilibrium benchmark to obtain the frequency deviation.

[0130] In this embodiment, frequency deviation refers to the numerical result obtained by calculating the difference between the activation frequency of each expert subnetwork and the target equilibrium benchmark during joint training operations. It is used to reflect the deviation of the actual activation level of the expert subnetwork from the overall equilibrium level.

[0131] Specifically, the activation frequency information set of each expert subnetwork is first obtained. And read the target equilibrium benchmark obtained through average calculation. Then, the difference calculation operation is performed sequentially according to the index order of the expert subnetworks. The activation frequency of each expert subnetwork is subtracted from the target equilibrium benchmark. The calculation formula is as follows: Where di represents the frequency deviation corresponding to the i-th expert subnetwork, and fi represents the activation frequency of the expert subnetwork. This represents the target equilibrium benchmark.

[0132] S504: Construct an expert balance constraint loss function based on the frequency deviation, apply load balancing constraints to the activation process of the expert sub-network, update the joint training parameters under the load balancing constraint, and obtain the joint trained model parameters.

[0133] In this embodiment, the expert balance constraint loss function is a constraint function that uses frequency deviation as an input variable and measures the degree to which the activation frequency of each expert sub-network deviates from the target balance benchmark through a mathematical function. The load balance constraint is to introduce this loss function into the optimization objective so that the joint training process can adjust the activation ratio of each expert sub-network when updating parameters. The model parameters after joint training refer to the final set of parameters obtained after multiple rounds of iterative optimization under the condition of load balance constraint.

[0134] Specifically, frequency deviations are read and used as input variables for loss function construction. During the loss function construction phase, the squares of each frequency deviation are summed to measure the overall imbalance. The calculation formula is as follows:

[0135] Where Lbal represents the expert equilibrium constraint loss value, N represents the number of expert subnetworks, and di represents the frequency deviation of the i-th expert subnetwork. Subsequently, this constraint loss is weighted and combined with the main task loss function in the joint training. Let the main task loss be Lmain, then the joint optimization objective can be expressed as: ,in, These are adjustable weighting coefficients used to control the strength of the balance constraint's influence on the overall training objective. They are adjusted during training. The value of the parameter is used to balance the relationship between the main task optimization and the load constraint. Under the condition of applying the load balancing constraint, based on the above joint optimization objective, the joint training parameters are updated using a gradient descent-type optimization algorithm. The gradient direction of the loss function with respect to each parameter is calculated through backpropagation, and the weight parameters and gating parameters of each expert subnetwork in the model are gradually adjusted in multiple iterations to make the activation frequency gradually approach a balanced state. The parameter set obtained after the training process converges is the model parameter after joint training.

[0136] In one embodiment, such as Figure 5 As shown, in step S504, an expert balancing constraint loss function is constructed based on the frequency deviation, and a load balancing constraint is applied to the activation process of the expert sub-network. Under the condition of applying the load balancing constraint, the joint training parameters are updated to obtain the jointly trained model parameters, including:

[0137] S5041: Calculate the load deviation value of each expert subnetwork based on the frequency deviation.

[0138] In this embodiment, the load deviation value refers to the quantitative index calculated based on the frequency deviation in the joint training operation. It is used to represent the degree of deviation of the actual activation load of each expert sub-network from the target balanced load. Its value reflects the intensity of the load imbalance of the expert sub-network in the activation distribution.

[0139] Specifically, when performing this step, the frequency deviation set corresponding to each expert subnetwork is first obtained from the previous steps. And set the load mapping coefficient. As an adjustment parameter, the quantization sensitivity of the bias is controlled. Then, the absolute value of the frequency bias of each expert subnetwork is taken and multiplied by the mapping coefficient. The calculation formula is as follows: Where li represents the load deviation value of the i-th expert subnetwork, and di represents the frequency deviation of the expert subnetwork. This represents the deviation mapping coefficient, which is adjusted... The value of can be adjusted during training to optimize the sensitivity of load bias calculation. During calculation, to avoid excessive gradients caused by abnormal biases, the frequency bias can be clipped and limited before calculation. When |di| exceeds a preset upper limit... At that time, perform a truncation operation: This operation ensures that the calculated load deviation remains numerically stable. Subsequently, all calculated load deviation values ​​are vectorized and stored, forming a sequence of load deviation values ​​arranged in expert index order. In the matrix implementation, the calculation process can be completed in parallel through broadcasting and element-wise operations, thereby enabling load bias calculation under batch sampling. Each output *li* represents the load bias value of the corresponding expert subnetwork.

[0140] S5042: Based on the load deviation value, generate the balance coefficient, introduce the balance coefficient into the expert balance constraint loss function, and perform a weighted combination of the expert balance constraint loss function to obtain the loss function expression containing the load balance constraint term.

[0141] In this embodiment, the balance coefficient refers to the weighting factor calculated based on the load deviation value, which is used to adjust the contribution ratio of each expert subnetwork in the balance constraint loss. The load balance constraint term refers to the additional term introduced into the expert balance constraint loss function with the balance coefficient as the adjustment variable, which is used to suppress the situation where the activation frequency difference is too large during the loss optimization process.

[0142] Specifically, when performing this step, the set of load deviation values ​​is first obtained from the previous steps. And set smooth adjustment parameters. To control the smoothness of the distribution of the balance coefficient, the corresponding balance coefficient is generated by performing a reverse normalization operation on the load deviation value. The calculation formula is as follows: Where wi represents the balancing coefficient of the i-th expert subnetwork, and li represents the load deviation value of the expert subnetwork. To smoothly adjust the parameters and prevent the gradient vanishing problem caused by excessively small expert coefficients due to large load deviations, the equilibrium coefficients are calculated and then introduced into the expert equilibrium constraint loss function. A weighted correction is applied to each frequency deviation squared term of the original constraint terms, resulting in the following expression for the corrected loss function:

[0143] in, This represents the expert balance constraint loss function after adding the balance coefficient, where di represents the frequency deviation of the corresponding expert sub-network. Through dynamic weighting with the balance coefficient, the proportion of the less loaded expert sub-network in the loss calculation is relatively increased, while the constraint strength of the more loaded expert sub-network is correspondingly weakened, thereby achieving overall load balancing adjustment. Subsequently, the loss function after balance coefficient correction will be introduced. The loss function expression, which includes a load balancing constraint term, is formed by weighting and combining the loss Lmain from the main task in the joint training.

[0144] Where Ltotal represents the total loss function, To control the weighting coefficients of the load balancing constraint strength, by adjusting... The value of can balance the impact between the main task optimization and the expert equilibrium constraint.

[0145] S5043: Calculate gradient information during joint training based on the loss function expression, perform parameter updates on the activation process of the expert sub-network, and output the model parameters after joint training.

[0146] In this embodiment, gradient information refers to the direction of parameter change obtained by backpropagation of the loss function expression, which is used to guide the optimization and update process of each parameter of the model. The model parameters after joint training refer to the set of convergent parameters obtained after multiple rounds of gradient updates and optimization iterations.

[0147] Specifically, firstly based on the loss function expression Backpropagation is performed on all learnable parameters participating in the joint training. The partial derivatives of the loss function with respect to each parameter are expanded and accumulated according to the chain rule to obtain the gradient information set. Each gradient component g_i represents a corresponding learnable parameter. The sensitivity to changes in the loss function is calculated using the following formula: During backpropagation, gradient calculation and gradient accumulation operations are performed on the attention layer parameters, gate layer parameters, and weight matrices in the expert subnetwork of the Transformer coding network to ensure that each layer parameter receives effective feedback from the loss signal. To prevent numerical overflow or gradient explosion during gradient propagation, a gradient clipping threshold tau is set. When the gradient magnitude exceeds this threshold, the following clipping calculation is performed: This process limits the magnitude of the gradient while maintaining its direction, ensuring the stability of subsequent parameter updates. After obtaining the corrected gradient information, a parameter update operation is performed, using an optimization algorithm based on an adaptive learning rate (such as the Adam optimizer) to update each parameter. The parameter update calculation formula is as follows: ,in, This represents the parameter value at the t-th iteration. This represents the adaptive learning rate for the current iteration, dynamically adjusted based on the exponentially weighted average of the sum of squared historical gradients to achieve a smooth decay of the learning rate. After the parameters are updated, forward propagation and loss calculation are re-executed on the new parameter set, and the backpropagation and update process is repeated until the validation set loss converges or the preset number of iterations is met. At this point, the output set of all parameters represents the parameters of the jointly trained model.

[0148] In one embodiment, such as Figure 6 As shown, in step S60, based on the jointly trained model parameters, text correction inference is performed, and the corrected text result is output, including...

[0149] S601: Obtain the input text information to be corrected, perform word embedding and position embedding operations on the input text information, and obtain the embedded representation of the text to be corrected.

[0150] In this embodiment, the input text information to be corrected enters the embedding stage after unified input processing. The word embedding and position embedding operations are kept consistent with those in the training stage to ensure the consistency of the correspondence in the encoding space.

[0151] Specifically, the input text information to be corrected is input into the embedding layer in word order. An index mapping operation is performed by looking up the embedding matrix to map each word number xi to its corresponding continuous vector ei. The calculation process is as follows: Here, WE is the word embedding matrix learned during the training phase. Subsequently, to maintain the integrity of the sequence's positional information, a positional encoding vector is calculated for each word position index pi. The calculation method remains consistent with that of the training phase. A vector sequence that can distinguish different indices is generated using sine and cosine functions to obtain the position embedding matrix. After calculating word embeddings and positional embeddings, they are added element-wise to achieve linear fusion of semantic and positional information, resulting in the embedding representation matrix of the text to be corrected. This embedding represents the input features of the subsequent Transformer coding network, used to perform forward computation in the text error correction inference stage.

[0152] S602: Input the embedded representation into the Transformer encoding network loaded with the parameters of the jointly trained model, perform forward computation, and obtain the context-related representation features of the text to be corrected.

[0153] In this embodiment, the context-related representation feature of the text to be corrected refers to the high-dimensional vector representation formed in the Transformer coding network after multi-layer self-attention structure and feedforward layer calculation, which can comprehensively reflect the inter-word dependency relationship and context semantic dependence within the input text. This representation has adaptability to language modeling and text correction tasks under the action of model parameters after joint training.

[0154] Specifically, the embedding representation matrix Einput is input into a Transformer encoding network loaded with the jointly trained model parameters. Within each encoding unit, a multi-head self-attention operation based on the query vector, key vector, and value vector is first performed, and obtained through linear mapping:

[0155] Where WQ, WK, and WV are the weight matrices learned during the joint training phase. Then, the attention weight matrix Ah for each attention head is calculated: Where dk is the dimension of the key vector. The multi-head attention output is concatenated and linearly mapped before being input to the feedforward layer, where linear transformations and non-linear activation operations are performed to non-linearly reconstruct and extract information from the context features. Through multi-layered stacked attention computation and feedforward operations, the model progressively strengthens the association weights between words in the input text in each layer's output, enabling the representation features to express the semantic consistency and dependencies of the context. After all encoding layers have been computed, the top-level output is used as the context-related representation features of the text to be corrected.

[0156] S603: Input the context-related representation features into the feedforward layer based on the multi-expert hybrid mechanism, generate the gated probability distribution based on the jointly trained model parameters, and activate the corresponding expert sub-network set to obtain the fused feature representation.

[0157] In this embodiment, the fusion feature representation refers to the comprehensive feature result obtained by performing multi-path nonlinear mapping and gating selection on context-related representation features through a multi-expert hybrid mechanism. This feature establishes a weighted fusion relationship among the response outputs of multiple expert sub-networks and is used to express the composite representation capability of text in terms of semantics, syntax and context dependence.

[0158] Specifically, the context-related representation features of the text to be corrected are input into the multi-expert hybrid feedforward layer defined by the jointly trained model parameters. In the feedforward layer, the gating probability distribution is first calculated based on the gating network parameters stored in the model. The activation probability vector of each expert sub-network is obtained through two-level linear transformation and nonlinear function mapping. , where N is the number of expert subnetworks, and each pi represents the activation strength of the i-th expert subnetwork. The generation process of the gating probability distribution can be formally represented as: Where H represents the context-dependent representation features of the input, and W1, W2 and b1, b2 are the weights and biases of the gating layer in the jointly trained model parameters. For activation function, A learnable temperature parameter is used to adjust the smoothness of the distribution. After obtaining the gating probability distribution, a set of expert subnetworks that meet a preset threshold condition are selected according to their probability magnitude. ,in The activation threshold is used. The selected expert subnetworks are sequentially input with the feature H, and each expert subnetwork performs an independent nonlinear feedforward operation: Where Wi and bi are the weights and biases of the i-th expert subnetwork, respectively. This is the nonlinear activation function within the expert group. Subsequently, a weighted fusion calculation is performed on the outputs of each activated expert according to the gating probability distribution weights, yielding the fused feature representation:

[0159] .

[0160] S604: Input the fused feature representation into the output head corresponding to the text correction task, perform inference calculation, obtain the error correction prediction result, and generate the corrected text result based on the error correction prediction result.

[0161] Specifically, the fusion feature representation The input is fed into the output head corresponding to the text correction task. First, a matrix multiplication operation is performed in the linear mapping layer of the output head to map the feature dimension D to the vocabulary dimension V, resulting in the prediction score matrix Z. The calculation formula is as follows: in, , Output matrix Then, a softmax operation is performed along the vocabulary dimension to normalize the prediction score matrix, resulting in the error correction prediction probability matrix Pout: By selecting the candidate word meta-index with the highest probability at each input position The predicted sequence is obtained. Based on the index correspondence between the predicted sequence and the input text sequence, a lexical mapping table is established. Lexical terms predicted as incorrect are replaced, and structural adjustments are made when insertion or deletion markers are present, generating an updated text sequence. After sequence replacement and adjustment, reverse word segmentation and character concatenation operations are performed on the predicted index sequence to restore the internal numbered sequence to natural language text form. Finally, the corrected text result is output.

[0162] In one embodiment, such as Figure 7 As shown, in step S604, which generates the corrected text result based on the error correction prediction result, the following is included:

[0163] S6041: Determine the candidate error correction word sequence based on the prediction score information in the error correction prediction results.

[0164] Specifically, the prediction score information corresponding to each input position in the error correction prediction result is read. All candidate words at each position are sorted in descending order of score. Several candidate words with the highest scores are selected as a candidate set. During the sorting process, the scores are normalized to ensure the comparability of scores between different input positions. After sorting and filtering, the candidate words selected at each input position are arranged in the order of the original text to generate a candidate error correction word sequence.

[0165] S6042: Perform a probability screening operation on the candidate error correction word sequence to select the target error correction word sequence that meets the preset confidence conditions.

[0166] In this embodiment, the pre-set confidence condition refers to the screening criteria used to determine whether each candidate lexical unit in the candidate error correction word sequence has sufficient reliability.

[0167] Specifically, based on the prediction confidence information of each candidate word in the error correction prediction results, the candidate error correction word sequence is traversed position by position, and the confidence value of each candidate word is read. Then, each confidence value is compared with the preset confidence condition. If the value is higher than the preset threshold, the candidate word is judged to have sufficient prediction reliability at that position and is retained; otherwise, it is marked as failing and removed from the candidate set. When multiple candidate words meet the threshold condition, a comprehensive comparison is made based on contextual relevance and the smoothness of confidence at adjacent positions. Words with higher confidence values ​​and better semantic continuity in the context are selected as the target output word at that position. After completing the probability screening of all positions, the sequence is reorganized according to the word order of the input text to form the target error correction word sequence that meets the preset confidence condition.

[0168] S6043: Generate error correction mapping information based on the correspondence between the target error correction word sequence and the input text information.

[0169] In this embodiment, the error correction mapping information refers to the structured mapping data generated by comparing the one-to-one correspondence between the target error correction word sequence and each word in the input text information.

[0170] Specifically, the input text information and the target error-correcting word sequence are first aligned and compared one by one according to the word order to determine the correspondence between them in sequence length and position. For each comparison position, the original word in the input text and the predicted word at the corresponding position in the target error-correcting word sequence are read, and it is determined whether they are the same. If the predicted word and the original word are the same, they are marked as "keep" in the mapping table. If the predicted word and the original word are different, the replacement operation information is recorded in the mapping table, including the replacement position index, the original word content, and the target word content. When an insertion or deletion marker is detected in the target error-correcting word sequence, "insert" or "delete" operation entries are added to the mapping table according to their position markers in the sequence. The insertion entry contains the target insertion position and the inserted word content, and the deletion entry contains the position index of the word to be deleted. Finally, all operation information is integrated in the original order of the input text to generate error correction mapping information.

[0171] S6044: Based on the error correction mapping information, replace the corresponding positions in the input text information to obtain the corrected text result.

[0172] Specifically, the parsing is performed sequentially according to the word position indexes recorded in the error correction mapping information. For operations marked "replace," the corresponding word position is located in the input text, the original word is replaced with the target word content, and the index positions of adjacent words are readjusted after the replacement. For operations marked "insertion," the target word is inserted before the specified position according to the position index in the mapping table, while maintaining the original order structure of the text sequence. For operations marked "deletion," the original word is removed at the corresponding index position, and the position information of its subsequent words is updated to maintain sequence continuity. After completing all mapping operations, the adjusted word sequence is reordered and concatenated to restore a continuous text form, ensuring that the connections between words conform to the language structure requirements. Finally, a unified encoding format check is performed on the concatenated text to ensure that the error-corrected text result is consistent with the input text in terms of character set and encoding, thus obtaining an error-corrected text result that can be directly output or used for downstream task processing.

[0173] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0174] In one embodiment, a text correction device based on a multi-expert hybrid mechanism is provided, which corresponds one-to-one with the text correction methods based on a multi-expert hybrid mechanism described in the above embodiments. For example... Figure 8As shown, this text correction device based on a multi-expert hybrid mechanism includes a text embedding processing module, a multi-head attention encoding module, a multi-expert gating feedforward module, an expert fusion computing module, a multi-task joint training module, and a text correction inference module. Detailed descriptions of each functional module are as follows:

[0175] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A text correction method based on a multi-expert hybrid mechanism, characterized in that, The text correction method based on a multi-expert hybrid mechanism includes: Obtain the input text information, perform word embedding and position embedding operations on the input text information to obtain the embedded representation containing semantic vectors and position information; The embedded representation is input into a Transformer encoding network containing a multi-layer encoding structure. In the Transformer encoding network, multi-head self-attention computation based on query vector, key vector and value vector is performed to obtain context-dependent representation features. In the feedforward stage of the Transformer coding network, a feedforward layer based on a multi-expert hybrid mechanism is constructed. Two-layer nonlinear gating computation is performed on each context-related representation feature to obtain the gating probability distribution for the expert sub-network set. The step of performing a two-layer nonlinear gating computation on each context-dependent representation feature to obtain a gating probability distribution for the expert subnetwork set includes: Each context-dependent representation feature is input into the first linear transformation layer, and a linear mapping operation is performed on the context-dependent representation features to obtain the first transformation result; The first transformation result is input into the activation function layer, and nonlinear activation processing is performed on the first transformation result to obtain the activation output; The activation output is input to the second linear transformation layer to obtain the second transformation result; The softmax operation is performed on the second transformation result. A learnable temperature parameter is introduced into the softmax operation to generate a gated probability distribution. Based on the gating probability distribution, expert subnetworks that meet the preset conditions are activated in the expert subnetwork set to participate in the calculation, and the corresponding expert subnetwork output results are obtained. The expert subnetwork output results are then subjected to weighted fusion and normalization processing according to the gating probability distribution to obtain the fusion feature representation. The fused feature representation is input into the multi-task output head structure, and joint training operations of language modeling, text correction and named entity recognition tasks are performed. In the joint training operation, an expert balance constraint loss function is constructed based on the activation frequency of each expert sub-network to perform load balancing constraint on the activation process of the expert sub-network, and the model parameters after joint training are obtained. In the joint training operation, an expert equilibrium constraint loss function is constructed based on the activation frequency of each expert sub-network to apply load balancing constraints to the activation process of the expert sub-networks, thereby obtaining the model parameters after joint training, including: In the joint training operation, the activation frequency information of each expert subnetwork is obtained by counting the number of activations of each expert subnetwork in multiple training samples. The average activation frequency is calculated based on the activation frequency information to obtain the target equilibrium benchmark. The frequency deviation is obtained by calculating the difference between the activation frequency of each expert subnetwork and the target equilibrium benchmark. Based on the frequency deviation, an expert load balancing constraint loss function is constructed, and a load balancing constraint is applied to the activation process of the expert sub-network. Under the condition of applying the load balancing constraint, the joint training parameters are updated to obtain the joint training model parameters. Based on the jointly trained model parameters, text error correction inference is performed, and the corrected text result is output.

2. The text correction method based on a multi-expert hybrid mechanism according to claim 1, characterized in that, The process involves activating expert subnetworks that meet preset conditions within the expert subnetwork set based on the gating probability distribution to participate in the computation, obtaining the corresponding expert subnetwork output results, and performing weighted fusion and normalization processing on the expert subnetwork output results according to the gating probability distribution to obtain the fused feature representation, including: Based on the numerical values ​​of the gating probability distribution, the expert subnetworks in the expert subnetwork set are sorted to obtain the sorting results; Based on the ranking results, determine the set of target expert subnetworks whose gating probability distributions satisfy preset conditions; The input data, consisting of context-dependent representation features, is sequentially fed into each target expert subnetwork in the target expert subnetwork set to obtain the output results of each target expert subnetwork. The output of the target expert subnetwork is matched with the weight information corresponding to the gating probability distribution to obtain the matching result. A weighted operation is performed based on the matching result to obtain the weighted result. The weighted result is then normalized to obtain the fused feature representation.

3. The text correction method based on a multi-expert hybrid mechanism according to claim 1, characterized in that, The process involves constructing an expert load balancing constraint loss function based on frequency deviation, applying load balancing constraints to the activation process of the expert sub-network, and updating the joint training parameters under the applied load balancing constraints to obtain the jointly trained model parameters, including: Calculate the load deviation value of each expert subnetwork based on the frequency deviation; Based on the load deviation value, a balance coefficient is generated. The balance coefficient is then introduced into the expert balance constraint loss function. The expert balance constraint loss function is then weighted and combined to obtain a loss function expression that includes load balance constraint terms. The gradient information during the joint training process is calculated based on the loss function expression. The activation process of the expert sub-network is then updated with parameters, and the joint-trained model parameters are output.

4. The text correction method based on a multi-expert hybrid mechanism according to claim 1, characterized in that, The process of performing text correction inference based on the jointly trained model parameters and outputting the corrected text result includes: Obtain the input text information to be corrected, perform word embedding and position embedding operations on the input text information to obtain the embedded representation of the text to be corrected; The embedded representation is input into a Transformer encoding network loaded with the parameters of the jointly trained model, and forward computation is performed to obtain the context-related representation features of the text to be corrected. The context-related representation features are input into the feedforward layer based on the multi-expert hybrid mechanism. The gated probability distribution is generated based on the jointly trained model parameters, and the corresponding expert sub-network set is activated to obtain the fused feature representation. The fused feature representation is input into the output header of the text correction task, inference calculation is performed to obtain the error correction prediction result, and the corrected text result is generated based on the error correction prediction result.

5. The text correction method based on a multi-expert hybrid mechanism according to claim 4, characterized in that, The generation of the corrected text result based on the error correction prediction result includes: Based on the prediction score information in the error correction prediction results, candidate error correction word sequences are determined; Perform a probability screening operation on the candidate error correction word sequences and select the target error correction word sequences that meet the preset confidence conditions; Based on the correspondence between the target error correction word sequence and the input text information, error correction mapping information is generated; Based on the error correction mapping information, the corresponding positions in the input text information are replaced to obtain the corrected text result.

6. A text correction device based on a multi-expert hybrid mechanism, characterized in that, The text correction device based on a multi-expert hybrid mechanism includes: The text embedding processing module is used to acquire input text information, perform word embedding and position embedding operations on the input text information, and obtain an embedded representation containing semantic vectors and position information. The multi-head attention encoding module is used to input the embedded representation into the Transformer encoding network, which contains a multi-layer encoding structure. In the Transformer encoding network, multi-head self-attention computation based on query vector, key vector and value vector is performed to obtain context-related representation features. The multi-expert gated feedforward module is used to construct a feedforward layer based on a multi-expert hybrid mechanism in the feedforward stage of the Transformer coding network. It performs two-layer nonlinear gated computation on each context-related representation feature to obtain the gated probability distribution for the set of expert sub-networks. The multi-expert gated feedforward module includes: Each context-dependent representation feature is input into the first linear transformation layer, and a linear mapping operation is performed on the context-dependent representation features to obtain the first transformation result; The first transformation result is input into the activation function layer, and nonlinear activation processing is performed on the first transformation result to obtain the activation output; The activation output is input to the second linear transformation layer to obtain the second transformation result; The softmax operation is performed on the second transformation result. A learnable temperature parameter is introduced into the softmax operation to generate a gated probability distribution. The expert fusion computing module is used to activate expert subnetworks that meet preset conditions in the expert subnetwork set to participate in the calculation according to the gating probability distribution, obtain the corresponding expert subnetwork output results, and perform weighted fusion and normalization processing on the expert subnetwork output results according to the gating probability distribution to obtain the fusion feature representation; The multi-task joint training module is used to input the fused feature representation into the multi-task output head structure and perform joint training operations for language modeling, text correction and named entity recognition tasks. In the joint training operation, an expert balance constraint loss function is constructed based on the activation frequency of each expert sub-network to perform load balancing constraints on the activation process of the expert sub-networks and obtain the model parameters after joint training. The multi-task joint training module includes: In the joint training operation, the activation frequency information of each expert subnetwork is obtained by counting the number of activations of each expert subnetwork in multiple training samples. The average activation frequency is calculated based on the activation frequency information to obtain the target equilibrium benchmark. The frequency deviation is obtained by calculating the difference between the activation frequency of each expert subnetwork and the target equilibrium benchmark. Based on the frequency deviation, an expert load balancing constraint loss function is constructed, and a load balancing constraint is applied to the activation process of the expert sub-network. Under the condition of applying the load balancing constraint, the joint training parameters are updated to obtain the joint training model parameters. The text correction inference module is used to perform text correction inference based on the jointly trained model parameters and output the corrected text results.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the text correction method based on a multi-expert hybrid mechanism as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the text correction method based on a multi-expert hybrid mechanism as described in any one of claims 1 to 5.

Images