Training method of natural language processing model based on Transformer architecture

Abstract

The present invention provides a training method for a natural language processing model based on a Transformer architecture, which belongs to the technical field related to big data processing, and comprises the steps of obtaining input text data and preprocessing it, sending the preprocessed text into a natural language processing model based on a Transformer network architecture for training, training the feedforward network of the network using a dynamic adaptive neural optimization algorithm, using the Transformer network architecture to perform context-aware semantic analysis on the input text, and then using a multi-layer extreme learning machine classification algorithm based on path integrals to establish a probability amplitude model from input features to output classification results using the Feynman path integral strategy. The present invention improves the accuracy of the model in task understanding and semantic analysis by optimizing the core algorithm of the network, and the method can capture complex semantic relationships, making the processing of natural language more intelligent.

Description

Technical Field

[0001] The present invention mainly relates to the technical field related to data processing, and specifically to a training method for a natural language processing model based on a Transformer architecture. Background Art

[0002] Natural language processing technology has made significant progress in recent years. In particular, driven by deep learning models, large-scale pre-trained models such as Transformer (such as BERT, GPT, and T5) have significantly improved computers' ability to understand human language. Such models can process long texts and capture complex semantic dependencies through the self-attention mechanism, greatly enhancing the ability to understand natural language in human-computer interaction. However, existing natural language processing technology still has certain limitations in practical applications.

[0003] In the prior art, traditional natural language processing models based on the Transformer network architecture perform poorly when processing long texts and complex semantics. They find it difficult to accurately capture global semantic relationships and the core intent of complex instructions, and are prone to task parsing errors. In the natural language training process, the traditional gradient descent method and its variant algorithms are prone to gradient vanishing, gradient exploding, or falling into local optimal solutions in high-dimensional parameter spaces, affecting the training efficiency and performance of the model. At the same time, the fixed learning rate method cannot dynamically adapt to the model training process.

[0004] In the existing technology, the natural language processing model based on the Transformer network architecture has weak generalization ability for new data and complex tasks during task classification and intent parsing. The model is likely to perform well in specific fields, but the effect decreases when dealing with broader and more diverse tasks. Summary of the invention

[0005] In order to address the deficiencies of current technology, the present invention combines existing technologies and, starting from practical applications, provides a training method for a natural language processing model based on the Transformer architecture. By optimizing the core algorithm of the Transformer, the accuracy of the model in task understanding and semantic analysis is improved. The method can capture complex semantic relationships, making the processing of natural language more intelligent.

[0006] The technical solution of the present invention is as follows:

[0007] The training method of a natural language processing model based on a Transformer architecture comprises the steps of obtaining input text data and preprocessing it, sending the preprocessed text to a natural language processing model based on a Transformer network architecture for training, and training a feedforward network of the Transformer network architecture using a dynamic adaptive neural optimization algorithm, and the steps are as follows:

[0008] S11, setting initial parameters of the feedforward network, the initial parameters including initial weights and initial biases, and defining control parameters that affect adjustment of the feedforward network parameters;

[0009] S12, performing iterative training of the feedforward network, dynamically adjusting the control parameters according to the number of iterations, and using a decay function to simulate energy dissipation in the physical system;

[0010] S13, using the current feedforward network parameters and control parameters, calculating the state variables through a dynamic adaptive function;

[0011] S14, transforming the state variables through nonlinear functions to generate new parameter update directions, and at the same time, implementing random interference using a random pulse interference method strategy;

[0012] S15, updating the feedforward network parameter weights and biases according to the converted state variables and the established learning rate, and at the same time, using an adaptive threshold dynamic adjustment mechanism to automatically adjust the update thresholds of key parameters according to the current performance of the model;

[0013] S16. After each iteration, the performance of the current model on the validation set is evaluated by calculating the loss function of the feedforward network.

[0014] S17. Adjust the expansion coefficient and the learning rate of the feedforward network weights based on the results of the performance evaluation to optimize the performance of subsequent iterations.

[0015] Further, in step S12, the control parameter adjustment formula is expressed as follows:

[0016]

[0017] In the formula, is the smoothing coefficient; is the control parameter of the feedforward network at the tth iteration; is the current iteration number, is the initial value of the control parameter of the feedforward network;

[0018] In step S13, the dynamic adaptive function calculates the state variable in the following manner:

[0019]

[0020] In the formula, Indicates The state variables of the iteration; is the hyperbolic tangent function; is the expansion factor; For the The weights of the feedforward network for iteration .

[0021] Further, in step S14, the state variables are transformed through a nonlinear function to generate a new parameter update direction, specifically:

[0022] The state variables are used to calculate the next weight update direction, expressed as:

[0023]

[0024] In the formula, The weight update direction for the feedforward network; is the learning rate of the weights of the feedforward network; For the The pulse interference term of the iteration;

[0025] The calculation method of the pulse interference term is as follows:

[0026]

[0027] In the formula, is the interference intensity coefficient, is a random variable with a value of {0, 1}, according to the probability generate; is the probability threshold; is a sign function, The role of is to ensure that the disturbance direction has the same sign as the state variable to maintain the consistency direction of the update.

[0028] Further, in step S15, the feedforward network parameter weights and biases are updated according to the converted state variables and the established learning rate, and the updating method is expressed as follows:

[0029]

[0030]

[0031] In the formula, It is The loss function of the feedforward network for this iteration is, is the learning rate of the feedforward network bias, is the gradient of the loss function with respect to the bias; For the The weights of the feedforward network at iteration ; For the The bias of the feedforward network for the iteration; For the The bias of the feedforward network for the iteration; For the Adaptive threshold for iterations;

[0032] The adaptive threshold calculation method is as follows:

[0033]

[0034] In the formula, is the target loss value, is a hyperparameter that adjusts the threshold sensitivity;

[0035] In step S16, the loss function of the feedforward network is expressed as follows:

[0036]

[0037] In the formula, It is The true value of the sample input to the feedforward network, It is The iteration model pair is The predicted value of the sample input to the feed-forward network, is the number of samples input to the feedforward network in the current batch.

[0038] Further, in step S17, the learning rate adjustment method of the expansion coefficient and the weight of the feedforward network is expressed as follows:

[0039]

[0040]

[0041] In the formula, For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for iterations; For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for the iteration; is the first performance adjustment factor, is the second performance adjustment factor.

[0042] Furthermore, after using the Transformer network architecture to perform context-aware semantic analysis on the input text, the user instructions are classified into specific task types through a task intention classification algorithm. The task intention classification algorithm adopts a multi-layer extreme learning machine classification algorithm based on path integral. On the basis of the traditional extreme learning machine, the Feynman path integral strategy is used to establish a probability amplitude model from input features to output classification results.

[0043] Furthermore, the training steps of the multi-layer extreme learning machine classification algorithm based on path integral are as follows:

[0044] S21, initializing and setting a multi-layer extreme learning machine, randomly initializing the weights and biases of the multi-layer extreme learning machine, and using a chaotic mapping function to increase the diversity of parameters and the robustness of the model;

[0045] S22, input the features output by the Transformer network architecture into the multi-layer extreme learning machine classifier;

[0046] S23. Using the Feynman path integral method, calculate the probability amplitude of all possible paths from the input feature to the output result, sum these paths, and obtain the overall probability amplitude;

[0047] S24. In a multi-layer extreme learning machine, the output of each layer is calculated in turn by forward propagation, and the weight of the output layer is adjusted according to the input data;

[0048] S25. Calculate the loss function of the multi-layer extreme learning machine to measure the gap between the model prediction results and the true labels. The loss function of the multi-layer extreme learning machine takes into account the classification error and the phase information of the Feynman path integral.

[0049] S26. Use the topological soliton optimization algorithm to optimize the output layer weights of the model. By simulating the propagation of solitons in nonlinear media, find the optimal parameter configuration. The topological soliton equation is calculated based on the nonlinear Schrödinger equation, and the soliton solution is used to update the parameters of the multi-layer extreme learning machine.

[0050] S27, repeat the above steps until the preset stop iteration condition is met and the training is completed.

[0051] Further, in step S21, the chaos mapping function is expressed as follows:

[0052]

[0053] In the formula, For the The chaotic variable of the iteration; For the The chaotic variable of the iteration; is the control parameter of the chaotic map, is the modulation parameter; is the number of iterations;

[0054] The weights are initialized using chaotic variables, expressed as:

[0055]

[0056] In the formula, For multi-layer extreme learning machine The weight of the layer; is the scaling factor of the multi-layer extreme learning machine; is the weight offset of the multi-layer extreme learning machine;

[0057] In step S23, the path integral probability amplitude of the sample is calculated as follows:

[0058]

[0059] In the formula, For the The path integral probability amplitude of samples; is an imaginary unit; is the scaling constant; is the action function; is the path of system state change with the current iteration number; The number of iterations of the current multi-layer extreme learning machine training; represents the integration over all possible paths.

[0060] Further, in step S24, for the Layer learning machine, its output is expressed as:

[0061]

[0062] In the formula, For the The output vector of the layer; For the The output vector of the layer, is the activation function of the multi-layer extreme learning machine, For multi-layer learning machine Bias of the layer;

[0063] The activation function of the multi-layer extreme learning machine is calculated as follows:

[0064]

[0065] In the formula, To input plural numbers; and They are The real and imaginary parts of is a real-valued activation function; is the phase parameter, is an imaginary unit;

[0066] The phase parameter is associated with the input feature and is calculated as follows:

[0067]

[0068] In the formula, is the adaptive phase parameter activation function, which is calculated as follows:

[0069]

[0070] In the formula, is a learnable parameter; is the Sigmoid activation function.

[0071] Further, in step S25, the loss function of the multi-layer extreme learning machine is calculated as follows:

[0072]

[0073] In the formula, is the loss function of the multi-layer extreme learning machine; For the The true labels of samples; is the trade-off coefficient of the multi-layer extreme learning machine; is the target path integral probability amplitude; is the regularization coefficient of the multi-layer extreme learning machine, is the total number of training samples, For the The predicted probability density of samples is calculated as follows:

[0074]

[0075] In the formula, for The complex conjugate of For the The wave function of the samples;

[0076] In step S26, the topological soliton equation is calculated based on the nonlinear Schrödinger equation, which is expressed as follows:

[0077]

[0078] In the formula, is the wave function; is a virtual space variable; is the current iteration number; is the nonlinear coefficient, Is an imaginary unit.

[0079] Beneficial effects of the present invention:

[0080] 1. The training method provided by the present invention adopts a pre-trained model of the Transformer architecture for deep semantic parsing. In the feedforward network training process of the Transformer, a dynamic adaptive neural optimization algorithm is adopted to dynamically adjust the network parameters by simulating the steady-state dynamic system in nature, thereby effectively avoiding the gradient vanishing and local optimal problems in the traditional gradient descent method, thereby improving the training efficiency and generalization ability of the model;

[0081] In order to prevent the model from falling into the local optimal solution during the optimization process, the system adopts a random pulse interference strategy. By introducing random interference during the training process, the model's search ability in different parameter spaces is enhanced, thereby improving the model's global search performance.

[0082] By optimizing the core algorithm of Transformer to improve the accuracy of the model in task understanding and semantic analysis, the system can capture complex semantic relationships and make natural language processing more intelligent.

[0083] 2. The training method provided by the present invention proposes a multi-layer extreme learning machine algorithm based on Feynman path integral in the process of task intent classification. By calculating the probability amplitude of all possible paths of input features and combining the path integral principle in quantum mechanics to guide classification, it can better capture the potential correlation of complex task instructions and improve the accuracy of task classification, so that complex multi-step tasks can be automatically decomposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is a flow chart of the parameter optimization method during feedforward network training of the present invention.

[0085] Figure 2 It is the Transformer network model architecture.

[0086] Figure 3 The present invention is a training flow chart of the multi-layer extreme learning machine classification algorithm based on path integral. DETAILED DESCRIPTION

[0087] The present invention will be further described with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the application.

[0088] This embodiment provides a training method for a natural language processing model based on a Transformer architecture, and the main steps of the method are as follows.

[0089] Natural language acquisition and preprocessing

[0090] Users can input relevant instructions through speech or text. If it is voice input, it can first be converted into text through the voice recognition algorithm; after obtaining the text input, the system will perform preliminary processing on the input text, such as word segmentation, stop word removal, syntactic analysis, etc. The goal is to optimize the text input and ensure the accuracy of subsequent processing.

[0091] Natural language understanding and intent analysis

[0092] The preprocessed text will be sent to a large-scale pre-trained model based on the Transformer architecture (such as GPT, BERT, T5, etc.) for training for deep semantic parsing.

[0093] The Transformer model mainly processes the input text sequence through the self-attention mechanism and extracts global dependencies to accurately understand user intentions.

[0094] This embodiment adjusts the optimization algorithm based on the traditional Transformer model to improve its training performance. The Transformer network consists of an encoder and a decoder. The encoder maps the input features into a series of context-aware vector representations, and the decoder gradually decodes these vector representations into target features. Its architecture is as follows: Figure 2 shown.

[0095] Specifically, the encoder part is composed of The encoder is composed of 2 identical encoding layers, each of which consists of two sub-layers: a multi-head attention mechanism sub-layer and a feed-forward network sub-layer. The encoder adds residual connections and normalization operations after each sub-layer.

[0096] The multi-head attention mechanism sublayer is implemented through the multi-head self-attention mechanism, which is the core module of the encoder. It is responsible for learning the importance of each position from the input features and combining the information of all positions to obtain a global context representation, thereby improving the performance and generalization ability of the model.

[0097] Through the multi-head self-attention mechanism, Transformer can focus on different parts of the input text at the same time and capture complex semantic relationships. For example, in natural language-based office processing, for the instruction "schedule a team meeting at 3 pm tomorrow", the model will recognize "tomorrow" and "3 pm" as time parameters and "schedule a meeting" as the core task.

[0098] Specifically, the implementation of the multi-head self-attention mechanism is that the input features pass through three different linear transformation layers to generate "query", "key" and "value" vectors respectively, and the linear transformation is achieved by multiplying the weight matrix; further, the generated query, key and value vectors are divided into multiple smaller parts, namely "heads", so that the model can process multiple different representation subspaces in parallel, thereby capturing the diversity information in the input data; further, for each head, the dot product of the query and key vectors is calculated respectively to obtain the attention score, which reflects the relative importance of different positions in the sequence, and in order to make the numerical value of the attention score stable, the score is divided by a scaling factor, which can be the square root of the key vector dimension; further, the Softmax function is used to process the attention score to obtain the attention weight of each position. The Softmax function ensures that all weights add up to 1, so that it can be processed as a probability distribution, expressed as:

[0099]

[0100] In the formula, For the The attention weight of each head, For the The query vector of the head, For the The key vector of the head, For the The transpose of the key vector of the head, is the dimension of the input features, is the number of attention heads, For the A vector of values ​​for the heads.

[0101] The obtained attention weights are used to perform weighted summation on the value vector to obtain the output vector; further, the output vectors of all heads are concatenated together, and then the output of the final multi-head self-attention layer is generated through a linear transformation layer.

[0102] In the Transformer network architecture, the feedforward network sublayer consists of two fully connected layers and an activation function, which is used to perform nonlinear transformation on the vector at each position. In this sublayer, the input vector first passes through the first fully connected layer for linear transformation, then passes through a ReLU activation function for nonlinear transformation, and finally passes through the second fully connected layer for linear transformation to improve the expressiveness and generalization capabilities of the model, so that the model can better handle complex input data.

[0103] Specifically, the decoder consists of The decoder layer is composed of the same decoding layers. Unlike the encoding layer, each decoding layer has an additional multi-head attention mechanism sublayer, that is, the decoding layer contains two multi-head attention mechanism sublayers and one feedforward network mechanism sublayer. In addition, the second multi-head attention mechanism sublayer of the decoding layer is implemented by the multi-head cross attention mechanism. The multi-head cross attention mechanism is a mechanism for information exchange between the encoder and the decoder. It takes the output features of the encoder and the input features of the decoder as input, and uses the attention mechanism to perform cross-attention calculations on the input, thereby obtaining a set of weighted decoder output features.

[0104] It should be noted that in the encoder and decoder of the Transformer network, the output of each sub-layer is the result of layer normalization and residual connection, expressed as:

[0105]

[0106] In the formula, is the layer normalization function, represents the output of the residual connected sublayer, Represents the input of the sublayer.

[0107] It should also be noted that the feedforward network of the Transformer network is a neural network structure composed of neurons, which plays a role in enhancing the nonlinear processing capabilities of the model.

[0108] The neural network structure includes weights and bias parameters. In order to avoid the gradient vanishing and gradient descent phenomena caused by the traditional neural network using the gradient descent method to update parameters, and to avoid the phenomenon that the parameters fall into the local optimal solution after the training is completed, this embodiment adopts a dynamic adaptive neural optimization algorithm in the training process of the feedforward network. Different from the traditional gradient descent optimization algorithm, the dynamic adaptive neural optimization algorithm is inspired by the steady-state dynamic system in nature, and realizes efficient optimization of the feedforward network parameters by simulating the conversion between steady state and unstable state in the nonlinear dynamic system.

[0109] The core of the dynamic adaptive neural optimization algorithm is to build a dynamic system, in which each update of the feedforward network parameters is similar to the transfer of the system state. This state transfer not only depends on the value of the current parameters, but is also affected by a dynamic adjustment mechanism that changes over time. This mechanism imitates the process of dynamic systems reaching steady state in nature. By adopting dynamic adaptive functions, the algorithm can explore more extensively in the parameter space and avoid the local optimal problem common in traditional optimization methods.

[0110] Specifically, the parameter optimization steps during feedforward network training are as follows:

[0111] S11, set the initial parameters of the feedforward network, and set the initial weight of the feedforward network to , the initial bias of the feedforward network is , the weight of the feedforward network is , the bias of the feedforward network is , the initialization method is expressed as:

[0112]

[0113]

[0114] In the formula, It means that it obeys a specific distribution. represents a normal distribution, is the variance of the preset weights and biases. Preferably, Set to 0.01.

[0115] Furthermore, the control parameters that affect the adjustment of the feedforward network parameters are defined, and their initialization is expressed as:

[0116]

[0117] In the formula, is the control parameter of the feedforward network, is the initial value of the control parameter of the feedforward network. Preferably, Set to 0.3.

[0118] S12, iterative training of the feedforward network is performed, the control parameters are dynamically adjusted according to the number of iterations, and the attenuation function is used to simulate the energy dissipation in the physical system. The calculation method is expressed as:

[0119]

[0120] In the formula, is the smoothing coefficient; is the control parameter of the feedforward network for the iteration; is the current iteration number. Preferably, Set to .

[0121] S13, using the current feedforward network parameters and control parameters, the state variables are calculated through a dynamic adaptive function, and the calculation method is expressed as:

[0122]

[0123] In the formula, is the state variable; Indicates The state variables of the iteration; is the hyperbolic tangent function; is the expansion factor; For the The weights of the feedforward network for iteration .

[0124] S14, transform the state variables through nonlinear functions to generate new parameter update directions. The calculation method is expressed as:

[0125] Use state variables to calculate the next weight update direction , expressed as:

[0126]

[0127] In the formula, The weight update direction for the feedforward network; is the learning rate of the weights of the feedforward network; For the The impulse interference term of the iteration.

[0128] In one embodiment, in order to avoid falling into the local optimum during the parameter optimization process, a random pulse interference method strategy is used to implement random interference to enhance the algorithm's search ability for different areas. The calculation method of the pulse interference term is expressed as:

[0129]

[0130] In the formula, is the interference intensity coefficient, is a random variable with a value of {0, 1}, according to the probability generate; is the probability threshold; is a sign function, The role of is to ensure that the interference direction has the same sign as the state variable to maintain the consistency of the update direction. Preferably, Set to 0.1.

[0131] Furthermore, the generation method of random variables is expressed as:

[0132]

[0133] S15. Update the feedforward network parameter weights and biases according to the converted state variables and the established learning rate. The update method is expressed as:

[0134]

[0135]

[0136] In the formula, It is The loss function of the feedforward network for this iteration is, is the bias learning rate of the feedforward network, is the gradient of the loss function with respect to the bias; For the The weights of the feedforward network at iteration ; For the The bias of the feedforward network for the iteration; For the The bias of the feedforward network for the iteration; For the Adaptive threshold for iterations.

[0137] In one embodiment, an adaptive threshold dynamic adjustment mechanism is used to automatically adjust the update threshold of key parameters according to the current performance of the model, so as to more finely control the adaptability of parameters in the learning process. The calculation method of the adaptive threshold is expressed as follows:

[0138]

[0139] In the formula, is the target loss value, is a hyperparameter that adjusts the threshold sensitivity. Preferably, Set to 0.1, Set to 0.05.

[0140] S16. After each iteration, the performance of the current model on the validation set is evaluated by calculating the loss function of the feedforward network, which is expressed as:

[0141]

[0142] In the formula, It is The true value of the sample input to the feedforward network, It is The iteration model pair is The predicted value of the sample input to the feedforward network, is the number of samples input to the feedforward network in the current batch.

[0143] S17. Based on the results of the performance evaluation, the expansion coefficient and the learning rate of the weight of the feedforward network are adjusted to optimize the performance of subsequent iterations. The adjustment method is expressed as:

[0144]

[0145]

[0146] In the formula, For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for the iteration; For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for the iteration; is the first performance adjustment factor, is the second performance adjustment coefficient. Preferably, Set to 0.3, Set to 0.5.

[0147] In the above technical solution of this embodiment, a dynamic adaptive neural optimization algorithm is adopted to dynamically adjust the network parameters by simulating the steady-state dynamic system in nature, thereby effectively avoiding the gradient vanishing and local optimal problems in the traditional gradient descent method, thereby improving the training efficiency and generalization ability of the model. At the same time, a random pulse interference strategy is adopted to enhance the search ability of the model in different parameter spaces by introducing random interference during the training process, thereby improving the global search performance of the model.

[0148] Task intent classification and parameter extraction

[0149] After using the Transformer model to perform context-aware semantic analysis on the instructions, the system will classify the user instructions into specific task types through the task intent classification algorithm. For example, in the automatic office process based on natural language, the instruction "help me generate a project summary report" will be mapped to the document generation task class, while "help me send an email to Zhang San" will be mapped to the communication task class.

[0150] In this embodiment, the task intention classification algorithm is specifically a multi-layer extreme learning machine classification algorithm based on path integral. On the basis of the traditional extreme learning machine, the Feynman path integral strategy is used to establish a probability amplitude model from input features to output classification results.

[0151] Specifically, the training process of the multi-layer extreme learning machine classification algorithm based on path integral is as follows:

[0152] S21. Initialize and set up a multi-layer extreme learning machine. Suppose the number of layers of the multi-layer extreme learning machine is , the number of nodes in each layer is ,in ;

[0153] Randomly initialize the weights of a multi-layer extreme learning machine and bias , using the chaotic mapping function to increase the diversity of parameters and the robustness of the model, expressed as:

[0154]

[0155] In the formula, For the The chaotic variable of the iteration; For the The chaotic variable of the iteration; is the control parameter of the chaotic map, satisfying ; is the modulation parameter; is the number of iterations. Preferably, Set to 0.3.

[0156] Furthermore, the chaotic variables are used to initialize the weights, which can be expressed as:

[0157]

[0158] In the formula, For multi-layer extreme learning machine The weight matrix of the layer; is the scaling factor of the multi-layer extreme learning machine; is the weight offset of the multi-layer extreme learning machine. Preferably, Set to 0.1.

[0159] S22. Input the features output by the Transformer model into the multi-layer extreme learning machine classifier. Suppose the features output by the Transformer model are , where is the total number of training samples.

[0160] S23. Use the Feynman path integral method to calculate the probability amplitude of all possible paths from input features to output results. By summing these paths, the overall probability amplitude is obtained to guide the prediction of the model. Specifically, for the samples, the path integral is expressed as:

[0161]

[0162] In the formula, For the The path integral probability amplitude of samples; is an imaginary unit; is the scaling constant; is the action function; is the path of system state change with the current iteration number; The number of iterations of the current multi-layer extreme learning machine training; represents the integration of all possible paths. Preferably, The value is Planck's constant.

[0163] Furthermore, the calculation method of the action function is expressed as:

[0164]

[0165] In the formula, is the quality parameter; Status For the current iteration The derivative of is the potential energy function, reflecting the intrinsic dynamic characteristics of the system; and is the number of iterations to start and end. Preferably, Set to 1.

[0166] Furthermore, the calculation method of the change path of the system state with the current number of iterations is expressed as:

[0167]

[0168] In the formula, For the The output of the layer at the current iteration; is the loss function of the multi-layer extreme learning machine.

[0169] Furthermore, the potential function is defined as a nonlinear function related to the input features and parameters to capture the complexity of the output features of the Transformer model. The calculation method is expressed as:

[0170]

[0171] In the formula, is the first potential energy learnable parameter, is the second potential energy learnable parameter, is the third potential energy learnable parameter, is the fourth potential energy learnable parameter.

[0172] Furthermore, since it is difficult to directly calculate the path integral, an approximate calculation method of the path integral is used to solve it. The calculation method of the path integral probability amplitude of the sample is expressed as:

[0173]

[0174] In the formula, To make the action The classical path of minimization satisfies the Euler-Lagrange equation and is calculated as follows:

[0175]

[0176] In the formula, is the Lagrangian.

[0177] Furthermore, the calculation method of the Lagrangian is expressed as:

[0178]

[0179] S24. In a multi-layer extreme learning machine, the output of each layer is calculated in turn by forward propagation. Since the hidden layer parameters of the extreme learning machine are fixed, the weight of the output layer needs to be adjusted according to the input data. Specifically, for the first The output of the layer is represented as:

[0180]

[0181] In the formula, For the The output vector of the layer; is the activation function of the multi-layer extreme learning machine.

[0182] Furthermore, the activation function of the multi-layer extreme learning machine adopts an adaptive complex-valued activation function, combined with the phase information of the Feynman path integral, and the calculation method is expressed as:

[0183]

[0184] In the formula, To input plural numbers; and They are The real and imaginary parts of is a real-valued activation function; is a phase parameter, which can be learned and adjusted. In one embodiment, the real-valued activation function adopts a Sigmoid activation function.

[0185] Furthermore, the correlation between the phase parameter and the input feature is calculated as follows:

[0186]

[0187] In the formula, is the adaptive phase parameter activation function.

[0188] Furthermore, the calculation method of the adaptive phase parameter activation function is expressed as:

[0189]

[0190] In the formula, is a learnable parameter; is the Sigmoid activation function.

[0191] S25. Calculate the loss function of the multi-layer extreme learning machine to measure the gap between the model prediction results and the true labels. The loss function of the multi-layer extreme learning machine not only considers the classification error, but also considers the phase information of the Feynman path integral to capture the intrinsic correlation of the data. The calculation process is expressed as:

[0192]

[0193] In the formula, For the The predicted output of samples; For the The wave function of the samples; For the The output vector of the layer.

[0194] Furthermore, the probability density of the prediction of the multi-layer extreme learning machine is expressed as:

[0195]

[0196] In the formula, For the The predicted probability density of samples; for The complex conjugate of .

[0197] Furthermore, the calculation method of the loss function of the multi-layer extreme learning machine is expressed as:

[0198]

[0199] In the formula, is the loss function of the multi-layer extreme learning machine; For the The true labels of samples; is the trade-off coefficient of the multi-layer extreme learning machine; is the target path integral probability amplitude; is the regularization coefficient of the multi-layer extreme learning machine; preferably, Set to 0.3.

[0200] S26. The output layer weights of the model are optimized using the topological soliton optimization algorithm. By simulating the propagation of solitons in nonlinear media, the optimal parameter configuration is found to improve the classification performance of the model. The topological soliton equation is calculated based on the nonlinear Schrödinger equation and is expressed as:

[0201]

[0202] In the formula, is the wave function; is a virtual space variable; is the current iteration number; is the nonlinear coefficient.

[0203] Furthermore, using the soliton solution To update the parameters of the multi-layer extreme learning machine, it is expressed as:

[0204]

[0205]

[0206] In the formula, is the topological soliton learning rate; and is the gradient of the loss function with respect to weights and biases; is a soliton shape function, providing direction information for parameter updating. Preferably, Set to 0.01.

[0207] Furthermore, the soliton shape function is calculated in the form of a single soliton solution, expressed as:

[0208]

[0209] In the formula, is the amplitude of the soliton shape function; is the wave number of the soliton shape function; is the soliton velocity as a function of the soliton shape; is the frequency parameter of the soliton shape function.

[0210] Furthermore, in order to adapt to the model training process, the soliton parameters are adaptively adjusted according to the gradient information, and the adjustment method is expressed as:

[0211]

[0212]

[0213]

[0214] In the formula, is the first soliton parameter learning rate, is the second soliton parameter learning rate, is the third soliton parameter learning rate; Update the symbol for the parameter.

[0215] S27, repeat the above steps until the preset stop iteration condition is met, which means that the model training is completed. In one embodiment, the preset stop iteration condition is reaching a preset maximum number of iterations, preferably, the preset maximum number of iterations is set to 1000 times.

[0216] Task decomposition and plan generation

[0217] After clarifying the task type and parameters, the system can convert the task into a series of specific operations. For complex multi-step tasks, the system can decompose the task. Specifically, by decomposing complex natural language instructions into multiple subtasks, the system can call different methods or modules to complete them. For example, in an automatic office scenario based on natural language, the instruction "help me generate a project summary and send it to everyone" can be decomposed into two steps: "generate a project summary" and "send an email". After the task is completed, the execution results (such as the generated documents, reports or task status) can be fed back to the user.

[0218] In the above technical solution of this embodiment, a multi-layer extreme learning machine algorithm based on Feynman path integral is proposed. By calculating the probability amplitude of all possible paths of input features and combining the path integral principle in quantum mechanics to guide classification, it can better capture the potential correlation of complex task instructions, improve the accuracy of task classification, and greatly improve the classification accuracy of the natural language processing model.

Claims

1. A training method for a natural language processing model based on a Transformer architecture, comprising the steps of obtaining input text data and preprocessing it, and sending the preprocessed text to a natural language processing model based on a Transformer network architecture for training, characterized in that: The training of the feedforward network of the Transformer network architecture uses a dynamic adaptive neural optimization algorithm, and the steps are as follows: S11, setting initial parameters of the feedforward network, the initial parameters including initial weights and initial biases, and defining control parameters that affect adjustment of the feedforward network parameters; S12, performing iterative training of the feedforward network, dynamically adjusting the control parameters according to the number of iterations, and using a decay function to simulate energy dissipation in the physical system; S13, using the current feedforward network parameters and control parameters, calculating the state variables through a dynamic adaptive function; S14, transforming the state variables through nonlinear functions to generate new parameter update directions, and at the same time, implementing random interference using a random pulse interference method strategy; S15, updating the feedforward network parameter weights and biases according to the converted state variables and the established learning rate, and at the same time, using an adaptive threshold dynamic adjustment mechanism to automatically adjust the update threshold of key parameters according to the current performance of the model; S16. After each iteration, the performance of the current model on the validation set is evaluated by calculating the loss function of the feedforward network. S17, based on the results of the performance evaluation, the expansion coefficient and the learning rate of the feedforward network weight are adjusted to optimize the performance of subsequent iterations; in step S12, the control parameter adjustment formula is expressed as follows: ， In the formula, is the smoothing coefficient; is the control parameter of the feedforward network at the tth iteration; is the current iteration number, is the initial value of the control parameter of the feedforward network; In step S13, the dynamic adaptive function calculates the state variable in the following manner: ， In the formula, Indicates The state variables of the iteration; is the hyperbolic tangent function; is the expansion factor; For the The weights of the feedforward network at iteration ; In step S14, the state variables are transformed through a nonlinear function to generate a new parameter update direction, specifically: The state variables are used to calculate the next weight update direction, expressed as: ， In the formula, The weight update direction for the feedforward network; is the learning rate of the weights of the feedforward network; For the The pulse interference term of the iteration; The calculation method of the pulse interference term is as follows: ， In the formula, is the interference intensity coefficient, is a random variable with a value of {0, 1}, according to the probability generate; is the probability threshold; is a sign function, The role of is to ensure that the disturbance direction has the same sign as the state variable to maintain the consistency direction of the update; In step S15, the feedforward network parameter weights and biases are updated according to the converted state variables and the established learning rate. The updating method is as follows: ’ ‘ In the formula, It is The loss function of the feedforward network for this iteration is, is the learning rate of the feedforward network bias, is the gradient of the loss function with respect to the bias; For the The weights of the feedforward network at iteration ; For the The bias of the feedforward network for the iteration; For the The bias of the feedforward network for the iteration; For the Adaptive threshold for iterations; The adaptive threshold calculation method is as follows: ， In the formula, is the target loss value, is a hyperparameter that adjusts the threshold sensitivity; In step S16, the loss function of the feedforward network is expressed as follows: ， In the formula, It is The true value of the sample input to the feedforward network, It is The iteration model pair is The predicted value of the sample input to the feedforward network, is the number of samples input to the feedforward network in the current batch; In step S17, the learning rate adjustment method of the expansion coefficient and the weight of the feedforward network is expressed as follows: ， ， In the formula, For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for iterations; For the The expansion factor of the iteration, For the The learning rate of the weights of the feedforward network for iterations; is the first performance adjustment factor, is the second performance adjustment factor.

2. The training method of the natural language processing model based on the Transformer architecture according to claim 1, characterized in that: After using the Transformer network architecture to perform context-aware semantic analysis on the input text, the user instructions are classified into specific task types through the task intention classification algorithm. The task intention classification algorithm adopts a multi-layer extreme learning machine classification algorithm based on path integral. On the basis of the traditional extreme learning machine, the Feynman path integral strategy is used to establish a probability amplitude model from input features to output classification results.

3. The training method of the natural language processing model based on the Transformer architecture according to claim 2, characterized in that: The training steps of the multi-layer extreme learning machine classification algorithm based on path integral are as follows: S21, initializing and setting a multi-layer extreme learning machine, randomly initializing the weights and biases of the multi-layer extreme learning machine, and using a chaotic mapping function to increase the diversity of parameters and the robustness of the model; S22, input the features output by the Transformer network architecture into the multi-layer extreme learning machine classifier; S23. Using the Feynman path integral method, calculate the probability amplitude of all possible paths from the input feature to the output result, sum these paths, and obtain the overall probability amplitude; S24. In a multi-layer extreme learning machine, the output of each layer is calculated in turn by forward propagation, and the weight of the output layer is adjusted according to the input data; S25. Calculate the loss function of the multi-layer extreme learning machine to measure the gap between the model prediction results and the true labels. The loss function of the multi-layer extreme learning machine takes into account the classification error and the phase information of the Feynman path integral. S26. Use the topological soliton optimization algorithm to optimize the output layer weights of the model. By simulating the propagation of solitons in nonlinear media, find the optimal parameter configuration. The topological soliton equation is calculated based on the nonlinear Schrödinger equation, and the soliton solution is used to update the parameters of the multi-layer extreme learning machine. S27, repeat the above steps until the preset stop iteration condition is met and the training is completed.

4. The training method of the natural language processing model based on the Transformer architecture according to claim 3, characterized in that: In step S21, the chaos mapping function is expressed as follows: ， In the formula, For the The chaotic variable of the iteration; For the The chaotic variable of the iteration; is the control parameter of the chaotic map, is the modulation parameter; is the number of iterations; The weights are initialized using chaotic variables, expressed as: ， In the formula, For multi-layer extreme learning machine The weight of the layer; is the scaling factor of the multi-layer extreme learning machine; is the weight offset of the multi-layer extreme learning machine; In step S23, the path integral probability amplitude of the sample is calculated as follows: ， In the formula, For the The path integral probability amplitude of samples; is an imaginary unit; is the scaling constant; is the action function; is the path of system state change with the current iteration number; The number of iterations of the current multi-layer extreme learning machine training; represents the integration over all possible paths.

5. The training method of the natural language processing model based on the Transformer architecture according to claim 4, characterized in that: In step S24, for the Layer learning machine, its output is expressed as: ， In the formula, For the The output vector of the layer; For the The output vector of the layer, is the activation function of the multi-layer extreme learning machine, For multi-layer learning machine Bias of the layer; The activation function of the multi-layer extreme learning machine is calculated as follows: ， In the formula, To input plural numbers; and They are The real and imaginary parts of is a real-valued activation function; is the phase parameter, is an imaginary unit; The phase parameter is associated with the input feature and is calculated as follows: ， In the formula, is the adaptive phase parameter activation function, which is calculated as follows: ， In the formula, is a learnable parameter; is the Sigmoid activation function.

6. The training method of the natural language processing model based on the Transformer architecture according to claim 5, characterized in that: In step S25, the loss function of the multi-layer extreme learning machine is calculated as follows: ， In the formula, is the loss function of the multi-layer extreme learning machine; For the The true labels of samples; is the trade-off coefficient of the multi-layer extreme learning machine; is the target path integral probability amplitude; is the regularization coefficient of the multi-layer extreme learning machine, is the total number of training samples, For the The predicted probability density of samples is calculated as follows: ， In the formula, for The complex conjugate of For the The wave function of the samples; In step S26, the topological soliton equation is calculated based on the nonlinear Schrödinger equation, which is expressed as follows: ， In the formula, is the wave function; is a virtual space variable; is the current iteration number; is the nonlinear coefficient, Is an imaginary unit.

Images