A standard text classification method based on machine learning

Abstract

The present invention discloses a standard text classification method based on machine learning, which belongs to the field of data recognition technology, including adopting the BERT algorithm based on constrained attention to vectorize the standard text; adopting the generative adversarial network algorithm based on semantic focus to generate samples, thereby realizing the expansion of the standard text data after vectorization; and adopting the neural network algorithm based on perturbation fuzzy fusion to extract features and classify the standard text data after vectorization. The present invention uses an interactive attention mechanism and a dynamic context perception mechanism, so that the model can not only capture the semantic association within the text, but also better understand the relationship between the standard text and the classification task instructions, significantly improving the classification model's ability to understand the context. A generative adversarial network based on semantic focus is used to generate additional sample data, and the semantic quality and consistency of the generated data are ensured through a semantic refocusing mechanism, solving the problem of insufficient samples in the standard text classification task.

Description

Technical Field

[0001] The present invention belongs to the technical field of data recognition, and in particular relates to a standard text classification method based on machine learning. Background Art

[0002] Standard texts are widely found in technical specifications, industry guidelines and operation manuals. Their content structure is complex, semantics is rigorous and there are many specialized terms, which makes it difficult for traditional text classification methods to effectively handle them. The number of various standard documents in the industry is growing day by day. These documents not only contain complex language and structure, but also have strong context dependence and professionalism, which brings additional difficulty to the classification task.

[0003] In the prior art, a Chinese invention patent with publication number CN107346433A proposes a text data classification method and server, the method comprising: the server analyzes a first training set through a support vector machine (SVM) algorithm, and performs a classification test on a first test set according to a first classification model obtained by the analysis, the first training set and the first test set both contain multiple support vectors, each support vector contains K weight factors corresponding to K feature words, and each feature factor corresponds to a feature word; the relative weight of each feature word in the K feature words in the target support vector is calculated according to the weight factor of the target support vector and the parameters in the first classification model; the second training set is analyzed through an SVM algorithm, and a classification test is performed on a second test set according to a second classification model obtained by the analysis, the support vectors in the second training set and the second test set both contain weight factors corresponding to feature words among the K feature words except the target feature words, which can reduce the classification error rate of the classification model.

[0004] The Chinese invention patent with publication number CN111488826A proposes a text recognition method, device, electronic device and storage medium to realize text recognition in different languages ​​in an image and improve recognition accuracy. The method includes: extracting features of the image to be recognized and obtaining a feature map corresponding to the image to be recognized, wherein the image to be recognized contains text of at least one language category; based on the feature map, performing position detection on the text in the image to be recognized and obtaining position information of the text in the image to be recognized, and performing language recognition on the text in the image to be recognized and obtaining language information of the text in the image to be recognized; based on the obtained position information and language information, recognizing the text in the image to be recognized, being able to predict the language of each text in the image to be recognized without user specification, and flexibly recognizing each text according to the recognition method corresponding to each recognized language, with higher recognition accuracy.

[0005] The Chinese invention patent with publication number CN111554268A proposes a language recognition method, text classification method and device, computer equipment and storage medium based on a language model, including: obtaining training word vectors corresponding to training sentences, inputting them into the first model to be trained and the trained second model respectively, obtaining the feature matrix output by each first network layer of the first model, and the feature matrix output by each second network layer of the second model; the first network layer and the second network layer correspond one to one, and the number of network layers of the first model is less than the number of network layers of the second model; performing similarity calculation on the feature matrix output by each first network layer and the feature matrix output by the second network layer corresponding to each first network layer to obtain each similarity, adjusting the model parameters of the first model based on each similarity until the updated target similarity meets the convergence condition, and obtaining the trained first model, and performing language recognition by the first model can improve the model training efficiency.

[0006] The existing technology still has the following problems: First, the text vectorization method in the existing technology can only do simple vectorization of words or phrases, and cannot capture the complex dependencies and contextual information in the standard text, which leads to a deviation in the understanding of the standard content; Second, the traditional data expansion method cannot effectively solve the problem of semantic consistency, resulting in the model's poor generalization performance due to insufficient training data and easy overfitting; Third, the existing neural network feature extraction method is prone to feature redundancy, which affects the model classification effect. In the case of strong correlation between features, it is difficult to further improve the model performance; Fourth, the traditional machine learning model learning rate adjustment method, such as static learning rate or simple learning rate decay strategy, cannot dynamically adapt to the local characteristics of the loss surface, resulting in low model training efficiency and easy gradient disappearance or gradient explosion problems. Summary of the invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a standard text classification method based on machine learning.

[0008] The technical solution adopted to solve the above technical problems is: a standard text classification method based on machine learning, including the following steps:

[0009] S1, input the collected standard text data into the BERT model for BERT model training, and use the constrained attention-based BERT algorithm to vectorize the standard text; S2, use the semantic focus-based generative adversarial network algorithm to generate samples, and then realize the expansion of the vectorized standard text data; S3, after the training of the vectorized standard text data expansion model is completed, the trained vectorized standard text data expansion model is used to increase the number of samples, and the expanded vectorized standard text data set is used to train the standard text classification model, and the standard text classification model is a fully connected neural network model; S4, use the perturbation fuzzy fusion-based neural network algorithm to extract and classify the features of the vectorized standard text data.

[0010] Furthermore, the training process of the semantic focus-based generative adversarial network algorithm in step S2 is as follows:

[0011] S201, initialize the generator and discriminator of the generative adversarial network. The generator is responsible for generating vectorized standard text data, and the discriminator determines whether the input vectorized standard text data is real or generated by the generator. The initialization method is expressed as:

[0012] ,

[0013] In the formula, is the generator parameter, It means that it obeys a specific distribution. represents a normal distribution, is the initialization variance of the generator parameters, are the discriminator parameters, is the initialization variance of the discriminator parameters;

[0014] S202, in each training cycle of the training process, the generator generates preliminary vectorized standard text data samples, which are input into the discriminator for authenticity evaluation. The update of the generator depends on the feedback provided by the discriminator, and its parameters are adjusted to better simulate the distribution of the vectorized standard text data, which is expressed as:

[0015] ,

[0016] In the formula, and are the parameters of the generator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, is the gradient of the loss function of the generative adversarial network with respect to the generator parameters, Adjusting the increments for semantic refocusing of generator parameters;

[0017] is the loss function of the generative adversarial network, Express expectations, is the standard text data after real vectorization. is the standard text data distribution after real vectorization, is the discriminator, For the generator, is the input noise distribution, is the noise vector sampled from the input noise distribution, Dynamically adjust weights for features, is the discriminator function, is the generator function;

[0018] The semantic focus mechanism is used to improve the semantic quality and relevance of the generated standard text. The semantic focus in the generation process is dynamically adjusted to ensure the accuracy and richness of the generated content. The semantic focus mechanism is implemented through the semantic recognition module, which is responsible for real-time monitoring and evaluation of the semantic focus of the generated standard text and adjusting it according to the preset target semantic focus. The calculation method of semantic recognition and scoring is expressed as follows:

[0019] ,

[0020] In the formula, is the function for calculating the semantic score vector, To generate the semantic score vector of the sample, is the Softmax function, and are the weight and bias of the semantic recognition module respectively, A function for extracting semantic features, specifically a 2-layer fully connected neural network. Generate samples for the generator;

[0021] Based on the semantic score vector of the generated sample, the semantic refocusing adjustment increment of the generator parameters is calculated as:

[0022] ,

[0023] In the formula, is the gradient with respect to the generator parameters, is the learning rate of the generative adversarial network, is the adjustment factor used to control the strength of semantic adjustment, is the Kullback-Leibler divergence, which is used to measure the difference between the semantic distribution of generated samples and the semantic distribution of real samples. is the semantic distribution of the real sample, calculated by the preset Softmax function;

[0024] The dynamic adjustment weight of the feature makes the discriminator give higher penalties to samples with large distribution differences of the real vectorized standard text data when evaluating the generated samples, which promotes the generator to produce higher quality vectorized standard text data. The calculation method of the sample penalty factor is expressed as:

[0025] ,

[0026] In the formula, for norm, and They are the mean and variance of the standard text data distribution after vectorization;

[0027] S203, in the process of generating the vectorized standard text data, the random noise that may appear in the generation process is reduced by a self-stable noise reduction strategy, and the implementation method of the noise reduction strategy is expressed as follows:

[0028] ,

[0029] In the formula, The original vectorized standard text data sample generated by the generator. is a vectorized standard text data sample with noise reduction, is the noise reduction function, is the self-stationary noise reduction adjustment coefficient, Set to 0.3 to control the strength of noise reduction. is the hyperbolic tangent function, is a tuning parameter that controls the sensitivity of the nonlinear transformation, Set to 2, To generate the variance of the vectorized standard text data sample, is the variance function;

[0030] S204, during the training process, the loss function not only considers the difference in statistical characteristics between the generated vectorized standard text data and the real vectorized standard text data, but also uses the evaluation indicators of semantic coherence and sample quality to promote the discriminator to more accurately guide the generator to generate high-quality vectorized standard text data. The calculation method is expressed as:

[0031] ,

[0032] In the formula, is the cross entropy loss for generating adversarial networks, Adjust the weights for the first GAN loss function, The weights are adjusted for the second GAN loss function, the weights are adjusted for the first GAN loss function, and the weights are adjusted for the cross entropy loss and semantic coherence cost of the GAN loss function. To generate the semantic coherence difference between the sample and the real sample, is a standard text data sample after real vectorization;

[0033] The semantic coherence difference is combined with the difference measurement of semantic depth features, which not only evaluates the surface difference of the standard text, but also goes deep into the semantic level to ensure the semantic coherence and practicality of the generated standard text. The calculation method of the semantic coherence difference between the generated sample and the real sample is expressed as:

[0034] ,

[0035] In the formula, is the number of elements in the sample, It is a multi-layer neural network representation function in the deep learning model, which is used to extract the deep semantic features of the vectorized standard text data. and are the i-th elements in the generated samples and the real samples respectively;

[0036] The gradient is normalized before each gradient update so that the update step size remains within a reasonable range, expressed as:

[0037] ,

[0038] In the formula, is the gradient of the unprocessed discriminator parameters, The gradient of the loss function of the generative adversarial network with respect to the discriminator parameters; is the gradient of the normalized discriminator parameters, is the expectation of the squared gradient of the unprocessed discriminator parameters, used to estimate the average magnitude of the gradient during the entire training process, is a small constant to prevent the denominator from being zero, Set to 0.0001;

[0039] are the parameters of the discriminator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, Set to 0.01;

[0040] S205, repeat the above steps until a preset stop iteration condition is met. The preset stop iteration condition is reaching a preset maximum number of iterations, which means that the model training is completed.

[0041] Furthermore, the training process of the BERT algorithm based on constrained attention in step S1 is:

[0042] S101, pre-training the BERT model to enable it to understand the basic structure of language. During the training process, standard text instructions are first segmented and converted into a format that the model can understand;

[0043] S102, based on the pre-training, the model is further trained through the interactive attention mechanism, so that the model learns how to adjust its attention distribution according to the task instructions, and adopts an attention enhancement factor to enhance the model's attention to the part of the standard text related to the task instructions. The calculation method of the attention enhancement factor is as follows:

[0044] ,

[0045] In the formula, is the attention enhancing factor, is the ReLU activation function, is the weight matrix of the trainable attention enhancement module, is the characteristic representation of standard text instructions, is the bias term of the attention enhancement module;

[0046] A dynamic context-aware mechanism is used to improve the model's ability to understand the context of standard text instructions. The dynamic context-aware mechanism enhances the model's attention mechanism through an additional context-aware network. Let U be the output of the context-aware network, which dynamically generates context weights based on the current attention distribution and standard text features, expressed as:

[0047] ,

[0048] In the formula, It is a preset context-aware network;

[0049] The context weight is used to adjust the original attention score to obtain a new context-aware attention score, which is expressed as:

[0050] ,

[0051] In the formula, is the updated attention enhancement factor, i.e., the new context-aware attention score, For one with All-1 vectors of the same dimension, is the context weight, so when When When , the model will pay less attention to non-critical information;

[0052] The context weight is calculated as follows:

[0053] ,

[0054] In the formula, is the LeakyReLU activation function, is the weight matrix of the context-aware network, is the bias term of the context-aware network;

[0055] Define the initialization method of the context-aware network. Suppose the weight matrix of the context-aware network is initialized in the following way:

[0056] ,

[0057] Where V is the dimension of the input feature of the context-aware network, H is the dimension of the hidden layer of the context-aware network, and the bias term of the context-aware network is initialized to a zero vector. Glorot initialization function is a commonly used weight initialization method that aims to keep the variance of input and output consistent;

[0058] S103, adopt the constrained attention adjustment strategy to further optimize the attention weight of the model. The constraint term is calculated as follows:

[0059] ,

[0060] In the formula, is the constraint term of the constrained attention adjustment strategy, The function is used to convert the updated attention enhancement factor into a probability distribution to ensure the reasonable distribution of attention weights. and are the weight and bias parameters of constrained attention adjustment, respectively;

[0061] S104, dynamically adjust the model parameters according to the feedback during the training process through the adaptive adjustment mechanism, and the calculation method is as follows:

[0062] ,

[0063] In the formula, is the parameter update operation, is the Adam optimizer function, N is the number of samples input to the BERT model in the current batch, is the learning rate of the BERT model, is the weight update amount of the BERT model;

[0064] The BERT model parameters are optimized using the following objective function:

[0065] ,

[0066] Where L is the loss function of the BERT model, is the prediction output of the BERT model for the i-th sample, obtained by the preset Softmax classification function. for norm;

[0067] S105, repeating the above steps until a preset stop iteration condition is met, where the preset stop iteration condition is reaching a preset maximum number of iterations.

[0068] Furthermore, the fully connected neural network model in step S3 includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, and an output layer. The input vector is connected to the first hidden layer as the input of the fully connected network, the output of the first hidden layer is used as the input of the second hidden layer, the output of the second hidden layer is used as the input of the third hidden layer, the output of the third hidden layer is used as the input of the fourth hidden layer, and the output of the fourth hidden layer is used as the input of the output layer.

[0069] Furthermore, the training process of the neural network algorithm based on disturbance fuzzy fusion in step S4 is as follows:

[0070] S401, initializing the neural network model including the weights and biases of the neural network, the initialization method is expressed as:

[0071] ,

[0072] In the formula, is the number of nodes in the input layer of the neural network, is sampled from a normal distribution with mean 0 and unit variance, is a normal distribution, is the identity matrix, is the weight of the neural network, is the bias of the neural network;

[0073] S402, during the training process of the neural network, each layer of the network is constrained through a specific regularization strategy to reduce the correlation between features;

[0074] The regularization strategy weakens the collinearity between features to ensure that the network can capture more independent and discriminative features. The calculation method is expressed as:

[0075] ,

[0076] In the formula, is the regularization function, is the regularization coefficient, which is used to control the influence strength of the regularization term. for norm, and are the i-th and j-th rows in the neural network weight matrix;

[0077] is the total loss of the neural network, is the cross entropy loss, It is a regularization term of the neural network. Its function is to reduce the similarity between different feature vectors by calculating the loss during the training process. Its calculation helps to enhance the orthogonality between feature vectors, thereby enhancing the feature discrimination ability of the model.

[0078] S403, using random perturbation and fuzzy processing to enhance the generalization ability of various changes and uncertainties that may be encountered by the vectorized standard text data, and prevent the overfitting of the neural network model. The implementation method of adding random perturbation in the feature extraction process is expressed as:

[0079] ,

[0080] In the formula, is the perturbation input feature vector of the neural network, represents the input feature vector of the neural network, is the perturbation intensity of the neural network, controlling the magnitude of the noise, Set to 0.001, Generate standard normally distributed noise of the same size as the input feature vector of the neural network, represents the size measurement function, is a random function, It is a function adjusted according to the characteristics of the neural network input;

[0081] The function adjusted according to the input characteristics of the neural network adjusts the intensity of the noise based on the variance of the input features, so that the disturbance is reduced in the area where the features are relatively stable, and the disturbance is increased in the area where the features vary greatly, so as to more effectively adapt to the uncertainty of the vectorized standard text data. The calculation method of the function adjusted according to the input characteristics of the neural network is expressed as:

[0082] ,

[0083] In the formula, To calculate the variance of the neural network input, is the variance function, is a preset standard for adjusting disturbance sensitivity;

[0084] S404, during the forward propagation process, the neural network extracts key features based on the standard text data after training vectorization, and continuously adjusts the network parameters in the back propagation to minimize the loss function. The forward propagation and parameter optimization of the neural network are expressed as follows:

[0085] ,

[0086] In the formula, Input feature vectors to the layers of the neural network, is the output feature vector of the neural network layer, is the ReLU activation function, For the neural network The dynamic threshold of the standard text data after the statistical vectorization of the layer at the tth iteration;

[0087] The back propagation algorithm is used to update the parameters of the neural network. The incremental calculation method of the update is expressed as:

[0088] ,

[0089] In the formula, is the weight update increment of the neural network, is the bias update increment of the neural network, is the learning rate of the neural network, is the symbol of partial derivative, Set to 0.001;

[0090] The dynamic threshold of the standard text data after statistical vectorization is dynamically adjusted for the activation output of each layer in the neural network. By analyzing the statistical characteristics of the output of each layer in each iteration, the threshold of the activation function is automatically adjusted to optimize the network's response to the input features, thereby improving the classification accuracy and robustness of the overall model. The calculation method of the dynamic threshold of the standard text data after statistical vectorization is expressed as follows:

[0091] ,

[0092] In the formula, For the The threshold of the layer at the tth iteration, is the threshold attenuation factor, which is used to balance the impact of the historical iteration threshold and the current iteration threshold. It is the median of the output of the previous neural network layer, that is, the median of the output of the l-1 layer neural network, which is used to reflect the typical threshold of the current activation state; is the median calculation function;

[0093] S405, using a quantum-inspired adaptive learning rate adjustment strategy to dynamically adjust the learning rate of the neural network, defining a quantum probability distribution to characterize the probability weights of different learning rates, setting is a set of learning rates, each learning rate corresponds to a quantum state, and the calculation method of the quantum state is expressed as:

[0094] ,

[0095] In the formula, is the quantum state corresponding to the i-th learning rate in the learning rate set, is the parameter of the quantum state corresponding to the i learning rate, is the first ground state of the quantum state, is an imaginary unit, is the learning rate phase, which is used to increase the flexibility of adjustment by utilizing the phase differences between different learning rates. is the second ground state of the quantum state;

[0096] The choice of learning rate is determined by measuring the quantum state, which corresponds to The probability of the state is calculated as:

[0097] ,

[0098] In the formula, is the probability of selecting the i-th learning rate in the learning rate set, for and The inner product of

[0099] Select the learning rate value with the largest probability in the learning rate set as the learning rate of the neural network for this iteration;

[0100] S406, repeat the above steps until a preset stop iteration condition is met. The preset stop iteration condition is reaching a preset maximum number of iterations, which means that the model training is completed.

[0101] The beneficial effects of the present invention are as follows: (1) The present invention adopts the BERT model based on constrained attention for vectorization. Through the interactive attention mechanism and the dynamic context perception mechanism, the model not only captures the semantic association within the text, but also better understands the relationship between the standard text and the classification task instructions, significantly improving the classification model's ability to understand the context. (2) The present invention adopts a generative adversarial network based on semantic focus to generate additional sample data, and ensures the semantic quality and consistency of the generated data through the semantic refocusing mechanism, solving the problem of insufficient samples in the standard text classification task. Compared with the traditional simple data augmentation method, the samples generated by the proposed method have higher semantic consistency and richness. (3) The present invention adds random perturbations and fuzzy processing strategies during the neural network training process, and improves the generalization ability of the model and the adaptability to different standard texts by reducing feature redundancy and increasing feature independence, thereby improving the stability and accuracy of the model in processing complex text classification tasks. (4) The present invention selects the learning rate through quantum state measurement, and the strategy of dynamically adjusting the learning rate enhances the adaptability of the model to the local loss surface. The quantum-inspired learning rate adjustment improves the training efficiency, which helps the model to be trained at the optimal learning rate at different stages to avoid gradient explosion or disappearance. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0103] Standard text data comes from industry standard publishing organizations and group standard writing organizations. The collection of standard text data is achieved through automated web crawler technology, which performs structured crawling on publicly released content from the above organizations. All collected standard text data is stored in JSON format. Each piece of standard text data contains standard text content, source link, release date, etc.

[0104] The obtained standard document text is manually annotated. Take a safety operation standard document of the power industry as an example. The safety operation standard document of the power industry stipulates the safety standards for high-voltage equipment operation. To simplify the description, the following is an example of a standard text data:

[0105] {

[0106] "Title of Standard Document": "Safety Standard for Operation of High Voltage Equipment",

[0107] "Standard content": "When performing maintenance and operation on high-voltage equipment, all circuit breakers must be in the disconnected state.",

[0108] "Annotation": "Operation steps"

[0109] }

[0110] The Word2Vec algorithm is used to vectorize the standard text in the standard text format. The Word2Vec algorithm is a commonly used vectorization algorithm in this field. According to a preset large-scale corpus, the standard text to be vectorized is scanned, and each word is represented as a one-hot encoded vector. The dimension of the one-hot encoded vector is equal to the size of the vocabulary in the corpus.

[0111] The standard text classification method based on machine learning in this embodiment includes the following steps:

[0112] S1, input the collected standard text data into the BERT model for BERT model training, and use the constrained attention-based BERT algorithm to vectorize the standard text.

[0113] This embodiment uses BERT (Bidirectional Encoder Representations from Transformers) model training to vectorize standard text. The Chinese explanation is: "Bidirectional Encoder is a bidirectional encoder, and Bidirectional Encoder Representations from Transformers is a bidirectional encoder representation based on Transformer. The traditional BERT model adopts the Transformer architecture to capture the dependencies in the standard text through the self-attention mechanism. This embodiment adopts an interactive attention mechanism, which not only considers the dependencies within the standard text, but also considers the interactive relationship between the standard text and the task instructions, so as to more accurately capture the features related to the standard topic.

[0114] Constrained attention is a strategy that accurately allocates attention weights by calculating specific constraints, ensuring that the model can effectively focus on key information related to task instructions while reducing attention to irrelevant content.

[0115] The training process of the BERT algorithm based on constrained attention is:

[0116] S101, pre-training the BERT model to enable it to understand the basic structure of the language. During the training process, the standard text instructions are first segmented and converted into a format that the model can understand. For the standard text "When performing maintenance and operation of high-voltage equipment, it is necessary to ensure that all circuit breakers are in the disconnected state. After segmentation, it may become ["When" "performing" "maintenance" and" "operation" of" "high-voltage equipment", "it is necessary to ensure" that" all" circuit breakers" are "in the" disconnected state".], the standard text after segmentation is converted into the corresponding word vector and then input into the model.

[0117] S102, based on the pre-training, the model is further trained through the interactive attention mechanism, so that the model learns how to adjust its attention distribution according to the task instructions, and adopts an attention enhancement factor to enhance the model's attention to the part of the standard text related to the task instructions. The calculation method of the attention enhancement factor is as follows:

[0118] ,

[0119] In the formula, is the attention enhancing factor, is the ReLU activation function, is the weight matrix of the trainable attention enhancement module, is the characteristic representation of standard text instructions, is the bias term of the attention enhancement module.

[0120] A dynamic context-aware mechanism is used to improve the model's ability to understand the context of standard text instructions. The dynamic context-aware mechanism means that the model generates context weights through a context-aware network. The weights are based on the current attention distribution and text features. After being processed by a nonlinear activation function, the model's attention score is dynamically adjusted, so that the model can increase its attention to key context information and reduce its attention to irrelevant information, forming a context-aware attention score. The dynamic context-aware mechanism enhances the model's attention mechanism through an additional context-aware network. The context-aware network can dynamically adjust the attention weights according to the context of standard text instructions, thereby more accurately capturing features related to task instructions. Let U be the output of the context-aware network, which dynamically generates context weights according to the current attention distribution and standard text features, expressed as:

[0121] ,

[0122] In the formula, It is a preset context-aware network.

[0123] The role of the dynamic context-aware mechanism is to improve the model's understanding of the context of standard text and task instructions, so that the model can dynamically adjust its attention distribution according to specific task instructions. Specifically, the context weights related to the task instructions are generated by the context-aware network. The context-aware network will analyze the current text features and combine the task instructions to generate dynamic context weights to guide the model's attention distribution.

[0124] The context weight is used to adjust the original attention score to obtain a new context-aware attention score, which is expressed as:

[0125] ,

[0126] In the formula, is the updated attention enhancement factor, i.e., the new context-aware attention score, For one with All-1 vectors of the same dimension, is the context weight, so when When , the model will pay less attention to non-critical information.

[0127] The context weight is calculated as follows:

[0128] ,

[0129] In the formula, is the LeakyReLU activation function, is the weight matrix of the context-aware network, is the bias term of the context-aware network.

[0130] Define the initialization method of the context-aware network. Suppose the weight matrix of the context-aware network is initialized in the following way:

[0131] ,

[0132] Where V is the dimension of the input feature of the context-aware network, H is the dimension of the hidden layer of the context-aware network, and the bias term of the context-aware network is initialized to a zero vector. The Glorot initialization function is a commonly used weight initialization method that aims to keep the variance of the input and output consistent.

[0133] S103, adopting a constrained attention adjustment strategy to further optimize the attention weight of the model, and using constraints to ensure that the attention weight does not deviate from the key information related to the task instructions. The constraint is calculated as follows:

[0134] ,

[0135] In the formula, is the constraint term of the constrained attention adjustment strategy, The function is used to convert the updated attention enhancement factor into a probability distribution to ensure the reasonable distribution of attention weights. and are the weight and bias parameters for constrained attention adjustment, respectively.

[0136] S104, dynamically adjust the model parameters according to the feedback during the training process through the adaptive adjustment mechanism, and the calculation method is as follows:

[0137] ,

[0138] In the formula, is the parameter update operation, is the Adam optimizer function, N is the number of samples input to the BERT model in the current batch, is the learning rate of the BERT model, is the weight update amount of the BERT model.

[0139] The BERT model parameters are optimized using the following objective function:

[0140] ,

[0141] Where L is the loss function of the BERT model, is the prediction output of the BERT model for the i-th sample, obtained by the preset Softmax classification function. for Norm.

[0142] S105, repeating the above steps until a preset stop iteration condition is met, the preset stop iteration condition is reaching a preset maximum number of iterations, and the preset maximum number of iterations is set to 1000 times.

[0143] S2 uses a semantically focused generative adversarial network algorithm to generate samples, thereby expanding the vectorized standard text data.

[0144] The collection, acquisition, annotation and preprocessing of vectorized standard text data are time-consuming and labor-intensive, and insufficient training samples can easily lead to poor model generalization ability and affect the accuracy of the model.

[0145] Semantic focus refers to the model's attention and emphasis on specific semantic content or topics in the process of generating text data, ensuring that the generated data can maintain consistency with the real data at the semantic level. Specifically, the generator monitors the semantic content of the generated text in real time through the semantic recognition module, converts the semantic features in the text into semantic score vectors, and dynamically adjusts the parameters of the generator based on the semantic score vectors of the generated samples, so as to focus on the important semantic features in the current generation task and make the generated text semantically consistent with the real data.

[0146] The training process of the semantic focus-based generative adversarial network algorithm is as follows:

[0147] S201, initialize the generator and discriminator of the generative adversarial network. The generator is responsible for generating vectorized standard text data, and the discriminator determines whether the input vectorized standard text data is real or generated by the generator. The initialization method is expressed as:

[0148] ,

[0149] In the formula, is the generator parameter, It means that it obeys a specific distribution. represents a normal distribution, is the initialization variance of the generator parameters, are the discriminator parameters, is the initialization variance of the discriminator parameters.

[0150] S202, in each training cycle of the training process, the generator generates preliminary vectorized standard text data samples, which are input into the discriminator for authenticity evaluation. The update of the generator depends on the feedback provided by the discriminator, and its parameters are adjusted to better simulate the distribution of the vectorized standard text data, which is expressed as:

[0151] ,

[0152] In the formula, and are the parameters of the generator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, is the gradient of the loss function of the generative adversarial network with respect to the generator parameters, Adjusting the increments for semantic refocusing of generator parameters;

[0153] is the loss function of the generative adversarial network, Express expectations, is the standard text data after real vectorization. is the standard text data distribution after real vectorization, is the discriminator, For the generator, is the input noise distribution, is the noise vector sampled from the input noise distribution, Dynamically adjust weights for features, is the discriminator function, is a generator function.

[0154] The role of is: in the iterative process of generating adversarial networks, when calculating the loss function, higher penalties are given to samples that are significantly different from the distribution of the true vectorized standard text data, so as to promote the generator to produce higher quality vectorized standard text data.

[0155] The semantic focus mechanism is used to improve the semantic quality and relevance of the generated standard text. The semantic focus in the generation process is dynamically adjusted to ensure the accuracy and richness of the generated content. The semantic focus mechanism is implemented through the semantic recognition module, which is responsible for real-time monitoring and evaluation of the semantic focus of the generated standard text and adjusting it according to the preset target semantic focus. The calculation method of semantic recognition and scoring is expressed as follows:

[0156] ,

[0157] In the formula, is the function for calculating the semantic score vector, To generate the semantic score vector of the sample, is the Softmax function, and are the weight and bias of the semantic recognition module respectively, A function for extracting semantic features, specifically a 2-layer fully connected neural network. Generate samples for the generator;

[0158] Based on the semantic score vector of the generated sample, the semantic refocusing adjustment increment of the generator parameters is calculated as:

[0159] ,

[0160] In the formula, is the gradient with respect to the generator parameters, is the learning rate of the generative adversarial network, is the adjustment factor used to control the strength of semantic adjustment, is the Kullback-Leibler divergence, which is used to measure the difference between the semantic distribution of generated samples and the semantic distribution of real samples. is the semantic distribution of the real sample, which is calculated by the preset Softmax function.

[0161] The dynamic adjustment weight of the feature makes the discriminator give higher penalties to samples with large distribution differences of the real vectorized standard text data when evaluating the generated samples, which promotes the generator to produce higher quality vectorized standard text data. The calculation method of the sample penalty factor is expressed as:

[0162] ,

[0163] In the formula, for norm, and They are the mean and variance of the standard text data distribution after vectorization.

[0164] S203, in the process of generating the vectorized standard text data, the random noise that may appear in the generation process is reduced by a self-stable noise reduction strategy, so as to improve the quality and consistency of the generated vectorized standard text data. The implementation method of the noise reduction strategy is expressed as follows:

[0165] ,

[0166] In the formula, The original vectorized standard text data sample generated by the generator. is a vectorized standard text data sample with noise reduction, is the noise reduction function, is the self-stationary noise reduction adjustment coefficient, Set to 0.3 to control the strength of noise reduction. is the hyperbolic tangent function, is a tuning parameter that controls the sensitivity of the nonlinear transformation, Set to 2, To generate the variance of the vectorized standard text data sample, is the variance function.

[0167] S204, during the training process, the loss function not only considers the difference in statistical characteristics between the generated vectorized standard text data and the real vectorized standard text data, but also uses the evaluation indicators of semantic coherence and sample quality to promote the discriminator to more accurately guide the generator to generate high-quality vectorized standard text data. The calculation method is expressed as:

[0168] ,

[0169] In the formula, is the cross entropy loss for generating adversarial networks, Adjust the weights for the first GAN loss function, The weights are adjusted for the second GAN loss function, the weights are adjusted for the first GAN loss function, and the weights are adjusted for the cross entropy loss and semantic coherence cost of the GAN loss function. To generate the semantic coherence difference between the sample and the real sample, This is a standard text data sample after real vectorization.

[0170] The semantic coherence difference is combined with the difference measurement of semantic depth features, which not only evaluates the surface difference of the standard text, but also goes deep into the semantic level to ensure the semantic coherence and practicality of the generated standard text. The calculation method of the semantic coherence difference between the generated sample and the real sample is expressed as:

[0171] ,

[0172] In the formula, is the number of elements in the sample, It is a multi-layer neural network representation function in the deep learning model, which is used to extract the deep semantic features of the vectorized standard text data. and are the i-th element in the generated sample and the real sample respectively.

[0173] In the process of updating the parameters of the discriminator, the problem of training instability caused by gradient explosion or disappearance is usually encountered. In this embodiment, the gradient is normalized before each gradient update so that the update step size is kept within a reasonable range, which is expressed as:

[0174] ,

[0175] In the formula, is the gradient of the unprocessed discriminator parameters, The gradient of the loss function of the generative adversarial network with respect to the discriminator parameters;

[0176] is the gradient of the normalized discriminator parameters, is the expectation of the squared gradient of the unprocessed discriminator parameters, used to estimate the average magnitude of the gradient during the entire training process, is a small constant to prevent the denominator from being zero, Set to 0.0001;

[0177] are the parameters of the discriminator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, Set to 0.01.

[0178] S205, repeat the above steps until a preset stop iteration condition is met. The preset stop iteration condition is reaching a preset maximum number of iterations. The preset maximum number of iterations is set to 1000 times, which means that the model training is completed.

[0179] S3, after the training of the vectorized standard text data expansion model is completed, the trained vectorized standard text data expansion model is used to increase the number of samples, and the expanded vectorized standard text data set is used to train the standard text classification model, which is a fully connected neural network model.

[0180] Assume that the original collected samples are 800, and the vectorized standard text data expansion model generates 200 samples by expansion. Then the expanded vectorized standard text data set contains 1000 samples.

[0181] The fully connected neural network model includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, and an output layer. The input vector is connected to the first hidden layer as the input of the fully connected network, the output of the first hidden layer is used as the input of the second hidden layer, the output of the second hidden layer is used as the input of the third hidden layer, the output of the third hidden layer is used as the input of the fourth hidden layer, and the output of the fourth hidden layer is used as the input of the output layer.

[0182] (1) Input layer: The input layer receives input vectorized standard text data. The vectorized standard text data is the data in the expanded data set. Since the length of the input vector is uncertain, all input vectorized standard text data must be padded or truncated to unify the input length. In one embodiment, the number of neurons in the input layer is set to 5000, that is, the dimension of the input vector of each sample entering the neural network is 5000.

[0183] (2) First hidden layer: The first hidden layer is a fully connected layer with 1280 nodes, that is, this layer contains 1280 neurons. The activation function is the ReLU activation function, which maps the input vector to the new feature space to form a feature representation with a length of 1280, and performs nonlinear feature transformation through ReLU activation.

[0184] (3) Second hidden layer: The second hidden layer is a fully connected layer with 640 nodes, that is, it contains 640 neurons and the activation function is the ReLU activation function, which maps the input vector to the new feature space to form a feature representation with a length of 640, and performs nonlinear feature transformation through ReLU activation;

[0185] (4) The third hidden layer: The third hidden layer is a fully connected layer with 320 nodes, that is, this layer contains 320 neurons. The activation function is the ReLU activation function, which maps the input vector to the new feature space to form a feature representation with a length of 320, and performs nonlinear feature transformation through ReLU activation;

[0186] (5) The fourth hidden layer is a fully connected layer with 160 nodes, that is, this layer contains 160 neurons. The activation function is the ReLU activation function, which maps the input vector to the new feature space to form a feature representation with a length of 160, and performs nonlinear feature transformation through ReLU activation;

[0187] (6) Output layer: The output layer is a fully connected layer. The number of nodes is the same as the number of categories. For example, if the number of categories is 10, the output layer contains 10 neurons. The activation function of the output layer is the Softmax activation function, which is used for multi-classification tasks and outputs the probability of each category. The output of the output layer is a probability vector, in which each element represents the probability that the input sample belongs to the corresponding category, and the sum of the probabilities of all categories is 1.

[0188] S4, uses a neural network algorithm based on perturbation fuzzy fusion to perform feature extraction and classification of vectorized standard text data.

[0189] In the prior art, some solutions use neural networks for feature extraction and classification. In some neural network structures, they may encounter problems such as gradient vanishing, gradient explosion, or falling into local optimal solutions, which affects the stability of training and the performance of the model. This embodiment optimizes the performance of the standard text classification model by reducing the redundancy between features, improving the independence and expressiveness of features, and using a perturbation fuzzy fusion strategy to improve the robustness and generalization ability of features, making the model more adaptable to different vectorized standard text data feature distributions.

[0190] Perturbation fuzzy fusion is achieved by introducing random perturbations based on standard normal distribution into the vectorized text features. Specifically, the perturbation intensity is adaptively adjusted according to the variance of the input features. The perturbation is reduced in areas where the features are stable, and increased in areas where the features vary greatly, so as to better simulate the uncertainty and diversity in real data, enable the neural network to better capture independent and discriminative features, prevent overfitting and enhance the performance of the model in test data.

[0191] The training process of the neural network algorithm based on perturbation fuzzy fusion is:

[0192] S401, initializing the neural network model including the weights and biases of the neural network, the initialization method is expressed as:

[0193] ,

[0194] In the formula, is the number of nodes in the input layer of the neural network, is sampled from a normal distribution with mean 0 and unit variance, is a normal distribution, is the identity matrix, is the weight of the neural network, is the bias of the neural network.

[0195] S402, during the training process of the neural network, each layer of the network is constrained through a specific regularization strategy to reduce the correlation between features.

[0196] The regularization strategy weakens the collinearity between features to ensure that the network can capture more independent and discriminative features. The calculation method is expressed as:

[0197] ,

[0198] In the formula, is the regularization function, is the regularization coefficient, which is used to control the influence strength of the regularization term. for norm, and are the i-th and j-th rows in the neural network weight matrix;

[0199] is the total loss of the neural network, is the cross entropy loss, It is a regularization term of the neural network. Its function is to reduce the similarity between different feature vectors by performing loss calculation during the training process. Its calculation helps to enhance the orthogonality between feature vectors, thereby enhancing the feature discrimination ability of the model.

[0200] S403, using random perturbation and fuzzy processing to enhance the generalization ability of various changes and uncertainties that may be encountered by the vectorized standard text data, and prevent the overfitting of the neural network model. The implementation method of adding random perturbation in the feature extraction process is expressed as:

[0201] ,

[0202] In the formula, is the perturbation input feature vector of the neural network, represents the input feature vector of the neural network, is the perturbation intensity of the neural network, controlling the magnitude of the noise, Set to 0.001, Generate standard normally distributed noise of the same size as the input feature vector of the neural network, represents the size measurement function, is a random function, It is a function adjusted according to the characteristics of the neural network input.

[0203] The function adjusted according to the input characteristics of the neural network adjusts the intensity of the noise based on the variance of the input features, so that the disturbance is reduced in the area where the features are relatively stable, and the disturbance is increased in the area where the features vary greatly, so as to more effectively adapt to the uncertainty of the vectorized standard text data. The calculation method of the function adjusted according to the input characteristics of the neural network is expressed as:

[0204] ,

[0205] In the formula, To calculate the variance of the neural network input, is the variance function, A preset standard for adjusting disturbance sensitivity.

[0206] S404, during the forward propagation process, the neural network extracts key features based on the standard text data after training vectorization, and continuously adjusts the network parameters in the back propagation to minimize the loss function. The forward propagation and parameter optimization of the neural network are expressed as follows:

[0207] ,

[0208] In the formula, Input feature vectors to the layers of the neural network, is the output feature vector of the neural network layer, is the ReLU activation function, is the standard text data dynamic threshold after statistical vectorization of the lth layer of the neural network at the tth iteration.

[0209] The back propagation algorithm is used to update the parameters of the neural network. The incremental calculation method of the update is expressed as:

[0210] ,

[0211] In the formula, is the weight update increment of the neural network, is the bias update increment of the neural network, is the learning rate of the neural network, is the symbol of partial derivative, Set to 0.001.

[0212] The dynamic threshold of the standard text data after statistical vectorization is dynamically adjusted for the activation output of each layer in the neural network. By analyzing the statistical characteristics of the output of each layer in each iteration, the threshold of the activation function is automatically adjusted to optimize the network's response to the input features, thereby improving the classification accuracy and robustness of the overall model. The calculation method of the dynamic threshold of the standard text data after statistical vectorization is expressed as follows:

[0213] ,

[0214] In the formula, is the threshold of the lth layer at the tth iteration, is the threshold attenuation factor, which is used to balance the impact of the historical iteration threshold and the current iteration threshold. It is the median of the output of the previous neural network layer, that is, the median of the output of the l-1 layer neural network, which is used to reflect the typical threshold of the current activation state; The median calculation function.

[0215] S405, the learning rate adjustment of the traditional neural network often cannot accurately adapt to the local characteristics of the loss surface. This embodiment adopts a quantum-inspired adaptive learning rate adjustment strategy to dynamically adjust the learning rate of the neural network, defines a quantum probability distribution to characterize the probability weights of different learning rates, and sets is a set of learning rates, each learning rate corresponds to a quantum state, and the calculation method of the quantum state is expressed as:

[0216] ,

[0217] In the formula, is the quantum state corresponding to the i-th learning rate in the learning rate set, is the parameter of the quantum state corresponding to the i learning rate, is the first ground state of the quantum state, is an imaginary unit, is the learning rate phase, which is used to increase the flexibility of adjustment by utilizing the phase differences between different learning rates. It is the second ground state of the quantum state.

[0218] The choice of learning rate is determined by measuring the quantum state, which corresponds to The probability of the state is calculated as:

[0219] ,

[0220] In the formula, is the probability of selecting the i-th learning rate in the learning rate set, for and The inner product of .

[0221] The learning rate value with the largest probability in the learning rate set is selected as the learning rate of the neural network for this iteration.

[0222] S406, repeat the above steps until a preset stop iteration condition is met. The preset stop iteration condition is reaching a preset maximum number of iterations. The preset maximum number of iterations is set to 1000 times, which means that the model training is completed.

[0223] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A standard text classification method based on machine learning, characterized in that: The following steps are involved: S1, input the collected standard text data into the BERT model for BERT model training, and use the constrained attention-based BERT algorithm to vectorize the standard text; S2, uses a semantic focus-based generative adversarial network algorithm to generate samples, thereby achieving vectorized standard text data expansion; The training process of the semantic focus-based generative adversarial network algorithm in step S2 is as follows: S201, initialize the generator and discriminator of the generative adversarial network. The generator is responsible for generating vectorized standard text data, and the discriminator determines whether the input vectorized standard text data is real or generated by the generator. The initialization method is expressed as: ， In the formula, is the generator parameter, It means that it obeys a specific distribution. represents a normal distribution, is the initialization variance of the generator parameters, are the discriminator parameters, is the initialization variance of the discriminator parameters; S202, in each training cycle of the training process, the generator generates preliminary vectorized standard text data samples, which are input into the discriminator for authenticity evaluation. The update of the generator depends on the feedback provided by the discriminator, and its parameters are adjusted to better simulate the distribution of the vectorized standard text data, which is expressed as: ， In the formula, and are the parameters of the generator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, is the gradient of the loss function of the generative adversarial network with respect to the generator parameters, Adjusting the increments for semantic refocusing of generator parameters; is the loss function of the generative adversarial network, Express expectations, is the standard text data after real vectorization. is the standard text data distribution after real vectorization, is the discriminator, For the generator, is the input noise distribution, is the noise vector sampled from the input noise distribution, Dynamically adjust weights for features, is the discriminator function, is the generator function; The semantic focus mechanism is used to improve the semantic quality and relevance of the generated standard text. The semantic focus in the generation process is dynamically adjusted to ensure the accuracy and richness of the generated content. The semantic focus mechanism is implemented through the semantic recognition module, which is responsible for real-time monitoring and evaluation of the semantic focus of the generated standard text and adjusting it according to the preset target semantic focus. The calculation method of semantic recognition and scoring is expressed as follows: ， In the formula, is the function for calculating the semantic score vector, To generate the semantic score vector of the sample, is the Softmax function, and are the weight and bias of the semantic recognition module respectively, A function for extracting semantic features, specifically a 2-layer fully connected neural network. Generate samples for the generator; Based on the semantic score vector of the generated sample, the semantic refocusing adjustment increment of the generator parameters is calculated as: ， In the formula, is the gradient with respect to the generator parameters, is the learning rate of the generative adversarial network, is the adjustment factor used to control the strength of semantic adjustment, is the Kullback-Leibler divergence, which is used to measure the difference between the semantic distribution of generated samples and the semantic distribution of real samples. is the semantic distribution of the real sample, calculated by the preset Softmax function; The dynamic adjustment weight of the feature makes the discriminator give higher penalties to samples with large distribution differences of the real vectorized standard text data when evaluating the generated samples, which promotes the generator to produce higher quality vectorized standard text data. The calculation method of the dynamic adjustment weight of the feature is expressed as: ， In the formula, for norm, and They are the mean and variance of the standard text data distribution after vectorization; S203, in the process of generating the vectorized standard text data, the random noise that may appear in the generation process is reduced by a self-stable noise reduction strategy, and the implementation method of the noise reduction strategy is expressed as follows: ， In the formula, The original vectorized standard text data sample generated by the generator. is a vectorized standard text data sample with noise reduction, is the noise reduction function, is the self-stationary noise reduction adjustment coefficient, Set to 0.3 to control the strength of noise reduction. is the hyperbolic tangent function, is a tuning parameter that controls the sensitivity of the nonlinear transformation, Set to 2, To generate the variance of the vectorized standard text data sample, is the variance function; S204, during the training process, the loss function not only considers the difference in statistical characteristics between the generated vectorized standard text data and the real vectorized standard text data, but also uses the evaluation indicators of semantic coherence and sample quality to promote the discriminator to more accurately guide the generator to generate high-quality vectorized standard text data. The calculation method is expressed as: ， In the formula, is the cross entropy loss for generating adversarial networks, Adjust the weights for the first GAN loss function, The weights are adjusted for the second GAN loss function, the weights are adjusted for the first GAN loss function, and the weights are adjusted for the cross entropy loss and semantic coherence cost of the GAN loss function. To generate the semantic coherence difference between the sample and the real sample, is a standard text data sample after real vectorization; The semantic coherence difference is combined with the difference measurement of semantic depth features, which not only evaluates the surface difference of the standard text, but also goes deep into the semantic level to ensure the semantic coherence and practicality of the generated standard text. The calculation method of the semantic coherence difference between the generated sample and the real sample is expressed as: ， In the formula, is the number of elements in the sample, It is a multi-layer neural network representation function in the deep learning model, which is used to extract the deep semantic features of the vectorized standard text data. and are the first elements; The gradient is normalized before each gradient update so that the update step size remains within a reasonable range, expressed as: ， In the formula, is the gradient of the unprocessed discriminator parameters, The gradient of the loss function of the generative adversarial network with respect to the discriminator parameters; is the gradient of the normalized discriminator parameters, is the expectation of the squared gradient of the unprocessed discriminator parameters, used to estimate the average magnitude of the gradient during the entire training process, is a small constant to prevent the denominator from being zero, Set to 0.0001; are the parameters of the discriminator at the tth and t+1th iterations, respectively. is the learning rate of the generative adversarial network, Set to 0.01; S205, repeating the above steps until a preset stop iteration condition is met, where the preset stop iteration condition is reaching a preset maximum number of iterations, indicating that the model training is completed; S3, after the training of the vectorized standard text data expansion model is completed, the number of samples is increased by using the trained vectorized standard text data expansion model, and the expanded vectorized standard text data set is used to train a standard text classification model, where the standard text classification model is a fully connected neural network model; S4, uses a neural network algorithm based on perturbation fuzzy fusion to perform feature extraction and classification of vectorized standard text data.

2. The standard text classification method based on machine learning according to claim 1, characterized in that: The training process of the BERT algorithm based on constrained attention in step S1 is: S101, pre-training the BERT model to enable it to understand the basic structure of language. During the training process, standard text instructions are first segmented and converted into a format that the model can understand; S102, based on the pre-training, the model is further trained through the interactive attention mechanism, so that the model learns how to adjust its attention distribution according to the task instructions, and adopts an attention enhancement factor to enhance the model's attention to the part of the standard text related to the task instructions. The calculation method of the attention enhancement factor is as follows: , In the formula, is the attention enhancing factor, is the ReLU activation function, is the weight matrix of the trainable attention enhancement module, is the characteristic representation of standard text instructions, is the bias term of the attention enhancement module; A dynamic context-aware mechanism is used to improve the model's ability to understand the context of standard text instructions. The dynamic context-aware mechanism enhances the model's attention mechanism through an additional context-aware network. Let U be the output of the context-aware network, which dynamically generates context weights based on the current attention distribution and standard text features, expressed as: , In the formula, It is a preset context-aware network; The context weight is used to adjust the original attention score to obtain a new context-aware attention score, which is expressed as: , In the formula, is the updated attention enhancement factor, i.e., the new context-aware attention score, For one with All-1 vectors of the same dimension, is the context weight, so when When When , the model will pay less attention to non-critical information; The context weight is calculated as follows: , In the formula, is the LeakyReLU activation function, is the weight matrix of the context-aware network, is the bias term of the context-aware network; Define the initialization method of the context-aware network. Suppose the weight matrix of the context-aware network is initialized in the following way: , Where V is the dimension of the input feature of the context-aware network, H is the dimension of the hidden layer of the context-aware network, and the bias term of the context-aware network is initialized to a zero vector. Glorot initialization function is a commonly used weight initialization method that aims to keep the variance of input and output consistent; S103, adopt the constrained attention adjustment strategy to further optimize the attention weight of the model. The constraint term is calculated as follows: , In the formula, is the constraint term of the constrained attention adjustment strategy, The function is used to convert the updated attention enhancement factor into a probability distribution to ensure the reasonable distribution of attention weights. and are the weight and bias parameters of constrained attention adjustment, respectively; S104, dynamically adjust the model parameters according to the feedback during the training process through the adaptive adjustment mechanism, and the calculation method is as follows: , In the formula, is the parameter update operation, is the Adam optimizer function, N is the number of samples input to the BERT model in the current batch, is the learning rate of the BERT model, is the weight update amount of the BERT model; The BERT model parameters are optimized using the following objective function: , Where L is the loss function of the BERT model, The BERT model is The predicted output of samples is obtained by the preset Softmax classification function. for norm; S105, repeating the above steps until a preset stop iteration condition is met, where the preset stop iteration condition is reaching a preset maximum number of iterations.

3. The standard text classification method based on machine learning according to claim 1, characterized in that: The fully connected neural network model in step S3 includes an input layer, a first hidden layer, a second hidden layer, a third hidden layer, a fourth hidden layer, and an output layer. The input vector is connected to the first hidden layer as the input of the fully connected network, the output of the first hidden layer is used as the input of the second hidden layer, the output of the second hidden layer is used as the input of the third hidden layer, the output of the third hidden layer is used as the input of the fourth hidden layer, and the output of the fourth hidden layer is used as the input of the output layer.

4. The standard text classification method based on machine learning according to claim 1, characterized in that: The training process of the neural network algorithm based on disturbance fuzzy fusion in step S4 is as follows: S401, initializing the neural network model including the weights and biases of the neural network, the initialization method is expressed as: , In the formula, is the number of nodes in the input layer of the neural network, is sampled from a normal distribution with mean 0 and unit variance, is a normal distribution, is the identity matrix, is the weight of the neural network, is the bias of the neural network; S402, during the training process of the neural network, each layer of the network is constrained through a specific regularization strategy to reduce the correlation between features; The regularization strategy weakens the collinearity between features to ensure that the network can capture more independent and discriminative features. The calculation method is expressed as: ， In the formula, is the regularization function, is the regularization coefficient, which is used to control the influence strength of the regularization term. for norm, and is the weight matrix of the neural network Row and OK; is the total loss of the neural network, is the cross entropy loss, It is a regularization term of the neural network. Its function is to reduce the similarity between different feature vectors by calculating the loss during the training process. Its calculation helps to enhance the orthogonality between feature vectors, thereby enhancing the feature discrimination ability of the model. S403, using random perturbation and fuzzy processing to enhance the generalization ability of various changes and uncertainties that may be encountered by the vectorized standard text data, and prevent the overfitting of the neural network model. The implementation method of adding random perturbation in the feature extraction process is expressed as: ， In the formula, is the perturbation input feature vector of the neural network, represents the input feature vector of the neural network, is the perturbation intensity of the neural network, controlling the magnitude of the noise, Set to 0.001, Generate standard normally distributed noise of the same size as the input feature vector of the neural network, represents the size measurement function, is a random function, It is a function adjusted according to the characteristics of the neural network input; The function adjusted according to the input characteristics of the neural network adjusts the intensity of the noise based on the variance of the input features, so that the disturbance is reduced in the area where the features are relatively stable, and the disturbance is increased in the area where the features vary greatly, so as to more effectively adapt to the uncertainty of the vectorized standard text data. The calculation method of the function adjusted according to the input characteristics of the neural network is expressed as: ， In the formula, To calculate the variance of the neural network input, is the variance function, is a preset standard for adjusting disturbance sensitivity; S404, during the forward propagation process, the neural network extracts key features based on the standard text data after training vectorization, and continuously adjusts the network parameters in the back propagation to minimize the loss function. The forward propagation and parameter optimization of the neural network are expressed as follows: ， In the formula, Input feature vectors to the layers of the neural network, is the output feature vector of the neural network layer, is the ReLU activation function, For the neural network The dynamic threshold of the standard text data after the statistical vectorization of the layer at the tth iteration; The back propagation algorithm is used to update the parameters of the neural network. The incremental calculation method of the update is expressed as: ， In the formula, is the weight update increment of the neural network, is the bias update increment of the neural network, is the learning rate of the neural network, is the symbol of partial derivative, Set to 0.001; The dynamic threshold of the standard text data after statistical vectorization is dynamically adjusted for the activation output of each layer in the neural network. By analyzing the statistical characteristics of the output of each layer in each iteration, the threshold of the activation function is automatically adjusted to optimize the network's response to the input features, thereby improving the classification accuracy and robustness of the overall model. The calculation method of the dynamic threshold of the standard text data after statistical vectorization is expressed as follows: ， In the formula, For the The threshold of the layer at the tth iteration, is the threshold attenuation factor, which is used to balance the impact of the historical iteration threshold and the current iteration threshold. It is the median of the output of the previous neural network layer, that is, the median of the output of the l-1 layer neural network, which is used to reflect the typical threshold of the current activation state; is the median calculation function; S405, using a quantum-inspired adaptive learning rate adjustment strategy to dynamically adjust the learning rate of the neural network, defining a quantum probability distribution to characterize the probability weights of different learning rates, setting is a set of learning rates, each learning rate corresponds to a quantum state, and the calculation method of the quantum state is expressed as: , In the formula, is the first in the learning rate set The quantum state corresponding to the learning rate is For the The parameters of the quantum state corresponding to the learning rate, is the first ground state of the quantum state, is an imaginary unit, is the learning rate phase, which is used to increase the flexibility of adjustment by utilizing the phase differences between different learning rates. is the second ground state of the quantum state; The choice of learning rate is determined by measuring the quantum state, which corresponds to The probability of the state is calculated as: , In the formula, is the first in the learning rate set The probability of selecting a learning rate, for and The inner product of Select the learning rate value with the largest probability in the learning rate set as the learning rate of the neural network for this iteration; S406, repeat the above steps until a preset stop iteration condition is met. The preset stop iteration condition is reaching a preset maximum number of iterations, which means that the model training is completed.

Images