Text prompt classification method based on improved BERT-WWM

Abstract

The invention discloses a text mentioning and classifying method based on improved BERT-WWM. The method comprises the following steps: collecting power infrastructure text data; constructing a BERT-WWM model, introducing sparse attention to replace multi-head self-attention in the BERT-WWM model, and adaptively adjusting a mask proportion in the sparse attention to obtain an improved BERT-WWM model; training the improved BERT-WWM model by adopting the data set, and optimizing parameters of the improved BERT-WWM model at the same time, so as to obtain an optimized improved BERT-WWM model; inputting power infrastructure text data to be classified into the optimized improved BERT-WWM model to obtain a classification result; according to the method, through the uniquely designed and optimized improved BERT-WWM model, the classification accuracy and the processing efficiency of the power system engineering text are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text classification, and in particular to a text prompt classification method based on an improved BERT-WWM. Background Art

[0002] With the increasing construction of power system projects, project management plays a vital role in ensuring the smooth implementation of projects, improving project quality and controlling costs. The management of power system engineering information covers detailed information on numerous equipment and facilities from power stations to substations, such as technical parameters of transformers, line layout, equipment capacity, etc. This information is usually saved in the form of Word documents, containing a large amount of complex professional data and descriptive text. In current technical means, project managers generally extract and manage information through manual methods, general text extraction tools or basic regular expressions. Although these methods have a certain role in preliminary data processing, they still have significant shortcomings.

[0003] From the perspective of large background technology, traditional engineering information management methods are difficult to achieve intelligent, rapid and efficient information, which poses a huge challenge to power engineering project management. In the case of huge amounts of information and frequent data processing needs, power system engineering requires more efficient technical means to improve the accuracy of data extraction and management, reduce the uncertainty caused by human factors, and have flexible expansion capabilities to adapt to new technologies and new needs; from the perspective of small background technology, the specific methods of power system engineering document processing are still mainly based on the following technologies: Manual entry method: Staff manually enter key information into Excel or database by reading Word documents. This method has high requirements for personnel, heavy data entry work and low efficiency, and manual operations are often prone to errors. General text extraction tools: Common OCR or PDF conversion software can convert documents into text, but these general tools lack the ability to understand professional terms in the power system, cannot accurately identify or process data containing professional vocabulary, and are more sensitive to format changes. The extraction results need to be cleaned up and sorted twice. Regular expression matching: Use regular expressions to identify and extract data in a specific format from a document. This method is effective when dealing with relatively simple text structures, but for complex formats and unstructured text content, the recognition accuracy is low and it is difficult to meet the needs of power engineering information extraction.

[0004] The defects of current existing technology:

[0005] 1. Limited text processing capabilities: Existing text processing methods mainly rely on simple regular expression matching or basic text extraction tools, which cannot effectively handle long-distance text dependencies. Especially when processing power system engineering documents containing complex professional terms, problems such as inaccurate extraction or missing important information often occur.

[0006] 2. Insufficient model generalization ability: Traditional text classification models are prone to overfitting when processing new samples, especially in highly professional scenarios such as the power system field. The model often finds it difficult to accurately understand and classify unseen professional terms and expressions.

[0007] 3. Inefficient parameter optimization: In the process of model training, the existing methods have relatively simple parameter optimization strategies, often using a fixed learning rate and a unified parameter update method. It is difficult to adaptively adjust the parameter features at different levels, resulting in slow model convergence and unstable training results.

[0008] 4. Hyperparameter adjustment lacks intelligence: Traditional hyperparameter adjustment methods mainly rely on manual experience or simple grid search. This method is not only time-consuming and labor-intensive, but also difficult to find the optimal parameter combination. Especially when processing large-scale power system text data, it often requires repeated attempts to obtain better results. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention provides a text prompt classification method based on an improved BERT-WWM, which aims to solve the problems mentioned in the background technology.

[0010] To achieve the above object, the present invention provides the following technical solution: a text prompt classification method based on an improved BERT-WWM, comprising the following steps: Step S1: collecting power infrastructure text data for preprocessing and constructing a data set, where the power infrastructure text data consists of sentences of different lengths; Step S2: Build a BERT-WWM model, introduce sparse attention to replace the multi-head self-attention in the BERT-WWM model, and adaptively adjust the mask ratio in the sparse attention to obtain an improved BERT-WWM model; Step S3: using the data set to train the improved BERT-WWM model, and optimizing the parameters of the improved BERT-WWM model to obtain an optimized improved BERT-WWM model; Step S4: input the power infrastructure text data to be classified into the optimized improved BERT-WWM model to obtain the classification result; The specific process of optimizing and improving the parameters of the BERT-WWM model is as follows: Combine Dropout regularization and DropConnect regularization to regularize the parameters of the improved BERT-WWM model; The LAMB optimizer is used as the optimizer for improving the BERT-WWM model; An LSTM-based meta-learning model is used to optimize the hyperparameters during training of the improved BERT-WWM model.

[0011] Furthermore, the mask ratio in sparse attention is adaptively adjusted, expressed as: ; In the formula, Indicates that the sentence length in the power infrastructure text data is The mask ratio when and It is an adjustable parameter.

[0012] Furthermore, the specific process of using the dataset to train the improved BERT-WWM model is: dividing the dataset into a training set and a validation set in proportion, using the training set to train the improved BERT-WWM model, and using the validation set to validate the improved BERT-WWM model.

[0013] Furthermore, the specific process of combining Dropout regularization and DropConnect regularization to regularize the parameters of the improved BERT-WWM model is as follows: By weight factor The dropout probability of Dropout regularization and DropConnect regularization is weighted combined to obtain the joint regularization probability , expressed as: ; In the formula, and Represent the dropout probability of Dropout regularization and DropConnect regularization respectively; The drop probability is dynamically adjusted according to the gradient information of each layer of the improved BERT-WWM model, expressed as: ; In the formula, is the basic drop probability, To improve the BERT-WWM model The gradient of the layer weights, To improve the maximum value of all layer gradients of the BERT-WWM model; Indicates the improved BERT-WWM model The dropout regularization dropout probability of the layer; Calculate the regularization strength, expressed as: ; In the formula, Indicates that in improving the BERT-WWM model Layers, training rounds The regularization strength when ; To improve the total number of layers of the BERT-WWM model; is the total number of training rounds; is a preset initial regularization strength; is a hyperparameter that controls the decay rate; Indicates the current training round; Design an improved gradient update rule for the BERT-WWM model; in the case of joint regularization probability, the gradient update formula for the improved BERT-WWM model is: ; In the formula, To improve the BERT-WWM model Layer weights; is the learning rate; is the loss function For Partial derivatives of layer weights; and They are the mask matrices for Dropout regularization and DropConnect regularization respectively; Based on the improved performance of the BERT-WWM model on the validation set, dynamically adjust and : ; ; In the formula, Indicates the change in validation set loss; is the adjustment factor; is the loss of the current validation set; for During training ; For the During training ; for During training ; for During training .

[0014] Furthermore, the specific process of using the LSTM-based meta-learning model to optimize the hyperparameters during training of the improved BERT-WWM model is as follows: Set the improved BERT-WWM model on the dataset The training task is ,Record The hyperparameters and improved BERT-WWM model completed the Performance data after training tasks ; Learn the hyperparameters and performance data of historical training tasks and generate the hyperparameter selection of the current training task, expressed as: ; ; In the formula, Indicates that The network generated Hyperparameter selection for each training task; Indicates that The network generated Hyperparameter selection for each training task; The process of network generation hyperparameter selection is expressed as: ; ; In the formula, represents the hyperparameter selection strategy; and is the weight matrix of hyperparameters and performance data, and is the activation function of the LSTM network; express The weight matrix in the network; and is the bias term; according to , train the improved BERT-WWM model in the current training task, and introduce an adaptive mechanism to adjust the hyperparameters according to the performance data of the improved BERT-WWM model.

[0015] Further, according to , train the improved BERT-WWM model in the current training task, and introduce an adaptive mechanism to adjust the hyperparameters according to the performance data of the improved BERT-WWM model. The specific process is as follows: Minimize the loss function of the training task by adjusting the hyperparameters , that is, selecting a set of optimal hyperparameter configurations in each training task to achieve the best improved BERT-WWM model performance; the loss function of the training task It is expressed as: ; In the formula, Indicates that the improved BERT-WWM model is The output of the following; It is a training task corresponding losses; Optimize the loss function for all training tasks , find the optimal hyperparameter configuration, expressed as: ; In the formula, is the total number of training tasks;

[0016] Introducing real-time feedback information of training tasks improves the accuracy change or loss change of the BERT-WWM model and dynamically adjusts the selection of hyperparameters, which can be expressed as: ; In the formula, is the current learning rate; is the adjustment factor; yes The amount of change; is the current loss value; Minimize the sum of the loss functions of all training tasks and obtain the hyperparameters of the improved BERT-WWM model after optimization of the LSTM-based meta-learning model , expressed as: ;

[0017] In the formula, Represents the hyperparameters of minimizing the improved BERT-WWM model .

[0018] Furthermore, the LAMB optimizer is used as the optimizer for improving the BERT-WWM model, which is expressed as: ; In the formula, and Respectively represent the estimated values ​​of the mean and variance of the improved BERT-WWM model gradient; is a constant; Indicates level correction.

[0019] Furthermore, the text data of the power infrastructure is collected and preprocessed, and the preprocessing includes: Text cleaning, using regular expressions to clean power infrastructure text data; Information extraction, extracting key information from text data of power infrastructure; Text vectorization: vectorize the power infrastructure text data after text cleaning.

[0020] Compared with the existing technology, the present invention has the following beneficial effects:

[0021] (1) This paper significantly improves the classification accuracy and processing efficiency of power system engineering texts through a uniquely designed and optimized improved BERT-WWM model. Test results show that the classification accuracy of this scheme on the standard test set reaches 95%, and the processing speed is significantly improved compared with the original BERT model. It can also effectively identify and classify complex professional terms and long text information.

[0022] (2) The present invention introduces a sparse attention mechanism into the BERT-WWM model and adaptively adjusts the mask ratio in the sparse attention, thereby effectively reducing the computational complexity, greatly improving the performance of the model in processing long texts, and at the same time improving the ability to capture long-distance dependencies; by adopting joint regularization and LAMB optimizer to optimize and improve the BERT-WWM model, the generalization ability of the model is significantly enhanced, the overfitting phenomenon of the model is reduced, the training convergence speed is increased by about, and the training process is more stable; by adopting an LSTM-based meta-learning model to optimize and improve the hyperparameters during BERT-WWM model training, the hyperparameter search time is greatly shortened, while ensuring the stability of the model performance.

[0023] (3) Compared with the prior art, the present invention breaks through the limitation of traditional text classification methods that rely on simple feature extraction, and realizes deep semantic understanding of power system professional texts through the designed optimized improved BERT-WWM model; in practical applications, the present invention can automatically adapt to different types of engineering documents and complete high-quality classification tasks without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0025] like Figure 1 As shown, the present invention provides a technical solution: a text prompt classification method based on an improved BERT-WWM, comprising the following steps:

[0026] Step S1: Collect power infrastructure text data for preprocessing and construct a data set. The power infrastructure text data consists of sentences of different lengths.

[0027] The collected power infrastructure text data may come from reports, documents or other records, and contain a lot of unstructured information, so preprocessing operations are required. The preprocessing includes:

[0028] Text cleaning: Use regular expressions to clean the power infrastructure text data, remove useless characters, spaces, special symbols, etc. in the power infrastructure text data, and improve data quality.

[0029] Information extraction: extract key information from power infrastructure text data to ensure that the data retains meaningful content.

[0030] Text vectorization: vectorize the cleaned power infrastructure text data, for example, by converting the text into processable numerical data through word embedding (such as Word2Vec, TF-IDF or BERT vectorization).

[0031] Step S2: Build a BERT-WWM model, introduce sparse attention to replace the multi-head self-attention in the BERT-WWM model, and adaptively adjust the mask ratio in the sparse attention to obtain an improved BERT-WWM model.

[0032] Among them, sparse attention reduces the amount of computation by limiting the number of matches between queries and keys, which can be expressed as: (1); In the formula, Represents query, key and value respectively; represents the dimension of the key vector; express The transpose of Represents the mask matrix, which is used to preserve the calculation within the local window; represents element-wise product; express function; Indicates sparse attention.

[0033] Among them, the mask ratio in sparse attention is adaptively adjusted, expressed as: (2); In the formula, Indicates that the sentence length in the power infrastructure text data is The mask ratio when and is an adjustable parameter for adaptive adjustment of sentence length; in this way, the improved BERT-WWM model can more flexibly adjust the ratio of the mask according to the length or complexity of the sentence, thereby better capturing the semantic relationship in the sentence.

[0034] Step S3: using the data set to train the improved BERT-WWM model, and optimizing the parameters of the improved BERT-WWM model to obtain an optimized improved BERT-WWM model;

[0035] Step S3.1: Divide the dataset into a training set and a validation set in proportion, use the training set to train the improved BERT-WWM model, and use the validation set to validate the improved BERT-WWM model.

[0036] Step S3.2: Combine Dropout regularization and DropConnect regularization to regularize the parameters of the improved BERT-WWM model.

[0037] Step S3.21: Joint probability initialization; In order to take advantage of both Dropout regularization and DropConnect regularization, the weight factor The dropout probability of Dropout regularization and DropConnect regularization is weighted combined to obtain the joint regularization probability , expressed as: (3); In the formula, and They represent the dropout probabilities of Dropout regularization and DropConnect regularization respectively.

[0038] Step S3.22: Adaptive probability adjustment; In order to better control the drop probability of each layer of the improved BERT-WWM model, the drop probability is dynamically adjusted according to the gradient information of each layer of the improved BERT-WWM model, which is expressed as: (4); In the formula, is the basic drop probability, To improve the BERT-WWM model The gradient of the layer weights, To improve the maximum value of all layer gradients of the BERT-WWM model; Indicates the improved BERT-WWM model Dropout regularization dropout probability of the layer.

[0039] Step S3.23: regularization strength calculation; The regularization strength is dynamically adjusted according to the depth of the improved BERT-WWM model layer and the current training round, expressed as: (5); In the formula, Indicates that in improving the BERT-WWM model Layers, training rounds The regularization strength when , which controls the influence of the regularization term in the improved BERT-WWM model, is usually used to prevent overfitting; To improve the total number of layers of the BERT-WWM model; is the total number of training rounds; It is a preset initial regularization strength, which is used to control the basic value of the regularization strength and determines the degree of regularization of the improved BERT-WWM model at the beginning; It is a hyperparameter that controls the decay rate, which determines how the regularization strength decreases as the number of layers of the improved BERT-WWM model increases; Indicates the current training round.

[0040] Step S3.24: Design a gradient update rule for the improved BERT-WWM model; In the case of joint regularization probability, the gradient update formula of the improved BERT-WWM model is: (6); In the formula, To improve the BERT-WWM model Layer weights; is the learning rate; is the loss function For The partial derivative of the layer weights, indicating the direction of weight update; and They are the mask matrices for Dropout regularization and DropConnect regularization, respectively.

[0041] Step S3.25: Adaptive adjustment mechanism; Based on the improved performance of the BERT-WWM model on the validation set, dynamically adjust and : (7); (8); In the formula, Indicates the change in validation set loss; is the adjustment factor; is the loss of the current validation set; for During training ; For the During training ; for During training ; for During training .

[0042] Step S3.26: Regularization effect evaluation; Finally, calculate the output of the regularized improved BERT-WWM model: (9); In the formula, is the activation function, represents an element-wise multiplication operation, To improve the BERT-WWM model The output of the layer; To improve the BERT-WWM model The output of the layer.

[0043] Combining Dropout regularization and DropConnect regularization can further enhance the robustness of the model. Dropout regularization is used to discard neurons, while DropConnect regularization is used to discard weight connections. Through this combination strategy, the sparsity of the network can be increased, so that the model has better generalization ability. This combination usually performs well in large-scale deep neural networks, especially pre-trained models like BERT.

[0044] Step S3.3: In large-scale deep learning models, especially in large-scale pre-trained models like BERT, conventional optimizers such as Adam or SGD are prone to gradient explosion or gradient vanishing problems during training, especially when the parameters of deep neural networks are updated unevenly; LAMB optimizer (Layer-wise AdaptiveMoments optimizer for BERT) is used as the optimizer for the improved BERT-WWM model; LAMB optimizer effectively solves this problem by introducing a layer-wise adaptive adjustment strategy to adaptively adjust the learning rate of each layer according to its characteristics during the training process of each layer.

[0045] The LAMB optimizer is represented as: (10); In the formula, Represents the hyperparameters of the improved BERT-WWM model, usually the weights or biases of the improved BERT-WWM model; and Respectively represent the estimated values ​​of the mean and variance of the improved BERT-WWM model gradient, is a small constant to prevent division by zero errors; Represents layer-level correction; the LAMB optimizer is a specific adjustment for each layer, making the updates of different layers more reasonable.

[0046] Step S3.4: Traditional hyperparameter optimization methods such as grid search and random search usually have huge computational overhead and are not efficient enough when facing complex high-dimensional hyperparameter spaces. The LSTM-based meta-learning model (Meta-Learner LSTM) is used to optimize the hyperparameters of the improved BERT-WWM model training. Specifically:

[0047] Step S3.41: Historical task data collection: Set the first The training task is ,Record The hyperparameters and improved BERT-WWM model completed the Performance data after training tasks .

[0048] Step S3.42: Generate hyperparameter adjustment strategy: learn the hyperparameters and performance data of historical training tasks, and generate the hyperparameter selection of the current training task, expressed as: (11); (12); In the formula, Indicates that The network generated Hyperparameter selection for each training task; Indicates that The network generated Hyperparameter selection for a training task.

[0049] The process of network generation hyperparameter selection is expressed as: (13); (14); In the formula, represents the hyperparameter selection strategy; and is the weight matrix of hyperparameters and performance data, and is the activation function of the LSTM network (for example, tanh function or sigmoid function); express The weight matrix in the network is used to select the hyperparameter strategy Weighted, and the weighted information is passed to the activation function ; and is the bias term, Used to control the hyperparameter selection strategy The activation value of , i.e., it directly affects the generation process of hyperparameter selection; It is the final adjustment item of the LSTM network output hyperparameter selection strategy, which directly affects the final output of the LSTM network. It changes the activation value of the LSTM network output so that the LSTM network can generate suitable hyperparameters according to the needs of the current training task.

[0050] Step S3.43: Hyperparameter update: According to , train the improved BERT-WWM model in the current training task, and introduce an adaptive mechanism to adjust the hyperparameters according to the performance data of the improved BERT-WWM model.

[0051] Minimize the loss function of the training task by adjusting the hyperparameters , that is, selecting a set of optimal hyperparameter configurations in each training task to achieve the best improved BERT-WWM model performance; the loss function of the training task It is expressed as: (15); In the formula, Indicates that the improved BERT-WWM model is The output of the following; It is a training task The corresponding loss.

[0052] The overall goal of the LSTM-based meta-learning model (Meta-LearnerLSTM) is to optimize the loss function of all training tasks , find the optimal hyperparameter configuration, expressed as: (16); In the formula, is the total number of training tasks.

[0053] Among them, during the training process of the improved BERT-WWM model, the learning rate can be adjusted according to the training progress of the improved BERT-WWM model (loss function By introducing real-time feedback information of the training task (improving the accuracy change or loss change of the BERT-WWM model), the selection of hyperparameters can be dynamically adjusted to make the hyperparameters more adaptable to the current state of the task, expressed as: (17); In the formula, is the current learning rate; is the adjustment factor; yes The amount of change; is the current loss value.

[0054] Minimize the sum of the loss functions of all training tasks and obtain the hyperparameters of the improved BERT-WWM model after optimization of the LSTM-based meta-learning model , expressed as: (18); In the formula, Represents the hyperparameters of minimizing the improved BERT-WWM model .

[0055] Through the LSTM-based meta-learning model, the automatic optimization of hyperparameters can be effectively achieved, so that each new training task can quickly select the most appropriate hyperparameter configuration. This not only reduces the computational overhead of hyperparameter search, but also reduces manual intervention through automation, thereby improving optimization efficiency and model performance.

[0056] Step S4: Input the power infrastructure text data to be classified into the optimized improved BERT-WWM model to obtain the classification result.

[0057] In order to prove the effectiveness of the improved BERT-WWM model after optimization in the present invention, an experimental comparison is carried out between it and the traditional support vector machine (SVM), Naive Bayes classifier and BERT model on standard test sets (such as AG News dataset, 20 Newsgroups dataset). The comparison results are shown in Table 1. Table 1 Comparison of different model experiments As shown in the results in Table 1, the classification accuracy of the improved BERT-WWM model optimized by the present invention reaches 95%, which is significantly higher than that used in traditional methods such as support vector machine (SVM) and Naive Bayes classifier, proving the advantage of the improved BERT-WWM model optimized by the present invention in understanding and classifying complex texts; the F1 score of the improved BERT-WWM model optimized by the present invention is also better than other models, which shows that the improved BERT-WWM model optimized by the present invention can better balance precision and recall when processing complex and rare categories; the support vector machine (SVM) and Naive Bayes classifier are far superior to the improved BERT-WWM model optimized by the present invention in processing speed, but the processing speed of the improved BERT-WWM model optimized by the present invention is still significantly higher than that of the BERT model; based on this result, the true effectiveness of the improved BERT-WWM model optimized by the present invention is proved, and its competitiveness in text classification tasks is demonstrated.

[0058] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text prompt classification method based on improved BERT-WWM, characterized in that: The steps include: Step S1: collecting power infrastructure text data for preprocessing and constructing a data set, where the power infrastructure text data consists of sentences of different lengths; Step S2: Build a BERT-WWM model, introduce sparse attention to replace the multi-head self-attention in the BERT-WWM model, and adaptively adjust the mask ratio in the sparse attention to obtain an improved BERT-WWM model; Step S3: using the data set to train the improved BERT-WWM model, and optimizing the parameters of the improved BERT-WWM model to obtain an optimized improved BERT-WWM model; Step S4: input the power infrastructure text data to be classified into the optimized improved BERT-WWM model to obtain the classification result; The specific process of optimizing and improving the parameters of the BERT-WWM model is as follows: Combine Dropout regularization and DropConnect regularization to regularize the parameters of the improved BERT-WWM model; The LAMB optimizer is used as the optimizer for improving the BERT-WWM model; An LSTM-based meta-learning model is used to optimize the hyperparameters during training of the improved BERT-WWM model.

2. The text prompting classification method based on improved BERT-WWM according to claim 1, characterized in that: Adaptively adjust the mask ratio in sparse attention, expressed as: ； In the formula, Indicates that the sentence length in the power infrastructure text data is The mask ratio when and It is an adjustable parameter.

3. The text prompting classification method based on improved BERT-WWM according to claim 2, characterized in that: The specific process of using the dataset to train the improved BERT-WWM model is: divide the dataset into a training set and a validation set in proportion, use the training set to train the improved BERT-WWM model, and use the validation set to validate the improved BERT-WWM model.

4. The text prompting classification method based on improved BERT-WWM according to claim 3, characterized in that: The specific process of combining Dropout regularization and DropConnect regularization to regularize the parameters of the improved BERT-WWM model is as follows: By weight factor The dropout probability of Dropout regularization and DropConnect regularization is weighted combined to obtain the joint regularization probability , expressed as: ； In the formula, and Represent the dropout probability of Dropout regularization and DropConnect regularization respectively; The drop probability is dynamically adjusted according to the gradient information of each layer of the improved BERT-WWM model, expressed as: ； In the formula, is the basic drop probability, To improve the BERT-WWM model The gradient of the layer weights, To improve the maximum value of all layer gradients of the BERT-WWM model; Indicates the improved BERT-WWM model The dropout regularization dropout probability of the layer; Calculate the regularization strength, expressed as: ； In the formula, Indicates that in improving the BERT-WWM model Layers, training rounds The regularization strength when ; To improve the total number of layers of the BERT-WWM model; is the total number of training rounds; is a preset initial regularization strength; is a hyperparameter that controls the decay rate; Indicates the current training round; Design an improved gradient update rule for the BERT-WWM model; in the case of joint regularization probability, the gradient update formula for the improved BERT-WWM model is: ； In the formula, To improve the BERT-WWM model Layer weights; is the learning rate; is the loss function For Partial derivatives of layer weights; and They are the mask matrices for Dropout regularization and DropConnect regularization respectively; Based on the improved performance of the BERT-WWM model on the validation set, dynamically adjust and : ； ； In the formula, Indicates the change in validation set loss; is the adjustment factor; is the loss of the current validation set; for During training ; For the During training ; for During training ; for During training .

5. The text prompting classification method based on improved BERT-WWM according to claim 4, characterized in that: The specific process of optimizing the hyperparameters of the improved BERT-WWM model training using the LSTM-based meta-learning model is as follows: Set the improved BERT-WWM model on the dataset The training task is ,Record The hyperparameters and improved BERT-WWM model completed the Performance data after training tasks ; Learn the hyperparameters and performance data of historical training tasks and generate the hyperparameter selection of the current training task, expressed as: ； ； In the formula, Indicates that The network generated Hyperparameter selection for each training task; Indicates that The network generated Hyperparameter selection for each training task; The process of network generation hyperparameter selection is expressed as: ； ； In the formula, represents the hyperparameter selection strategy; and is the weight matrix of hyperparameters and performance data, and is the activation function of the LSTM network; express The weight matrix in the network; and is the bias term; according to , train the improved BERT-WWM model in the current training task, and introduce an adaptive mechanism to adjust the hyperparameters according to the performance data of the improved BERT-WWM model.

6. The text prompting classification method based on improved BERT-WWM according to claim 5, characterized in that: according to , train the improved BERT-WWM model in the current training task, and introduce an adaptive mechanism to adjust the hyperparameters according to the performance data of the improved BERT-WWM model. The specific process is as follows: Minimize the loss function of the training task by adjusting the hyperparameters , that is, selecting a set of optimal hyperparameter configurations in each training task to achieve the best improved BERT-WWM model performance; the loss function of the training task It is expressed as: ； In the formula, Indicates that the improved BERT-WWM model is The output of the following; It is a training task corresponding losses; Optimize the loss function for all training tasks , find the optimal hyperparameter configuration, expressed as: ； In the formula, is the total number of training tasks; Introducing real-time feedback information of training tasks improves the accuracy change or loss change of the BERT-WWM model and dynamically adjusts the selection of hyperparameters, which can be expressed as: ； In the formula, is the current learning rate; is the adjustment factor; yes The amount of change; is the current loss value; Minimize the sum of the loss functions of all training tasks and obtain the hyperparameters of the improved BERT-WWM model after optimization of the LSTM-based meta-learning model , expressed as: ； In the formula, Represents the hyperparameters of minimizing the improved BERT-WWM model .

7. The text prompting classification method based on improved BERT-WWM according to claim 6, characterized in that: The LAMB optimizer is used as the optimizer for the improved BERT-WWM model, expressed as: ； In the formula, and Respectively represent the estimated values ​​of the mean and variance of the improved BERT-WWM model gradient; is a constant; Indicates level correction.

8. The text prompting classification method based on improved BERT-WWM according to claim 7, characterized in that: Collect power infrastructure text data for preprocessing, including: Text cleaning, using regular expressions to clean power infrastructure text data; Information extraction, extracting key information from text data of power infrastructure; Text vectorization: vectorize the power infrastructure text data after text cleaning.

Images