A text classification method based on a layer-by-layer thawing progressive training model

Abstract

The application discloses a text classification method based on a layer-by-layer unfreezing progressive training model, and relates to the technical field of text classification. The method comprises the following steps: obtaining news text data to be classified; inputting the news text data to be classified into a layer-by-layer unfreezing progressive training model to obtain a classification result; the layer-by-layer unfreezing progressive training model is trained by using a sequential progressive fine-tuning strategy and / or a combined progressive fine-tuning strategy; and the layer-by-layer unfreezing progressive training model comprises an embedding layer, an attention layer, a feedforward layer, a hidden layer and a dense layer. The model is trained by using at least one training strategy, i.e., the sequential progressive fine-tuning strategy and / or the combined progressive fine-tuning strategy, so that the potential of the model itself can be tapped, and the purpose of obtaining a model with optimal training effect is achieved. Therefore, accurate classification processing of the news text data to be classified can be realized.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, and in particular to a text classification method based on a progressively unfreezing training model. Background Technology

[0002] Currently, with the development of model research, the number of network layers and parameters in models is gradually increasing, and the complexity of the model network is also increasing, making it difficult to directly train the model to achieve optimal performance. Therefore, it is difficult to achieve sufficiently accurate results in the classification of textual data. Summary of the Invention

[0003] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically by providing a text classification method based on a layer-by-layer unfreezing progressive training model, as detailed below:

[0004] 1) In a first aspect, the present invention provides a text classification method based on a layer-by-layer unfreezing progressive training model, the specific technical solution of which is as follows:

[0005] Obtain the news text data to be classified, input the news text data to be classified into the progressively unfrozen training model, and obtain the classification result;

[0006] The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies.

[0007] The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer.

[0008] The beneficial effects of the text classification method based on a layer-by-layer unfreezing progressive training model provided by this invention are as follows:

[0009] This solution employs at least one training strategy, namely a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies, to train the model and unlock its potential, thereby achieving the goal of training a model with optimal performance. Based on this, accurate classification processing of news text data to be classified can be realized.

[0010] Based on the above solution, the present invention can be further improved as follows.

[0011] Furthermore, the calculation process of the embedding layer is as follows:

[0012] y1=x*W1 T +b1;

[0013] Where x is the input data of the embedding layer, W1 is the weight matrix of the embedding layer, b1 is the bias matrix of the embedding layer, and y1 is the output result of the embedding layer.

[0014] Furthermore, the calculation process of the attention layer is as follows:

[0015]

[0016] rms is the standard deviation of the input data in the embedding layer, and y2 is the output of the attention layer.

[0017] Furthermore, the calculation process of the feedforward layer is as follows:

[0018] y3=(y2*W2 T +b2)*W3 T +b3;

[0019] Where W2 and W3 are the two weight matrices of the feedforward layer, T represents the transpose operation of the matrix, b2 and b3 are the two bias matrices of the feedforward layer, and y3 is the output of the feedforward layer.

[0020] Furthermore, the calculation process of the hidden layer is as follows:

[0021] y4=y3*W4 T +b4;

[0022] Where W4 is the weight matrix of the hidden layer, b4 is the bias matrix of the hidden layer, and y4 is the output of the hidden layer.

[0023] Furthermore, the calculation process for the dense layer is as follows:

[0024] y5=y4*W5 T +b5;

[0025] Where W5 is the weight matrix of the dense layer, b5 is the bias matrix of the dense layer, and y5 is the output of the dense layer.

[0026] 2) Secondly, the present invention also provides a text classification system based on a layer-by-layer unfreezing progressive training model, the specific technical solution of which is as follows:

[0027] The classification module is used to: acquire news text data to be classified, input the news text data to be classified into a progressively unfrozen training model, and obtain the classification result;

[0028] The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies.

[0029] The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer.

[0030] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to perform any of the above methods.

[0031] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0032] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0033] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0034] Figure 1 This is a flowchart illustrating a text classification method based on a progressively unfreezing training model according to an embodiment of the present invention.

[0035] Figure 2 This is a structural framework diagram of an electronic device according to the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0037] like Figure 1 As shown, an embodiment of the present invention provides a text classification method based on a layer-by-layer unfreezing progressive training model, comprising the following steps:

[0038] S1, Obtain the news text data to be classified, input the news text data to be classified into the progressive training model of layer-by-layer unfreezing, and obtain the classification result;

[0039] The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies.

[0040] The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer.

[0041] The beneficial effects of this invention are as follows:

[0042] This solution employs at least one training strategy, namely a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies, to train the model and unlock its potential, thereby achieving the goal of training a model with optimal performance. Based on this, accurate classification processing of news text data to be classified can be realized.

[0043] Unclassified news text data refers to news content recorded in text form, typically including information such as headlines, body text, date, and location. This data can be obtained through various channels, such as news websites, social media, and news applications.

[0044] Methods for obtaining news text data:

[0045] Web crawling: Using web crawling tools such as BeautifulSoup or Scrapy libraries in Python, you can find and download news text content from web pages.

[0046] API: Many news websites and services provide API interfaces, through which specific news text data can be obtained.

[0047] Open datasets: Some organizations and institutions provide datasets containing news text data, which can be found through internet searches.

[0048] Database query: If you have permission to access the database, you can write SQL query statements to extract news text data.

[0049] User input: Obtain news text data input by users through interaction with them.

[0050] Manual collection: When there are no other means, news text data can be manually collected, organized and saved from different sources.

[0051] The categorization results can be in various categories such as finance, lottery, real estate, stocks, home furnishings, education, technology, society, fashion, current affairs, sports, horoscope, games, and entertainment.

[0052] Based on the above solution, the present invention can be further improved as follows.

[0053] Furthermore, the calculation process of the embedding layer is as follows:

[0054] y1=x*W1 T +b1;

[0055] Where x is the input data of the embedding layer, W1 is the weight matrix of the embedding layer, b1 is the bias matrix of the embedding layer, and y1 is the output result of the embedding layer.

[0056] Furthermore, the calculation process of the attention layer is as follows:

[0057]

[0058] rms is the standard deviation of the input data in the embedding layer, and y2 is the output of the attention layer.

[0059] Furthermore, the calculation process of the feedforward layer is as follows:

[0060] y3=(y2*W2 T +b2)*W3 T +b3;

[0061] Where W2 and W3 are the two weight matrices of the feedforward layer, T represents the transpose operation of the matrix, b2 and b3 are the two bias matrices of the feedforward layer, and y3 is the output of the feedforward layer.

[0062] Furthermore, the calculation process of the hidden layer is as follows:

[0063] y4=y3*W4 T +b4;

[0064] Where W4 is the weight matrix of the hidden layer, b4 is the bias matrix of the hidden layer, and y4 is the output of the hidden layer.

[0065] Furthermore, the calculation process for the dense layer is as follows:

[0066] y5=y4*W5 T +b5;

[0067] Where W5 is the weight matrix of the dense layer, b5 is the bias matrix of the dense layer, and y5 is the output of the dense layer.

[0068] Example 1: The current architecture is divided into two parts: the model part and the evaluation architecture. The model part is divided into three main parts: pretrain_model, hidden_layer, and dense_layer. The pretrain_model further includes three parts: embedding_layer, attention_layer, and feedforward_layer.

[0069] Specifically, the embedding_layer, attention_layer, and feedforward_layer perform text-to-quantization, attention weight calculation, and non-linear mapping of the attention weight calculation results, respectively. The hidden_layer performs further feature extraction calculations on the results calculated by the pretrain_model, and the dense_layer maps the results of the hidden_layer to the true results.

[0070] The specific calculations for the model layer are as follows:

[0071] pretrain_model:

[0072] embedding_layer:

[0073] y1=x*W1 T +b1

[0074] Where x is the input data of the current layer, W1 is the weight matrix of the embedding_layer layer, and b1 is the bias matrix of the embedding_layer layer.

[0075] attention_layer:

[0076]

[0077] Where: rms is the standard deviation of the input data of the embedding_layer layer.

[0078] feedforward_layer:

[0079] y3=(y2*W2 T +b2)*W3 T +b3

[0080] Where W2 and W3 are the two weight matrices of the feedforward_layer layer, T represents the transpose of the matrix, and b2 and b3 are the two bias matrices of the feedforward_layer layer.

[0081] hidden_layer:

[0082] y4=y3*W4 T +b4

[0083] Where y3 is the calculation result of the feedforward_layer layer, W4 is the weight matrix of the hidden_layer layer, and b4 is the bias matrix of the hidden_layer layer.

[0084] dense_layer:

[0085] y5=y4*W5 T +b5

[0086] Where y4 is the calculation result of the hidden_layer layer, W5 is the weight matrix of the dense_layer layer, and b5 is the bias matrix of the dense_layer layer.

[0087] evaluate_part:

[0088] This is the evaluation module, used to call the intermediate trained and saved model to verify the specified test set and statistically analyze the model metrics.

[0089] Two strategies were employed when training the model: 1) sequentially fine-tuning the model; 2) combining progressive fine-tuning of the model.

[0090] For strategy 1) sequential fine-tuning model:

[0091] First, train the `embedding_layer` by freezing the weights of all other model layers (initial weights) and updating only the parameters of `embedding_layer`. After training, obtain the weights of this model layer. Then, call the `evaluate_part` module (which stores test data) to evaluate the accuracy (acc) of the currently trained model using pre-constructed test data. Then record and save the acc;

[0092] After the training and statistics are completed, the next training of the model begins. During this training, the embedding_layer and attention_layer are trained, the weights of all other model layers are frozen, and only the parameters of the embedding_layer and attention_layer are updated. After training, a model weight is obtained. Then, the evaluate_part module is called to evaluate the accuracy (acc) of the currently trained model using pre-constructed test data.

[0093] After completing the statistics, the embedding_layer, attention_layer and feedforward_layer layers are trained again, and the training method is the same as above.

[0094] This process is applied sequentially to all other model layers, progressively increasing the number of model layers trained while simultaneously freezing the other model layers. At the same time, the evaluation metrics of the model are recorded after each training session, and the training method with the highest evaluation metric is selected as the optimal model training method to improve the model's performance in the current task.

[0095] In another embodiment, after training the embedding_layer, when training both the embedding_layer and attention_layer, the updated parameters described above can be used for subsequent processing. Alternatively, the updated parameters can be omitted during the training of the embedding_layer and attention_layer, meaning both layers can be trained simultaneously to obtain their respective model layer weights. Then, the evaluate_part module is called to evaluate the accuracy (acc) of the currently trained model using pre-constructed test data.

[0096] By comparing all accuracies (acc), the optimal acc (i.e. the one with the largest acc value) is determined, and the weights and parameters corresponding to this acc are used as the optimal parameters of the model, thus obtaining the optimal model.

[0097] It should be further noted that the parameters of the untrained model layers are all basic parameters.

[0098] For strategy 2) combined progressive fine-tuning model

[0099] For the existing model layers embedding_layer, attention_layer, feedforward_layer, hidden_layer, and dense_layer, iterate through all combinations between model layers, such as [embedding_layer, attention_layer], [embedding_layer, feedforward_layer], [embedding_layer, hidden_layer], etc. Train the weights of the model layers in each combination pair in turn, and freeze the weights of other model layers. After each training, evaluate the model's performance and record the accuracy value. After this iteration, select the model training method with the highest evaluation index as the optimal model training method to improve the model's performance on the current task.

[0100] In another embodiment of this scheme, the combination pair may also include three or four layers, that is, three or four layers can be trained simultaneously, and the accuracy acc of all combination pairs is recorded and compared to determine the optimal parameters.

[0101] Furthermore, it should be noted that if two strategies are used for model training, the training results of the same layers corresponding to the two strategies need to be weighted before updating the parameters. The specific process is as follows:

[0102] In Strategy 1), the target model layer corresponding to the second training iteration, along with the obtained target parameters and target accuracy (acc), are used as the first pair. Based on the target model layer in the first pair, Strategy 2) searches for target combination pairs corresponding to the model layer names (i.e., using the model layer names and number in the target model layer, searching for target combination pairs that match that number and model layer name). The target parameter results and target accuracy (acc) of the first pair and the target combination pairs are then weighted and summed to obtain the weighted parameter result and weighted accuracy. This weighted accuracy is used as the training result for that number and that model layer name. This significantly reduces the workload when comparing accuracy and also merges data of the same category, improving processing accuracy.

[0103] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0104] This invention also provides a text classification system based on a layer-by-layer unfreezing progressive training model, the specific technical solution of which is as follows:

[0105] The classification module is used to: acquire news text data to be classified, input the news text data to be classified into a progressively unfrozen training model, and obtain the classification result;

[0106] The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies.

[0107] The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer.

[0108] It should be noted that the beneficial effects of the text classification system based on a layer-by-layer unfreezing progressive training model provided in the above embodiments are the same as those of the text classification method based on a layer-by-layer unfreezing progressive training model described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0109] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330, which is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above-mentioned methods. Specifically:

[0110] The electronic device 300 can vary considerably due to differences in configuration or performance. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. The memories 310 store at least one computer program 330, which is loaded and executed by the processors 320 to enable the electronic device 300 to implement the text classification method based on a layer-by-layer unfreezing progressive training model provided in the above embodiments. Of course, the electronic device 300 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. It may also include other components for implementing device functions, which will not be elaborated upon here.

[0111] An embodiment of the present invention provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described methods.

[0112] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0113] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the methods described above.

[0114] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0115] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product in one or more computer-readable media containing computer-readable program code.

[0116] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0117] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A text classification method based on a layer-by-layer unfreezing progressive training model, characterized in that, include: Obtain the news text data to be classified, input the news text data to be classified into the progressively unfrozen training model, and obtain the classification result; The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies. The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer. The sequential fine-tuning strategy includes: The `embedding_layer` is trained by freezing the initial weights of all other model layers and updating only the parameters of `embedding_layer`. After training, the weights of this model layer are obtained. Then, the `evaluate_part` module is called to evaluate the accuracy `acc` of the trained model using pre-constructed test data, where `acc` = ... Then record and save the acc; After the training and statistics are completed, the next training of the model begins, namely training the embedding_layer and attention_layer. The weights of all other model layers are frozen, and only the parameters of the embedding_layer and attention_layer are updated. After training, a model weight is obtained. Then, the evaluate_part module is called to evaluate the accuracy acc of the currently trained model using pre-constructed test data. After completing the statistics, the embedding_layer, attention_layer and feedforward_layer layers are trained again; This process is applied sequentially to all other model layers, progressively increasing the number of model layers to be trained while simultaneously freezing the other model layers. At the same time, the evaluation metrics of the model are recorded after each training session, and the training method with the highest evaluation metric is selected as the optimal model training method. The combined progressive fine-tuning strategy is specifically as follows: For the existing model layers embedding_layer, attention_layer, feedforward_layer, hidden_layer, and dense_layer, iterate through all combinations between model layers, train the weights of the model layers in the combination pair in turn, and freeze the weights of other model layers. After each training, evaluate the model's performance, record the accuracy value, and select the model training method with the highest evaluation index as the optimal model training method.

2. The text classification method based on a layer-by-layer unfreezing progressive training model according to claim 1, characterized in that, The calculation process for the embedding layer is as follows: y1=x* ; Where x is the input data of the embedding layer, It is the weight matrix of the embedding layer. y1 is the bias matrix of the embedding layer, and y1 is the output of the embedding layer.

3. The text classification method based on a layer-by-layer unfreezing progressive training model according to claim 2, characterized in that, The calculation process for the attention layer is as follows: y2= ； rms is the standard deviation of the input data in the embedding layer, and y2 is the output of the attention layer.

4. The text classification method based on a layer-by-layer unfreezing progressive training model according to claim 3, characterized in that, The calculation process for the feedforward layer is as follows: y3 = (y2* )* ; in, and 3 are the two weight matrices of the feedforward layer, and T represents the transpose operation of the matrix. and These are the two bias matrices of the feedforward layer, and y3 is the output of the feedforward layer.

5. The text classification method based on a layer-by-layer unfreezing progressive training model according to claim 4, characterized in that, The calculation process for the hidden layer is as follows: y4=y3* ; in, It is the weight matrix of the hidden_layer layer. y4 is the bias matrix of the hidden layer, and y4 is the output of the hidden layer.

6. The text classification method based on a layer-by-layer unfreezing progressive training model according to claim 5, characterized in that, The calculation process for the dense layer is as follows: y5=y4* ; in, It is the weight matrix of the dense_layer layer. y5 is the bias matrix of the dense layer, and y5 is the output of the dense layer.

7. A text classification system based on a layer-by-layer unfreezing progressive training model, employing the text classification method based on a layer-by-layer unfreezing progressive training model as described in claim 1, characterized in that... include: The classification module is used to: acquire news text data to be classified, input the news text data to be classified into a progressively unfrozen training model, and obtain the classification result; The progressive training model for layer-by-layer unfreezing includes training through a sequential progressive fine-tuning strategy and / or a combination of progressive fine-tuning strategies. The progressive training model with layer-by-layer unfreezing includes an embedding layer, an attention layer, a feedforward layer, a hidden layer, and a dense layer.

8. A text classification system based on a layer-by-layer unfreezing progressive training model according to claim 7, characterized in that, The calculation process for the embedding layer is as follows: y1=x* ; Where x is the input data of the embedding layer, It is the weight matrix of the embedding layer. y1 is the bias matrix of the embedding layer, and y1 is the output of the embedding layer.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to perform the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to perform the method as described in any one of claims 1 to 6.

Images