A multi-label chemical industry news classification method based on transfer learning and deep learning

Abstract

The application discloses a multi-label chemical industry news classification method based on transfer learning and deep learning, comprising the following steps: acquiring news data; segmenting the news data and adding labels to establish a data set sequence M; converting each text in the data set sequence M into a vector sequence to obtain a combined vector of each text; selecting a pre-training model and training the combined vector to obtain a text vector; constructing a multi-label chemical industry news classification model by using transfer learning to capture local features and global semantic information in the text vector; optimizing the multi-label chemical industry news classification model; and analyzing the performance of the multi-label chemical industry news classification model. The application solves the problems in the prior art that the existing news classification method cannot realize a multi-label classification task, performs poorly on a multi-label classification task, needs a large amount of text data for training, and needs a large storage space for storing word vectors.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a multi-label chemical industry news classification method based on transfer learning and deep learning. Background Technology

[0002] News classification technology can better automate the organization, management, and analysis of large amounts of news text, and is widely used in news recommendation, sentiment analysis, and event tracking. Furthermore, news classification technology can also be used to detect online misinformation and automate intelligent question answering, helping to improve the efficiency and accuracy of information processing.

[0003] In existing technologies, news classification methods mainly include Text-CNN and LSTM models, but these methods all have some problems. Text-CNN does not capture the semantic information of long texts completely and cannot perform multi-label classification tasks; while LSTM models require a large amount of text data for training and are prone to gradient explosion.

[0004] Besides Text-CNN and LSTM models, there are other existing methods for news classification. For example, word embedding-based methods map each word into a low-dimensional vector space, using these vectors to represent the semantic information of the entire text. However, this method often requires significant storage space to store word vectors and may be affected by differences between languages ​​and domains. Additionally, machine learning-based classification methods (such as Naive Bayes and Support Vector Machines) can also be used for news classification tasks, but these methods require manual feature selection and feature engineering, and perform poorly on multi-label classification tasks. Summary of the Invention

[0005] Purpose of the invention: The purpose of this invention is to provide a multi-label news classification method for the chemical industry based on transfer learning and deep learning, which is suitable for small sample news data and has high news classification accuracy.

[0006] Technical Solution: To achieve the above objectives, the present invention provides a multi-label chemical industry news classification method based on transfer learning and deep learning, comprising the following steps:

[0007] Step S1: Obtain news data;

[0008] Step S2: Segment and label the news data to create a dataset sequence M;

[0009] Step S3: Convert each text segment in the dataset sequence M into a vector sequence to obtain a combined vector of each text segment;

[0010] Step S4: Select a pre-trained model and train it using combined vectors to obtain text vectors;

[0011] Step S5: Utilize transfer learning to construct a multi-label chemical news classification model to capture local features and global semantic information in text vectors;

[0012] Step S6: Optimize the multi-label chemical industry news classification model;

[0013] Step S7: Perform performance analysis on the multi-label chemical news classification model.

[0014] The news data mentioned in step S1 refers to the collected domestic and international chemical industry news.

[0015] The segmentation and tagging described in step S2 include the following sub-steps:

[0016] Step S201: Segment all news data according to the stop word Stop_word = {' / n', '。'}, where ' / n' represents a newline character, forming n text segments;

[0017] Step S202: Add a label to each text segment, that is, compare each text segment with the keywords in the label list LABEL one by one. If the keywords match, it is recorded as 1, and if they do not match, it is recorded as 0, forming a data list of 1 or 0, which serves as the label of the text segment.

[0018] Step S203: Each obtained text segment is denoted as: Para = {id, content, label}, where id = 1, 2, ..., n, and content represents the text content;

[0019] Step S204: Obtain the dataset sequence M: M = {Para1, Para2, Para3, ..., Para...} n}

[0020] The label list LABEL is as follows:

[0021] LABEL = {'Industry News', 'Policy Interpretation', 'Company News', 'Market Analysis', 'Technological Innovation', 'Useless Information'}.

[0022] Step S3, which involves converting each text segment in the dataset sequence M into a vector sequence to obtain a combined vector for each text segment, includes the following sub-steps:

[0023] Step S301: Perform a text classification task on each text segment in the dataset sequence M, that is, add a label [CLS] before each text segment to indicate that the text is used for classification task, and add a label [SEP] at the end to separate each text segment;

[0024] Step S302: Segment each text segment into words to obtain the vector representation of each word, i.e., word vector; merge multiple groups of words into a whole vector representation to obtain segment vector; assign a vector to each segment using different sine and cosine functions to represent its relative position in the sequence to obtain position vector;

[0025] Step S303: Calculate the sum of the word vector, fragment vector and position vector of each text segment to obtain the combined vector of each text segment.

[0026] Each word segment is assigned a vector to obtain a position vector, specifically:

[0027] Let the length of a segment of text be Z, then the position vector PE(i,j) of the i-th word in this segment of text is:

[0028]

[0029] Where d is the dimension of the embedding vector, and j is each dimension, j∈(1,d).

[0030] Step S4, which involves selecting a pre-trained model and training it using combined vectors to obtain text vectors, includes the following sub-steps:

[0031] Step S401: Select the bert-base-chinese pre-trained model;

[0032] Step S402: Input the combined vector of each text segment in the dataset sequence M into the pre-trained model to train the model;

[0033] Step S403: After multiple rounds of training, until the value of the loss function no longer changes, a series of text vectors with rich semantic information are obtained. This vector is a high-dimensional vector, where each dimension represents different semantic information.

[0034] Step S5, which involves constructing a multi-label chemical news classification model, refers to adding convolutional and pooling layers to a CNN neural network model to capture local features and global semantic information in text vectors. This includes the following sub-steps:

[0035] Step S501: Use the text vector as the input feature vector with dimensions [batch_size, sequence_length, hidden_size], where batch_size is the batch size, sequence_length is the sequence length, and hidden_size is the dimension of the text vector.

[0036] Step S502: Add three convolutional layers to the CNN neural network model, namely Conv1, Conv2, and Conv3, to perform convolution operations on the text vectors and extract local features. The output dimension of the convolutional layer is [batch_size, sequence_length - filter_size + 1, num_filters], where filter_size is the size of the convolutional kernel and num_filters is the number of convolutional kernels.

[0037] Step S503: Add a pooling layer to the CNN neural network model, perform pooling operation on the output of each convolutional kernel, extract the maximum value of the convolutional layer output, and further compress the feature dimension;

[0038] Step S504: Repeat steps S501 to S503 above until all text vector feature information and global semantic information are extracted.

[0039] The optimization of the multi-label chemical news classification model described in S6 includes the following sub-steps:

[0040] Step S601: Flatten the three-dimensional feature matrix output by the pooling layer and convert it into a two-dimensional feature vector to prepare for the subsequent fully connected layer.

[0041] Step S602: By using the ReLU activation function, the neural network can better perform nonlinear fitting. The fitting formula is: f(x) = max(0,x), that is, when the signal is less than 0, the output is 0, and when it is greater than 0, the output is the signal itself. At the same time, Dropout is used to prevent the neural network from overfitting and improve the model's generalization ability.

[0042] Step S603: Use the Adam algorithm to adaptively adjust the learning rate on different dimensions of each parameter to adapt to the characteristics of different parameters, thereby improving the training effect of the model;

[0043] Step S604: Add an output layer, typically using the Sigmoid function as the activation function, to perform binary classification on each label and obtain the predicted probability for each label. The Sigmoid function is:

[0044]

[0045] The performance analysis of the multi-label chemical news classification model described in step S7 uses the following evaluation metrics:

[0046] Accuracy:

[0047] This represents the proportion of samples that are actually positive out of all samples predicted as positive by the multi-label chemical news classification model; where TP represents the number of correctly classified positive examples, i.e., the number of instances that are actually positive and are classified as positive by the multi-label chemical news classification model, and FP represents the number of incorrectly classified positive examples, i.e., the number of instances that are actually negative but are classified as positive by the multi-label chemical news classification model.

[0048] Recall rate:

[0049] This represents the proportion of samples that are actually positive but were predicted as positive by the multi-label chemical news classification model out of all samples that are actually positive; where FN represents the number of instances that were incorrectly classified as negative, i.e., the number of instances that were actually positive but were classified as negative by the multi-label chemical news classification model.

[0050] The harmonic mean F1 score of precision and recall:

[0051]

[0052] This indicates the combined performance of the multi-label chemical news classification model based on precision and recall.

[0053] Beneficial effects: The present invention has the following advantages: 1. The classification method adopted in the present invention is based on the bert-base-chinese network structure as a pre-trained model. It takes advantage of the network structure in pre-training on text data, that is, the learned general language representation can be used in various specific natural language processing tasks without repeated training. The network structure is transferred to the classification of chemical news to obtain a series of text vectors to make up for the lack of text data used for training in the chemical field.

[0054] 2. This method constructs a multi-label chemical news classification model by adding convolutional and pooling layers to the CNN neural network model, so as to capture local features and global semantic information in the text vector, which can better improve the accuracy of text classification.

[0055] 3. This method achieves more reasonable news classification by segmenting the initial news text data and adding labels. Compared with other text classification models such as CNN, LSTM, and BERT, the multi-label chemical news classification constructed in this invention has significant improvements in accuracy, recall, and F1 score. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0057] Figure 2 This is a schematic diagram illustrating the process of segmenting and tagging news data according to the present invention;

[0058] Figure 3 This is a schematic diagram illustrating how the present invention converts text into a vector sequence;

[0059] Figure 4 This is the label classification standard for the present invention. Detailed Implementation

[0060] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0061] like Figure 1 As shown, the present invention provides a multi-label chemical industry news classification method based on transfer learning and deep learning, comprising the following steps:

[0062] Step S1: Obtain news data: refers to the collection of domestic and international chemical industry news.

[0063] Step S2: As Figure 2 As shown, segmenting and labeling the news data to create a dataset sequence M involves the following sub-steps:

[0064] Step S201: Segment all news data according to the stop word Stop_word = {' / n', '。'}, where ' / n' represents a newline character, forming n text segments;

[0065] Step S202: Add a label to each text segment, that is, compare each text segment with the keywords in the label list LABEL one by one. If the keywords match, it is recorded as 1, and if they do not match, it is recorded as 0, forming a data list of 1 or 0, which serves as the label of the text segment.

[0066] Step S203: Each obtained text segment is denoted as: Para = {id, content, label}, where id = 1, 2, ..., n, and content represents the text content;

[0067] Step S204: Obtain the dataset sequence M: M = {Para1, Para2, Para3, ..., Para...} n}

[0068] The label list LABEL is as follows:

[0069] LABEL = {'Industry News', 'Policy Interpretation', 'Company News', 'Market Analysis', 'Technological Innovation', 'Useless Information'}.

[0070] Step S3: As Figure 3 As shown, the process of converting each text segment in the dataset sequence M into a vector sequence to obtain a combined vector for each text segment includes the following sub-steps:

[0071] Step S301: Perform a text classification task on each text segment in the dataset sequence M, that is, add a label [CLS] before each text segment to indicate that the text is used for classification task, and add a label [SEP] at the end to separate each text segment;

[0072] Step S302: Segment each text segment into words to obtain the vector representation of each word, i.e., word vector; merge multiple groups of words into a whole vector representation to obtain segment vector; assign a vector to each segment using different sine and cosine functions to represent its relative position in the sequence to obtain position vector;

[0073] Step S303: Calculate the sum of the word vector, fragment vector and position vector of each text segment to obtain the combined vector of each text segment.

[0074] Each word segment is assigned a vector to obtain a position vector, specifically:

[0075] Let the length of a segment of text be Z, then the position vector PE(i,j) of the i-th word in this segment of text is:

[0076]

[0077] Where d is the dimension of the embedding vector, and j is each dimension, j∈(1,d).

[0078] Step S4: Select a pre-trained model and train it using combined vectors to obtain text vectors, including the following sub-steps:

[0079] Step S401: Select the bert-base-chinese pre-trained model;

[0080] Step S402: Input the combined vector of each text segment in the dataset sequence M into the pre-trained model to train the model;

[0081] Step S403: After multiple rounds of training, until the value of the loss function no longer changes, a series of text vectors with rich semantic information are obtained. This vector is a high-dimensional vector, where each dimension represents different semantic information.

[0082] Step S5: Constructing a multi-label chemical news classification model using transfer learning to capture local features and global semantic information in text vectors. This involves adding convolutional and pooling layers to a CNN neural network model to construct a multi-label chemical news classification model, which captures local features and global semantic information in text vectors. This includes the following sub-steps:

[0083] Step S501: Use the text vector as the input feature vector with dimensions [batch_size, sequence_length, hidden_size], where batch_size is the batch size, sequence_length is the sequence length, and hidden_size is the dimension of the text vector.

[0084] Step S502: Add three convolutional layers to the CNN neural network model, namely Conv1, Conv2, and Conv3, to perform convolution operations on the text vectors and extract local features. The output dimension of the convolutional layer is [batch_size, sequence_length - filter_size + 1, num_filters], where filter_size is the size of the convolution kernel. In this embodiment, filter_size is 2, 3, and 4 respectively, and num_filters is the number of convolution kernels. In this embodiment, num_filters is 512.

[0085] Step S503: Add a pooling layer to the CNN neural network model, perform pooling operation on the output of each convolutional kernel, extract the maximum value of the convolutional layer output, and further compress the feature dimension;

[0086] Step S504: Repeat steps S501 to S503 above until all text vector feature information and global semantic information are extracted.

[0087] Step S6: Optimize the multi-label chemical industry news classification model, including the following sub-steps:

[0088] Step S601: Flatten the three-dimensional feature matrix output by the pooling layer and convert it into a two-dimensional feature vector to prepare for the subsequent fully connected layer.

[0089] Step S602: By using the ReLU activation function, the neural network can better perform nonlinear fitting. The fitting formula is: f(x) = max(0,x), that is, when the signal is less than 0, the output is 0, and when it is greater than 0, the output is the signal itself. At the same time, Dropout is used to prevent the neural network from overfitting and improve the model's generalization ability.

[0090] Step S603: Use the Adam algorithm to adaptively adjust the learning rate on different dimensions of each parameter to adapt to the characteristics of different parameters, thereby improving the training effect of the model;

[0091] Step S604: Add an output layer, typically using the sigmoid function as the activation function, to perform binary classification on each label, such as... Figure 4As shown, this is the label classification standard of the present invention, which obtains the predicted probability of each label. The Sigmoid function is:

[0092]

[0093] Step S7: Perform performance analysis on the multi-label chemical news classification model. The evaluation metrics used are:

[0094] Accuracy:

[0095] This represents the proportion of samples that are actually positive out of all samples predicted as positive by the multi-label chemical news classification model; where TP represents the number of correctly classified positive examples, i.e., the number of instances that are actually positive and are classified as positive by the multi-label chemical news classification model, and FP represents the number of incorrectly classified positive examples, i.e., the number of instances that are actually negative but are classified as positive by the multi-label chemical news classification model.

[0096] Recall rate:

[0097] This represents the proportion of samples that are actually positive but were predicted as positive by the multi-label chemical news classification model out of all samples that are actually positive; where FN represents the number of instances that were incorrectly classified as negative, i.e., the number of instances that were actually positive but were classified as negative by the multi-label chemical news classification model.

[0098] The harmonic mean F1 score of precision and recall:

[0099]

[0100] This indicates the combined performance of the multi-label chemical news classification model based on precision and recall.

[0101] This invention provides a multi-label chemical industry news classification method based on transfer learning and deep learning. The method uses the BERT-base-Chinese network structure as a pre-trained model and leverages the advantages of this network in the field of natural language processing. That is, this network structure learns a general language representation by pre-training on large-scale text data. This general representation can be used in various specific natural language processing tasks without repeated training. The network structure is transferred to the classification of chemical industry news to obtain a large number of text vectors to make up for the lack of text data for training in the chemical industry.

[0102] Furthermore, compared to fine-tuning the bert-base-chinese network structure, this method constructs a multi-label chemical news classification model by adding convolutional and pooling layers to the CNN neural network model. This captures local features and global semantic information in the text vectors, thus improving the accuracy of text classification.

[0103] Meanwhile, by segmenting the initial news text data and adding labels, a more reasonable news classification is achieved. Compared with other text classification models such as CNN, LSTM, and BERT, the multi-label chemical news classification constructed in this invention has significant improvements in accuracy, recall, and F1 score. As shown in Table 1, the accuracy, recall, and F1 score of the CNN, LSTM, and BERT network models and the label chemical news classification model constructed in this invention are compared with those of the present invention.

[0104] Table 1 shows the accuracy, recall, and F1 score of CNN, LSTM, BERT network models and the labeled chemical industry news classification model constructed in this invention.

[0105] Model Accuracy Recall rate F1 value LSTM 0.85 0.86 0.8549 CNN 0.88 0.89 0.8849 Bert 0.9418 0.9417 0.9417 Method of the present invention 0.9467 0.9467 0.9467 .

Claims

1. A multi-label chemical field news classification method based on transfer learning and deep learning, characterized in that, It comprises the following steps: Step S1: obtaining news data; Step S2: segmenting and adding labels to the news data, and establishing a data set sequence M; Step S3: converting each text in the data set sequence M into a vector sequence to obtain a combined vector of each text; Step S4: selecting a pre-trained model and training it using the combined vector to obtain a text vector; Step S5: constructing a multi-label chemical news classification model using transfer learning to capture local features and global semantic information in the text vector; Step S6: optimizing the multi-label chemical news classification model; Step S7: analyzing the performance of the multi-label chemical news classification model; The step S5 constructing a multi-label chemical news classification model refers to adding a convolution layer and a pooling layer in the CNN neural network model to construct a multi-label chemical news classification model to capture local features and global semantic information in the text vector, which comprises the following sub-steps: Step S501: taking the text vector as an input feature vector with a dimension of [batch_size, sequence_length, hidden_size], wherein batch_size is the batch size, sequence_length is the sequence length, and hidden_size is the dimension of the text vector; Step S502: adding three convolution layers, namely Conv1, Conv2 and Conv3, in the CNN neural network model to perform convolution operation on the text vector to extract local features, and the output dimension of the convolution layer is [batch_size, sequence_length-filter_size+1, num_filters], wherein filter_size is the size of the convolution kernel, and num_filters is the number of convolution kernels; Step S503: adding a pooling layer in the CNN neural network model to perform pooling operation on the output of each convolution kernel to extract the maximum value of the convolution layer output and further compress the feature dimension; Step S504: repeating the above steps S501 to S503 until all feature information and global semantic information of the text vector are extracted. 2.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The news data of step S1 refers to the collected domestic and foreign chemical news. 3.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The segmenting and adding labels of step S2 comprise the following sub-steps: Step S201: segment all news data by stop words, wherein a newline character is formed to form a paragraph of text; Step S202: adding a label label to each text, that is, comparing each text with the keywords in the label list LABEL one by one, marking 1 if it matches the keywords, and 0 if it does not match, forming a data list about 1 or 0 as the label label of this text; Step S203: Each piece of text obtained is recorded as: wherein content represents the text content; Step S204: Obtain the sequence of data sets M: . 4.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 3, characterized in that, The label list LABEL is: 。 5.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The step S3 converting each text in the data set sequence M into a vector sequence to obtain a combined vector of each text comprises the following sub-steps: Step S301: performing a text classification task on each text in the data set sequence M, that is, adding a flag [CLS] at the beginning of each text to indicate that the text is used for classification task, and adding a flag [SEP] at the end to separate each text; Step S302: segment each piece of text to obtain a vector representation of each word, i.e., a word vector; merge multiple groups of words into a whole vector representation to obtain a segment vector; assign a vector to each segmented word using different sine and cosine functions to represent its relative position in the sequence to obtain a position vector; Step S303: calculate the sum of the word vector, segment vector and position vector of each piece of text to obtain a combined vector of each piece of text. 6.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 5, characterized in that, The position vector is obtained by assigning a vector to each segmented word, specifically: Let the length of one segment of text be Z, then the position vector of the first word in this segment of text is: ,​​ where d is the dimension of the embedding vector, is each dimension, . 7.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The step S4 of selecting a pre-training model and training it using the combined vector to obtain a text vector includes the following sub-steps: Step S401: select a bert-base-chinese pre-training model; Step S402: input the combined vector of each piece of text in the data set sequence M into the pre-training model to train the model; Step S403: after multiple rounds of training, until the value of the loss function loss no longer changes, a series of text vectors with rich semantic information are obtained, and the vector is a high-dimensional vector, where each dimension represents different semantic information. 8.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The step S6 of optimizing the multi-label chemical news classification model includes the following sub-steps: Step S601: perform a flattening operation on the three-dimensional feature matrix output by the pooling layer to convert it into a two-dimensional feature vector, preparing for the subsequent fully connected layer; Step S602: pass through a RELU activation function, so that the neural network can better perform nonlinear fitting, and the fitting formula is: That is, output 0 when the signal is less than 0, and output the signal itself when it is greater than 0; at the same time, pass through a Dropout to prevent overfitting of the neural network and improve the generalization ability of the model; Step S603: use the Adam algorithm to adaptively adjust the learning rate in different dimensions of each parameter to adapt to the characteristics of different parameters, thereby improving the training effect of the model; Step S604: add an output layer using the Sigmoid function as the activation function to perform binary classification on each label to obtain the prediction probability of each label, and the Sigmoid function is: 。 9.The multi-label chemical field news classification method based on transfer learning and deep learning according to claim 1, wherein, The step S7 of performance analysis of the multi-label chemical news classification model uses the following evaluation indicators: Precision: , The ratio of the number of samples that are actually positive examples and are predicted by the multi-label chemical news classification model to be positive examples to the total number of samples predicted by the multi-label chemical news classification model to be positive examples; wherein TP represents the number of correctly classified positive examples, i.e., the number of instances that are actually positive examples and are classified as positive examples by the multi-label chemical news classification model, and FP represents the number of incorrectly classified positive examples, i.e., the number of instances that are actually negative examples but are classified as positive examples by the multi-label chemical news classification model; Recall rate: , The ratio of the number of samples that are actually positive examples and are predicted by the multi-label chemical news classification model to be positive examples to the total number of samples that are actually positive examples; wherein FN represents the number of incorrectly classified negative examples, i.e., the number of instances that are actually positive examples but are classified as negative examples by the multi-label chemical news classification model; Harmonic mean F1 value of precision and recall: ， The comprehensive performance of the multi-label chemical news classification model based on precision and recall.

Images