BERT-based multi-classification method and system for smart government text

Abstract

The present invention belongs to the field of text classification, and provides a BERT-based smart government text multi-classification method and system. The BERT-based smart government text multi-classification method includes obtaining government text and converting it into a feature vector, wherein the feature vector is composed of a word vector, a segmentation vector and a position vector, and the beginning and end of each sentence in the text are marked; the feature vector is input into a trained BERT model, and the classification result of the government text is output; wherein, in the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector.

Description

Technical Field

[0001] The present invention belongs to the field of text classification, and in particular relates to a BERT-based smart government text multi-classification method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] When processing messages from the public on the online government inquiry platform, the messages are classified so that they can be subsequently assigned to the corresponding functional departments. At present, most e-government systems still rely on manual processing based on experience, which has problems such as large workload, low efficiency, and high error rate.

[0004] The inventors found that most existing text classifications use word vectors, and most word vectors are trained using methods such as Word2Vec and GloVe. The problem is that the word vectors obtained through training not only ignore the contextual relationship of the words, but also have a large number of words in some texts, resulting in a slow training speed and an accuracy rate affected by word segmentation (i.e., dividing a sequence of Chinese characters into individual words). Summary of the invention

[0005] In order to solve the above problems, the present invention provides a BERT-based intelligent government text multi-classification method and system, which can quickly and accurately classify government texts. BERT stands for Bidirectional Encoder Representations from Transformers, which is a language model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a BERT-based smart government text multi-classification method.

[0008] A BERT-based smart government text multi-classification method, including:

[0009] Obtain government text and convert it into a feature vector, which consists of a word vector, a segment vector, and a position vector, and marks the beginning and end of each sentence in the text;

[0010] Input the feature vector into the trained BERT model and output the classification result of government affairs text;

[0011] In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector.

[0012] The second aspect of the present invention provides a BERT-based smart government text multi-classification system.

[0013] A BERT-based intelligent government text multi-classification system, including:

[0014] A feature conversion module is used to obtain government text and convert it into a feature vector, which consists of a word vector, a segment vector and a position vector, and marks the beginning and end of each sentence in the text;

[0015] The classification module in this paper is used to input feature vectors into the trained BERT model and output the classification results of government documents;

[0016] In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the BERT-based smart government text multi-classification method as described above.

[0019] A fourth aspect of the present invention provides a computer device.

[0020] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the BERT-based smart government text multi-classification method as described above are implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The BERT-based smart government text multi-classification method and system proposed in the present invention uses a bidirectional Transformer encoding layer in the feature conversion module, which extracts text features. Each extracted feature includes information about the left and right contexts, overcoming the problem of the prior art ignoring the contextual relationship of words. By setting the number of training steps to obtain a model for verification and saving the highest-scoring model, the highest-scoring model is used for testing, which has the beneficial effects of high test accuracy, strong reliability and relatively stable performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] FIG. 1( a ) is a flow chart of the BERT model training method of the present invention;

[0025] FIG1( b ) is a flow chart of a BERT model testing method of the present invention;

[0026] Figure 2 The BERT input representation graph of the present invention;

[0027] Figure 3 This is a structural diagram of the BERT model of the present invention;

[0028] Figure 4 This is a structural diagram of the Encoder in the Transformer encoding layer of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Embodiment 1

[0033] The data used in this example contains a total of 9210 messages from the public, each of which includes a number, user, topic, time, message details and classification label. The 9210 samples in the data set belong to seven different categories: urban and rural construction, environmental protection, transportation, education, culture and sports, labor and social security, commerce and tourism, and health and family planning. The data distribution is shown in Table 1.

[0034] Table 1 Text data of public messages

[0035]

[0036] This embodiment provides a BERT-based smart government text multi-classification method, the method comprising the following steps:

[0037] Step 1: Obtain government text and convert it into a feature vector, which consists of a word vector, a segment vector, and a position vector, while marking the beginning and end of each sentence in the text.

[0038] Among them, the government text includes the message number, user, subject, time, message details and classification label.

[0039] In the process of converting feature vectors, the features converted from the message details in the sample are composed of three parts: word vector, segment vector and position vector, such as Figure 2 As shown, the classification labels of text types are converted into numerical types as label_id={0, 1, 2, 3, 4, 5, 6};

[0040] Since the BERT model is used for text classification, the text information is transformed according to the input requirements of the model. The word vector represents the text information with a vector. For the text classification task, a [CLS] is inserted before the text, and the output vector corresponding to the symbol is used as the semantic representation of the text for text classification; the segmentation vector is because of the prediction task of the next sentence in BERT, so there will be two sentences spliced ​​together, and the upper and lower sentences correspond to different segmentation vectors. Both sentences have [CLS] at the beginning and [SEP] at the end of the sentence; since the model is used for text classification tasks, only one sentence is used, so there is only one set of [CLS] and [SEP].

[0041] The position vector is because the words at different positions in the text express different semantic information, so different vectors are added to the words at different positions to distinguish them.

[0042] The text data of the input training set is converted into features, which consists of three parts: word vector, segment vector and position vector. The word vector represents the encoding of the current word, the segment vector represents the position encoding of the sentence where the current word is located, and the position vector represents the position encoding of the current word. Each sentence uses CLS and SEP as the beginning and end markers.

[0043] Step 2: Input the feature vector into the trained BERT model and output the classification result of the government text;

[0044] In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector. The Encoder structure is as follows: Figure 4 shown.

[0045] Specifically, in the training phase, as shown in Figure 1(a):

[0046] Step 1: Data reading. First, the original data is randomly divided into training set, validation set and test set in a ratio of 9:1. When reading the data of the training set and validation set, only the two important columns of message details and classification labels in the table are obtained for model training and testing.

[0047] Get the given data set, including the message number, user, topic, time, message details and classification label. Shuffle the data in the data set and divide it into training set and validation set in a ratio of 9:1. For the problem of mass message classification, use the two methods in the data reading module to read the message details and primary classification columns in the data set respectively. One method is to obtain the data of the training set and the other is to obtain the data of the validation set.

[0048] Step 2: Feature conversion: the features of the message details in the sample are converted into word vectors, segment vectors and position vectors.

[0049] Step 3: Train the model and input the feature data into the bidirectional Transformer encoding, the most important part of the BERT model structure. This layer extracts text features using the Transformer Encoder feature extractor, such as Figure 3 As shown. Encoder consists of a self-attention mechanism and a feed-forward neural network. Its structure is as follows Figure 3 shown.

[0050] The core of Encoder is self-attention. It can find the relationship between each word and other words in the sentence, and there is no distance limit. The relationship between dozens or even hundreds of words can still be found, so that the left and right context information of each word can be fully mined, so that the two-way representation of the word can be obtained, which makes it easier to obtain a high-scoring training model. The pre-trained model used in the experiment is BERT-Base, Chinese. This model uses a 12-layer Transformer with a hidden size of 768 and a multi-head attention mechanism parameter of 12. The pre-trained model is directly loaded and run when training the model.

[0051] When training the model, if the number of training steps exceeds the preset number, such as 1000 times, it will be verified on the validation set and the accuracy will be given. If the accuracy is greater than the previous highest score model, the model will be saved. If the number of epochs for training all training samples exceeds the previously set number of training times, the iteration stops.

[0052] Step 4: Save the high-scoring model. When the number of training steps exceeds the preset number, for example, 1000, the model will be verified on the validation set and the accuracy will be given. If the accuracy is greater than the previous highest-scoring model, the model will be saved. If the number of epochs for training all training samples exceeds the previously set number of training, the iteration stops. When training the model, the batch size (batch size) is 12 and the number of epochs (epochs) is 3.

[0053] The evaluation index for saving the highest score model is the accuracy Acc (accuracy). Accuracy refers to the ratio of the number of samples of multiple categories correctly predicted by the model to the total number of samples. The calculation method of the evaluation index is as shown in formula (1).

[0054]

[0055] Among them, TP represents the number of true positive samples, FP represents the number of false positive samples, FN represents the number of false negative samples, and TN represents the number of true negative samples.

[0056] In the testing phase, as shown in Figure 1(b):

[0057] Step 1: Read the test set data.

[0058] Step 2: Perform feature conversion on the test set data.

[0059] Step 3: Load the high-scoring training model saved during the training phase and test it using the test set.

[0060] Step 4: Use scoring indicators to score the test results. The scoring results of the seven types of text data are shown in Table 2.

[0061] The evaluation of the test results is based on four evaluation indicators, namely, accuracy Acc (accuracy), precision P (precision), recall R (recall) and F value F1 (f1-score). Accuracy is also the search precision, that is, the ratio of the number of truly correct results in the results returned after the search to the total number of results. Recall is also the search completeness, that is, the ratio of the number of truly correct results in the search results to the number of truly correct results in the entire data set (searched and unsearched). Because accuracy and recall sometimes contradict each other, the F value is used to comprehensively consider the two evaluation indicators. The calculation methods of these four indicators are shown in formulas (1) to (4):

[0062]

[0063]

[0064]

[0065] Table 2 Test data results

[0066]

[0067] As shown in Table 2, the test results show that the accuracy of the model reaches 97%. After testing, it can be seen that the BERT model has certain reliability and robustness, which is quite satisfactory.

[0068] This embodiment uses a bidirectional Transformer encoding layer in the feature conversion module, which extracts text features. Each extracted feature includes information about the left and right contexts, overcoming the problem of the prior art ignoring the contextual relationship of words. By setting the number of training steps to obtain a model for verification and saving the highest-scoring model, the highest-scoring model is used for testing, which has the beneficial effects of high test accuracy, strong reliability and stability.

[0069] Embodiment 2

[0070] This embodiment provides a BERT-based smart government text multi-classification system, including:

[0071] (1) A feature conversion module, which is used to obtain government text and convert it into a feature vector. The feature vector consists of a word vector, a segment vector, and a position vector, and also marks the beginning and end of each sentence in the text.

[0072] In the specific implementation, the word vector represents the encoding of the current word, the segment vector represents the position encoding of the sentence where the current word is located, and the position vector represents the position encoding of the current word. Each sentence uses CLS and SEP as the beginning and end markers. The Encoder feature extractor consists of a self-attention mechanism and a feedforward neural network. Among them, the government text includes the message number, user, topic, time, message details and classification label.

[0073] (2) The text classification module is used to input the feature vector into the trained BERT model and output the classification results of government documents;

[0074] In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector.

[0075] Encoder consists of a self-attention mechanism and a feedforward neural network. Its structure is as follows: Figure 3 shown.

[0076] The core of Encoder is self-attention. It can find the relationship between each word and other words in the sentence, and there is no distance limit. The relationship between dozens or even hundreds of words can still be found, so that the left and right context information of each word can be fully mined, so that the two-way representation of the word can be obtained, which makes it easier to obtain a high-scoring training model. The pre-trained model used in the experiment is BERT-Base, Chinese. This model uses a 12-layer Transformer with a hidden size of 768 and a multi-head attention mechanism parameter of 12. The pre-trained model is directly loaded and run when training the model.

[0077] When training the model, if the number of training steps exceeds the preset number, such as 1000 times, it will be verified on the validation set and the accuracy will be given. If the accuracy is greater than the previous highest score model, the model will be saved. If the number of epochs for training all training samples exceeds the previously set number of training times, the iteration stops.

[0078] During the training of the BERT model, set the test result evaluation indicators and save the highest score model as the trained BERT model.

[0079] When training a model, if the number of training steps exceeds the preset number, for example, 1000, it will be verified on the validation set and the accuracy will be given. If the accuracy is greater than the previous highest-scoring model, the model will be saved. If the number of epochs for training all training samples exceeds the previously set number of training, the iteration stops. When training the model, the batch size (batchsize) is 12 and the number of epochs is 3.

[0080] The evaluation index for saving the highest score model is the accuracy Acc (accuracy). Accuracy refers to the ratio of the number of samples of multiple categories correctly predicted by the model to the total number of samples. The calculation method of the evaluation index is as shown in formula (1).

[0081]

[0082] Among them, TP represents the number of true positive samples, FP represents the number of false positive samples, FN represents the number of false negative samples, and TN represents the number of true negative samples.

[0083] The evaluation of the test results is based on four evaluation indicators, namely, accuracy Acc (accuracy), precision P (precision), recall R (recall) and F value F1 (f1-score). Accuracy is also the search precision, that is, the ratio of the number of truly correct results in the results returned after the search to the total number of results. Recall is also the search completeness, that is, the ratio of the number of truly correct results in the search results to the number of truly correct results in the entire data set (searched and unsearched). Because accuracy and recall sometimes contradict each other, the F value is used to comprehensively consider the two evaluation indicators. The calculation methods of these four indicators are shown in formulas (1) to (4):

[0084]

[0085]

[0086]

[0087] This embodiment uses a bidirectional Transformer encoding layer in the feature conversion module, which extracts text features. Each extracted feature includes information about the left and right contexts, overcoming the problem of the prior art ignoring the contextual relationship of words. By setting the number of training steps to obtain a model for verification and saving the highest-scoring model, the highest-scoring model is used for testing, which has the beneficial effects of high test accuracy, strong reliability and stability.

[0088] Embodiment 3

[0089] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the BERT-based smart government text multi-classification method as described in Embodiment 1.

[0090] This embodiment uses a bidirectional Transformer encoding layer in the feature conversion module, which extracts text features. Each extracted feature includes information about the left and right contexts, overcoming the problem of the prior art ignoring the contextual relationship of words. By setting the number of training steps to obtain a model for verification and saving the highest-scoring model, the highest-scoring model is used for testing, which has the beneficial effects of high test accuracy, strong reliability and stability.

[0091] Embodiment 4

[0092] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the BERT-based smart government text multi-classification method as described in Example 1 are implemented.

[0093] This embodiment uses a bidirectional Transformer encoding layer in the feature conversion module, which extracts text features. Each extracted feature includes information about the left and right contexts, overcoming the problem of the prior art ignoring the contextual relationship of words. By setting the number of training steps to obtain a model for verification and saving the highest-scoring model, the highest-scoring model is used for testing, which has the beneficial effects of high test accuracy, strong reliability and stability.

[0094] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0095] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0096] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0097] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0098] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A BERT-based smart government text multi-classification method, characterized by: include: Obtain government text and convert it into a feature vector, which consists of a word vector, a segment vector, and a position vector, and marks the beginning and end of each sentence in the text; Input the feature vector into the trained BERT model and output the classification result of government affairs text; In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector; The Encoder feature extractor consists of a self-attention mechanism and a feedforward neural network; When the number of training steps exceeds the preset number during model training, the model is verified on the validation set and the accuracy is given. If the accuracy is greater than the previous highest-scoring model, the model is saved. During the training of the BERT model, set the test result evaluation index and save the model with the highest score as the trained BERT model; Among them, the accuracy is expressed by Acc, and its expression is: TP represents the number of true positive samples, FP represents the number of false positive samples, FN represents the number of false negative samples, and TN represents the number of true negative samples.

2. The BERT-based smart government text multi-classification method as claimed in claim 1, characterized in that: The word vector represents the encoding of the current word, the segment vector represents the encoding of the position of the sentence where the current word is located, and the position vector represents the encoding of the position of the current word. Each sentence uses CLS and SEP as the beginning and end markers.

3. The BERT-based smart government text multi-classification method as claimed in claim 1, characterized in that: The government text includes the message number, user, subject, time, message details and category label.

4. The BERT-based smart government text multi-classification method as claimed in claim 1, characterized in that: In the process of training the BERT model, the pre-trained model BERT-Base-Chinese model is used, and the pre-trained model is directly loaded and run when training the model.

5. The BERT-based smart government text multi-classification method as claimed in claim 1, characterized in that: The test result evaluation indicators include accuracy, precision, recall and F1 value; Among them, the precision is represented by P, and its expression is: ; The recall rate is represented by R, and its expression is: ; The F1 value is expressed as F1, and its expression is: .

6. A BERT-based intelligent government text multi-classification system, characterized by: include: A feature conversion module is used to obtain government text and convert it into a feature vector, which consists of a word vector, a segment vector and a position vector, and marks the beginning and end of each sentence in the text; The classification module in this paper is used to input feature vectors into the trained BERT model and output the classification results of government documents; In the process of training the BERT model, the Encoder feature extractor in the bidirectional Transformer encoding is used to extract text features in the feature vector; During the training of the BERT model, set the test result evaluation index and save the model with the highest score as the trained BERT model; Among them, the accuracy is expressed by Acc, and its expression is: TP represents the number of true positive samples, FP represents the number of false positive samples, FN represents the number of false negative samples, and TN represents the number of true negative samples.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the BERT-based smart government text multi-classification method as described in any one of claims 1-5 are implemented.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the BERT-based smart government text multi-classification method as described in any one of claims 1-5 are implemented.

Images