Method and system for classifying messages on online government inquiry platforms based on deep neural networks

Abstract

The present invention discloses a method and system for classifying messages on an online government inquiry platform based on a deep neural network, the method comprising the following steps: obtaining a plurality of training government message data and corresponding labels; extracting non-repeated characters in the training government message data, and obtaining the vector representation of each character to obtain a government text vector dictionary; obtaining the vector representation of the plurality of training government message data based on the vector dictionary; encoding the label data to obtain the vector representation of each label; training a government text classification model using a machine learning model based on the vector representation of the government message data and the corresponding labels; and classifying government messages based on the government text classification model. The present invention creates a dictionary based on a large amount of government message text, and vectorizes the training data based on the dictionary, thereby ensuring the performance of the model and having higher accuracy for government text classification.

Description

Technical Field

[0001] The present invention belongs to the technical field of text data processing, and in particular to a method and system for classifying messages on an online government inquiry platform based on a deep neural network. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In recent years, as online government inquiry platforms such as WeChat, Weibo, Mayor's Mailbox, and Sunshine Hotline have gradually become important channels for the government to understand public opinion, gather public wisdom, and gather public sentiment, the amount of text data related to various social conditions and public opinions has continued to rise, which has brought great challenges to the work of relevant departments that used to rely mainly on manual work to divide messages and organize hot spots. At the same time, with the development of technologies such as big data, cloud computing, and artificial intelligence, the establishment of a smart government affairs system based on natural language processing technology has become a new trend in the innovative development of social governance, which has a great driving force in improving the government's management level and governance efficiency.

[0004] In terms of text classification technology, traditional text representation is high-dimensional and sparse, and its feature expression ability is weak. Traditional machine learning classification algorithms consume a lot of manpower and material resources in feature engineering. In addition, for government texts, due to their strong domain characteristics, large data volume, and complex content, the classification accuracy is not high when using existing feature extraction methods. Summary of the invention

[0005] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a method and system for classifying messages on an online government inquiry platform based on a deep neural network. The method builds a dictionary based on a large amount of government message data and is more suitable for the classification of government texts.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A method for classifying messages on an online government inquiry platform based on a deep neural network comprises the following steps:

[0008] Obtain multiple training government message data and corresponding labels;

[0009] Extract non-repeated words from the training government message data, and obtain the vector representation of each word to obtain the government text vector dictionary;

[0010] Based on the vector dictionary, vector representations of the plurality of training government message data are obtained; the label data is encoded to obtain vector representations of each label;

[0011] Based on the vector representation of government message data and corresponding labels, a machine learning model is used to train a government text classification model;

[0012] Government message classification is performed based on the government text classification model.

[0013] One or more embodiments provide a method for classifying messages on an online government inquiry platform based on a deep neural network, including:

[0014] A data acquisition module, configured to acquire a plurality of training government message data and corresponding labels;

[0015] The dictionary building module is configured to extract non-repeated words in the training government message data and obtain the vector representation of each word to obtain the government text vector dictionary;

[0016] A vector representation module is configured to obtain vector representations of the plurality of training government message data based on a vector dictionary; encode the label data to obtain vector representations of each label;

[0017] A model training module is configured to train a government text classification model using a machine learning model based on the government message data and the vector representation of the corresponding labels;

[0018] The message classification module classifies government messages based on the government text classification model.

[0019] One or more embodiments provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for classifying messages on an online government inquiry platform based on a deep neural network is implemented.

[0020] One or more embodiments provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for classifying messages on an online government inquiry platform based on a deep neural network.

[0021] One or more of the above technical solutions have the following beneficial effects:

[0022] The dictionary involved in the above technical solution is not a commonly used dictionary in the industry, but is created during the training phase based on a large amount of government text message data. A vector representation is constructed for each word in the government text. Based on the dictionary, the training text is vectorized and word embedded and input into the deep learning model, which can improve the accuracy of government text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 A flowchart of a method for classifying messages on an online government inquiry platform based on a deep neural network in one or more embodiments of the present invention;

[0025] Figure 2 This is a model structure diagram of a message classification system for an online government inquiry platform based on a deep neural network in one or more embodiments of the present invention. DETAILED DESCRIPTION

[0026] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0027] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0028] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0029] Embodiment 1

[0030] This embodiment discloses a method for classifying messages on an online government inquiry platform based on a deep neural network. The method includes two stages: classification model construction and message classification.

[0031] 1. Classification model construction

[0032] Step 1: Obtain the government text training data to be classified with labels;

[0033] Specifically, the training data for government text classification is provided by the official website of the competition. There are 9,210 sample data in total. The specific attribute columns of the data include message number, message user, message subject, message time, message details and first-level labels. The first-level labels are the labels corresponding to the sample data, namely urban and rural construction, environmental protection, transportation, education, culture and sports, labor and social security, commerce and tourism, and health and family planning.

[0034] Specifically, the text cleaning includes removing empty message text data, removing duplicate message text data, and removing special symbols in the message text data, including some common punctuation marks, special characters, etc., which are matched using regular expressions. In addition, considering that if there is a message with only some of the above-mentioned special symbols, an empty string will be obtained after all of them are removed, it is necessary to remove the empty string, remove stop words, and other operations.

[0035] Step 2: All non-repeated characters in the training data are combined into a government text vocabulary. Use BERT (Bidirectional Encoder Representation from Transformers) and the semantic model to generate vector representations for each character in the vocabulary, and then concatenate the character vectors generated by the two to form a vector representation of a single character. For the labels in the training data, one-hot encoding is used.

[0036] The BERT model, i.e., the encoder of the bidirectional Transformer, has a model architecture based on multi-layer bidirectional transformation decoding. The main feature of the model is that it uses two methods, Masked LM and Next Sentence Prediction, to capture word and sentence level representations respectively. The BERT model uses the Transformer, which is more efficient than other models, can capture longer-distance dependencies, and can capture true text context information.

[0037] Sememes refer to the smallest indivisible semantic unit in linguistics. Linguists believe that the meaning of all words can be represented by a limited number of combinations of sememes. This is done by pre-defining a set of sememes and annotating many words with the sememes to form a sememe knowledge base. HowNet is a well-known sememe knowledge base, which defines about 2,000 semantically unrelated sememes and about 90 dynamic roles, and annotates more than 100,000 Chinese and English words with these sememes and dynamic roles. Sememes can be used to learn the representation of different meanings of a word and capture the relationship between words in a more detailed manner.

[0038] The government text vocabulary is constructed, specifically, by extracting the "message details" data in the training data using a slicing method, extracting all non-repeated characters in the text data to construct a vocabulary, and obtaining a government text vocabulary of size 6000.

[0039] For each word in the government text vocabulary, BERT and the semantic model are used to generate word vectors respectively, and then the word vectors generated by the two are concatenated to form a vector representation of a single word. Each word in the vocabulary and its corresponding concatenated vector representation constitute the government text vector dictionary of this embodiment.

[0040] Specifically, the BERT dimension is set to 768 and the semantic model dimension is set to 200. For each non-repeating word in the government text vocabulary, BERT and the semantic model are used to generate a word vector for each word. Then, for the same word, the word vector generated by BERT and the word vector generated by the semantic model are concatenated to obtain a 968-dimensional vector representation for each word in the vocabulary.

[0041] The maximum sentence length of the text data is set to 700. The text is added to the maximum length according to the sentence length of the text data, and zeros are filled in the insufficient parts to ensure the same vector dimension so as to effectively process the text in batches.

[0042] For each piece of text data in the government text, the vector representation of each word in the text data is obtained from the government text vocabulary. In this way, the spatial vector representation of each piece of text data is obtained and used as the input data of the text classification model Text-CNN.

[0043] Encode the labels in the sample data, establish the category and ID index, and map each word to the dictionary size. Specifically, let categories = [0, 1, 2, 3, 4, 5], and get categories_id = {'urban and rural construction': 0, 'environmental protection': 1, 'transportation': 2, 'education style': 3, 'labor and social security': 4, 'commerce and tourism': 5, 'health and family planning': 6}, the dictionary size is 7, and finally each label is uniquely encoded, the label vector matrix dimension is 7, set to 1 in the corresponding position, and fill the remaining positions with zeros, for example, the label "education style" is set to: education style = [0, 0, 0, 1, 0, 0, 0].

[0044] The dictionary used in this embodiment is not a commonly used dictionary in the industry, but is created during the training phase based on a large amount of government text message data. A vector representation is constructed for each word in the government text. The training text is vectorized and word embedded based on the dictionary and input into the deep learning model, which can improve the accuracy of government text classification.

[0045] In addition, the vector representation of each word is generated by BERT and the semantic model, and then the word vectors generated by the two are concatenated to form the vector representation of a single word, which is input into the model. The BERT model can better extract text features, and the word vectors generated by the semantic model incorporate knowledge into the features. The combination of the two includes both text features and semantic knowledge features, which can fully integrate the advantages of knowledge and data, better represent word vectors, make model training better, and have higher classification accuracy.

[0046] Step 3: Based on the deep neural network model, train the government text classification model to obtain the model parameters.

[0047] Figure 2 This is a model structure diagram of the first embodiment of the present invention

[0048] This embodiment uses the Text-CNN model to train a government text classification model, and the model classification accuracy rate reaches 91%. The Text-CNN model framework includes a word embedding layer, a convolution layer, a maximum pooling layer, a fully connected layer, and a Relu layer.

[0049] The word embedding layer embeds words into low-dimensional vectors; the maximum length of the sentence of the input sample data is 700, and it is padded with zeros when it is insufficient. The embedding vector dimension of each word is 968. The embedding layer has the following two features: Force the operation to be performed on the CPU. By default, TensorFlow will try to place the operation on the GPU (if available), but the embedded implementation currently has no GPU support and will cause an error if placed on the GPU. Create a name scope named "embedding". The scope adds all operations to a top-level node named "embedding" to obtain a good hierarchy when visualizing the network in TensorBoard.

[0050] The convolutional layer convolves the embedded word vectors with multiple filter sizes, iterates the tensors of different shapes produced by each convolution, and merges the results into a large feature vector.

[0051] The maximum pooling layer generates a long feature vector and adds dropout regularization;

[0052] Relu layer to classify the results.

[0053] A pooled_outputs is established to save the results of each convolution, and pooled is obtained after convolution, relu activation function and max-pooling operations with different convolution kernel sizes.

[0054] The result after each convolution is taken out, and the Dropput layer is performed to prevent overfitting. Finally, a fully connected layer and a Relu layer are added to map the features into probabilities of different categories.

[0055] Model training includes:

[0056] (1) Define the model parameters of the model: placeholders for input and output, and dropout ratio. We feed it to the network when executing it on the training set or test set. Since dropout is only used during training, the probability of keeping a neuron in the dropout layer is also an input to the network.

[0057] The method to initialize parameters is: before model training, use the global_variables_initializer function to define the initial value of the variable and use the pre-trained value for initial embedding.

[0058] (2) Compute the cross entropy loss for each class, given the scores and the correct input labels, and average the losses.

[0059] (3) Define the loss function of the optimization network and use the Adam optimizer.

[0060] (4) The specific process of the training loop is: iterate the batches of data and call the train_step function for each batch.

[0061] The method to save model parameters is: Use Checkpointing to save the model parameters for later recovery. Checkpoints can be used to continue training at a later time.

[0062] Step 4: Based on the test data, use the trained government text classification model to perform classification and evaluation.

[0063] When using evaluation indicators for evaluation, precision P, recall R and F value are used as evaluation indicators:

[0064]

[0065]

[0066]

[0067] The higher the F value, the better the information extraction effect.

[0068] In this embodiment, the data set is the data provided by the official website, wherein Attachment 2 contains 495 sample data, including message time, message details, and primary classification. The given Attachment 2 data set is used to use the Text-CNN model to complete the message classification task. This embodiment aims to classify government texts.

[0069] 2. Governmental Document Classification Stage

[0070] After model training to obtain model parameters and model accuracy evaluation, the above model can be deployed on the online government inquiry platform to classify messages on the platform. That is, it also includes:

[0071] Step 5: Obtain the message data from the online government inquiry platform and classify it based on the government text classification model.

[0072] The use of the Text-CNN model to train a government text classification model has high accuracy and execution efficiency. The present invention uses network text analysis and data mining technology to manage messages on the online government inquiry platform in an informationized manner, which can improve the government's management level and policy efficiency.

[0073] Embodiment 2

[0074] On the basis of the first embodiment, the purpose of this embodiment is to provide a system for classifying messages on an online government inquiry platform based on a deep neural network, including:

[0075] A data acquisition module, configured to acquire a plurality of training government message data and corresponding labels;

[0076] A dictionary building module is configured to build a vector dictionary, wherein the vector dictionary includes a vector representation of each word in the training government text data;

[0077] A vector representation module is configured to obtain vector representations of the plurality of training government message data based on a vector dictionary; encode the label data to obtain vector representations of each label;

[0078] A model training module is configured to train a government text classification model using a machine learning model based on the government message data and the vector representation of the corresponding labels;

[0079] The message classification module classifies government messages based on the government text classification model.

[0080] Embodiment 3

[0081] The purpose of this embodiment is to provide an electronic device.

[0082] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for classifying messages on an online government inquiry platform based on a deep neural network as described in Example 1 is implemented.

[0083] Embodiment 4

[0084] The purpose of this embodiment is to provide a computer-readable storage medium.

[0085] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the method for classifying messages on an online government inquiry platform based on a deep neural network as described in Example 1.

[0086] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0087] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0088] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for classifying messages on an online government inquiry platform based on a deep neural network, characterized in that: The following steps are involved: Obtain multiple training government message data and corresponding labels; All non-repeated characters in the training data are combined into a government text vocabulary. For each character in the government text vocabulary, BERT and the semantic model are used to generate word vectors respectively. The word vectors generated by the two are concatenated to form a vector representation of a single character. Each character in the vocabulary and its corresponding concatenated vector representation constitute a government text vector dictionary. Obtaining vector representations of the plurality of training government message data based on a vector dictionary; A unique number is set for each label, and one-hot encoding is performed. The dimension of the label vector matrix is ​​the number of categories. The corresponding position of the category is set to 1, and the remaining positions are filled with zeros to obtain the vector representation of each label. Based on the vector representation of government message data and corresponding labels, a machine learning model is used to train a government text classification model; Government message classification is performed based on the government text classification model.

2. The method for classifying messages on an online government inquiry platform based on a deep neural network as claimed in claim 1 is characterized in that: After obtaining a plurality of training government message data, the training government message data is also preprocessed by text cleaning.

3. The method for classifying messages on an online government inquiry platform based on a deep neural network as claimed in claim 1, characterized in that: The machine learning model adopts a text convolutional neural network, including a word embedding layer, a convolution layer, a maximum pooling layer, a fully connected layer and a Relu layer.

4. The method for classifying messages on an online government inquiry platform based on a deep neural network as claimed in claim 1, characterized in that: The method further includes: based on the test data, using the trained government text classification model to perform classification and evaluation.

5. A system for classifying messages on online government inquiry platforms based on deep neural networks, characterized in that: include: A data acquisition module, configured to acquire a plurality of training government message data and corresponding labels; The dictionary construction module is configured to form a government text vocabulary from all non-repeated characters in the training data. For each character in the government text vocabulary, BERT and the semantic model are used to generate word vectors respectively. The word vectors generated by the two are concatenated to form a vector representation of a single character. Each character in the vocabulary and its corresponding concatenated vector representation constitute a government text vector dictionary. A vector representation module, configured to obtain vector representations of the plurality of training government message data based on a vector dictionary; A unique number is set for each label, and one-hot encoding is performed. The dimension of the label vector matrix is ​​the number of categories. The position where the category belongs is set to 1, and the remaining positions are filled with zeros to obtain the vector representation of each label. A model training module is configured to train a government text classification model using a machine learning model based on the government message data and the vector representation of the corresponding labels; The message classification module classifies government messages based on the government text classification model.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the method for classifying messages on an online government inquiry platform based on a deep neural network as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the method for classifying messages on an online government inquiry platform based on a deep neural network as described in any one of claims 1 to 4.

Images