Text classification method, device, electronic device and storage medium

Abstract

The present application relates to the field of computers, and in particular to the field of artificial intelligence technology, and discloses a text classification method, device, electronic device, and storage medium. The method includes: obtaining text information to be identified, inputting the text information into a trained first text classification model, and obtaining a target word vector matrix; then performing semantic mining processing on each target word vector to obtain corresponding semantic features, and finally obtaining a target predicted classification result based on each semantic feature; and the first text classification model is obtained after parameter adjustment based on a first loss value and a second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between the two predicted classification results. The two loss values ​​are used to adjust the parameters of the first text classification model, so that the predicted classification result of the first text classification model approaches the actual classification result and another predicted classification result, further improving the classification accuracy of the model.

Description

Background Art

[0002] With the advent of the big data era, there is a huge amount of text data from different data sources on the Internet. For example, it is necessary to determine whether text data such as web pages and public account articles are illegal texts, so as to conduct reasonable filtering and supervision.

[0003] In order to meet the needs of text classification tasks, a text classification model can be used to classify text data from different types of data sources. However, due to differences in expression methods, text length, etc., the classification effect of the text classification model is relatively poor. Alternatively, multiple text classification models can be used to classify text data from different types of data sources. Each text classification model can only classify text data from a single category of data source. In this case, the number of parameters of the entire text classification model is very large, and storing multiple text classification models will also consume more memory space. Summary of the invention

[0004] Embodiments of the present application provide a text classification method, device, electronic device, and storage medium to solve the problem of difficulty in classifying text data from different types of data sources.

[0005] The present invention provides a text classification method, which includes:

[0006] Obtain text information to be recognized;

[0007] Inputting the text information into a trained first text classification model to obtain a target word vector matrix corresponding to the text information;

[0008] Perform semantic mining on each target word vector to obtain the corresponding semantic features;

[0009] Based on each semantic feature, obtaining a target prediction classification result of the text information;

[0010] Among them, the first text classification model is obtained after parameter adjustment based on the first loss value and the second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between two predicted classification results.

[0011] An embodiment of the present application provides a text classification device, the device comprising:

[0012] An acquisition unit, used for acquiring text information to be recognized;

[0013] A processing unit, configured to input the text information into a trained first text classification model to obtain a target word vector matrix corresponding to the text information;

[0014] Perform semantic mining on each target word vector to obtain the corresponding semantic features;

[0015] Based on each semantic feature, obtaining a target prediction classification result of the text information;

[0016] Among them, the first text classification model is obtained after parameter adjustment based on the first loss value and the second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between two predicted classification results.

[0017] Optionally, if the semantic mining sub-model is a long short-term memory unit, the training unit is used to:

[0018] Based on the long short-term memory model, second context vectors corresponding to the respective word vectors are calculated, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a second context vector represents the correlation between a corresponding word vector and other word vectors from which the second context vector has been extracted;

[0019] Feature extraction is performed on each second context vector to obtain semantic features corresponding to each word vector.

[0020] Optionally, if the semantic mining sub-model is a convolutional model, the training unit is used to:

[0021] The convolution model is used to perform feature extraction on each word vector to obtain semantic features corresponding to each word vector.

[0022] Optionally, the training unit is used to:

[0023] Determine the gradient of the parameters on the first text classification model based on the total loss value;

[0024] Parameters of the first text classification model are adjusted according to the direction of the gradient.

[0025] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned text classification methods.

[0026] An embodiment of the present application provides a computer-readable storage medium, which includes a program code. When the program code is executed on an electronic device, the program code is used to enable the electronic device to execute the steps of any one of the above-mentioned text classification methods.

[0027] The beneficial effects of this application are as follows:

[0028] The embodiment of the present application provides a text classification method, device, electronic device and storage medium. In the embodiment of the present application, the text information to be identified is obtained, and the text information is input into the trained first text classification model to obtain the target word vector matrix; then the semantic mining process is performed on each target word vector to obtain the corresponding semantic features, and finally the target prediction classification result is obtained based on each semantic feature; and the first text classification model is obtained after parameter adjustment based on the first loss value and the second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between the two predicted classification results. During the training process, the first text classification model is based on the first loss value between the predicted classification result and the actual classification result, and the second loss value between the two predicted classification results. The parameters are adjusted so that the predicted classification result of the first text classification model is close to the actual classification result and another predicted classification result. In this way, the trained first text classification model can classify text information from different types of data sources, while reducing the model parameters, and also improving the classification accuracy of the first text classification model.

[0029] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 This is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a process for training a first text classification model in an embodiment of the present application;

[0033] Figure 3 This is a schematic diagram of the process of training the second text classification model X in an embodiment of the present application;

[0034] Figure 4a Schematic diagram of the structure of the attention model in the embodiment of the present application;

[0035] Figure 4b A schematic diagram of generating a query vector, a key vector, and a value vector in an embodiment of the present application;

[0036] Figure 4cA schematic diagram of generating a first context vector corresponding to a word vector Y in an embodiment of the present application;

[0037] Figure 4d A schematic diagram of the structure of the long short-term memory model implemented in this application;

[0038] Figure 4e Schematic diagram of the structure of the convolution model in the embodiment of the present application;

[0039] Figure 5 This is a flowchart of text classification in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of a text classification device in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0042] Figure 8 Schematic diagram of the structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0044] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0045] Some of the terms used in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0046] 1. Artificial Intelligence (AI):

[0047] AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0048] AI is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies; artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning and other major directions. With the research and progress of artificial intelligence technology, AI has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, etc. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0049] 2. Natural Language Processing (NLP):

[0050] NLP is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0051] 3. Machine Learning (ML):

[0052] ML is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. ML is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0053] 4. Multi-task learning:

[0054] Multi-task learning is a transfer learning algorithm that helps improve the performance of the target model by using the knowledge contained in multiple similar tasks. By sharing knowledge between similar tasks, the learning efficiency of the target model can be accelerated and the generalization of the target model can be improved.

[0055] 5. Knowledge Distillation:

[0056] When using a large amount of data sets to train a neural network (hereinafter referred to as a model), in order to handle complex data distribution, one approach is to build a complex model, such as a model containing hundreds of layers of residual networks, which often contains millions of parameters; another approach is to use the same data set to train multiple large-scale models (i.e., models containing a large number of parameters, hereinafter referred to as large models), and then combine the output results of multiple large models to obtain the final classification result. However, the second approach will cause the following two problems: first, the above-mentioned multiple large models need to be retrained for new application scenarios, which is costly and time-consuming; second, due to the excessive number of parameters contained in each large model, the entire model is too large and difficult to deploy on a large scale. In order to solve the problems caused by the above two methods, a new method of building and training models is needed, which can reduce the structure of the model and the number of parameters of the model without significantly reducing the model effect.

[0057] The core idea of ​​knowledge distillation is to use the knowledge learned by the large model as prior knowledge, and then use the prior knowledge to train the small model. The small model with prior knowledge only needs a small amount of new scene data to converge. In addition, the number of parameters contained in the small model itself is far less than that of the large model, which is more conducive to model convergence, shortens the model training time, improves the model training speed, and also makes it convenient to deploy the trained small model in the application scenario.

[0058] 6. Bidirectional Encoder Representations from Transformer (BERT):

[0059] BERT is a natural language processing model that pre-trains deep bidirectional representations by jointly regulating the context in all layers. Therefore, the pre-trained BERT representation can be fine-tuned through an additional output layer, making it suitable for model construction for a wide range of tasks without significantly modifying the model's architecture for specific tasks.

[0060] 7. Tranformer:

[0061] Tranformer is a language sequence processing model based on the attention mechanism. Transformer uses the attention mechanism to mine the correlation between words in a sentence, allowing the model to learn the context vector of the word, improving the output effect of the model. It can also use the attention mechanism to achieve fast parallelism, improving the shortcoming of slow training of recurrent neural network (RNN).

[0062] 8. Long Short-Term Memory (LSTM):

[0063] LSTM is a time-recursive model designed to solve the gradient vanishing and gradient exploding problems in long sequence training.

[0064] 9. Text Convolution Neural Network (TextCNN):

[0065] TextCNN is a model that uses CNN to classify text information.

[0066] 10. Fast Gradient Method (FGM):

[0067] The fast gradient descent method adjusts the parameters of the second text classification model in the direction of gradient ascent, and continuously optimizes the second text classification model so that the second predicted classification result output by the model continuously approaches the actual classification result and the first predicted classification result.

[0068] The following is a brief introduction to the design concept of the embodiment of the present application:

[0069] In order to respond to the needs of text classification tasks, a text classification model can be used to classify text data from different types of data sources. However, due to differences in expression methods, text length, etc., the classification effect of the text classification model is relatively poor. Alternatively, multiple text classification models can be used to classify text data from different types of data sources. Each text classification model can only classify text data from a single category of data source. In this case, the number of parameters of the entire text classification model is very large, and storing multiple text classification models will also consume more memory space. In view of this, the present application embodiment proposes a new text classification method.

[0070] The method includes: obtaining text information to be identified, inputting the text information into a trained first text classification model to obtain a target word vector matrix; then performing semantic mining processing on each target word vector to obtain corresponding semantic features, and finally obtaining a target prediction classification result based on each semantic feature; and the first text classification model is obtained after parameter adjustment based on a first loss value and a second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between the two predicted classification results. During the training process, the first text classification model is parameter adjusted based on the first loss value between the predicted classification result and the actual classification result, and the second loss value between the two predicted classification results, so that the predicted classification result of the first text classification model approaches the actual classification result and another predicted classification result. In this way, the trained first text classification model can classify text information from different types of data sources, while reducing model parameters, and also improving the classification accuracy of the first text classification model.

[0071] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0072] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 130, and the application operation interface 120 can be logged in through the terminal device 110. The terminal device 110 and the server 130 can communicate through a communication network.

[0073] In an embodiment of the present application, the terminal device 110 is an electronic device used by a user, and the electronic device may be a personal computer, mobile phone, tablet computer, notebook, e-book reader, smart home or other computer device. Each terminal device 110 communicates with the server 130 via a communication network. In an optional embodiment, the communication network is a wired network or a wireless network, so the terminal device 110 and the server 130 may be directly or indirectly connected via wired or wireless communication, and the present application does not limit this. The server 130 may be an independent physical server 130, or a server 130 cluster or distributed system composed of multiple physical servers 130, or a cloud server 130 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0074] Among them, the text classification model in the embodiment of the present application can be deployed on the terminal device 110, and can also be deployed on the server 130. When the text classification model is deployed on the terminal device 110, the terminal device 110 obtains the text information to be classified, classifies the text information to be classified, and presents the final classification result to the user. When the text classification model is deployed on the server 130, the server 130 receives the text information to be classified sent by the terminal device 110, classifies the text information to be classified, and returns the final classification result to the terminal device 110, so that the terminal device 110 presents the classification result to the user. The text information to be classified can come from a browser web page, text information matching the query information input by the user, or a data source such as a public account article, and this application does not limit it here.

[0075] In the embodiment of the present application, the trained first text classification model is used to classify the text. For ease of understanding, refer to Figure 2 The flowchart shown first introduces the training process of the first text classification model.

[0076] S201: Obtain a first training sample from a first training data set, the first training data set comprising first training samples from different types of data sources, a first training sample comprising first text information, a first actual classification result of the first text information, and a second predicted classification result outputted after the first text information is input into a second text classification model; wherein each first text information corresponds to a type of data source, and each type of data source corresponds to a second text classification model.

[0077] The first training data set contains a large number of first training samples. For ease of reading, the first training data set can be divided into multiple batches, and the number of samples contained in each batch is the same. In the embodiment of the present application, after each batch of first training samples is read, it is determined that the second text classification model has completed a round of iteration.

[0078] Each type of data source corresponds to a second text classification model. In the embodiment of the present application, the output results of each second text classification model are used as the first training sample to train the first text classification model. Therefore, before executing step 201, each second text classification model needs to be trained first. For ease of understanding and description, refer to Figure 3 The flowchart shown takes a second text classification model X as an example to introduce its training process.

[0079] S2011: Acquire a second training sample from a second training data set, where the second training data set includes second training samples from the same data source, and one second training sample includes second text information and a second actual classification result of the second text information.

[0080] S2012: Input the second text information into the second text classification model X to obtain a corresponding third predicted classification result.

[0081] S2013: Adjust parameters of the second text classification model X based on a third loss value between the third predicted classification result and the second actual classification result.

[0082] S2014: Determine whether the preset convergence condition is met, if so, execute step 2015; otherwise, return to step 2011.

[0083] S2015: Output the trained second text classification model X.

[0084] Each data source needs to build and train a corresponding second text classification model, but the text information of different data sources differs in terms of expression method, text length, etc. In order to speed up the training speed of each second text classification model, the embodiment of the present application utilizes the structural characteristics of BERT and generates a second text classification model corresponding to each data source by fine-tuning the parameters of BERT.

[0085] BERT is a natural language processing model, which includes an input layer, an output layer, and a hidden layer composed of multiple Transformers. That is to say, Transformers are the core modules that make up BERT, and the attention mechanism is the most crucial part of Transformers. Generally speaking, the semantic meaning expressed by a word or term in a text is related to its context. For example, just seeing the character "鹄" may not enable one to understand its meaning, but after seeing its context "鸿鹄之志", one can understand that "鹄" in this sentence means "swan". Therefore, the context information of words or terms helps to enhance their semantic representation. At the same time, different words or terms in the context often play different roles in enhancing the semantic representation. Continuing with the example of "鸿鹄之志", the character "鸿" has the greatest impact on understanding the character "鹄", while the character "之" has a relatively smaller impact. So, the attention mechanism is mainly used to distinguish the influence of each input word vector on text classification and focus the attention on a part of the word vectors that have a greater impact on text classification.

[0086] S202: Input the first text information into the first text classification model to obtain the corresponding first predicted classification result.

[0087] The first text classification model includes an input layer, an output layer, and a semantic mining sub-model. Among them, one data source corresponds to one input layer and one output layer, but the semantic mining sub-model is shared. In this way, the semantic mining sub-model can learn the characteristics of text data from different data sources, while improving the classification accuracy of the model, it can also reduce the number of parameters of the model, facilitating the deployment of the first text classification model in the application scenario. From the above introduction, it can be seen that determining the influence of the context information of words or terms on text classification is very important. Sequence learning models such as Transformers, LSTMs, and TextCNNs can be used to perform semantic mining processing on each word vector to obtain the corresponding semantic features.

[0088] First, input the first text information into the first text classification model, and perform word segmentation and vectorization processing on the first text information in sequence to obtain the corresponding word vector matrix.

[0089] First, clean the text information to remove redundant and invalid text information, then use a word segmentation tool to segment the text information, and finally use the random initialization method, or methods such as the bag-of-words method and Word2Vector to represent the text information in vector form to obtain the word vector corresponding to each word.

[0090] Secondly, perform semantic mining processing on each word vector based on the semantic mining sub-model to obtain the corresponding semantic features, where one semantic feature represents the confidence of a word vector in text classification;

[0091] Finally, the first predicted classification result is obtained based on each semantic feature.

[0092] (I) If the semantic mining sub-model is an attention model, the semantic mining process is as follows:

[0093] First, the first context vectors corresponding to each word vector are calculated based on the attention model, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a first context vector represents the correlation between the word vector and each word vector belonging to the same sentence;

[0094] Secondly, feature extraction is performed on each word vector and the corresponding first context vector to obtain the semantic features corresponding to each word vector.

[0095] The attention model in the embodiment of the present application is Transformer, see Figure 4a The Transformer structure diagram shown in the figure shows that Transformer usually consists of four parts: input layer, output layer, encoder and decoder. In the text classification task, only the input layer, output layer and encoder are used, and the encoder contains two parts: attention mechanism and feedforward neural network.

[0096] In the attention mechanism, see Figure 4b As shown, the word vector matrix obtained by converting the same sentence is used as the input matrix, and the linear transformation matrix W is used Q , W K , W V , get the query vector, key vector and value vector corresponding to each word vector. Figure 4c As shown in the figure, taking a word vector Y as an example, the process of determining its corresponding first context vector is to multiply the query vector with each key vector respectively, use the product result as the attention weight of the word vector Y to each word vector, and then multiply each value vector with the corresponding attention weight respectively, and output the product result as the first context vector. In the process of generating the first context vector, the value vector of the word vector Y and the value vectors of each word vector belonging to the same sentence are weighted and fused, reflecting the correlation between the word vector Y and each word vector belonging to the same sentence.

[0097] (II) If the semantic mining sub-model is a long short-term memory model, the semantic mining process is as follows:

[0098] First, the second context vectors corresponding to each word vector are calculated based on the long short-term memory model, where the context represents the semantics of a word corresponding to a word vector in a sentence, and a second context vector represents the correlation between a corresponding word vector and other word vectors from which the second context vector has been extracted;

[0099] Secondly, feature extraction is performed on each second context vector to obtain the semantic features corresponding to each word vector.

[0100] The long short-term memory model in the embodiment of the present application is LSTM, which is a special recurrent neural network (RNN). Figure 5 As shown in the structural diagram, LSTM includes an input layer, an output layer and a hidden layer. Multiple neurons are set on each hidden layer. Each word vector corresponds to a neuron. Each neuron includes a forget gate, an input gate, an output gate and a tanh layer. The gate structure is used to eliminate or increase the ability of information to neurons, so that each LSTM can remember long-term information and overcome the long-term dependency (Long-Term Dependencies) problem of traditional RNN.

[0101] RNN is a recurrent neural network, which is widely used in computer vision technology, natural language processing technology and other fields. The essential feature of this network is that there are both internal feedback connections and feedforward connections between processing units. Compared with feedforward neural networks, RNN has stronger dynamic behavior and computing power in the calculation process. It is precisely because of this characteristic of RNN that RNN is prone to long-term dependency problems - that is, the current system state is not only affected by the system state at the previous moment, but also by the system state a long time ago. For example, if RNN is asked to predict the next word from the sentence "Does this rock candy taste good?", it is easy to get the word "sweet"; if RNN is asked to predict the next word from the sentence "He took a bite of the dish, and tears came out of his eyes because of the spiciness, and his face turned red. The person next to him quickly poured him a glass of cold water, and he drank two gulps of it before gradually returning to normal. He said angrily: 'Does this dish taste good?'" It is difficult for RNN to output the word "spicy".

[0102] See also Figure 4d The structural diagram shown introduces the use of any LSTM neuron.

[0103] First, the second context vector h output by the previous neuron t-1 And the word vector x of the current neuron input t As input, the memory cell state C of the previous neuron is obtained through the forget gate t-1 The probability of being forgotten is f t , f t The calculation method of is shown in formula (1), where 1 means completely retained, and 0 means completely discarded.

[0104]

[0105] Secondly, the second context vector h output by the previous neuron t-1 And the word vector x of the current neuron input t As input, the memory cell state C' of the current neuron is obtained through the input gate t The probability of being retained t , and use i t Calculate the memory cell state C' of the current neuron t , specifically, i t The calculation formula of C' is shown in formula (2). t The calculation formula is shown in formula (3).

[0106] Using C t-1 and C' t Update the memory cell state of the current neuron, and the updated memory cell state is C t , C t The calculation formula of is shown in formula (4).

[0107]

[0108] C' t =tanh(W C *[h t-1 ,x t ]+b C ) formula (3);

[0109] C t =f t *C t-1 +i t *C' t Formula (4);

[0110] Finally, the second context vector h output by the previous neuron is t-1 And the word vector x of the current neuron input t As input, C is obtained through the output gate t The probability of the state of the memory cell being preserved is o t , o t The calculation formula of is shown in formula (5);

[0111] And, C t After being processed by the activation function tanh and o t Multiply to get the second context vector h of the current neural output t ,h t The calculation method of is shown in formula (6).

[0112]

[0113] h t =o t *tanh(C t ) formula (4);

[0114] (III) If the semantic mining sub-model is a convolutional model, the semantic mining process is as follows:

[0115] Based on the convolutional model, feature extraction is performed on each word vector to obtain the semantic features corresponding to each word vector.

[0116] The convolution model in the embodiment of the present application is TextCNN, see Figure 4e As shown in the structural diagram, TextCNN includes input layer, convolution layer, maximum pooling layer and Softmax layer. First, use the word segmentation tool to segment the text information, then use the random initialization method, or use the bag-of-words method, Word2Vector and other methods to represent the text information in vector form to obtain the word vector corresponding to each word; input the word vector matrix into the convolution layer, and the convolution kernels of each neuron on the convolution layer extract features from it respectively to obtain the corresponding semantic features; in order to reduce the risk of overfitting of the model and reduce the model parameters, and improve the calculation speed of the model, the maximum pooling layer can also be used to reduce the dimension of the semantic features; finally, the Softmax layer and various semantic features are used to obtain the prediction and classification results of the text information.

[0117] S203: Determine a total loss value based on a first loss value between the first predicted classification result and the first actual classification result, and a second loss value between the first predicted classification result and the second predicted classification result; and adjust parameters of the first text classification model based on the total loss value.

[0118] The first text classification model is parameter-adjusted based on the first loss value and the second loss value, wherein the first loss value is the error value between the first predicted classification result of the first text classification model and the first actual classification result, and the second loss value is the error value between the first predicted classification result and the second predicted classification result of the second text classification model. Since text data from different types of data sources often have similar text classification task requirements, in the training process, the output results of multiple second text classification models are used as first training samples to train the first text classification model, and the features of similar text classification tasks are shared in the first text classification model, which can accelerate the learning efficiency of the first text classification model, improve the generalization of the first text classification model, and reduce the structure of the first text classification model and the number of parameters of the model without significantly reducing the classification accuracy; in addition, adjusting the parameters of the first text classification model based on the two loss values ​​can make the first predicted classification result output by the first text classification model approach the first actual classification result and the second predicted classification result, so as to better learn the features of text data from different types of data sources and further improve the classification accuracy of the first text classification model.

[0119] Optionally, the process of adjusting parameters of the first text classification model based on the total loss value is as follows:

[0120] First, the total loss value is used to determine the gradient of the parameters on the first text classification model;

[0121] Secondly, the parameters of the first text classification model are adjusted according to the direction of the gradient.

[0122] S204: Determine whether the preset convergence condition is met, if so, execute step 205; otherwise, return to step 201.

[0123] The convergence conditions mentioned in step 204 and step 2014 are similar. The convergence condition mentioned in step 204 can be any one of the following:

[0124] When the total loss value converges to the preset expected value; or, the set number of iterations is reached; or, all first training samples are read.

[0125] The convergence condition mentioned in step 2041 may be any of the following:

[0126] When the third loss value converges to a preset expected value; or, the set number of iterations is reached; or, all second training samples are read.

[0127] S205: Output the trained first text classification model.

[0128] The trained first text classification model can recognize text data from different types of data sources. There is no need to train a corresponding classification model separately for each data source. While improving the classification accuracy of the model, it also reduces the number of model parameters, making it easier to deploy the trained first text classification model in application scenarios.

[0129] See also Figure 5 As shown in the flowchart, the process of classifying text using the trained first text classification model is as follows.

[0130] S501: Acquire text information to be recognized;

[0131] S502: Input the text information into the trained first text classification model to obtain a target word vector matrix corresponding to the text information;

[0132] S503: Perform semantic mining on each target word vector to obtain corresponding semantic features;

[0133] S504: Obtain a target predicted classification result of the text information based on each semantic feature; wherein the first text classification model is obtained after parameter adjustment based on a first loss value and a second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between two predicted classification results.

[0134] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a structural schematic diagram of a text classification device. Figure 6 As shown, the text classification device 600 may include:

[0135] An acquisition unit 601 is used to acquire text information to be recognized;

[0136] A processing unit 602 is used to input the text information into a trained first text classification model to obtain a target word vector matrix corresponding to the text information;

[0137] Perform semantic mining on each target word vector to obtain the corresponding semantic features;

[0138] Based on each semantic feature, obtaining a target prediction classification result of the text information;

[0139] Among them, the first text classification model is obtained after parameter adjustment based on the first loss value and the second loss value, the first loss value is the error value between the predicted classification result and the actual classification result, and the second loss value is the error value between two predicted classification results.

[0140] Optionally, the device further includes a training unit 603, and the training unit 603 is specifically configured to:

[0141] Acquire a first training sample from a first training data set, wherein the first training data set includes first training samples from different types of data sources, wherein one first training sample includes first text information, a first actual classification result of the first text information, and a second predicted classification result outputted after inputting the first text information into a second text classification model; wherein each first text information corresponds to a type of data source, and each type of data source corresponds to a second text classification model;

[0142] Iteratively training the first text classification model based on the obtained first training sample until a preset convergence condition is met, and outputting the trained first text classification model, wherein one iteration process includes:

[0143] Inputting the first text information into the first text classification model to obtain a corresponding first prediction classification result;

[0144] Determining a total loss value based on a first loss value between the first predicted classification result and the first actual classification result, and a second loss value between the first predicted classification result and the second predicted classification result;

[0145] Parameters of the first text classification model are adjusted based on the total loss value.

[0146] Optionally, the apparatus further includes a training data generating unit 604, wherein the training data generating unit 604 is specifically configured to:

[0147] Acquire a second training sample from a second training data set, wherein the second training data set includes second training samples from the same data source, and a second training sample includes second text information and a second actual classification result of the second text information;

[0148] Iteratively train a second text classification model based on the obtained second training sample until a preset convergence condition is met, and output a trained second text classification model, wherein one iteration process includes:

[0149] Inputting the second text information into the one second text classification model to obtain a corresponding third prediction classification result;

[0150] Based on a third loss value between the third predicted classification result and the second actual classification result, parameters of the second text classification model are adjusted.

[0151] Optionally, the second text classification model includes a semantic mining sub-model, and the training unit 603 is used to:

[0152] Inputting the text information contained in the first training sample into the second text classification model, performing word segmentation and vectorization processing on the text information in sequence, and obtaining a corresponding word vector matrix;

[0153] Using the semantic mining sub-model to perform semantic mining processing on each word vector to obtain corresponding semantic features, wherein a semantic feature represents the confidence of a word vector for text classification;

[0154] Each second prediction classification result is obtained based on each semantic feature.

[0155] Optionally, if the semantic mining sub-model is an attention model, the training unit 603 is used to:

[0156] Calculate first context vectors corresponding to each word vector using the attention model, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a first context vector represents the correlation between the word vector and each word vector belonging to the same sentence;

[0157] Feature extraction is performed on each of the word vectors and the corresponding first context vector to obtain semantic features corresponding to each of the word vectors.

[0158] Optionally, if the semantic mining sub-model is a long short-term memory unit, the training unit 603 is used to:

[0159] Calculate the second context vectors corresponding to the word vectors respectively using the long short-term memory model, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a second context vector represents the correlation between a corresponding word vector and other word vectors from which the second context vector has been extracted;

[0160] Feature extraction is performed on each second context vector to obtain semantic features corresponding to each word vector.

[0161] Optionally, if the semantic mining sub-model is a convolutional model, the training unit 603 is used to:

[0162] The convolution model is used to perform feature extraction on each word vector to obtain semantic features corresponding to each word vector.

[0163] Optionally, the training unit 603 is used to:

[0164] Determine the gradient of the parameters of the second text classification model using the total loss value;

[0165] Parameters of the second text classification model are adjusted according to the direction of the gradient.

[0166] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0167] After introducing the method and device for training a text classification model according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0168] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0169] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application, referring to Figure 7 As shown, the electronic device 700 may include at least a processor 701 and a memory 702. The memory 702 stores program codes, and when the program codes are executed by the processor 701, the processor 701 executes the steps of any one of the above-mentioned text classification methods.

[0170] In some possible implementations, the computing device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor performs the steps of text classification according to various exemplary implementations of the present application described above in this specification. For example, the processor may perform the following steps: Figure 5 Follow the steps shown in .

[0171] Refer to the following Figure 8 800 according to this embodiment of the present application. Figure 8 As shown, the computing device 800 is in the form of a general computing device. The components of the computing device 800 may include but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the storage unit 802 and the processing unit 801).

[0172] Bus 803 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0173] The storage unit 802 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 8021 and / or a cache memory unit 8022 , and may further include a read-only memory (ROM) 8023 .

[0174] The storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0175] The computing device 800 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the computing device 800, and / or communicate with any device that enables the computing device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 805. In addition, the computing device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 806. As shown, the network adapter 806 communicates with other modules for the computing device 800 via a bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computing device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0176] Based on the same inventive concept as the above method embodiment, various aspects of the text classification method provided by the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the text classification method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device can execute the following steps: Figure 5 Follow the steps shown in .

[0177] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0178] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0179] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A text classification method, characterized in that: The method comprises: Obtain text information to be recognized; Inputting the text information into a trained first text classification model to obtain a target word vector matrix corresponding to the text information; Perform semantic mining on each target word vector to obtain the corresponding semantic features; Based on each semantic feature, obtaining a target prediction classification result of the text information; The first text classification model is obtained after parameter adjustment based on a first loss value and a second loss value, the first loss value is an error value between a first predicted classification result of a first training sample predicted by the first text classification model and a first actual classification result of the corresponding training sample, and the second loss value is an error value between two predicted classification results predicted by two text classification models for the same first training sample; Among them, the second text classification model is trained iteratively using the second training data set until a preset convergence condition is met, and a trained second text classification model is obtained, and each iteration includes: obtaining a second training sample of the second training data set, the second training data set contains second training samples from the same data source, and a second training sample contains second text information and a second actual classification result of the second text information; obtaining a third predicted classification result of the second text information predicted by the second text classification model, and adjusting parameters of the second text classification model based on a third loss value between the third predicted classification result and the second actual classification result.

2. The method according to claim 1, characterized in that The first text classification model is obtained after adjusting parameters based on the first loss value and the second loss value, and specifically includes: Acquire a first training sample from a first training data set, wherein the first training data set includes first training samples from different types of data sources, wherein one first training sample includes first text information, a first actual classification result of the first text information, and a second predicted classification result outputted after inputting the first text information into a second text classification model; wherein each first text information corresponds to a type of data source, and each type of data source corresponds to a second text classification model; Iteratively training the first text classification model based on the obtained first training sample until a preset convergence condition is met, and outputting the trained first text classification model, wherein one iteration process includes: Inputting the first text information into the first text classification model to obtain a corresponding first prediction classification result; Determining a total loss value based on a first loss value between the first predicted classification result and the first actual classification result, and a second loss value between the first predicted classification result and the second predicted classification result; Parameters of the first text classification model are adjusted based on the total loss value.

3. The method according to claim 2, characterized in that If the first text classification model includes a semantic mining sub-model, inputting the first text information into the first text classification model to obtain a corresponding first prediction classification result includes: Inputting the first text information into the first text classification model, performing word segmentation and vectorization processing on the first text information in sequence, and obtaining a corresponding word vector matrix; Performing semantic mining processing on each of the word vectors based on the semantic mining sub-model to obtain corresponding semantic features, wherein a semantic feature represents the confidence of a word vector for text classification; The first prediction classification result is obtained based on each semantic feature.

4. The method according to claim 3, characterized in that If the semantic mining sub-model is an attention model, semantic mining processing is performed on each word vector based on the semantic mining sub-model to obtain corresponding semantic features, including: Based on the attention model, first context vectors corresponding to the word vectors are respectively calculated, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a first context vector represents the correlation between the word vector and the word vectors belonging to the same sentence; Feature extraction is performed on each of the word vectors and the corresponding first context vector to obtain semantic features corresponding to each of the word vectors.

5. The method according to claim 3, characterized in that If the semantic mining sub-model is a long short-term memory unit, semantic mining processing is performed on each word vector based on the semantic mining model to obtain corresponding semantic features, including: Based on the long short-term memory model, second context vectors corresponding to the word vectors are respectively calculated, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a second context vector represents the correlation between a corresponding word vector and other word vectors from which the second context vector has been extracted; Feature extraction is performed on each second context vector to obtain semantic features corresponding to each word vector.

6. The method according to claim 3, characterized in that If the semantic mining sub-model is a convolutional model, semantic mining processing is performed on each word vector based on the semantic mining sub-model to obtain corresponding semantic features, including: The convolution model is used to perform feature extraction on each word vector to obtain semantic features corresponding to each word vector.

7. The method according to claim 2, characterized in that Adjusting parameters of the first text classification model based on the total loss value includes: Determine the gradient of the parameters on the first text classification model based on the total loss value; Parameters of the first text classification model are adjusted according to the direction of the gradient.

8. A text classification device, characterized in that: The device comprises: An acquisition unit, used for acquiring text information to be recognized; A processing unit, configured to input the text information into a trained first text classification model to obtain a target word vector matrix corresponding to the text information; Perform semantic mining on each target word vector to obtain the corresponding semantic features; Based on each semantic feature, obtaining a target prediction classification result of the text information; The first text classification model is obtained after parameter adjustment based on a first loss value and a second loss value, the first loss value is an error value between a first predicted classification result of a first training sample predicted by the first text classification model and a first actual classification result of the corresponding training sample, and the second loss value is an error value between two predicted classification results predicted by two text classification models for the same first training sample; Among them, the second text classification model is trained iteratively using the second training data set until a preset convergence condition is met, and a trained second text classification model is obtained, and each iteration includes: obtaining a second training sample of the second training data set, the second training data set contains second training samples from the same data source, and a second training sample contains second text information and a second actual classification result of the second text information; obtaining a third predicted classification result of the second text information predicted by the second text classification model, and adjusting parameters of the second text classification model based on a third loss value between the third predicted classification result and the second actual classification result.

9. The device according to claim 8, characterized in that The device further comprises a training unit, which is specifically used for: Acquire a first training sample from a first training data set, wherein the first training data set includes first training samples from different types of data sources, wherein one first training sample includes first text information, a first actual classification result of the first text information, and a second predicted classification result outputted after inputting the first text information into a second text classification model; wherein each first text information corresponds to a type of data source, and each type of data source corresponds to a second text classification model; Iteratively training the first text classification model based on the obtained first training sample until a preset convergence condition is met, and outputting the trained first text classification model, wherein one iteration process includes: Inputting the first text information into the first text classification model to obtain a corresponding first prediction classification result; Determining a total loss value based on a first loss value between the first predicted classification result and the first actual classification result, and a second loss value between the first predicted classification result and the second predicted classification result; Parameters of the first text classification model are adjusted based on the total loss value.

10. The device according to claim 9, characterized in that If the first text classification model includes a semantic mining sub-model, the training unit is used to: Inputting the first text information into the first text classification model, performing word segmentation and vectorization processing on the first text information in sequence, and obtaining a corresponding word vector matrix; Performing semantic mining processing on each of the word vectors based on the semantic mining sub-model to obtain corresponding semantic features, wherein a semantic feature represents the confidence of a word vector for text classification; The first prediction classification result is obtained based on each semantic feature.

11. The device according to claim 10, characterized in that If the semantic mining sub-model is an attention model, the training unit is used to: Based on the attention model, first context vectors corresponding to the word vectors are respectively calculated, wherein the context represents the semantics of a word corresponding to a word vector in a sentence, and a first context vector represents the correlation between the word vector and the word vectors belonging to the same sentence; Feature extraction is performed on each of the word vectors and the corresponding first context vector to obtain semantic features corresponding to each of the word vectors.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of any one of the methods in claims 1 to 7.

13. A computer-readable storage medium, characterized in that: The method comprises a program code. When the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of any method described in claims 1 to 7.

Images