Multi-level multi-class text classification method based on CBOW and skip-gram word vectors

Abstract

The present application discloses a multi-level multi-class text classification method based on CBOW and skip-gram word vectors. The present method comprises: constructing a patent text dataset containing multiple levels of labels, so as to provide rich annotated data for subsequent classification; using a CBOW model to preprocess a text to generate word frequency vectors, so as to reduce data dimensionality and extract key information; then aggregating the word frequency vectors by means of a BiLSTM network; capturing long-range dependencies in the text; using a skip-gram model to extract semantic features of the text; and finally performing weighted averaging on text representation vectors generated by BiLSTM and skip-gram so as to obtain a final representation of the text, the final representation being used for training a multi-level classification model. In the technology, by combining context information and semantic information, the accuracy of text classification is effectively improved, and the classification efficiency is improved.

Description

A multi-level multi-class text classification method based on CBOW and skip-gram word vectors

[0001] The present application claims priority to the Chinese patent application No. CN202411386146.4, filed on September 30, 2024, and entitled "A multi-level multi-class text classification method based on CBOW and skip-gram word vectors", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of natural language processing, and specifically relates to a multi-level multi-class text classification method based on CBOW and skip-gram word vectors. BACKGROUND

[0003] CBOW (Continuous Bag of Words) model is a neural network model for generating word vectors. The core idea of CBOW model is to predict the center word using the context or surrounding words. For a selected context, CBOW model will add (or use other aggregation methods) the word vectors of each word in the context to obtain a context vector. This context vector represents the semantic information of the current text.

[0004] Skip-gram technology is a method for learning word embeddings in the field of natural language processing (NLP). The core idea of Skip-gram model is to predict the center word based on the context, that is, given a word, the model tries to predict the context words around this word. For each word in the text, it is taken as the center word, and a certain number of words around it are selected as the context words. These center word and context word pairs constitute the training samples of the Skip-gram model. Use the Skip-gram model to train the training samples to learn the word vectors of each word. During training, the model tries to maximize the joint probability between the center word and its context words. Once the model is trained, the word vectors of each word can be used to represent the entire text. A common practice is to add (or take the average) the word vectors of all words in the text to obtain a fixed-length vector as the representation of the text.

[0005] Text classification technology is widely used in information retrieval, sentiment analysis, spam filtering and other fields. Traditional text classification methods mainly rely on keyword matching and simple statistical methods. With the development of deep learning, text classification methods based on neural networks have gradually become the mainstream. However, existing text classification techniques still have problems such as low classification accuracy and low training efficiency when dealing with large-scale, multi-level and multi-classification data. The current text classification algorithm is not efficient and accurate enough for the classification needs of patent text, which is a large amount of multi-level type.

[0006] Application content

[0007] The present application provides a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors to improve the accuracy and efficiency of patent text classification and meet the needs of practical applications.

[0008] In a first aspect, a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors is provided, the method comprising:

[0009] S1, constructing a patent text dataset containing multi-level labels;

[0010] S2, performing preliminary processing on the target patent text in the patent text dataset through the CBOW model to generate a word frequency vector, reducing the data dimension and extracting the key information of the target patent text;

[0011] S3, aggregating the generated word frequency vector through a BiLSTM network; wherein the BiLSTM processes the vector sequence of the context words and obtains the hidden state of the last time step as the aggregation result of each word frequency vector to represent the target patent text;

[0012] S4, extracting the semantic features of the target patent text through the skip-gram word vector model;

[0013] S5, weighting and averaging the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4 to obtain the final representation of the target patent text, and training the multi-level classification model of the patent text through the final representation of the target patent text.

[0014] Further, the construction of the patent text dataset containing multi-level labels in S1 further comprises:

[0015] Collecting patent text data, assigning one or more hierarchical labels to each text according to the category, subcategory and technical field of the patent; wherein the hierarchical labels include a pre-defined classification system or automatically generated labels based on the text content.

[0016] Further, in S2, the CBOW model is used to preliminarily process the target patent text in the patent text data set to generate a word frequency vector, reduce the data dimension, and extract the key information of the target patent text, and further includes:

[0017] The target patent text is subjected to word segmentation processing to decompose the text into words or lexical units;

[0018] The CBOW model is used to train the vector representation of each word; wherein the CBOW model predicts the center word according to the context words to learn the vector of each word;

[0019] After training, each word in the text is replaced by its corresponding CBOW vector, and these vectors are aggregated to form a word frequency vector representation of the text, thereby reducing the data dimension and extracting key information.

[0020] Further, in S3, the BiLSTM network is used to aggregate the generated word frequency vector, and further includes:

[0021] The CBOW-generated word frequency vector sequence is input into the BiLSTM model to process the forward and reverse context information simultaneously.

[0022] Further, in S4, the skip-gram word vector model is used to extract features from the target patent text to extract semantic features of the target patent text, and further includes:

[0023] The skip-gram model is used to train each word in the text to learn the vector representation of the word;

[0024] After training, the skip-gram vectors of all words in the text are aggregated to obtain a semantic feature vector of the text.

[0025] Further, in S5, the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4 are weighted and averaged to obtain the final representation of the target patent text, and further includes:

[0026] The weights of the text representation vectors generated by BiLSTM and skip-gram are determined;

[0027] After determining the weights, the two representation vectors are weighted and averaged to obtain a text representation vector that integrates context information and semantic information;

[0028] The text representation vector is used as an input feature to train a multi-level classification model. During the training process, the cross-entropy loss function and the RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect.

[0029] In a second aspect, a multi-level multi-classification text classification system based on CBOW and skip-gram word vectors is provided, and the system comprises:

[0030] A construction module is configured to construct a patent text dataset comprising multi-level labels.

[0031] A CBOW module is configured to perform preliminary processing on target patent text in the patent text dataset by a CBOW model, generate a word frequency vector, reduce data dimensions, and extract key information of the target patent text.

[0032] A BiLSTM module is configured to aggregate the generated word frequency vector by a BiLSTM network. The BiLSTM is used to process a vector sequence of context words and obtain a hidden state at a last time step as an aggregation result of each word frequency vector to represent the target patent text.

[0033] A skip-gram module is configured to extract semantic features of the target patent text by a skip-gram word vector model.

[0034] A target training module is configured to obtain a final representation of the target patent text by weighted averaging the aggregation result of the target patent text and the semantic features of the target patent text, and train a multi-level classification model of the patent text based on the final representation of the target patent text.

[0035] In a third aspect, an electronic device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the multi-level multi-classification text classification method of any one of the first aspect when executing the computer program.

[0036] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the multi-level multi-classification text classification method of any one of the first aspect.

[0037] In a fifth aspect, a computer program product is provided, comprising computer programs / instructions, which are executed by a processor to implement the multi-level multi-classification text classification method of any one of the first aspect.

[0038] The present application has the following advantages:

[0039] (1) Based on the multi-level label patent text dataset established by the researchers of the present case, the combination of the CBOW model and the skip-gram word vector effectively improves the accuracy of text classification.

[0040] (2) The multi-level classification method can adapt to large-scale, multi-level, and multi-classification patent data, and improves the classification efficiency. Attached Figure Description

[0041] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0042] The structures, proportions, sizes, etc. illustrated in this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed herein, and are not intended to limit the conditions under which this application can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and purposes that this application can produce, should still fall within the scope of the technical content disclosed in this application.

[0043] Figure 1 is a flowchart of the steps of a multi-level, multi-class text classification method based on CBOW and skip-gram word vectors provided in an embodiment of this application;

[0044] Figure 2 is a block diagram of a multi-level, multi-class text classification system based on CBOW and skip-gram word vectors provided in an embodiment of this application;

[0045] Figure 3 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In the description of this application, the terms “comprising,” “having,” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0048] The application aims to provide a multi-level multi-classification text classification technology based on CBOW and skip-gram word vectors, so as to improve the precision and efficiency of patent text classification and meet the needs of practical applications.

[0049] Please refer to FIG. 1, which shows a flowchart of a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors provided by the embodiments of the application, which can include the following steps:

[0050] S1, constructing a patent text dataset containing multi-level labels.

[0051] In this step, patent text data is collected, and one or more level labels are assigned to each text according to the category, subcategory and technical field of the patent; wherein the level label includes a pre-defined classification system or a label automatically generated based on the text content.

[0052] S2, performing preliminary processing on the target patent text in the patent text dataset through the CBOW model to generate a word frequency vector, reduce the data dimension and extract the key information of the target patent text.

[0053] In this step, the target patent text is processed by word segmentation, and the text is decomposed into words or lexical units;

[0054] The CBOW model is used to train the vector representation of each word; wherein the CBOW model predicts the center word according to the context word, thereby learning the vector of each word;

[0055] After training, each word in the text is replaced by its corresponding CBOW vector, and these vectors are aggregated to form the word frequency vector representation of the text, thereby reducing the data dimension and extracting the key information.

[0056] S3, aggregating the generated word frequency vector through the BiLSTM network.

[0057] Wherein, the BiLSTM is used to process the vector sequence of the context word, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text.

[0058] In this step, the CBOW generated word frequency vector sequence is input into the BiLSTM model, and the BiLSTM model processes the forward and reverse context information at the same time.

[0059] S4, extracting the semantic features of the target patent text through the skip-gram word vector model.

[0060] In this step, the skip-gram model is used to train each word in the text to learn the vector representation of the word; after the training is completed, the skip-gram vectors of all words in the text are aggregated to obtain the semantic feature vector of the text.

[0061] S5, the aggregation result of the target patent text in S3 and the semantic feature of the target patent text in S4 are weighted and averaged to obtain the final representation of the target patent text, and the multi-level classification model training of the patent text is performed through the final representation of the target patent text.

[0062] In this step, the weights of the text representation vectors generated by BiLSTM and skip-gram are determined; after the weights are determined, the two representation vectors are weighted and averaged to obtain a text representation vector that integrates context information and semantic information; the text representation vector is used as an input feature to train a multi-level classification model, and in the training process, the cross-entropy loss function and the RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect.

[0063] In the optional embodiment of the present application, the following technical means and measures are used:

[0064] (1) A patent text data set containing multi-level labels is established;

[0065] (2) For a given patent text, the CBOW model is used to preprocess the text to convert the text into a word frequency vector, reducing the dimensionality of the text data;

[0066] (3) The CBOW vectors of each word are aggregated by means of BiLSTM, that is, the vector sequence of the context words is processed by BiLSTM, and the hidden state of the last time step is obtained as the aggregation result of the CBOW word vectors to represent the text;

[0067] (4) The skip-gram word vector model is used to extract features from the given patent text data, obtain the semantic features of the text, and add the word vectors of each word to obtain a fixed-length vector to represent the entire text;

[0068] (5) The text representation vectors calculated in (3) and (4) are weighted and averaged to obtain the final representation of the text, so as to train the multi-level classification model of the patent text.

[0069] Please refer to FIG. 2, which shows a block diagram of a multi-level multi-classification text classification system based on CBOW and skip-gram word vectors according to an embodiment of the present application, which includes:

[0070] A construction module is configured to construct a patent text data set containing multi-level labels;

[0071] a CBOW module, configured to perform preliminary processing on the target patent text in the patent text dataset by a CBOW model, to generate a word frequency vector, to reduce data dimension, and to extract key information of the target patent text;

[0072] a BiLSTM module, configured to aggregate the generated word frequency vector by a BiLSTM network; wherein the BiLSTM is used to process a vector sequence of context words, and a hidden state of a last time step is obtained as an aggregation result of each word frequency vector to represent the target patent text;

[0073] a skip-gram module, configured to perform feature extraction on the target patent text by a skip-gram word vector model, to extract semantic features of the target patent text;

[0074] a target training module, configured to perform weighted average on the aggregation result of the target patent text and the semantic features of the target patent text to obtain a final representation of the target patent text, and to perform training of a multi-level classification model of patent text based on the final representation of the target patent text.

[0075] The specific definitions of the multi-level multi-classification text classification system can be referred to the definitions of the multi-level multi-classification text classification method based on CBOW and skip-gram word vectors in the above, which will not be repeated here. Each module in the above multi-level multi-classification text classification system can be realized by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0076] In one embodiment, an electronic device, which can be a computer, is provided, and an internal structure diagram of the electronic device can be as shown in FIG. 3. The electronic device includes a processor, a memory, and a network interface connected by a system bus. The processor of the device is configured to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for multi-level multi-classification text classification data. The network interface of the computer device is used for communication connection with external terminals through a network. The computer program is executed by the processor to implement a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors.

[0077] Those skilled in the art can understand that the structure as shown in FIG. 3 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In an embodiment of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of the above method for multi-level multi-class text classification based on CBOW and skip-gram word vectors.

[0079] In an embodiment of the present application, a computer program product is provided, and the computer program product includes computer programs / instructions. The computer programs / instructions are executed by a processor to implement the steps of the above method for multi-level multi-class text classification based on CBOW and skip-gram word vectors.

[0080] The computer readable storage medium and the computer program product provided in the embodiment have similar implementation principles and technical effects to the above method embodiments, and thus will not be described here.

[0081] Although the present application has been described in detail above with general description and specific embodiments, some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application shall fall within the scope of the present application.

Claims

1. A multi-level multi-class text classification method based on CBOW and skip-gram word vectors, characterized in that, The method comprises: S1, constructing a patent text data set containing multi-level labels; S2, performing preliminary processing on the target patent text in the patent text data set through a CBOW model to generate a word frequency vector, reduce the data dimension, and extract key information of the target patent text; S3, aggregating the generated word frequency vector through a BiLSTM network; wherein the BiLSTM processes the vector sequence of the context word and obtains the hidden state of the last time step as the aggregation result of each word frequency vector to represent the target patent text; S4, extracting the semantic features of the target patent text through a skip-gram word vector model; S5, obtaining the final representation of the target patent text by weighted averaging the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4, and training a multi-level classification model of the patent text through the final representation of the target patent text.

2. The multi-level multi-class text classification method of claim 1, wherein, In S1, the patent text data set containing multi-level labels is also constructed, which further comprises: Collecting patent text data, and assigning one or more hierarchical labels to each text according to the category, subcategory, and technical field of the patent; wherein the hierarchical labels include a pre-defined classification system or labels automatically generated based on the text content.

3. The multi-level multi-class text classification method of claim 1, wherein, In S2, the preliminary processing of the target patent text in the patent text data set through the CBOW model to generate the word frequency vector, reduce the data dimension, and extract the key information of the target patent text further comprises: Performing word segmentation processing on the target patent text to decompose the text into words or lexical units; Using the CBOW model to train the vector representation of each word; wherein the CBOW model predicts the center word based on the context word to learn the vector of each word; After training, replace each word in the text with its corresponding CBOW vector, and aggregate these vectors to form the word frequency vector representation of the text, thereby reducing the data dimension and extracting the key information.

4. The multi-level multi-class text classification method of claim 1, wherein, In S3, the aggregation of the generated word frequency vector through the BiLSTM network further comprises: Inputting the CBOW-generated word frequency vector sequence into the BiLSTM model to process the forward and reverse context information simultaneously through the BiLSTM model.

5. The multi-level multi-class text classification method of claim 1, wherein, In S4, the feature extraction of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text further comprises: Using the skip-gram model to train each word in the text to learn the vector representation of the word; After training, aggregate the skip-gram vectors of all words in the text to obtain the semantic feature vector of the text.

6. The multi-level multi-class text classification method of claim 1, wherein, In S5, the weighted averaging of the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4 to obtain the final representation of the target patent text further comprises: Determining the weights of the text representation vectors generated by BiLSTM and skip-gram; After determining the weights, weighted average the two representation vectors to obtain a text representation vector that integrates context information and semantic information; The text representation vector is taken as an input feature to train a multi-level classification model, and in the training process, a cross-entropy loss function and an RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect. The system comprises:

7. A multi-level multi-class text classification system based on CBOW and skip-gram word vectors, characterized in that, A construction module for constructing a patent text data set containing multi-level labels; A CBOW module for performing preliminary processing on target patent text in the patent text data set through a CBOW model to generate a word frequency vector, reduce data dimensions, and extract key information of the target patent text; A BiLSTM module for aggregating the generated word frequency vector through a BiLSTM network; wherein the BiLSTM processes a vector sequence of context words and obtains a hidden state at the last time step as an aggregation result of each word frequency vector to represent the target patent text; A skip-gram module for extracting semantic features of the target patent text through a skip-gram word vector model; A target training module for performing weighted averaging on the aggregation result of the target patent text and the semantic features of the target patent text to obtain a final representation of the target patent text, and training a multi-level classification model for patent text based on the final representation of the target patent text. A memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to implement the multi-level multi-classification text classification method of any one of claims 1-6.

8. An electronic device, comprising: A computer program is stored thereon, and the computer program is executed by a processor to implement the multi-level multi-classification text classification method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer program / instructions are executed by the processor to implement the multi-level multi-classification text classification method of any one of claims 1-6.

10. A computer program product comprising computer programs / instructions, characterized in that, ​

Images