Multi-level multi-classification text classification method based on CBOW and skip-gram word vectors
By combining the multi-level and multi-class classification method of CBOW, Bilstm, and SKIP-Gram models, the problems of large-scale, multi-level, and multi-class classification text category accuracy and low training efficiency in existing technologies have been solved. Accurate text classification.
Patent Information
- Application Number
- CN202411386146.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-21
AI Technical Summary
When existing text classification technology processes large-scale, multi-level, and multi-class data, the classification accuracy is not high and the training efficiency is low, making it difficult to meet the massive multi-level type classification needs of patent texts.
Adopt a multi-level and multi-class classification method based on CBOW and SKIP-GRAM word vector, generate frequency vector of word vector through the CBOW model, the Bilstm network aggregates vector, the SKIP-GRAM model extract semantic characteristics, and the average weighted text is obtained by the final representation of the text. Used to train multi -level classification models.
It improves the accuracy and efficiency of text classification, can adapt to large -scale, multi -level, and multi -class patented data, which significantly improves the classification efficiency.
Smart Images
Figure CN119336908A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a multi-level and multi-classification text classification method based on CBOW and skip-gram word vectors. Background Art
[0002] The CBOW (Continuous Bag of Words) model is a neural network model for generating word vectors. The core idea of the CBOW model is to use the context or surrounding words to predict the central word. For the selected context, the CBOW model adds the word vectors of each word in the context (or uses other aggregation methods) to obtain a context vector. This context vector represents the semantic information of the current text.
[0003] Skip-gram technology is a method for learning word embeddings in the field of natural language processing (NLP). The core idea of the Skip-gram model is to predict the central word based on the context, that is, given a word, the model tries to predict the context words around the word. For each word in the text, it is used as the central word, and a certain number of words around it are selected as context words. These central word and context word pairs constitute the training samples of the Skip-gram model. The training samples are trained using the Skip-gram model to learn the word vector of each word. During the training process, the model tries to maximize the joint probability between the central word and its context words. Once the model is trained, the word vector of each word can be used to represent the entire text. A common practice is to add (or average) the word vectors of all words in the text to obtain a fixed-length vector as a representation of the text.
[0004] Text classification technology is widely used in information retrieval, sentiment analysis, spam filtering and other fields. Traditional text classification methods mainly rely on keyword matching and simple statistical methods. With the development of deep learning, text classification methods based on neural networks have gradually become mainstream. However, when dealing with large-scale, multi-level, and multi-classified data, existing text classification technologies still have problems such as low classification accuracy and low training efficiency. The current text classification algorithms are not efficient and accurate enough for the classification needs of massive, multi-level types such as patent texts. Summary of the invention
[0005] Based on this, the embodiment of the present application provides a multi-level and multi-classification text classification method based on CBOW and skip-gram word vectors to improve the accuracy and efficiency of patent text classification and meet the needs of practical applications.
[0006] In a first aspect, a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors is provided, the method comprising:
[0007] S1, construct a patent text dataset containing multi-level labels;
[0008] S2, preliminarily processes the target patent text in the patent text dataset through the CBOW model, generates word frequency vectors, reduces the data dimension and extracts the key information of the target patent text;
[0009] S3, aggregate the generated word frequency vectors through the BiLSTM network; wherein, the vector sequence of the context words is processed using the BiLSTM, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text;
[0010] S4, extracting features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text;
[0011] S5, performing weighted averaging of the aggregation results of the target patent text in S3 and the semantic features of the target patent text in S4 to obtain the final representation of the target patent text, and performing multi-level classification model training of the patent text through the final representation of the target patent text.
[0012] Optionally, the patent text dataset containing multi-level labels is constructed in S1, and also includes:
[0013] Patent text data is collected, and one or more hierarchical labels are assigned to each text according to the patent category, subcategory, and technical field; the hierarchical labels include a predefined classification system or labels automatically generated based on the text content.
[0014] Optionally, S2 uses the CBOW model to preliminarily process the target patent text in the patent text dataset, generate a word frequency vector, reduce the data dimension and extract key information of the target patent text, and also includes:
[0015] Perform word segmentation on the target patent text and decompose the text into words or vocabulary units;
[0016] Use the CBOW model to train and get the vector representation of each word. The CBOW model predicts the central word based on the context words, thereby learning the vector of each word.
[0017] After training, each word in the text is replaced with its corresponding CBOW vector, and these vectors are aggregated to form a word frequency vector representation of the text, thereby reducing the data dimension and extracting key information.
[0018] Optionally, S3 aggregates the generated word frequency vectors through a BiLSTM network, and also includes:
[0019] The word frequency vector sequence generated by CBOW is input into the BiLSTM model, and the BiLSTM model processes the forward and reverse context information at the same time.
[0020] Optionally, S4 uses a skip-gram word vector model to extract features of the target patent text to extract semantic features of the target patent text, and further includes:
[0021] Use the skip-gram model to train each word in the text and learn the vector representation of the word;
[0022] After training, the skip-gram vectors of all words in the text are aggregated to obtain the semantic feature vector of the text.
[0023] Optionally, in S5, the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4 are weighted averaged to obtain the final representation of the target patent text, and further includes:
[0024] Determine the weights of the text representation vectors generated by BiLSTM and skip-gram;
[0025] After determining the weights, the two representation vectors are weighted averaged to obtain a text representation vector that integrates contextual information and semantic information;
[0026] The text representation vector is used as the input feature to train the multi-level classification model. During the training process, the cross entropy loss function and RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect.
[0027] In the second aspect, a multi-level multi-classification text classification system based on CBOW and skip-gram word vectors is provided, and the system includes:
[0028] A building module for constructing a patent text dataset containing multi-level labels;
[0029] The CBOW module is used to perform preliminary processing on the target patent text in the patent text dataset through the CBOW model, generate word frequency vectors, reduce data dimensions, and extract key information of the target patent text;
[0030] A BiLSTM module is used to aggregate the generated word frequency vectors through a BiLSTM network. The BiLSTM is used to process the vector sequence of the context words, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text.
[0031] The skip-gram module is used to extract features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text;
[0032] The target training module is used to perform weighted averaging of the aggregation results of the target patent text and the semantic features of the target patent text to obtain the final representation of the target patent text, and to train the multi-level classification model of the patent text through the final representation of the target patent text.
[0033] In a third aspect, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-level and multi-classification text classification method described in any one of the first aspects is implemented.
[0034] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the multi-level and multi-category text classification method described in any one of the first aspects is implemented.
[0035] In a fifth aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the multi-level and multi-classification text classification method described in any one of the first aspects.
[0036] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0037] (1) Based on the multi-level labeled patent text dataset established by the researchers in this case, the accuracy of text classification was effectively improved by combining the CBOW model and skip-gram word vectors;
[0038] (2) The multi-level classification method can adapt to large-scale, multi-level, and multi-classification patent data, thereby improving classification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.
[0040] Figure 1 A flowchart of a multi-level and multi-classification text classification method based on CBOW and skip-gram word vectors provided in an embodiment of the present application;
[0041] Figure 2A block diagram of a multi-level and multi-classification text classification system based on CBOW and skip-gram word vectors provided in an embodiment of the present application;
[0042] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0044] In the description of the present invention, the terms "comprises", "has" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may also include other steps or units that are not explicitly listed but are inherent to these processes, methods, products or apparatuses, or steps or units added based on further optimization schemes conceived by the present invention.
[0045] The present invention aims to provide a multi-level and multi-classification text classification technology based on CBOW and skip-gram word vectors to improve the accuracy and efficiency of patent text classification and meet the needs of practical applications.
[0046] Please refer to Figure 1 , which shows a flowchart of a multi-level and multi-classification text classification method based on CBOW and skip-gram word vectors provided in an embodiment of the present application, which may include the following steps:
[0047] S1, build a patent text dataset containing multi-level labels.
[0048] In this step, patent text data is collected, and one or more hierarchical labels are assigned to each text according to the category, subcategory, and technical field of the patent; wherein the hierarchical labels include a predefined classification system, or labels automatically generated based on the text content.
[0049] S2, preliminarily processes the target patent text in the patent text dataset through the CBOW model, generates word frequency vectors, reduces the data dimension and extracts the key information of the target patent text.
[0050] In this step, the target patent text is segmented to decompose the text into words or vocabulary units;
[0051] Use the CBOW model to train and get the vector representation of each word. The CBOW model predicts the central word based on the context words, thereby learning the vector of each word.
[0052] After training, each word in the text is replaced with its corresponding CBOW vector, and these vectors are aggregated to form a word frequency vector representation of the text, thereby reducing the data dimension and extracting key information.
[0053] S3, aggregate the generated word frequency vectors through the BiLSTM network.
[0054] Among them, BiLSTM is used to process the vector sequence of context words, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text.
[0055] In this step, the word frequency vector sequence generated by CBOW is input into the BiLSTM model, and the BiLSTM model processes the forward and reverse context information at the same time.
[0056] S4, extracting features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text;
[0057] In this step, the skip-gram model is used to train each word in the text to learn the vector representation of the word; after the training is completed, the skip-gram vectors of all words in the text are aggregated to obtain the semantic feature vector of the text.
[0058] S5, performing weighted averaging of the aggregation results of the target patent text in S3 and the semantic features of the target patent text in S4 to obtain the final representation of the target patent text, and performing multi-level classification model training of the patent text through the final representation of the target patent text.
[0059] In this step, the weights of the text representation vectors generated by BiLSTM and skip-gram are determined; after the weights are determined, the two representation vectors are weighted averaged to obtain a text representation vector that integrates contextual information and semantic information; the text representation vector is used as an input feature to train a multi-level classification model. During the training process, the cross entropy loss function and RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect.
[0060] In the optional embodiments of the present application, the following technical means and measures are used to achieve:
[0061] (1) Establish a patent text dataset containing multi-level labels;
[0062] (2) For a given patent text, the CBOW model is used to preprocess the text, convert the text into a word frequency vector, and reduce the dimension of the text data;
[0063] (3) Aggregate the CBOW vectors of each word with the help of BiLSTM, that is, use BiLSTM to process the vector sequence of context words and obtain the hidden state of the last time step as the aggregation result of each CBOW word vector to represent the text;
[0064] (4) Using the skip-gram word vector model to extract features from the given patent text data, obtain the semantic features of the text and use the word vector of each word to sum up to get a fixed-length vector to represent the entire text;
[0065] (5) The text representation vectors calculated in two ways for the same text in (3) and (4) are weighted averaged to obtain the final representation of the text, thereby training a multi-level classification model for the patent text.
[0066] Please refer to Figure 2 , which shows a block diagram of a multi-level multi-classification text classification system based on CBOW and skip-gram word vectors provided in an embodiment of the present application, including:
[0067] A building module for constructing a patent text dataset containing multi-level labels;
[0068] The CBOW module is used to perform preliminary processing on the target patent text in the patent text dataset through the CBOW model, generate word frequency vectors, reduce data dimensions, and extract key information of the target patent text;
[0069] A BiLSTM module is used to aggregate the generated word frequency vectors through a BiLSTM network. The BiLSTM is used to process the vector sequence of the context words, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text.
[0070] The skip-gram module is used to extract features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text;
[0071] The target training module is used to perform weighted averaging of the aggregation results of the target patent text and the semantic features of the target patent text to obtain the final representation of the target patent text, and to train the multi-level classification model of the patent text through the final representation of the target patent text.
[0072] For the specific definition of the multi-level multi-classification text classification system, please refer to the definition of the multi-level multi-classification text classification method based on CBOW and skip-gram word vectors in the above text, which will not be repeated here. Each module in the above multi-level multi-classification text classification system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0073] In one embodiment, an electronic device is provided. The electronic device may be a computer, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for multi-level multi-classification text classification data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-level multi-classification text classification method based on CBOW and skip-gram word vectors is implemented.
[0074] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0075] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-level and multi-classification text classification method based on CBOW and skip-gram word vectors are implemented.
[0076] In one embodiment of the present application, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned multi-level multi-classification text classification method based on CBOW and skip-gram word vectors.
[0077] The computer-readable storage medium and computer program product provided in this embodiment have similar implementation principles and technical effects to those of the above method embodiments, and are not described in detail here.
[0078] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in M forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SyMchliMk) DRAM (SLDRAM), memory bus (RaMbus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0079] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0080] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.
Claims
1. A multi-level multi-classification text classification method based on CBOW and skip-gram word vectors, characterized in that: The method comprises: S1, construct a patent text dataset containing multi-level labels; S2, preliminarily processes the target patent text in the patent text dataset through the CBOW model, generates word frequency vectors, reduces the data dimension and extracts the key information of the target patent text; S3, aggregate the generated word frequency vectors through the BiLSTM network; wherein, the BiLSTM is used to process the vector sequence of the context words, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text; S4, extracting features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text; S5, performing weighted averaging of the aggregation results of the target patent text in S3 and the semantic features of the target patent text in S4 to obtain the final representation of the target patent text, and performing multi-level classification model training of the patent text through the final representation of the target patent text.
2. The multi-level and multi-classification text classification method according to claim 1 is characterized in that: S1 builds a patent text dataset with multi-level labels, including: Patent text data is collected, and one or more hierarchical labels are assigned to each text according to the patent category, subcategory, and technical field; wherein the hierarchical labels include a predefined classification system or labels automatically generated based on the text content.
3. The multi-level and multi-classification text classification method according to claim 1 is characterized in that: In S2, the target patent text in the patent text dataset is preliminarily processed by the CBOW model to generate word frequency vectors, reduce the data dimension and extract key information of the target patent text, including: Perform word segmentation on the target patent text and decompose the text into words or vocabulary units; Use the CBOW model to train and get the vector representation of each word. The CBOW model predicts the central word based on the context words, thereby learning the vector of each word. After training, each word in the text is replaced with its corresponding CBOW vector, and these vectors are aggregated to form a word frequency vector representation of the text, thereby reducing the data dimension and extracting key information.
4. The multi-level and multi-classification text classification method according to claim 1 is characterized in that: In S3, the generated word frequency vectors are aggregated through the BiLSTM network, which also includes: The word frequency vector sequence generated by CBOW is input into the BiLSTM model, and the BiLSTM model processes the forward and reverse context information at the same time.
5. The multi-level and multi-classification text classification method according to claim 1 is characterized in that: In S4, the skip-gram word vector model is used to extract features of the target patent text to extract the semantic features of the target patent text, and also includes: Use the skip-gram model to train each word in the text and learn the vector representation of the word; After training, the skip-gram vectors of all words in the text are aggregated to obtain the semantic feature vector of the text.
6. The multi-level and multi-classification text classification method according to claim 1 is characterized in that: In S5, the aggregation result of the target patent text in S3 and the semantic features of the target patent text in S4 are weighted averaged to obtain the final representation of the target patent text, and also includes: Determine the weights of the text representation vectors generated by BiLSTM and skip-gram; After determining the weights, the two representation vectors are weighted averaged to obtain a text representation vector that integrates contextual information and semantic information; The text representation vector is used as the input feature to train the multi-level classification model. During the training process, the cross entropy loss function and RMSprop optimization algorithm are used to optimize the model parameters to achieve the best classification effect.
7. A multi-level multi-classification text classification system based on CBOW and skip-gram word vectors, characterized in that: The system comprises: A building module for constructing a patent text dataset containing multi-level labels; The CBOW module is used to perform preliminary processing on the target patent text in the patent text dataset through the CBOW model, generate word frequency vectors, reduce data dimensions, and extract key information of the target patent text; A BiLSTM module is used to aggregate the generated word frequency vectors through a BiLSTM network. The BiLSTM is used to process the vector sequence of the context words, and the hidden state of the last time step is obtained as the aggregation result of each word frequency vector to represent the target patent text. The skip-gram module is used to extract features of the target patent text through the skip-gram word vector model to extract the semantic features of the target patent text; The target training module is used to perform weighted averaging of the aggregation results of the target patent text and the semantic features of the target patent text to obtain the final representation of the target patent text, and to train the multi-level classification model of the patent text through the final representation of the target patent text.
8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the multi-level and multi-classification text classification method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the multi-level and multi-classification text classification method as described in any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-level and multi-classification text classification method as described in any one of claims 1 to 6 is implemented.