A method for classifying Chinese patent IPCs based on SBERT

Abstract

The application discloses a Chinese patent IPC classification method based on SBERT, comprising the following steps: corpus data preprocessing: extracting specific words and sentences from patent text to form first corpus data, and extracting term descriptions corresponding to categories from an IPC classification table according to categories to form second corpus data; data enhancement processing of the corpus data; text vectorization coding: inputting the first corpus data and the second corpus data into a first BERT pre-training model and a second BERT pre-training model under an SBERT framework respectively for vectorization coding to obtain vector representation of the patent text; similarity comparison: selecting a classification number according to a similarity ranking obtained by calculation. The application adopts SBERT with a twin structure as a framework of automatic classification of Chinese patents, takes patent text and term descriptions in an IPC classification table as two inputs of SBERT, and determines the patent category according to the closeness of two vectors after BERT vectorization coding, so that the calculation amount is reduced, the classification accuracy is improved, and multiple relevant IPC classification numbers are given.

Description

Technical Field

[0001] This invention relates to the field of patent IPC classification technology, and in particular to a Chinese patent IPC classification method based on SBERT. Background Technology

[0002] With the rapid development of technology, the number of global patent applications has been increasing year by year. Patent applications need to be categorized according to different fields for classification, statistics, and management. That is, each approved patent is classified into an International Patent Classification (IPC) code based on its technical content. Currently, this classification work still relies on manual methods, which places a huge workload on patent examiners. Therefore, it is necessary to study how to utilize natural language processing techniques to solve the problems related to automatic patent classification by mining semantic information from patent application texts.

[0003] Automatic patent classification is currently mainly achieved through deep learning networks. Figure 1 This paper demonstrates a classification structure used in existing technologies. This structure mainly consists of two parts: a pre-trained model and a Text-CNN classification head. The pre-trained model implements the vectorized representation of the patent text, while the Text-CNN classification head implements the classification of the patents.

[0004] Using BERT (Bidirectional Encoder Representation from Transformers) pre-trained models as the text vectorization representation method for patent classification is currently the best-performing approach. BERT employs a bidirectional Transformer encoder, acquiring semantic representations at the word and sentence levels through word masking and context relationship prediction, respectively. This training method gives BERT strong sentence and word representation capabilities. It exhibits excellent performance in both word-level NLP tasks, such as named entity recognition, and sentence-level NLP tasks, such as question answering. The vector representations encoded by BERT can reveal potential deep semantic and syntactic information. The pre-trained model is obtained by pre-training on massive amounts of text under unsupervised conditions, and its parameters can be fine-tuned for specific tasks.

[0005] Text-CNN is an architectural variant of CNN used for text classification tasks. For example... Figure 1As shown in the diagram, the right side illustrates the hierarchical structure of Text-CNN, primarily consisting of convolution, pooling, and fully connected layers. Text-CNN uses three convolutional kernel sizes, with the kernel width matching the word vector length, and kernel heights of 3, 4, and 5 respectively. The convolutional kernels slide from top to bottom across the word vectors to perform the convolution operation. The convolutional feature maps are then pooled and concatenated to obtain a one-dimensional representation of the text's features. Finally, a fully connected layer is used to classify the patent text.

[0006] Existing technologies employ the state-of-the-art BERT pre-trained model as the method for representing patent documents. This has yielded good results in patent classification performance. However, existing technologies still have the following shortcomings in achieving automatic patent IPC classification:

[0007] 1) On the one hand, existing methods only use historical patent data as prior knowledge, without utilizing the classification rules and terminology provided in the IPC classification table. The existing technology employs a typical black-box approach based on deep learning, where deep networks acquire a certain level of recognition ability through learning from a large amount of historical patent data. However, in addition to historical patent data, the IPC classification table and rule terminology provided by the State Intellectual Property Office remain unused as prior knowledge.

[0008] 2) On the other hand: Existing technologies only provide one primary classification number when automatically classifying patent texts. However, when identifying a specific patent, in addition to the primary classification number, 2-5 supplementary classification numbers are required. When an invention patent involves different types of technical subjects, and these technical subjects also constitute inventive information, multiple classifications should be performed based on the involved technical subjects, providing multiple classification numbers. The classification number that best represents the inventive information should be placed first as the primary classification number. Existing technologies cannot provide multiple classification numbers for a patent when it involves multiple technical subjects.

[0009] In view of the two shortcomings of the existing technology mentioned above, this invention proposes an automatic classification method for Chinese patented IPCs based on SBERT. Summary of the Invention

[0010] The purpose of this invention is to provide a Chinese patent IPC classification method based on SBERT. This method uses SBERT with a twin structure as the framework for automatic classification of Chinese patents. The patent text and the classification terms in the IPC classification table are used as two data inputs to SBERT. BERT is used to vectorize the two texts, and the similarity between the two vectors is used to determine the category of the real patent. This not only reduces the amount of computation and improves the classification accuracy, but also provides multiple IPC classification numbers.

[0011] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0012] A Chinese patent IPC classification method based on SBERT includes the following steps:

[0013] Corpus data preprocessing: Specific words and phrases are extracted from the patent text to form the first corpus data, and term descriptions that can express the corresponding class are extracted from the IPC classification table to form the second corpus data; the patent text can be the target patent text to be classified or training samples, and the training samples are obtained from historical patent texts;

[0014] The corpus data is subjected to data augmentation processing;

[0015] Corpus data preprocessing: Specific words and phrases are extracted from the patent text to form the first corpus data, and term descriptions that can express the corresponding class are extracted from the IPC classification table to form the second corpus data; the patent text can be the target patent text to be classified or training samples, and the training samples are obtained from historical patent texts;

[0016] The corpus data is subjected to data augmentation processing;

[0017] Text vectorization encoding: The first and second corpus data after data augmentation are respectively input into the first and second BERT pre-trained models under the SBERT framework for vectorization encoding to obtain the vector representation of the patent text. The vector representation of the patent text corresponding to the first corpus data constitutes the feature set U, and the vector representation of the patent text corresponding to the second corpus data constitutes the feature set V.

[0018] Similarity comparison: Calculate the similarity value between the feature set U and the feature set V to obtain the similarity value ranking corresponding to the term descriptions of different categories. Select the IPC classification number corresponding to the term description with the first similarity value as the main IPC classification number, and the IPC classification number corresponding to the term description with the Nth (N>1) similarity value as the optional alternative IPC classification number.

[0019] Furthermore, the data augmentation process for the corpus data specifically involves: during the SBERT model training process, the DropOut method is used to input the same text multiple times into the BERT pre-trained model to achieve data augmentation of the sample data.

[0020] Furthermore, the vectorization encoding process is as follows: each sentence text is individually vectorized and encoded to obtain a vector representation corresponding to each sentence text, and the vector representations of all sentence texts are combined to form the vector representation of the patent text.

[0021] Furthermore, the calculation of the similarity value between feature set U and feature set V specifically involves obtaining the similarity value by calculating the cosine similarity or Euclidean distance between feature set U and feature set V.

[0022] Furthermore, the text vectorization encoding process also includes average pooling processing on the vector representation of the patent text.

[0023] Furthermore, the specific phrases are the patent title and abstract.

[0024] By adopting the above solution, the present invention has the following beneficial effects:

[0025] 1. This invention utilizes historical patent text data as the first corpus data and fully leverages the term descriptions of the IPC classification table as the second corpus data. For sentence pair tasks, SBERT can reduce computational overhead compared to BERT. At the same time, it allows both texts to be input with the maximum text length, and each sentence text is individually vectorized and encoded. Therefore, its Chinese patent classification method has a significantly improved accuracy and greatly reduced computational load compared to the previous black-box method, and has the characteristics of faster and more accurate classification.

[0026] 2. Finally, this invention, through similarity comparison, provides not only the Top 1 main classification number but also Top N classification numbers as alternative classification numbers for the patent. These alternative classification numbers provide room for selection during manual processing. Furthermore, the Top N classification numbers with high similarity values ​​reveal that the patent involves different types of technical topics. When assigning patent classification numbers, in addition to providing the main classification number, classification numbers encompassing other topics should also be provided.

[0027] The advantages of this invention lie in its use of SBERT, a framework with a twin structure, for automatic classification of Chinese patents. This framework has two inputs: one input is the text (patent subject and abstract) corresponding to the patent to be classified, and the other input is the terminology from the IPC classification table. For the patent to be classified, the second input takes all categories of classification terms. Through similarity calculation, a ranking of similarity values ​​corresponding to the terminology descriptions of different categories is obtained. The top-ranked classification number is selected as the primary IPC classification number, and the top N classification numbers serve as alternative IPC classification numbers to indicate that the patent involves other topics. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other modifications can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a patent classification framework diagram based on existing technologies such as BERT and Text-CNN;

[0030] Figure 2 A flowchart of a Chinese patent IPC classification method based on SBERT according to an embodiment of the present invention;

[0031] Figure 3 This is a Chinese patent classification framework diagram based on SBERT according to an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a patented vector representation based on a pre-trained model. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The BERT described in this embodiment stands for Bidirectional Encoder Representation from Transformers, a pre-trained language representation model. It emphasizes that instead of using traditional unidirectional language models or shallow concatenation of two unidirectional language models for pre-training, it employs a new masked language model (MLM) to generate deep bidirectional language representations.

[0035] The SBERT described in this embodiment is a Siamese network, also known as Sentence BERT. All sub-networks of the SBERT model use the BERT model, and the two BERT models share parameters. When comparing the similarity of two sentences, A and B, A and B are input into two BERT networks respectively, and the outputs are two sets of vectors representing the sentences. Then, the similarity between the two is calculated. This principle can also be used to perform vector clustering to achieve unsupervised learning tasks.

[0036] like Figure 2As shown, this embodiment of the invention provides a Chinese patent IPC classification method based on SBERT, including the following steps:

[0037] S10. Corpus Data Preprocessing: Specific phrases are extracted from the patent text to form the first corpus data, and terminology descriptions that can express the corresponding category are extracted from the IPC classification table to form the second corpus data. The patent text can be the target patent text to be classified or training samples, and the training samples are obtained from historical patent texts. The specific phrases extracted from the first corpus data are generally selected from the patent title and abstract. For the second corpus data, taking Part A of the IPC classification table as an example, terminology was extracted from the Chinese definition column at the subclass level of Part A of the IPC classification table, and it was also merged and sorted. The processed results are shown in Table 1.

[0038] Table 1A: Terminology Descriptions of Subcategories

[0039]

[0040] S20. Perform data augmentation on the corpus data. During model training, when there are few or imbalanced training samples, data augmentation is needed to prevent overfitting and improve model accuracy. In this embodiment, the number of positive samples is far less than the number of negative samples, so data augmentation is required. Specifically, during SBERT model training, DropOut can be used to augment the training samples. The DropOut mask in the SBERT model is random, which means that the same text will output different vector representations after passing through the BERT model. Therefore, this embodiment achieves data augmentation of the sample data by inputting the same text into the BERT model multiple times. It is better than more complex data augmentation methods such as word deletion or replacement based on synonym or mask language models, because deletion or replacement operations may change the original meaning of the text, while the samples generated by the DropOut method are semantically identical to the original samples, only the generated vector representations are different.

[0041] S30. Text Vectorization Encoding: The first and second corpus data after data augmentation are respectively input into the first and second BERT pre-trained models under the SBERT framework for vectorization encoding. This vectorization encoding process vectorizes each sentence text separately to obtain the vector representation of each sentence text. The vector representations of all sentence texts constitute the vector representation of the patent text. The vector representation of the patent text corresponding to the first corpus data constitutes the feature set U, and the vector representation of the patent text corresponding to the second corpus data constitutes the feature set V.

[0042] See Figure 3 As shown, based on the SBERT framework, for sentence pair tasks, the SBERT model reduces computational overhead compared to the BERT model. Simultaneously, it allows both input texts (first and second corpora) to be of maximum length. When using the BERT model, the symbol [SEP] is used to concatenate two sentence texts to form a sentence pair, and the number of sentence pairs (i.e., the number of vectorization encodings) is the square of the number of sentences. However, with the SBERT model, each sentence text is vectorized and encoded separately, and similarity comparisons are performed using methods such as cosine similarity or Euclidean distance. Therefore, compared to the BERT model, this invention uses the SBERT model with fewer vectorization encodings.

[0043] Furthermore, since the two input texts are vectorized and encoded separately by BERT on two branches (the first BERT pre-trained model and the second BERT pre-trained model), both texts can be input at the maximum text length allowed by the BERT pre-trained model. In the SBERT model, sentence-level text vectorization representation can take several different forms. For example... Figure 4 As shown, each character in the patent text, after being input into the pre-trained model, obtains its own vector representation, R. [CLS] This represents the vector representation of the entire text; however, vector R... [CLS] When representing the entire text vector, it is often not precise enough. This embodiment further adopts the average pooling method, that is, taking the average value of all word vector representations, that is, using the method of adding an average pooling layer at the output to improve the quality of vectorization encoding.

[0044] S40. Similarity Comparison: Calculate the similarity values ​​between feature set U and feature set V to obtain a ranking of similarity values ​​corresponding to term descriptions of different categories. Select the IPC classification number corresponding to the term description with the highest similarity value (Top1) as the primary IPC classification number, and the IPC classification numbers corresponding to the term description with the Nth highest similarity value (TopN, where N is a positive integer and N>1) as optional alternative IPC classification numbers. Specifically, the similarity value can be obtained by calculating the cosine similarity or Euclidean distance between feature set U and feature set V.

[0045] Based on the above embodiments, this invention employs SBERT, which has a twin structure, as a framework for automatic classification of Chinese patents. This framework has two inputs: one input is the text (patent subject and abstract) corresponding to the patent to be classified, and the other input is the terminology descriptions from the IPC classification table. For the patent to be classified, the second input is the classification terms for all categories. Through similarity calculation, a ranking of similarity values ​​corresponding to the terminology descriptions of different categories is obtained. The top-ranked classification number can be used as the basis for patent classification and selected as the primary IPC classification number. The top N classification numbers can be used as alternative IPC classification numbers to indicate that the patent involves other topics.

[0046] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "optional embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0047] The embodiments described above do not constitute a limitation on the scope of protection of this technical solution. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the above embodiments should be included within the scope of protection of this technical solution.

Claims

1. A Chinese patent IPC classification method based on SBERT, characterized in that, Includes the following steps: Corpus data preprocessing: Specific words and phrases are extracted from the patent text to form the first corpus data, and term descriptions that can express the corresponding class are extracted from the IPC classification table to form the second corpus data; the patent text is the target patent text to be classified or training samples, and the training samples are obtained from historical patent texts; The data augmentation process for the corpus data is specifically performed as follows: During the training of the SBERT model, the DropOut method is used to input the same text multiple times into the BERT pre-trained model to achieve data augmentation of the sample data. Text vectorization encoding: The first and second corpus data after data augmentation are respectively input into the first and second BERT pre-trained models under the SBERT framework for vectorization encoding to obtain the vector representation of the patent text. The vector representation of the patent text corresponding to the first corpus data constitutes the feature set U, and the vector representation of the patent text corresponding to the second corpus data constitutes the feature set V. Similarity comparison: Calculate the similarity value between the feature set U and the feature set V to obtain the similarity value ranking corresponding to the descriptions of different categories of terms. Select the IPC classification number corresponding to the term description with the first similarity value as the main IPC classification number, and the IPC classification number corresponding to the term description with the Nth similarity value as the optional alternative IPC classification number, where N>1.

2. The SBERT-based Chinese patent IPC classification method as described in claim 1, characterized in that, The vectorization encoding process is as follows: each sentence text is individually vectorized and encoded to obtain a vector representation of each sentence text, and the vector representations of all sentence texts are combined to form the vector representation of the patent text.

3. The SBERT-based Chinese patent IPC classification method as described in claim 1, characterized in that: The similarity value between feature set U and feature set V is calculated by calculating the cosine similarity or Euclidean distance between feature set U and feature set V.

4. The SBERT-based Chinese patent IPC classification method as described in claim 1, characterized in that: The text vectorization encoding process also includes average pooling of the vector representation of the patent text.

5. The SBERT-based Chinese patent IPC classification method as described in claim 1, characterized in that: The specific phrases mentioned refer to the patent title and abstract.

Images