Method, device and storage medium for improving multi-level patent text classification effect

Abstract

The application provides a method for improving the multi-level patent text classification effect based on a BERT-ALMG model, an electronic device and a storage medium, comprising: constructing a label set corresponding to patent classification, and establishing a label feature vector for each label; obtaining a patent text to be classified; analyzing the patent text to obtain a text feature vector of the patent text; analyzing the relationship between the text feature vector and the label feature vector to obtain a label text vector of the patent text; and performing multi-granularity feature convolution operation on the label text vector to classify the patent text. For the multi-level classification task of Chinese patents, the BERT pre-training model is used to fully capture the context semantic features of long text, solving the problem that the length of Chinese patent text is long and the semantic of professional vocabulary is difficult to understand. The label attention module fully excavates the label semantic information, and uses a matrix to dynamically update the label text vector closely related to the hierarchical content, which can fully model the hierarchical structure.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, electronic device, and storage medium for improving the classification effect of multi-level patent texts based on the BERT-ALMG model. Background Technology

[0002] Patent classification is a multi-level problem. Currently, the main patent classification systems are the International Patent Classification (IPC) and the Cooperative Patent Classification (CPC). IPC is the internationally accepted patent classification method, used by many countries to classify invention and utility model patents according to their technical fields. The CPC classification system originates from IPC but is more detailed in its entry division. IPC classification includes four levels: Part, Class, Subclass, and Group. For multi-level patent classification, the classification number of the patent text corresponds to a node at each level, organized into a tree-like hierarchical structure. The entire problem can be viewed as a Hierarchical Multi-label Text Classification (HTMC) problem. The characteristics of patent classification require patent examiners to select appropriate classification numbers from a large number of similar categories, which places extremely high demands on their professional competence. Automatic classification of multi-level patents can reduce a significant amount of manual work and avoid patent infringement problems caused by technological duplication, thus having practical significance.

[0003] The above-mentioned traditional patent text classification methods have the following technical problems:

[0004] (1) Compared with traditional texts, patent texts, as a typical type of scientific and technological texts, have more standardized formats, more rigorous wording, and are more lengthy, making their contextual semantics more difficult to understand.

[0005] (2) Many studies usually train a local classifier to predict all hierarchical labels, and the hierarchical structure information and the rich semantic information of the upper and lower hierarchical classification labels are often ignored.

[0006] (3) Errors in the forward propagation process between upper and lower levels affect the model's expressive performance;

[0007] (4) Classification labels not only have commonalities but also differences between parent and child classes. Subclasses under the same parent class maintain differences while also having connections. However, the differences and diversity of hierarchy and label granularity are often ignored during classification prediction. Summary of the Invention

[0008] This application aims to provide a method, electronic device, and storage medium for improving the classification performance of multi-level patent texts based on the BERT-ALMG model, in order to solve the problem of poor patent text classification performance in the prior art.

[0009] On the one hand, this application provides a method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model, including:

[0010] Construct a set of tags corresponding to the patent classification, and create a tag feature vector for each tag;

[0011] Obtain the patent text to be classified;

[0012] The patent text is analyzed to obtain the text feature vector of the patent text;

[0013] Analyze the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text;

[0014] The patent text is classified by performing multi-granularity feature convolution operations on the labeled text vector.

[0015] Furthermore, the analysis of the patent text to obtain the text feature vector of the patent text includes:

[0016] The syntactic and semantic features of the patent text context are analyzed using the text feature representation layer of the BERT pre-trained model to obtain the text feature vector of the patent text.

[0017] Furthermore, the BERT pre-trained model includes two pre-training tasks: masked language modeling and next sentence prediction;

[0018] The masked language model masks the input word portion and restores the masked words through a bidirectional deep Transformer model, learning a context-sensitive text representation.

[0019] The next sentence prediction task learns the implication relationships between different sentences by judging whether they are adjacent, thereby obtaining the syntactic and semantic features of the patent text context.

[0020] Furthermore, the step of analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text includes:

[0021] The text feature vector and the label feature vector are mapped to a spatial dimension through a bridging vector to obtain the label attention matrix;

[0022] The text feature vector and the label force attention matrix are processed by an activation function to obtain the corresponding local text vector;

[0023] The upper-layer local text vector and the label feature vector are normalized, and then a matrix dot product operation is performed with the local text vector of this layer to obtain the label text vector.

[0024] Furthermore, the text feature vector and the label feature vector are mapped to a spatial dimension through a bridging vector to obtain the label-attention matrix, which includes:

[0025] The text feature vector is subjected to linear transformation by a single-layer perceptron, and then multiplied by the bridging vector to obtain the first relation matrix;

[0026] The label feature vector is linearly transformed by a single-layer perceptron and then multiplied by the bridging vector to obtain the second relation matrix.

[0027] The label-force attention matrix is ​​obtained by using the first relation matrix and the second relation matrix.

[0028] Furthermore, when performing multi-granularity feature convolution operations on the label text vectors, different granularity convolution kernels are used for different levels. Label text vectors at the first level use convolution kernels with a first width, and label text vectors at the second level use convolution kernels with a second width. The first level is higher than the second level, and the first width is greater than the second width.

[0029] Furthermore, the step of performing multi-granularity feature convolution on the labeled text vector to classify the patent text includes:

[0030] Perform multi-granularity feature convolution operation on the label text vector to obtain salient features;

[0031] Based on the salient features, the probability of the patent text belonging to the corresponding level is predicted, and then the patent text is classified.

[0032] Furthermore, the multi-granularity feature convolution operation performed on the label text vector to obtain salient features includes:

[0033] One-dimensional convolution is performed on the word window to obtain the corresponding features, and then max pooling is performed to obtain salient features.

[0034] On the one hand, this application also provides an electronic device, the electronic device comprising:

[0035] One or more processors;

[0036] Memory; and

[0037] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method described in any one of the first aspects.

[0038] In one aspect, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the method described in any of the first aspects.

[0039] Unlike existing technologies, this application provides a method, electronic device, and storage medium for improving the classification performance of multi-level patent texts based on the BERT-ALMG model, comprising: constructing a set of tags corresponding to patent classification and establishing a tag feature vector for each tag; obtaining the patent text to be classified; analyzing the patent text to obtain the text feature vector of the patent text; analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text; and performing multi-granularity feature convolution operation on the tag text vector to classify the patent text.

[0040] To address the multi-level classification task of Chinese patents, this paper proposes the BERT-ALMG model. The BERT structure in this model fully captures the contextual semantic features of long texts, solving the problems of lengthy Chinese patent texts and the difficulty in understanding the semantics of specialized vocabulary. The label attention module fully mines the semantic information of labels and introduces bridging vectors to fully capture the relationship between text feature vectors and label vectors, obtaining a label attention matrix. This matrix is ​​used to dynamically update the label text vectors closely related to the hierarchical content and pass them layer by layer. This module can fully model the hierarchical structure. Considering the commonalities and differences between hierarchical labels, a multi-granularity feature extraction module is designed to extract knowledge of different coarse and fine granularities for classification prediction. Extensive comparative experiments were conducted on a self-built patent dataset, and the experimental results show that the model outperforms other models. In summary, the BERT-ALMG model proposed in this application is significantly effective in solving the multi-level patent classification problem and has better practical value. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic diagram of a system for improving the classification performance of multi-level patent texts based on the BERT-ALMG model, as provided in the embodiments of this application.

[0043] Figure 2 This is a flowchart illustrating an embodiment of the method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model provided in this application.

[0044] Figure 3 This is a schematic diagram of an embodiment of the BERT-ALMG model provided in this application.

[0045] Figure 4 This is a schematic diagram of the overall structure of the BERT model provided in the embodiments of this application;

[0046] Figure 5 This is a schematic diagram illustrating how the tag attention layer in this application uses an attention mechanism to fully capture the relationship between hierarchical tags and text.

[0047] Figure 6 This is a schematic diagram illustrating the patent category statistics in the embodiments of this application;

[0048] Figure 7 This is a schematic diagram of the statistical characteristics of the dataset in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of the experimental results of the patent dataset in the embodiments of this application;

[0050] Figure 9 This is a schematic diagram of the ablation analysis of the BERT-ALMG model in the embodiments of this application;

[0051] Figure 10 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0054] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0055] It should be noted that since the method in this application embodiment is executed in an electronic device, the processing objects of each electronic device exist in the form of data or information, such as time, which is essentially time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the electronic device can process them. Specific details will not be elaborated here.

[0056] This application provides a method, electronic device, and storage medium for improving the classification effect of multi-level patent texts based on the BERT-ALMG model, which will be described in detail below.

[0057] Please see Figure 1 , Figure 1 This is a schematic diagram of a system for improving multi-level patent text classification based on the BERT-ALMG model, as provided in an embodiment of this application. The system may include an electronic device 100, which integrates a device for improving multi-level patent text classification methods, such as... Figure 1 Electronic devices in the system.

[0058] In this embodiment, the electronic device 100 can be a standalone server, a server network, or a server cluster. For example, the electronic device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0059] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include more than one application scenario. Figure 1 The number of more or fewer electronic devices shown, for example Figure 1 Only one electronic device is shown in the diagram. It is understood that the system may also include one or more other servers, which are not specified here.

[0060] In addition, such as Figure 1 As shown, the system may also include a memory 200 for storing data.

[0061] It should be noted that, Figure 1 The schematic diagram of the system shown is merely an example. The system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of the system and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0062] First, this application provides a method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model. The main body executing this method is a device for improving the classification performance of multi-level patent texts. This device is applied to an electronic device. The method for improving the classification performance of multi-level patent texts includes: constructing a set of tags corresponding to patent classifications and establishing a tag feature vector for each tag; obtaining the patent text to be classified; analyzing the patent text to obtain the text feature vector of the patent text; analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text; and performing a multi-granularity feature convolution operation on the tag text vector to classify the patent text.

[0063] See Figure 2 , Figure 2This is a schematic flowchart of one embodiment of the method provided in this application. (In conjunction with...) Figures 2-5 The method includes:

[0064] Step S201: Construct a set of tags corresponding to the patent classification, and establish a tag feature vector for each tag;

[0065] To address the aforementioned technical issues, this embodiment proposes a method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model. Through the following model design, the above technical problems are effectively solved, achieving the goal of improving the classification performance of patent texts and demonstrating good performance across multiple model evaluation metrics.

[0066] Specifically, the multi-level classification problem of patent texts is defined as follows: given patent text D = (d1, ..., d2) n Let Y be the total set of classification labels according to the IPC classification method, and let y1, y2, ..., y2 be the corresponding label subsets at different levels. L There are L layers, and Y follows a tree structure. The goal is to fully model existing patent texts and classification label information to predict the total set of labels to which a given patent belongs.

[0067] Step S202: Obtain the patent text to be classified;

[0068] Step S203: Analyze the patent text to obtain the text feature vector of the patent text;

[0069] In this embodiment, the BERT-ALMG model is used to analyze the patent text to obtain the text feature vector of the patent text.

[0070] The model structure of BERT-ALMG is as follows: Figure 3 As shown, the BERT-ALMG (BERT-Attention Based Label Multi-Granularity Feature Extraction) model consists of three modules: a text feature representation layer, an attention-based label layer (AL), and a multi-granularity feature extraction layer (MG).

[0071] In this embodiment, the text feature representation layer of the BERT pre-trained model is used to analyze the syntactic and semantic features of the patent text context to obtain the text feature vector of the patent text. The label attention layer fully captures the relationship between the label and the text, and at the same time, the generated label text vector is passed forward to the next layer. In the multi-granularity feature extraction layer, one-dimensional convolutional kernels of different coarse-grained sizes are used for different layers to fully perceive coarse and fine-grained features in order to improve the classification performance of the model.

[0072] In the Text Feature Representation Layer, a BERT pre-trained model is used as the text encoder to capture the syntactic and semantic features of the patent text. The BERT model is a deep bidirectional Transformer-based pre-trained model that includes two pre-training tasks: masking language modeling and next sentence prediction. The masking language model masks the input word parts, and the masked words are restored by the bidirectional deep Transformer model, learning a context-sensitive text representation. The next sentence prediction task is a binary classification task that learns the implication relationships between different sentences by judging whether they are adjacent, thereby obtaining the syntactic and semantic features of the patent text context.

[0073] The BERT model fully leverages large-scale unlabeled text to extract rich semantic information. Its internal structure consists of a deep bidirectional Transformer, which employs a bidirectional multi-head attention mechanism and a feedforward neural network. Its main structure is as follows: Figure 4 As shown.

[0074] The input to the BERT layer consists of word vectors, block vectors, and position vectors. [CLS] is added at the beginning of each sentence and [SEP] at the end of each block. The three vectors are concatenated and output to the Transformer block. Through a self-attention mechanism, the semantic relationships between words are fully learned. After residual connections and layer normalization, the hidden layer vector H for each layer is obtained.

[0075] X = x t +x s +x p

[0076] H = Transformer(X)

[0077] Where, x t x s x p These represent word vectors, block vectors, and position vectors, respectively.

[0078] The hidden layer vectors are continuously output to the Transformer, and after multiple layers are stacked, the context semantic representation vector E is obtained.

[0079] E = (Transformer(X)) j

[0080] Where j represents the number of Transformer layers in the BERT structure.

[0081] Step S204: Analyze the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text;

[0082] In this embodiment, the text feature vector and the label feature vector are mapped to a spatial dimension through a bridging vector to obtain a label force attention matrix; the text feature vector and the label force attention matrix are processed by an activation function to obtain the corresponding local text vector; the upper-layer local text vector and the label feature vector are normalized, and then matrix multiplication is performed with the local text vector of this layer to obtain the label text vector.

[0083] Specifically, the text feature vector is linearly transformed using a single-layer perceptron and then multiplied by the bridging vector to obtain a first relation matrix; the label feature vector is linearly transformed using a single-layer perceptron and then multiplied by the bridging vector to obtain a second relation matrix; the label attention matrix is ​​obtained through the first relation matrix and the second relation matrix.

[0084] Specifically, such as Figure 5 As shown, the label attention layer uses an attention mechanism to fully capture the relationship between hierarchical labels and text, and fully models the dependencies between hierarchical levels.

[0085] To better capture the inherent relationship between text and tags, a bridging vector B is introduced. l The text feature vector E l With the label feature vector Q l By bridging the vector and mapping it to a spatial dimension, a close relationship is formed between the two. Specifically, the text feature vector E is... l After a simple linear transformation of a single-layer perceptron and bridging vector B l Performing matrix multiplication yields the first relation matrix r. l The label feature vector Q l After linear transformation by a single-layer perceptron, and then with the bridging vector B l Matrix multiplication yields the second relation matrix c. l The specific calculation formula is as follows:

[0086] r l =f(El )·B l

[0087] c l =f(Q) l )·B l

[0088] Where l represents the level number of the module and f represents the single-layer perceptron operation.

[0089] The second relation matrix c l After processing with the activation function and the first relation matrix r l Jointly capture attention matrix to M l The relationship is as follows:

[0090] M l =softmax(σ(c) l )·(r l ) T )

[0091] M l With E l After processing by the activation function, the corresponding local text vector t is obtained. l The activation function used is the ReLU function, and then the upper-level local text vector t is... l-1 With Q l Perform normalization operation and then combine with this layer Perform a matrix dot product to obtain the label text vector T. l This module can fully capture the inherent relationship between hierarchical tags and text, and propagate information from the upper level to the lower level, thus fully modeling the hierarchical structure and tag information. Here, w and b are the weight parameter and bias term, respectively.

[0092] t l =σ(M l E l w+b)

[0093] T l =t l ·softmax((t l-1 ) T Q l ))

[0094] Step S205: Perform multi-granularity feature convolution operation on the label text vector to classify the patent text.

[0095] In this embodiment, a multi-granularity feature convolution operation is performed on the label text vector to obtain salient features; the probability of the patent text belonging to the corresponding level is predicted based on the salient features, and then the patent text is classified.

[0096] Specifically, when performing multi-granularity feature convolution operations on the label text vector, a one-dimensional convolution operation is performed on the word window to obtain the corresponding features, and then a max pooling operation is performed to obtain salient features.

[0097] Specifically, when performing multi-granularity feature convolution operation on the label text vector, different granularity convolution kernels are used for different levels. The label text vector at the first level uses a convolution kernel with a first width, and the label text vector at the second level uses a convolution kernel with a second width. The first level is higher than the second level, and the first width is greater than the second width.

[0098] Specifically, in order to better capture coarse and fine features at different levels in the multi-granularity feature extraction layer, different granularity convolution kernels are used for different levels. Wider convolution kernels are used for higher levels, while narrower convolution kernels are used for lower levels to extract fine-grained features.

[0099] This module uses the label text vector T generated by the label attention layer. l Perform multi-granularity feature convolution operation on it to obtain salient features s l Specifically, for the word window T l (n：n+k) Perform one-dimensional convolution operations to obtain the corresponding features. One-dimensional convolution is applied to each possible word window to obtain the feature vector. Then perform max pooling to obtain s l .

[0100] T l (n：n+j) =concat(T l n T l n+1 , ..., T l n+j )

[0101]

[0102]

[0103]

[0104] Among them, T l n T represents l The nth word vector window has a convolution kernel height of k, which varies at different levels, and its width is the same as the width of T. m b mrepresents the weight parameters and bias term of the m-th convolutional kernel, respectively, and maxpool is the max pooling layer. σ represents the activation function, which is the ReLU function.

[0105] In this embodiment, a combination of local and global methods is used to handle multi-level classification problems. Therefore, the model output includes two parts: local classification loss and global classification loss. The local classification loss is related to s... l The relationship is close, and the global classification loss is based on the local text vector t of the label attention layer. l It was pieced together.

[0106] The independent feature representations s obtained from the multi-granularity feature extraction layer l , This represents the probability that the i-th text in layer 1 is predicted to be of category j. The value L represents the category to which text i belongs (class j): 1 if it does, 0 otherwise. l The calculation formula is shown below.

[0107]

[0108]

[0109] Where w and b are the weight parameter and bias term, respectively, p is the number of layers, and q is the total number of samples in each layer.

[0110] The local text vector obtained from the label attention layer S is obtained by concatenating layers one by one, and the probability of the i-th text being predicted as category j is obtained by applying an activation function. Therefore, the global loss value L is calculated. g The details are as follows.

[0111]

[0112]

[0113] Where w and b are the weight parameter matrix and the bias term, respectively. The value of text i belonging to category j is 1 if it is, and 0 otherwise. p is the layer number, r is the total number of categories in each layer, and the total loss of the model is the sum of the two.

[0114] L = L l +L g

[0115] In practical application scenarios, experiments are conducted on the model to verify its effectiveness and obtain its evaluation.

[0116] The selected dataset comes from invention patent data on the Peking University Open Research Data Platform. This patent data has 16 attributes: application number, application date, publication number, patent name, patent classification, abstract, applicant, inventor, applicant address, applicant postal code, province code, city code, county code, legal status, and validity. This experiment selects the "abstract" and "patent classification" attributes, classifying the patents at the first two levels: department and category. Details are shown in Table 1. This application used a random method to select 12,000 data entries, encompassing all patent categories, as shown below. The data was randomly divided into training, validation, and test sets according to an approximate 4:1:1 ratio. The final dataset is shown below. Figure 6 As shown.

[0117] like Figure 7 The table shows the statistical characteristics of the dataset. The minimum number of words in the training set, validation set, and test set is less than 100, the maximum number of words is greater than 300, and the average number of words is 255, 254, and 255, respectively. Therefore, this application sets the number of words in the patent abstract to 256.

[0118] In practical applications, common performance evaluation metrics for classification problems include precision (P) and recall (R). The F1 score is the harmonic mean of P and R. AUPRC represents the area under the PR curve, and its specific calculation is as follows.

[0119]

[0120]

[0121]

[0122] Since model evaluation depends on model scores and thresholds, inappropriate threshold settings will affect model performance; however, this is unavoidable in the model evaluation process. Therefore, we choose an evaluation metric that has proven to be more effective in the HMTC task. The F1 score is used to evaluate model performance. In this multi-class classification problem, the precision P and recall R, given a threshold, are expressed as follows, where i represents the class and ALL is the total number of classes. Represented as The area under the curve.

[0123]

[0124]

[0125] The experimental code is based on Python 3.8.1, the deep learning framework is based on TensorFlow 1.4, and the system uses one Nvidia 3060 GPU with 12GB of RAM. The average number of words in the summaries across all samples was 256, therefore the maximum text length was set to 256. The BERT-ABCL model had 40 network epochs, a batch size of 31, a learning rate (lr) of 1e-3, and used the Adam Optimizer and the Bert Basic model. The number of coarse and fine-grained feature extraction filters was 128.

[0126] The BERT-ALMG model proposed in this embodiment is compared with mainstream models such as CNN, RNN, RCNN, RNN-Attention, Transformer, DPCNN, FastText, and LAHCN. All models underwent multiple experiments on a self-built patent dataset; detailed results are as follows... Figure 8 As shown.

[0127] In the multi-level classification task of Chinese patent texts, this method achieves F1 and F2 scores. The proposed method outperformed other methods, achieving scores of 0.6250 and 0.6670 respectively, representing improvements of 7.42 and 10.46 percentage points compared to LAHCN. Among other methods, the CNN structure alone achieved good performance, outperforming all methods except BERT-ALMG, demonstrating the superiority of CNN structures in handling complex and redundant textual information such as patent documents. Furthermore, while other methods besides LAHCN and BERT-ALMG treated all labels as flattened hierarchical structures for classification, the experimental results show that BERT-ALMG can fully exploit hierarchical structural features and model hierarchical dependency information through text vector updates in the label attention layer. The use of multi-granularity feature extraction modules for different levels demonstrates the superiority of this method in solving the hierarchical classification problem of Chinese patents.

[0128] Based on the above experiments, ablation experiments were conducted on a self-built patent dataset to verify the effectiveness of the BERT, AL, and MG modules respectively. The experimental results are as follows: Figure 9 As shown.

[0129] BERT indicates that only the BERT model is used to classify all labels simultaneously. Compared with BERT-ALMG, ALMG reduces BERT and only uses label attention and multi-granularity feature extraction to capture patent features to complete the classification task between levels. BERT-AL lacks MG, and the hierarchical granular features are not explicitly represented. The BERT-ALMG proposed in this application fully obtains the contextual semantic information of words and uses label attention to capture the intrinsic relationship between hierarchical labels and text, and uses this to update the label text vector. Finally, multi-granularity feature extraction is used for different levels for the classifier to make predictions.

[0130] The results of BERT and BERT-ALMG fully demonstrate the effectiveness of the ALMG module, with F1 scores improved by 20.39% and 26.67% respectively. It is evident that for complex problems such as multi-level Chinese patents, simply using the BERT pre-trained module to fully utilize the contextual semantic information of words is clearly insufficient, as it lacks explicit representation of the inherent relationships between levels.

[0131] Accordingly, based on the experimental results of ALMG and BERT-ALMG in the table, BERT-ALMG improved the F1 score by 8.74% and 12.09% respectively, indicating that the BERT module played an important role in the entire model. Without the BERT module, the model's ability to represent the contextual semantics of Chinese patent terms weakens, making it difficult to understand the semantics of specialized terms in Chinese patents, and the model's classification performance will also decline.

[0132] Compared to BERT-AL, BERT-ALMG is superior in F1 and These improvements were 1.7% and 3.57% respectively. MG uses multi-granularity features to extract knowledge of different coarse and fine granularities for different levels to provide for model classification and prediction, which to some extent solves the problem of model performance degradation caused by the commonality and difference of labels between levels. The experimental results also fully demonstrate the effectiveness of this module.

[0133] The beneficial effects of this invention are as follows: For the task of multi-level classification of Chinese patents, a BERT-ALMG model is proposed. The BERT structure in this model fully captures the contextual semantic features of long texts, solving the problems of lengthy Chinese patent texts and the difficulty in understanding the semantics of specialized vocabulary. The label attention module fully mines the semantic information of labels and introduces bridging vectors to fully capture the relationship between text feature vectors and label vectors, obtaining a label attention matrix. This matrix is ​​used to dynamically update the label text vectors closely related to the hierarchical content and pass them layer by layer. This module can fully model the hierarchical structure. Considering the commonalities and differences between hierarchical labels, a multi-granularity feature extraction module is designed to extract knowledge of different coarse and fine granularities for classification prediction. Extensive comparative experiments were conducted on a self-built patent dataset, and the experimental results show that the performance of this model is superior to other models. In summary, the BERT-ALMG model proposed in this application is significantly effective in solving the multi-level patent classification problem and has better practical value.

[0134] This application also provides an electronic device, such as... Figure 10 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0135] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0136] The processor 501 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502, and by calling data stored in the memory 502, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0137] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0138] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0139] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0140] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 runs the applications stored in the memory 502 to realize various functions, as follows: constructing a set of tags corresponding to the patent classification, and establishing a tag feature vector for each tag; obtaining the patent text to be classified; analyzing the patent text to obtain the text feature vector of the patent text; analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text; performing a multi-granularity feature convolution operation on the tag text vector to classify the patent text.

[0141] Furthermore, the analysis of the patent text to obtain the text feature vector of the patent text includes: analyzing the syntactic and semantic features of the patent text context using the text feature representation layer of the BERT pre-trained model to obtain the text feature vector of the patent text.

[0142] Furthermore, the BERT pre-trained model includes two pre-training tasks: masked language model and next sentence prediction. The masked language model masks the input word parts and restores the masked words through a bidirectional deep Transformer model to learn context-sensitive text representations. The next sentence prediction task learns the implication relationships between different sentences by judging whether they are adjacent, thereby obtaining the syntactic and semantic features of the patent text context.

[0143] Further, the step of analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text includes: mapping the text feature vector and the tag feature vector to a spatial dimension through a bridging vector to obtain a tag force attention matrix; processing the text feature vector and the tag force attention matrix through an activation function to obtain the corresponding local text vector; normalizing the upper-layer local text vector and the tag feature vector, and then performing a matrix dot product operation with the local text vector of the current layer to obtain the tag text vector.

[0144] Further, mapping the text feature vector and the label feature vector to a spatial dimension through a bridging vector to obtain the label attention matrix includes: performing a linear transformation of the text feature vector using a single-layer perceptron, and then multiplying it with the bridging vector to obtain a first relation matrix; performing a linear transformation of the label feature vector using a single-layer perceptron, and then multiplying it with the bridging vector to obtain a second relation matrix; and obtaining the label attention matrix through the first relation matrix and the second relation matrix.

[0145] Furthermore, when performing multi-granularity feature convolution operations on the label text vectors, different granularity convolution kernels are used for different levels. Label text vectors at the first level use convolution kernels with a first width, and label text vectors at the second level use convolution kernels with a second width. The first level is higher than the second level, and the first width is greater than the second width.

[0146] Furthermore, the step of performing multi-granularity feature convolution operation on the label text vector to classify the patent text includes: performing multi-granularity feature convolution operation on the label text vector to obtain salient features; predicting the probability that the patent text belongs to the corresponding level based on the salient features, and then classifying the patent text.

[0147] Furthermore, the step of performing multi-granularity feature convolution operation on the label text vector to obtain salient features includes: performing a one-dimensional convolution operation on the word window to obtain the corresponding features, and then performing a max pooling operation to obtain salient features.

[0148] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0149] To this end, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk, or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method provided in embodiments of this application. For example, the computer program loaded by the processor may execute the following steps: constructing a set of tags corresponding to a patent classification, and establishing a tag feature vector for each tag; obtaining the patent text to be classified; analyzing the patent text to obtain the text feature vector of the patent text; analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text; and performing a multi-granularity feature convolution operation on the tag text vector to classify the patent text.

[0150] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0151] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0152] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0153] The foregoing has provided a detailed description of a method, electronic device, and storage medium for improving the classification effect of multi-level patent texts based on the BERT-ALMG model, as provided in the embodiments of this application. Specific examples have been used in this application to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model, characterized in that, The BERT-ALMG model structure includes: a text feature representation layer, a label attention layer, and a multi-granularity feature extraction layer; The method for improving the classification performance of multi-level patent texts based on the BERT-ALMG model includes: Construct a set of tags corresponding to the patent classification, and create a tag feature vector for each tag; Obtain the patent text to be classified; The patent text is analyzed to obtain the text feature vector of the patent text; Analyzing the relationship between the text feature vector and the tag feature vector to obtain the tag text vector of the patent text includes: mapping the text feature vector and the tag feature vector to a spatial dimension through a bridging vector to obtain a tag force attention matrix; processing the text feature vector and the tag force attention matrix through an activation function to obtain the corresponding local text vector; normalizing the upper-layer local text vector and the tag feature vector, and then performing a matrix dot product operation with the local text vector of the current layer to obtain the tag text vector; The step of mapping the text feature vector and the label feature vector into a spatial dimension through a bridging vector to obtain the label attention matrix includes: performing a linear transformation of the text feature vector using a single-layer perceptron, and then multiplying it with the bridging vector to obtain a first relation matrix; performing a linear transformation of the label feature vector using a single-layer perceptron, and then multiplying it with the bridging vector to obtain a second relation matrix; and obtaining the label attention matrix through the first relation matrix and the second relation matrix. A multi-granularity feature convolution operation is performed on the label text vector to classify the patent text. When performing the multi-granularity feature convolution operation on the label text vector, different granularity convolution kernels are used for different levels. The label text vector at the first level uses a convolution kernel with a first width, and the label text vector at the second level uses a convolution kernel with a second width. The first level is higher than the second level, and the first width is greater than the second width.

2. The method as described in claim 1, characterized in that, The process of analyzing the patent text to obtain its text feature vector includes: using the text feature representation layer of a BERT pre-trained model to analyze the syntactic and semantic features of the patent text context to obtain the text feature vector of the patent text.

3. The method as described in claim 2, characterized in that, The BERT pre-trained model includes two pre-training tasks: masked language model and next sentence prediction. The masked language model masks the input word parts and restores the masked words through a bidirectional deep Transformer model, learning context-sensitive text representations. The next sentence prediction task learns the implication relationships between different sentences by judging whether they are adjacent, thereby obtaining the syntactic and semantic features of the patent text context.

4. The method as described in claim 1, characterized in that, The step of performing multi-granularity feature convolution operation on the tag text vector to classify the patent text includes: performing multi-granularity feature convolution operation on the tag text vector to obtain salient features; predicting the probability that the patent text belongs to the corresponding level based on the salient features, and then classifying the patent text.

5. The method as described in claim 4, characterized in that, The step of performing multi-granularity feature convolution on the labeled text vector to obtain salient features includes: performing a one-dimensional convolution operation on the word window to obtain the corresponding features, and then performing a max pooling operation to obtain salient features.

6. An electronic device, characterized in that, The electronic device includes: one or more processors; memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, It contains a computer program that is loaded by a processor to perform the steps of the method according to any one of claims 1 to 5.

Images