Claim level and pre-training model-based similarity calculation method and system

Abstract

The invention discloses a similarity calculation method and system based on a claim hierarchy and a pre-training model, and the method comprises the steps: splitting an obtained claim based on a claim hierarchy structure, constructing a technical combination sentence, and forming a recombined claim text; performing feature extraction on the recombined claim text based on a pre-training model with an adaptive module, and fusing the hidden state of each layer of network of the model by using a scalar mixing layer; the feature sequence output by the scalar mixing layer passes through the average pooling layer and the feature mixing layer in sequence, separation and recombination of claim level features are carried out, and text feature vectors used for similarity calculation are formed. According to the method, a claim technology combination sentence is constructed on the basis of a claim hierarchical structure, feature extraction is performed by using a multi-language pre-training model with an adaptive module, and aggregation and feature mixing are performed after feature extraction, so that aggregation features with the claim hierarchical structure are constructed, and patent similarity calculation is assisted.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular, relates to a patent similarity calculation method based on a claim hierarchy structure and a pre-training model. Background Art

[0002] Analyzing and mining patent data can not only improve the efficiency and quality of technological innovation, but also enhance the competitiveness of countries and enterprises in the international market. In recent years, due to its easy access, quantifiability and high generalization, patent data has become one of the preferred data for evaluating the technological research and development capabilities of countries, departments and enterprises, and has received increasing attention and utilization.

[0003] As an important tool for technological innovation management and strategic decision-making, patent analysis can realize the transformation of patent data into technical information and provide support for the management process. Patent similarity calculation, as a key step in patent analysis, is mainly used in multiple scenarios such as technology opportunity prediction, technology innovation prediction, competitor identification, technology theme division, patent layout analysis, patent infringement analysis, and patent value assessment. Its importance is obvious.

[0004] At present, the calculation of patent similarity is mainly based on patent citation relationships, patent IPC classification numbers, patent titles, keywords, specifications and claims, etc., supplemented by machine learning, deep learning and other methods. The calculation accuracy has gradually improved, but there is still room for improvement: (1) In the context of current technological progress and the diversity of R&D tools, most patent texts have data heterogeneity caused by multiple languages ​​and symbols. How to effectively unify and standardize these heterogeneous data to improve the accuracy and consistency of calculation results is a difficult problem that needs to be solved urgently. (2) Although machine learning and deep learning methods are widely used and relatively adaptable in patent similarity calculation, they often show poor generalization ability when processing new untrained data. (3) With the rapid growth of the number of patents, the existing calculation methods face challenges in calculation efficiency and scalability when processing large-scale patent data. (4) Most of the current patent similarity calculation methods rely on text features, but ignore the use of non-text information such as specific terms, professional vocabulary, special symbols, chemical formulas, etc. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a similarity calculation method and system based on claim hierarchy and pre-trained model. The method constructs a claim technical combination sentence based on the claim hierarchy structure, uses a multilingual pre-trained model with an adaptation module for feature extraction, and combines the features of each layer in the pre-trained model through a scalar mixing layer to increase the amount of information extracted from the feature; and after the feature extraction, aggregation and feature mixing are performed to construct an aggregated feature with a claim hierarchy structure, thereby assisting in the calculation of patent similarity.

[0006] According to one aspect of the present invention, a method for calculating similarity based on claim hierarchy and a pre-trained model is provided, comprising: Based on the claim hierarchy structure, the acquired claims are split and technical combination sentences are constructed to form a reorganized claim text; Based on the pre-trained model with the adaptation module, the feature extraction of the reorganized claim text is performed, and the hidden states of each layer of the model network are fused using a scalar mixing layer; The feature sequence output by the scalar mixing layer is sequentially passed through the average pooling layer and the feature mixing layer to separate and reorganize the claim level features to form a text feature vector for similarity calculation.

[0007] As a further technical solution, the method further includes: A pre-trained model with an adaptation module is constructed. Different tasks correspond to different adaptation modules. After training, the adaptation module is inserted into each layer of the pre-trained model in the form of a plug-in.

[0008] As a further technical solution, after the adaptation module is inserted into the pre-trained model, it includes: reducing the dimension of the original input through full connection, and then restoring the dimension after passing through the internal neural network to form a bottleneck structure.

[0009] As a further technical solution, feature extraction is performed on the reorganized claim text, and the hidden states of each layer of the network in the scalar mixed layer fusion model are used, including: A sequence of receiving technical combination sentence texts that are spliced ​​step by step; Encode the received sequence and use the self-attention mechanism to extract features; The hidden state vector generated by each layer of the pre-trained model is passed to the scalar mixing layer to form an aggregated hidden state sequence, wherein the aggregated hidden state sequence contains feature information of the technical combination sentence text sequence.

[0010] As a further technical solution, the average pooling layer is configured to average the text features of the technical combination sentences according to the time dimension of the sequence; the feature mixing layer is configured to mix the features output by the average pooling layer in a feature splicing manner to obtain a mixed and reorganized text feature vector.

[0011] As a further technical solution, based on the claim hierarchy, the obtained claims are split and technical combination sentences are constructed, including: Based on the location of the patent claims, the text of the claims is extracted and stored; The patent claims are extracted by defining regular expressions. All parsed claim numbers are stored in the claim_order list in order, recording the original order. Define a recursive function for combining dependent claims and the independent claims they reference, so that all references in each dependent claim are correctly parsed and concatenated; Reorganize the claims based on the claim text disassembled in the preamble; According to the original order in the claim_order list, the reorganized claims are sequentially constructed into a new list and output, completing the splitting of the entire claim and the construction of the technical combination sentence.

[0012] According to one aspect of the present invention, a similarity calculation system based on claim hierarchy and pre-trained model is provided, comprising: The text construction module is used to split the acquired claims and construct technical combination sentences based on the claim hierarchy structure to form a reorganized claim text; A feature extraction module, which is used to extract features from the reorganized claim text based on a pre-trained model with an adaptation module, and fuse the hidden states of each layer of the model using a scalar mixing layer; The feature combination module is used to separate and reorganize the claim level features by passing the feature sequence output by the scalar mixing layer through the average pooling layer and the feature mixing layer in sequence, so as to form a text feature vector for similarity calculation.

[0013] According to one aspect of the present specification, there is provided an electronic device, comprising a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for calculating the similarity based on the claim hierarchy and a pre-trained model.

[0014] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the steps of the method for calculating the similarity between the claim hierarchy and the pre-trained model.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention proposes a method for constructing technical combination sentences based on the hierarchical structure of claims: this method not only enhances the full expression of technical information, but also makes the calculation of patent similarity more accurate. By constructing technical combination sentences based on the reference relationship between claims, the present invention can capture more technical details and hierarchical features, and improve the model's ability to understand and extract features from patent texts.

[0016] (2) The present invention introduces the XLM-R pre-trained language model and adapter plug-in: The present invention innovatively introduces the multi-language pre-trained model XLM-R for feature extraction, and uses the Adapter plug-in for fine-tuning. The model fuses the hidden states of each layer through a scalar mixing layer, which improves the effect of feature extraction and reduces the complexity of training. The XLM-R pre-trained model has powerful multi-language feature extraction capabilities and can better handle complex information such as professional terms and symbols in patent texts.

[0017] (3) The design of the feature combination module and lossless fusion method of the present invention: The present invention proposes a new feature combination method, that is, the separation and reorganization of the technical combination sentence features are realized through the average pooling layer and the feature mixing layer. The present invention adopts a lossless fusion method of feature splicing, which can effectively retain all feature information in the technical combination sentence, thereby improving the similarity calculation effect of the model. This method ensures the full utilization of features at different levels and realizes efficient fusion of features.

[0018] (4) The present invention verifies the model effect through patents in the field of "quantum computing". Experiments show that the effect of the model of the present invention (accuracy coefficient 0.826, precision coefficient 0.856) can reach and exceed the classification effect of models such as TF-IDF (accuracy coefficient 0.772, precision coefficient 0.761), Doc2vec (accuracy coefficient 0.695, precision coefficient 0.725), Bert (accuracy coefficient 0.822, precision coefficient 0.844), Sbert (accuracy coefficient 0.799, precision coefficient 0.836), Simcse (accuracy coefficient 0.780, precision coefficient 0.789). Compared with other models, the patent feature extraction based on claim hierarchy and pre-training model proposed by the present invention as model input can effectively improve the model classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of a method for calculating similarity between claim hierarchy and a pre-trained model provided in an embodiment of the present invention.

[0021] Figure 2 It is a schematic diagram of the construction rules of the claims and technical combination sentences provided by the embodiment of the present invention.

[0022] Figure 3 It is a schematic diagram of encoding sequence features by the self-attention mechanism within the pre-trained language model XLM-R provided in an embodiment of the present invention.

[0023] Figure 4 It is a schematic diagram of a feature combination module provided by an embodiment of the present invention.

[0024] Figure 5 It is a framework diagram of a similarity calculation system based on claim hierarchy and pre-trained model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] It should be noted that: The patent claims referred to in the present specification may be regarded as the claims of a patent or the claims of a patent application.

[0026] The idea of ​​a pre-trained language model (referred to as a pre-trained model) is to use corpus data to pre-train the model in advance before the model is used for a certain task, so that the model has certain general linguistic knowledge. In this way, when the model is applied to a specific task, the model already has the knowledge of pre-training and can quickly adapt to different tasks. LM-R is a language model based on the BERT architecture, which uses the self-attention mechanism of the transformer model to calculate and extract features. It is composed of multiple layers of attention modules, each of which combines context information. Starting from the model input text sequence, each layer fuses the context information, and then passes the fused information to the next layer, repeating the previous work in the next layer to continue to fuse the context information of the input, and so on, until the output layer. The output layer model outputs a fused feature sequence, in which the feature of the first tag of the sequence is designed to fuse the features of the input text sequence, and the features of the remaining tags of the sequence correspond to each element of the input text sequence. Therefore, the present invention adopts the multilingual pre-training model XLM-R as a pre-training model, and uses its own multilingual characteristics and powerful feature extraction capabilities to extract features from patent texts.

[0027] Thanks to the progress of computer technology and the continuous improvement of hardware levels such as GPU, machine learning, especially deep learning, has developed rapidly in recent years driven by big data, and various models have been widely used in natural language processing, image processing and other fields. Traditional machine learning methods require a large scale of training data and high-quality annotations, and the test set and training set also need to come from the same field and meet the assumption of the same distribution. However, this is often difficult to achieve in practical applications. Sample annotation in certain fields requires a lot of manpower and material resources, and data in different fields must also have different distributions. Transfer learning targets this problem by simulating the ability of people to learn by analogy, transferring the knowledge learned from the source field to the target field, relaxing the restrictions on the data set, and can well solve the problem of a small number of labeled samples in the target field. Therefore, transfer learning has been widely studied and applied in the field of traditional machine learning. Therefore, the present invention optimizes the text data set using transfer learning.

[0028] The attention mechanism is the cornerstone of the transformer model. It is responsible for dynamically capturing the contextual information in the input sentence. In terms of implementation, the attention model learns the importance of each element from the sequence according to the problem that needs to be paid attention to, and then merges the elements according to the importance. Specifically, we can understand it by analogy from the question and content. The question is the topic we need to pay attention to, and the content is the corresponding sequence in the above text that contains all the information. The attention mechanism is to integrate the information according to the degree of match between the question and each piece of information in the content. The question is often referred to as Q (Query), while K (Key) is used to represent the content summary for the information, and V (Value) represents the actual information content. When calculating the attention, the importance of each piece of information to the question is calculated based on Q and K, and then the corresponding V is combined according to the importance. At the same time, multiple groups of attention can be used, which is called multi-head attention mechanism, in which each head of attention can focus on different content.

[0029] Specifically, given a matrix , which contains n d-dimensional query representations and m matrices of keys and values , the formula for calculating the scaled dot-product attention of a single head is:

[0030] in The transformation often uses the softmax function to map the rows of the matrix to a distribution, that is, .

[0031] Multi-head attention is calculated by calling the above formula for each head h:

[0032] in is a learnable linear transformation. The output of the multi-head attention concatenates the outputs of all k heads and then adds another learnable linear transformation :

[0033] The pre-trained language model selected by the present invention is implemented based on a multi-head self-attention mechanism. The characteristic of the self-attention mechanism is that Q, K, and V come from the same sequence, which can be understood as each element in the sequence paying attention to itself and other elements, thereby generating corresponding features. In the present invention, relying on the attention mechanism within the pre-trained model, the features between the levels of the claim technical combination sentence can pay attention to each other and refer to each other, thereby realizing the mutual fusion of the features of the two, thereby enriching the information content of the extracted features.

[0034] The present invention uses a pre-trained model. As mentioned above, the parameters of the pre-trained model are learned in the pre-training stage, so the pre-trained parameters can be used to save training time in the similarity calculation task. However, the number of parameters of the pre-trained model is also large, and a large amount of resources are still required to fine-tune the parameters. In addition, in order to adapt to the application requirements of different fields and specific tasks, a model usually needs to be fine-tuned for each task. In this case, a large amount of training resources need to be consumed, and the traditional pre-trained model fine-tuning method needs to be improved.

[0035] Adapter-tuning inserts modularly into the structure of the pre-trained model, adds a designed adapter module to each layer, and keeps the parameters of the original pre-trained model unchanged during fine-tuning, only updating the parameters of the adapter, thereby greatly reducing the amount of parameters that need to be saved and updated during the fine-tuning process. Another advantage of this module insertion method is that it is conducive to the migration of the model, because no changes are made to the parameters and structure of the original pre-trained model, so the adapter module trained on a certain task can be taken out separately for saving and sharing, just like a plug-in that can be easily embedded in the pre-trained model so that the pre-trained model can adapt to different tasks. Based on the ease of use and portability of the adapter module, it is very suitable for translation quality tasks, so it is applied to the pre-trained model in the model of the present invention to improve the model fine-tuning method. The amount of parameters that need to be trained for the improved model is significantly reduced by 57, thereby greatly reducing the training cost.

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0037] Based on the claim hierarchy structure and pre-training model and other methods, the present invention constructs a similarity calculation method based on the claim hierarchy and the pre-training model, which further improves the calculation accuracy of the similarity.

[0038] like Figure 1 As shown, a method for calculating similarity between claim hierarchy and pre-trained model proposed in an embodiment of the present invention includes the following steps: Step 1: Based on the claim hierarchy structure, the acquired claims are split and technical combination sentences are constructed to form a reorganized claim text.

[0039] The patent claims are one of the most important parts of the patent text, which defines the scope and content of patent protection. The claims are composed of claims that record the technical features of the invention or utility model, and have strict legal form requirements. The hierarchical structure of the claims usually includes main claims and dependent claims. The main claim describes the core technical features of the invention and is a required item, while the dependent claim further defines and refines the content of the main claim and is an optional item. It can be seen that the patent claims are an interrelated whole. The study and application of the claim text requires full consideration of the relationship between the claims. Understanding and utilizing this hierarchical structure is crucial for accurately calculating patent similarity. Therefore, the present invention uses the reorganized claims as the feature extraction text during feature extraction, thereby ensuring that the extracted features can fully characterize the detailed content of the text.

[0040] The text construction step is implemented by combining the writing rules and citation relationship of patent (or patent application) claims. Usually, a dependent claim expresses the reference to a claim with "according to the claim...". If a claim contains the expression "according to the claim...any claim" or "according to the claim...or...", it is regarded as multiple dependent claims and cites multiple claims. Combined with the rules constructed by Xiang Shuxuan et al. (Research on Patent Similarity Calculation Method Based on Claim Hierarchical Characteristics [J]. Data Analysis and Knowledge Discovery, 2024, 8(02):33-43): The citation relationship between claims gives each claim the characteristics of the number of citations and the level at which it is located. The number of citations refers to the number of claims that the claim has cumulatively cited, and the level at which it is located refers to the number of levels of citations through which the claim is directly related to the independent claim. The claim citation relationship forms the context of technical information. According to the claim citation relationship and the level formed by it, the claim text can be organized into a tree structure. The claim tree and technical combination sentence construction are as follows: Figure 2 shown.

[0041] In the embodiment of the present invention, the specific implementation process of this step is described as follows: Each path from the root node to the leaf node of the claim tree represents a technical information chain, forming a complete description of a certain protected content. The path includes the progression and interpretation relationship of each claim, as well as the superior and subordinate relationship between technical concepts. Therefore, reorganizing the claims according to the reference relationship to form a technical combination sentence can make the contained technical information more fully understood and applied.

[0042] (1) The present embodiment uses the python pandas library to read an Excel file storing patent claims from a specified path and stores its content in the df data frame. Then, based on the location of the patent claims, the text of the claims is extracted and stored in the claims_text variable. Based on the data storage format, each claim is separated by a line break, and the split('\n') method is used to split the claim text into multiple entries to form a claim list claims_list.

[0043] (2) The embodiment of the present invention extracts the claims of a patent by defining a regular expression. independent_pattern is used to match independent claims, and its form is usually "1. A...", where the number represents the claim number and "A" is the beginning mark of the claim description. dependent_pattern is used to match dependent claims, and its form is usually "According to claim X...", where X is the number of the referenced independent claim. When traversing the claims_list list, each claim text will be matched by a regular expression: if an independent claim is matched, the number and content of the claim are stored in the independent_claims dictionary, with the key being the claim number and the value being the complete text of the claim. If a dependent claim is matched, the number of the independent claim referenced by it and the content of the dependent claim are stored in the dependent_claims dictionary, with the key being the number of the dependent claim and the value being a tuple, which contains the referenced independent claim number and the text of the dependent claim. At the same time, all parsed claim numbers are stored in the claim_order list in order to record the original order.

[0044] (3) The embodiment of the present invention defines a recursive function combine_claims, which is used to combine dependent claims and the independent claims they reference. The function receives a claim number claim_number as a parameter, and first checks whether the number has been processed to avoid infinite recursion caused by circular references. Next, the function determines whether the current number belongs to an independent claim: if it is an independent claim, the complete text of the independent claim is directly returned. If it is a dependent claim, the combine_claims function is first recursively called to obtain the complete text of the referenced independent claim, and then the text is spliced ​​with the text of the current dependent claim to form a complete combination result. Through this recursive method, the function can gradually reference and combine dependent claims, ensuring that all references in each dependent claim are correctly parsed and spliced.

[0045] (4) The embodiment of the present invention reorganizes the claims based on the claims text disassembled in the previous order. First, by traversing the independent_claims dictionary, all independent claims are directly added to the combined_claims dictionary to ensure that the independent claims maintain their original structure. Then, the code traverses the dependent claims in the dependent_claims dictionary, calls the recursive function combine_claims for each dependent claim, combines the referenced independent claims and dependent claims together, and stores the combined results in the combined_claims dictionary. In this way, all claims - whether independent or dependent - are correctly combined and processed.

[0046] (5) The code of the embodiment of the present invention constructs the combined claims into a new list results in order according to the original order in the claim_order list. This ensures that the order of the claims is consistent with the order in the original document. The code then converts the results list into a new pandas data frame output_df and writes it to a new Excel file 'combined_patent_claims.xlsx', with the output path being the specified desktop folder. This process completes the parsing, combining and saving of the entire claim.

[0047] Step 2: Based on the pre-trained model with the adaptation module, feature extraction is performed on the reorganized claim text, and the hidden states of each layer of the network are fused using a scalar mixing layer.

[0048] The feature extraction step uses the pre-trained language model XLM-R to extract feature information from the technical combination sentence text. On the one hand, this step needs to make full use of XLM-R's powerful language representation ability and pre-trained linguistic knowledge to extract as many features as possible that are conducive to the similarity calculation effect. On the other hand, it is necessary to consider the fine-tuning cost problem caused by the huge number of parameters of XLM-R and reduce the training cost as much as possible. To achieve the above goals, the embodiment of the present invention uses the hierarchical structure of the technical combination sentence to be spliced ​​level by level as input, uses the hidden state of each layer of the network of the scalar mixed layer fusion model, and changes the traditional fine-tuning idea to use the Adapter plug-in training model.

[0049] In the embodiment of the present invention, the specific implementation process of this step is described as follows: (1) A multilingual pre-trained model that uses a sequence of technical combination sentences concatenated step by step as input.

[0050] The feature extraction part receives the sequence of technical combination sentence texts spliced ​​together level by level as input, and uses the XLM-R model as the encoder to encode the features of the sequence. Since the self-attention mechanism is used internally in XLM-R for feature extraction, the self-attention mechanism can be fully utilized by splicing the technical combination sentence texts together level by level, so that each level of the technical combination sentence pays attention to each other. The self-attention mechanism is used internally in XLM-R to capture the feature information of each token in the input sequence. For this model, XLM-R receives the sequence of technical combination sentence texts spliced ​​together level by level as input. Internally, each word in the text sequence integrates the feature information of all words in the sequence. Figure 3 This diagram describes the calculation process of a single word feature in a single-layer single-head attention. In the entire XLM-R model, there are hundreds of such structures. In each layer of XLM-R, a hidden state vector (i.e., Output in the figure) is generated and passed to the next layer as input.

[0051] (2) Adapter module.

[0052] While XLM-R provides powerful pre-trained knowledge and feature extraction capabilities, the large number of parameters in XLM-R requires more data and consumes more resources when training the model. To this end, this model adds an Adapter module to XLM-R. When training the model, only the parameters in the Adapter module can be updated, and other parameters in XLM-R remain unchanged. This module can learn the relevant features of downstream tasks during training, which can effectively improve the efficiency of training. The effect of the Adapter module is very close to the effect of updating all parameters of the XLM-R model, but the training cost required by the Adapter is smaller. The specific approach of the Adapter module is to first reduce the dimension of the original input through full connection to reduce the number of parameters, and then restore the dimension after passing through the internal neural network to form a bottleneck structure. When applying, you only need to add two adapter modules in the middle of each layer of the original pre-trained model. During the fine-tuning process, the parameters of the original pre-trained model are frozen and not updated, and only the parameters of the adapter module are updated, which greatly reduces the number of parameters that need to be updated and saved in the fine-tuning stage.

[0053] (3) Scalar Mix layer.

[0054] After the pre-trained model performs feature encoding on the technical combination sentence text sequence, all the hidden state vectors generated by each layer of the pre-trained model are passed to a scalar mixing layer. This module learns the weighted sum of the hidden states of each layer of the encoder to generate a new aggregated hidden state sequence. This aggregated hidden state sequence contains the feature information of the technical combination sentence text sequence.

[0055] Step 3: The feature sequence output by the scalar mixing layer is sequentially passed through the average pooling layer and the feature mixing layer to separate and reorganize the claim level features to form a text feature vector for similarity calculation.

[0056] Feature combination step ( Figure 4 ) receives the feature sequence obtained from the feature extraction module as input, and needs to split and reorganize the feature sequence of the hybrid technology combination sentence text features to generate the features used for similarity calculation. The feature combination step sequentially separates and reorganizes the text features of the technical combination sentence through the average pooling layer (Avg Pool) and the feature mixing layer (Mix Layer).

[0057] In the embodiment of the present invention, the specific implementation process of this step is described as follows: (1) Average pooling layer (Avg Pool).

[0058] The average pooling layer averages the text features of the technical combination sentences according to the time dimension of the sequence. Assuming that the feature dimension of a technical combination sentence is , where N represents the length of the technical combination sentence sequence, that is, the number of words in the technical combination sentence text, and d represents the dimension of each word tag in the sequence, that is, there are N words in the sequence, and the feature of each word is a d-dimensional vector. The average pooling layer averages each feature dimension of all words in the sequence to form a new feature. The feature dimension after pooling is The average pooling layer has two main functions. The first is to aggregate the features of each dimension of the sequence through the average value, so as to aggregate the features of the entire sentence and realize the conversion from the features of each word in the sequence to the features of the entire sentence. The second is to unify the features of the technical combination sequence into the same shape for easy recombination and input into the fully connected layer.

[0059] (2) Feature mixing layer (Mix Layer).

[0060] The feature combination group receives the technical combination sentence features of the average pooling layer as input, and combines them in a specific way to form the features required for the final model prediction. The original intention of designing this layer of neural network is to enable the model to comprehensively consider the features from each technical combination sentence. In order to enable the model to better learn how to use the features of the technical combination sentences, this model uses feature concatenation to mix the features at this layer. The feature concatenation method uses the method of concatenating the feature vectors of the technical combination sentences. It is the simplest and most direct feature combination method. Assume that the feature dimensions of the technical combination sentences before concatenation are all , then the concatenated feature dimension will become Feature concatenation can retain all feature information of the technical combination sentence text, losslessly combine the features, without any information loss in the combination, and retain the combination structure consistent with the model input.

[0061] The implementation basis of each embodiment of the present invention is realized by programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a similarity calculation system based on the claim hierarchy and the pre-trained model, which is used to execute the similarity calculation method based on the claim hierarchy and the pre-trained model in the above method embodiment.

[0062] See also Figure 5 The system includes: a text construction module, which is used to split the acquired claims and construct technical combination sentences based on the claim hierarchy structure to form a reorganized claim text; a feature extraction module, which is used to extract features from the reorganized claim text based on a pre-trained model with an adaptation module, and use a scalar mixing layer to fuse the hidden states of each layer of the model network; a feature combination module, which is used to sequentially pass the feature sequence output by the scalar mixing layer through an average pooling layer and a feature mixing layer to separate and reorganize the claim hierarchy features to form a text feature vector for similarity calculation The similarity calculation system based on claim hierarchy and pre-training model provided in the embodiment of the present invention adopts Figure 5 Several modules in the patent claim construct technical combination sentences based on the claim hierarchy structure, and use the multilingual XLM-R pre-trained model for feature extraction. The feature extraction part uses a pre-trained model with an Adapter module, and combines the features of each layer in the pre-trained model through a scalar mixing layer to increase the amount of information extracted from the feature. The feature combination module aggregates the extracted features and fuses the two parts of the features through the Mix Layer, thereby constructing an aggregated feature with a claim hierarchy structure, which in turn assists in the calculation of patent similarity.

[0063] In the embodiment of the present invention, the specific implementation process of the system is described as follows: The model is based on the TransQuest architecture design and consists of three modules. The first is the text construction module, which constructs technical combination sentences based on the reference relationship of the patent claim hierarchy structure, so that the technical information contained in it can be more fully understood and used, thereby improving the accuracy of similarity calculation; the second is the text feature extraction module, which uses the pre-trained language model XLM-R to extract the feature information of the technical combination sentence, uses the scalar mixing layer to fuse the hidden states of each layer of the model network, and changes the traditional fine-tuning idea to use the Adapter plug-in training model; the last is the feature combination module, which receives the feature sequence obtained from the feature extraction module as input, and the feature combination module successively separates and reorganizes the claim hierarchy features through the average pooling layer (AvgPool) and the feature mixing layer (Mix Layer), and finally forms a text feature vector to assist in similarity calculation.

[0064] It should be noted that the system embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference lies in the setting of corresponding functional modules, and the principles thereof are basically the same as the principles of the above-mentioned system embodiments provided by the present invention. As long as technical personnel in this field refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned system embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, they will improve the modules in the above-mentioned system embodiments to obtain corresponding system class embodiments for implementing the methods in other method class embodiments.

[0065] This invention uses patents in the field of quantum computing as the research object, and verifies the effectiveness of the method of the invention through patent classification comparison experiments and similarity calculations. In the data collection part, the invention uses the IncoPat database as the data source, "quantum computing" as the search keyword, "valid" as the patent validity, and "Chinese invention patent" as the patent type. Finally, there are 1459 patents. After deleting null values ​​and data that do not meet the calculation requirements, 1328 patent data are finally obtained for experiments. The data distribution is shown in Table 1.

[0066] Table 1 Patent distribution in the field of quantum computing

[0067] In terms of experimental verification, the present invention selects benchmark methods such as TF-IDF, Doc2vec, Bert, Sbert, and Simcse based on the original full text (Base) and technical combination sentences (TCS), as shown in Table 2, to compare the extraction effects of patent features using the method proposed in the present invention and similar benchmark methods.

[0068] Table 2 Comparison of benchmark methods

[0069] At the evaluation index level, the present invention uses the precision (P macro ), recall, R macro ) and F1 value (F1-score, F1-score macro ) Accuracy is used to evaluate the classification results. The specific calculation formula is as follows:

[0070] in, i Represents each category label; N Represents the number of tags; TP i (true positive) indicates the number of positive samples of this class predicted as positive examples; FP i (false positive) indicates the number of negative samples of this class that are predicted as positive examples; FN i (falsenegative) indicates the number of positive samples of this class that are predicted as negative examples, TN i (true negative) indicates the number of negative samples of this class that are predicted as negative examples.

[0071] From the analysis of Table 1, it can be seen that the patents in the field of quantum computing are mainly distributed in the G and H parts. In order to reduce the impact of sample imbalance, the present invention specifically defines the classification task as a binary classification task of the G and H parts. The data is screened and the screened data is split into a training set, a validation set and a test set in a ratio of 6:2:2. The present invention uses the patent representation as the input of the artificial neural network (ANN) with reference to the method of the article, and compares the classification effect of the model. In the experimental process, dropout and early stopping are used to prevent overfitting. The early stopping strategy is to stop training when the model has no Pearson improvement for 10 consecutive times during the training process (that is, the model may reach the optimal state and cannot learn new knowledge). During the training, the following parameters are set according to the effect. Use a batch size of 2 and a gradient accumulation of 4. In order to deal with the gradient explosion, a gradient clipping with a norm of 1 is used. The learning rates of 1.0e-4 and 3.0e-4 are used for the pre-training layer and other layer parameters, respectively, and the dropout value is set to 0.05. The results are shown in Table 3.

[0072] Table 3 Comparison of model effects

[0073] The experimental results show that combining the claims in a hierarchical structure and inputting them into the model can improve the classification effect of all models to a certain extent. Compared with other models, the patent feature extraction based on the claim hierarchical structure and pre-trained model proposed in the present invention as model input can effectively improve the model classification effect.

[0074] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides an electronic device, including a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for calculating the similarity based on the claim hierarchy and the pre-trained model.

[0075] Based on the same inventive concept as the above-mentioned embodiment, an embodiment of the present invention also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the steps of the method for calculating the similarity based on the claim hierarchy and the pre-trained model.

[0076] In summary, based on the features and characteristics of patent texts, the present invention proposes a similarity calculation method based on claim hierarchy and pre-trained models. The method constructs claim technical combination sentences based on the claim hierarchy structure, and uses a multilingual XLM-R pre-trained model for feature extraction and combination. The feature extraction part uses a pre-trained model with an Adapter module, and combines the features of each layer in the pre-trained model through a scalar mixing layer to increase the amount of information extracted from the feature. The feature combination part aggregates the extracted features and fuses the two parts of the features through a Mix Layer, thereby constructing an aggregated feature with a claim hierarchy structure, thereby assisting in the calculation of patent similarity.

[0077] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A similarity calculation method based on claim hierarchy and pre-training model, characterized in that: include: Based on the claim hierarchy structure, the acquired claims are split and technical combination sentences are constructed to form a reorganized claim text; Based on the pre-trained model with the adaptation module, the feature extraction of the reorganized claim text is performed, and the hidden states of each layer of the model network are fused using a scalar mixing layer; The feature sequence output by the scalar mixing layer is sequentially passed through the average pooling layer and the feature mixing layer to separate and reorganize the claim level features to form a text feature vector for similarity calculation.

2. The similarity calculation method based on claim hierarchy and pre-training model according to claim 1 is characterized in that: The method further comprises: A pre-trained model with an adaptation module is constructed. Different tasks correspond to different adaptation modules. After training, the adaptation module is inserted into each layer of the pre-trained model in the form of a plug-in.

3. The similarity calculation method based on claim hierarchy and pre-training model according to claim 2 is characterized in that: After the adaptation module is inserted into the pre-trained model, it includes: reducing the dimension of the original input through full connection, and then restoring the dimension after passing through the internal neural network to form a bottleneck structure.

4. The similarity calculation method based on claim hierarchy and pre-training model according to claim 1 is characterized in that: Feature extraction is performed on the reorganized claim text, and the hidden states of each layer of the model network are fused using a scalar mixing layer, including: A sequence of receiving technical combination sentence texts that are spliced ​​step by step; Encode the received sequence and use the self-attention mechanism to extract features; The hidden state vector generated by each layer of the pre-trained model is passed to the scalar mixing layer to form an aggregated hidden state sequence, where the aggregated hidden state sequence contains feature information of the technical combination sentence text sequence.

5. The similarity calculation method based on claim hierarchy and pre-training model according to claim 4 is characterized in that: The average pooling layer is configured to average the text features of the technical combination sentences according to the time dimension of the sequence; the feature mixing layer is configured to mix the features output by the average pooling layer in a feature splicing manner to obtain a mixed and reorganized text feature vector.

6. The similarity calculation method based on claim hierarchy and pre-training model according to claim 1 is characterized in that: Based on the claim hierarchy, the acquired claims are split and technical combination sentences are constructed, including: Based on the location of the patent claims, the text of the claims is extracted and stored; The patent claims are extracted by defining regular expressions. All parsed claim numbers are stored in the claim_order list in order, recording the original order. Define a recursive function for combining dependent claims and the independent claims they reference, so that all references in each dependent claim are correctly parsed and concatenated; Reorganize the claims based on the claim text disassembled in the preamble; According to the original order in the claim_order list, the reorganized claims are sequentially constructed into a new list and output, completing the splitting of the entire claim and the construction of the technical combination sentence.

7. A similarity calculation system based on claim hierarchy and pre-trained model, characterized in that: include: The text construction module is used to split the acquired claims and construct technical combination sentences based on the claim hierarchy structure to form a reorganized claim text; A feature extraction module, which is used to extract features from the reorganized claim text based on a pre-trained model with an adaptation module, and fuse the hidden states of each layer of the model using a scalar mixing layer; The feature combination module is used to separate and reorganize the claim level features by passing the feature sequence output by the scalar mixing layer through the average pooling layer and the feature mixing layer in sequence, so as to form a text feature vector for similarity calculation.

8. An electronic device, characterized in that: It includes a memory and a processor; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the steps of the method for calculating the similarity based on the claim hierarchy and the pre-trained model as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which enable the computer to execute the steps of the method for calculating the similarity between the claim hierarchy and the pre-trained model as described in any one of claims 1 to 6.

Images