Patent plagiarism judgment method based on graph attention network

Through a graph attention network based on the BERT-GAT model, the claim graph is constructed and global semantic features are extracted, and the problem of limited full text processing of patent claims in the prior art is solved, and the high accuracy of patent plagiarism relationship is achieved and Accurate identification.

CN118886408APending Publication Date: 2024-11-01GUANGDONG UNIVERSITY OF BUSINESS STUDIES
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411118820.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The prior art is limited by the BERT-base-chinese input length limitation when processing the full text of the patent claims, resulting in most patent plagiarism detection research being limited to the abstract and the first claim, and is unable to effectively process long texts.

Method used

The graph attention network based on the BERT-GAT model is adopted, and the problem of long text processing is overcome by constructing the claim graph, sentence-level semantic feature extraction and graph-based global semantic feature extraction, combined with the feedforward neural network to judge the plagiarism relationship.

Benefits of technology

It significantly improves the accuracy and accuracy of the patent plagiarism relationship, can effectively handle the full text of the claims, and provides a more accurate method for determining patent plagiarism.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118886408A_ABST
    Figure CN118886408A_ABST
Patent Text Reader

Abstract

According to the patent text similarity judgment method provided by the invention, a novel plagiarism patent recognition technology based on deep learning is constructed by researching and applying a graph attention mechanism. By integrating powerful semantic understanding of BERT and a graph attention network of GAT, the method can effectively capture complex semantic relation and structure information in patent texts. Compared with a traditional method, the accuracy and precision of the patent plagiarism relation are remarkably improved. The method has wide application prospects in the field of text similarity detection, has huge potential in the aspect of improving the text plagiarism recognition accuracy and precision, and has important application significance in patent review and related fields.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a patent plagiarism determination method based on a graph attention network. Background Art

[0002] Against the backdrop of growing awareness of global intellectual property protection, the patent system plays a key role in stimulating innovation and protecting the rights of inventors. However, in practice, plagiarism, patchwork, and falsification of data have seriously damaged the fairness and authority of the patent system and inhibited the progress of scientific and technological innovation. In view of this, it is urgent to use advanced scientific and technological means to accurately identify such problems.

[0003] In recent years, deep learning technology innovation, especially in the field of natural language processing (NLP), has evolved from early one-hot encoding, n-gram, TF-IDF statistical models to word embedding word2vec, glove, recurrent neural network RNN, long short-term memory network LSTM, text convolutional neural network TextCNN, and then to the Transformer series models based on self-attention mechanism (such as BERT, GPT), which have significantly improved the expressiveness of NLP tasks. On this basis, models adapted to specific tasks continue to emerge, such as the graph neural network (GNN) for graph data structures and its variants, graph convolutional neural network GCN, graph attention network GAT, the twin neural network Sentence-BERT (SBERT) for text pair similarity comparison, and models such as XLNet, Longformer, and Bigbird suitable for long text processing.

[0004] In the specific task of patent plagiarism identification, although a number of studies have made progress, such as the plagiarism detection system that uses space vector models and similarity algorithms, combines document style with n-gram feature representation, integrates BERT, LSTM and differential evolution algorithm (DE), as well as the PatentSBERTa model that focuses on patent claims and the method of calculating patent similarity based on the citation relationship between claims and technical combination sentences, these studies are limited by the input length limit of existing large language pre-training models such as BERT-base-chinese (usually no more than 512 characters) when processing the full text of patent claims. Therefore, most research on patent documents is limited to the patent abstract and the first claim.

[0005] To this end, this paper proposes an innovative patent plagiarism determination method based on the BERT-GAT model, which is specifically designed for the full text of the claims and uses a local to global analysis method to overcome the difficulty of long text processing. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a patent plagiarism determination method based on a graph attention network, and the patent plagiarism determination method comprises the following steps: Step 1: Graph construction of claims; In patent claims, there are usually multiple claims and reference relationships between them. The references between claims are often based on the key statement "according to claim x", where x is the number of the cited claim. Due to the special nature of claims, this paper takes each independent claim as a node in a graph data structure, and constructs its reference relationship as a directed edge in the graph data structure to form a claim reference network, which is called claim graph G in this paper, and its corresponding adjacency matrix is ​​recorded as A.

[0007] Step 2: Extract sentence-level semantic features; This article uses the Chinese version of the BERT pre-trained model, bert-base-chinese, to extract text semantic features. BERT includes a word embedding layer, an attention layer, a normalization layer, and a forward propagation layer. Each claim is segmented and input into BERT separately. During the input process, each node is first mapped to a low-dimensional feature vector representation through the word embedding layer. Then, through a multi-layer attention mechanism, the local relational features of each sentence are captured, and the semantic feature representation of each node is calculated. The sentence-level feature is denoted as X.

[0008] Step 3: Graph-based global semantic feature extraction: In the graph-based global semantic feature extraction, this paper uses the adjacency matrix A constructed in step 1 and the sentence-level feature X extracted in step 2 as the input of the graph model, and uses a three-layer graph multi-head attention network to calculate the connection between each node and reconstruct its features. Finally, the output global features are averaged along the node dimension to obtain a semantic feature vector with global information, recorded as Y.

[0009] Step 4: Patent plagiarism relationship judgment: In the present invention, firstly, the semantic feature vectors of the global information of the claims of the patent to be evaluated P1 and the comparative patent P2 are obtained through the above steps, respectively, and the absolute value of the difference between Y1 and Y2 is calculated as the feature vector for judging the similarity of the sentence pair. Then it is placed in the feedforward neural network layer and mapped into a two-dimensional plagiarism state probability distribution vector, and finally used The function is used to judge whether there is a plagiarism relationship. Through the above processing, the output of the model realizes the judgment of whether there is a plagiarism relationship between two patents.

[0010] The present invention provides a method for determining the similarity of patent texts. The method studies and uses the graph attention mechanism to construct a new deep learning-based plagiarized patent identification technology. By integrating the powerful semantic understanding of BERT and the graph attention network of GAT, the method can effectively capture the complex semantic relationships and structural information in the patent text. Compared with traditional methods, the accuracy and precision of patent plagiarism relationships have been significantly improved. The method has broad application prospects in the field of text similarity detection, has great potential in improving the accuracy and precision of text plagiarism identification, and has important application significance in patent review and related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the patent plagiarism determination method based on graph attention network of the present invention. DETAILED DESCRIPTION

[0012] Combine the following Figure 1 The present invention provides a method for determining patent plagiarism based on a graph attention network, the method comprising: Step 1: Construction of the graph of the claims; the graph of the claims in step 1 is constructed by the following method: The claims of a patent usually contain multiple claims and references between them. The references between claims are often based on the key statement "according to claim x", where x is the number of the cited claim. Due to the special nature of the claims, this paper treats each independent claim as a node in a graph data structure, and constructs its reference relationship as a directed edge in the graph data structure, forming a claim reference network, which is called the claim graph G in this paper, and its corresponding adjacency matrix is ​​recorded as A.

[0013] Step 1.1: This paper processes each claim in the claims into clauses and treats each claim as a separate graph node. The nodes of the graph are represented as Indicates the text of each claim after division.

[0014] Step 1.2: If V Cited in , then there exists point to The directed edge of Can Transmit information, and its edge set is represented as , indicating the connection between the various claims.

[0015] Step 1.3: Construct a claim node based on the edit and E and The size between The adjacency matrix A of is the number of nodes in the graph. The contents of each element in the matrix A are 1 and 0 respectively to indicate whether there is a connection between two nodes, as shown in the formula:

[0016] Step 2: sentence-level semantic feature extraction; the sentence-level semantic feature extraction in step 2 is constructed by the following method: This article uses the Chinese version of the BERT pre-trained model, bert-base-chinese, to extract text semantic features. BERT includes a word embedding layer, an attention layer, a normalization layer, and a forward propagation layer. Each claim is segmented and input into BERT.

[0017] Step 2.1: First, map each node to a low-dimensional feature vector representation through the word embedding layer.

[0018] Step 2.2: Then, through the multi-layer attention mechanism, the local relational features of each sentence are captured, and the semantic feature representation of each node is calculated. The process is shown in the following formula, where represents the input features, represents the word embedding matrix, Represents the text features after word embedding, The function represents the forward propagation calculation of the entire model. Indicates the output result.

[0019]

[0020]

[0021] Step 3: Extraction of global semantic features based on graphs; Step 3 is constructed by the following method: In the graph-based global semantic feature extraction, this paper takes the adjacency matrix A constructed in step 1 and the sentence-level feature Xoutput extracted in step 2 as the input of the graph model. Xoutput is represented in the graph attention network as { … }, where n is the number of items in the claim, which respectively represent the sentence-level semantic encoding of each numbered claim calculated by BERT. Then a three-layer graph multi-head attention network is used to calculate the connection between each node and reconstruct its features. Finally, the output global features are averaged along the node dimension to obtain a semantic feature vector with global information, recorded as Y.

[0022] Step 3.1: Attention coefficient calculation: for node pairs Calculate an attention score between each pair , this score reflects the node For Node The influence of The calculation is as shown in the following formula, where and The nodes are and nodes The embedding vector of Represents a splicing operation, is the learnable attention parameter, is the activation function:

[0023] Step 3.2: Attention mask calculation: Construct an attention mask matrix Amask based on the adjacency matrix A, in which the positions with the original value of 1 are set to 0, the positions with the original value of 0 are set to negative infinity, and Add it to Amask to get the masked attention matrix .

[0024] Step 3.3: Normalize attention weights: normalize the attention scores Transformed into normalized similarity weight, as shown in the following formula, where Representation Node A collection of neighbors:

[0025] Step 3.4: Feature aggregation: According to the normalized attention weights, the features of neighboring nodes are weighted and summed to obtain the node The new feature representation of is shown in the following formula, where Represents a nonlinear activation function:

[0026] Step 3.5: A three-layer graph multi-head attention network is used to calculate the connection between each node and reconstruct its features as , as shown in the following formula, where K represents the number of heads, represents the attention score of the K-th head.

[0027]

[0028] Step 3.5: The output global features are averaged along the node dimension to obtain a semantic feature vector with global information, which is used as the judgment basis Y for text plagiarism identification.

[0029] Step 4: Patent plagiarism relationship judgment; the patent plagiarism relationship judgment in step 4 is constructed by the following method: Step 4.1: Through the above steps, the semantic feature vectors of the global information of the claims of the evaluated patent P1 and the comparative patent P2 are obtained respectively, which are denoted as Y1 and Y2 respectively, and the absolute value of the difference between Y1 and Y2 is calculated.

[0030] Step 4.2: Put the result of step 4.1 into the feedforward neural network layer and map it into a two-dimensional plagiarism state probability distribution vector. Finally, use The function determines whether the relationship is plagiarism or not. The formula is as follows. The Sigmod function can map the calculation result to a value between 0 and 1, and use 0.5 as the threshold for judgment. If it is greater than or equal to 0.5, it is judged as plagiarism, and if it is less than 0.5, it is judged as non-plagiarism.

[0031]

Claims

1. A patent plagiarism determination method based on graph attention network, characterized in that: The method comprises the following steps: Step 1: Take each independent claim as a node in a graph data structure, and construct its reference relationship as a directed edge in the graph data structure to form a claim reference network, which is denoted as claim graph G, and its corresponding adjacency matrix is ​​denoted as A; Step 2: Extract sentence-level semantic features. Use the Chinese version of the BERT pre-trained model, bert-base-chinese, to extract text semantic features. BERT includes a word embedding layer, an attention layer, a normalization layer, and a forward propagation layer. Segment each claim into words and input them into BERT. During the input process, first map each node to a low-dimensional feature vector representation through the word embedding layer, and then capture the local relational features of each sentence through a multi-layer attention mechanism, and calculate the semantic feature representation of each node. Step 3: Extract global semantic features based on the graph; use the adjacency matrix A constructed in step 1 and the sentence-level feature X extracted in step 2 as the input of the graph model, use a three-layer graph multi-head attention network to calculate the connection between each node and reconstruct its features, and finally average the output global features along the node dimension to obtain a semantic feature vector with global information, recorded as Y; Step 4: Determine the relationship between patent plagiarism. Through the above steps, the semantic feature vectors of the global information of the claims of the patent to be evaluated P1 and the patent to be compared P2 are obtained, which are recorded as Y1 and Y2 respectively, and the absolute value of the difference between Y1 and Y2 is calculated as the feature vector for determining the similarity of the sentence pair. Then, it is placed in the feedforward neural network layer and mapped into a two-dimensional probability distribution vector of plagiarism status. Finally, it is used The function determines whether the relationship is plagiarized.

2. The patent plagiarism determination method according to claim 1, characterized in that: The sentence level semantic feature extraction further comprises: Step 2.1: Map each node to a low-dimensional feature vector representation through the word embedding layer; Step 2.2: Through the multi-layer attention mechanism, the local relational features of each sentence are captured, and the semantic feature representation of each node is calculated. ,in represents the input features, represents the word embedding matrix, Represents the text features after word embedding, The function represents the forward propagation calculation of the entire model. Indicates the output result.

3. The patent plagiarism determination method according to claim 1, characterized in that: The graph-based global semantic feature extraction is constructed by the following method: The adjacency matrix A constructed in step 1 and the sentence-level feature Xoutput extracted in step 2 are used as the input of the graph model. Xoutput is represented in the graph attention network as { … }, where n is the number of items in the claim, which represent the sentence-level semantic encoding of each numbered claim calculated by BERT, and then a three-layer graph multi-head attention network is used to calculate the connection between each node and reconstruct its features; finally, the output global features are averaged along the node dimension to obtain a semantic feature vector with global information, denoted as Y; Step 3.1: Attention coefficient calculation: for node pairs Calculate an attention score between each pair , this score reflects the node For Node The influence of Calculated, where and The nodes are and nodes The embedding vector of Represents a splicing operation, is the learnable attention parameter, is the activation function: Step 3.2: Attention mask calculation: Construct an attention mask matrix Amask based on the adjacency matrix A, in which the positions with the original value of 1 are set to 0, the positions with the original value of 0 are set to negative infinity, and Add it to Amask to get the masked attention matrix . Step 3.3: Normalize attention weights: normalize the attention scores Converted into normalized similarity weight ,in Representation Node A collection of neighbors: Step 3.4: Feature aggregation: According to the normalized attention weights, the features of neighboring nodes are weighted and summed to obtain the node New feature representation ,in Represents a nonlinear activation function: Step 3.5: Use a three-layer graph multi-head attention network to calculate the connection between each node and reconstruct its features, as shown in the following formula, where K represents the number of heads, represents the attention score of the K-th head. Step 3.5: The output global features are averaged along the node dimension to obtain a semantic feature vector with global information, which is used as the judgment basis Y for text plagiarism identification.