A visual patent retrieval method based on pre-trained language model

Abstract

The present invention relates to the technical field of natural language processing, and specifically to a visual patent search method based on a pre-trained language model. In a patent keyword search scenario, the present invention extracts keywords and represents word vectors from patent texts based on the pre-trained language model, reduces the dimensionality of patent features returned by the search, and visually displays the search results in the form of a scatter plot. Compared with search results displayed in a list form, the method is more intuitive and provides richer search information such as the degree of similarity between patents and patent clustering.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a visual patent retrieval method based on a pre-trained language model. Background Art

[0002] The primary task of patent search is to retrieve relevant patent texts from massive patent databases. Currently, the most commonly used patent text search strategy is keyword search, which constructs a search formula based on the keywords to be searched and retrieves relevant patent texts from the patent database.

[0003] Keyword search strategies can miss important patent texts with high technical similarity, making it difficult to achieve high-precision patent searches. To address these technical issues, patent CN112000783A discloses a patent recommendation method, apparatus, device, and storage medium based on text similarity analysis. This patent recommendation method obtains subject keywords and descriptive keywords, uses the subject keywords and all descriptive keywords as search terms to obtain a basic similar text set, and then uses the subject keywords and each descriptive keyword as search terms to obtain an extended similar text set. The extended similar text set is then traversed. For each extended similar text, the similarity between the extended similar text and the basic similar texts in the basic similar text set is calculated based on the text feature words of the extended similar text and the search terms corresponding to the extended similar text. When the similarity between the extended similar text and any basic similar text in the basic similar text set exceeds a predetermined threshold, the extended similar text is moved to the basic similar text set. This patent recommendation method can improve the recall rate of similar texts and reduce the miss detection rate.

[0004] Although the above existing technologies can improve the recall rate to a certain extent, the following problems still exist in actual use:

[0005] 1. The search results primarily display a rough ranking of similarities in digital form, making it difficult to intuitively display the differences between individual similarities. Furthermore, the similarity values ​​for each patent are included in the search results, which is not intuitive for long lists, requiring scrolling through the list to see the corresponding similarity values.

[0006] 2. In the displayed search result list, the similarity between the retrieved patent texts is not calculated. Therefore, for the search personnel, it is impossible to intuitively reflect the similarity between the patent texts.

[0007] In summary, there is still much room for improvement in the current patent search results. Summary of the Invention

[0008] To avoid and overcome the technical problems existing in the prior art, the present invention provides a visual patent search method based on a pre-trained language model. The present invention can clearly and intuitively display the similarity relationship between the search keyword and the retrieved patent text, as well as the similarity relationship between the retrieved patent texts.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A visual patent search method based on a pre-trained language model includes the following search steps:

[0011] S1. Using patent text annotated with existing keywords, train a RoBERTa+Bi-LSTM model to extract keywords from patents.

[0012] S2: Input the patent text in the patent database into the RoBERTa+Bi-LSTM model in S1 according to the specified format, extract multiple keywords and their word vectors, and add the multiple word vectors together to form a high-dimensional vector representation of the patent text;

[0013] S3: Input multiple keywords to be searched into the RoBERTa model in S1 according to the specified format to obtain the word vectors of the keywords. The vectors of multiple keywords are added together to form a high-dimensional vector representation of the query text.

[0014] S4, sequentially calculating the cosine similarity between the query text vector in S3 and the high-dimensional vectors of all patent texts in the patent database processed by S2; selecting patent texts with cosine similarity less than a set threshold as search candidate results;

[0015] S5. Input the query text vector in S3 and the text vector of the search candidate patent obtained in S4 into the Barnes-Hut t-SNE model for dimension reduction, and reduce the dimension to 2.

[0016] S6. The query text vector and candidate patent text vector after dimensionality reduction are presented in the form of points in a two-dimensional plane to form a scatter plot. The similarity relationship between the search results is visualized by the distance between the points in the scatter plot.

[0017] As a further solution of the present invention: before inputting the patent text in the patent database into the RoBERTa model, it is necessary to perform data processing on the patent text and convert the input format of the patent text into T1:

[0018] T1=([CLS],TITLE,[SEP],ABSTRACT,[SEP],IPC_TEXT,[SEP],MAIN_TEX T);

[0019] [CLS] is a placeholder marking the beginning of the text; [SEP] is a separator; TITLE indicates the location of the patent title after the text sequence in the patent text is expanded; ABSTRACT indicates the location of the patent abstract after the text sequence in the patent text is expanded; IPC_TEXT indicates the location of the patent IPC classification number after the text sequence in the patent text is expanded; and MAIN_TEXT indicates the location of the patent invention content after the text sequence in the patent text is expanded. This input format fully utilizes the structured information in the patent text.

[0020] As a further solution of the present invention: there are one or more keywords to be retrieved, and each keyword to be retrieved is arranged in sequence to form a keyword sequence to be retrieved, and the keyword sequence to be retrieved is input into the RoBERTa model; before inputting the keyword sequence to be retrieved into the RoBERTa model, it is necessary to perform data processing on the keywords to be retrieved, and convert the input format of the keywords to be retrieved into T2, T2 = ([CLS], IPC_TEXT, [SEP], KEYWORD_1, [SEP], KEYWORD_2, ..., [SEP], KEY WORD_N), wherein the IPC_TEXT here represents the position of the patent IPC classification number after the keyword sequence to be retrieved is expanded; KEYWORD_1 represents the position of the first keyword to be retrieved after the keyword sequence to be retrieved is expanded; KEYWORD_2 represents the position of the second keyword to be retrieved after the keyword sequence to be retrieved is expanded; KEYWORD_N represents the position of the Nth keyword to be retrieved after the keyword sequence to be retrieved is expanded.

[0021] As a further solution of the present invention: when using the RoBERTa model to perform corresponding splicing on the keywords in the patent text or the keywords to be retrieved in the keyword sequence to be retrieved, the vectors output by the last four hidden layers in the RoBERTa model are selected for splicing to obtain the corresponding spliced ​​vectors.

[0022] As a further solution of the present invention, the Barnes-Hut t-SNE algorithm is used in dimensionality reduction.

[0023] As a further solution of the present invention: the dimension of the word vector of each keyword in the sequence state vector and the word vector of each keyword to be retrieved is 3072. After dimensionality reduction processing by the Barnes-Hut t-SNE algorithm, the corresponding dimension is reduced from 3072 to 2.

[0024] As a further solution of the present invention: IPC_TEXT in the input format of the keyword to be searched can be retained or deleted according to the search requirements; when the IPC classification number needs to be limited, it is retained, otherwise it is deleted.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. In the patent keyword search scenario, the present invention extracts keywords and represents word vectors from patent texts based on a pre-trained language model, reduces the dimensionality of patent features returned by the search, and visualizes the search results in the form of a scatter plot. Compared with the search results displayed in list form, it is more intuitive and provides richer search information such as the degree of similarity between patents and patent clustering.

[0027] 2. After using a pre-trained model to extract the keywords of the patent, the present invention uses the word vectors corresponding to the keywords as the feature representation of the patent, rather than directly storing the keywords or further vectorizing the patent text. The same process is performed on the keywords to be retrieved. Using the word vectors of the keywords as the feature representation provides a basis for dimensionality reduction visualization on the one hand, and on the other hand, it also provides users with the function of controlling keyword weights and thus adjusting search preferences.

[0028] 3. This paper introduces the Barnes-Hutt-SNE algorithm to reduce the dimensionality of high-dimensional word vectors. Compared with the common PCA dimensionality reduction, the Barnes-Hut t-SNE algorithm does not seek dimensionality reduction through linear transformation of features. It has better visualization effect after reducing high-dimensional word vectors to 2 dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the main process of the present invention.

[0030] Figure 2 Flowchart for generating keyword vectors in the present invention.

[0031] Figure 3 This is a similarity calculation flow chart in the present invention.

[0032] Figure 4 It is a scatter plot in the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] See also Figure 1In an embodiment of the present invention, a visual patent search method based on a pre-trained language model is provided. The present invention aims to visualize patent search results and is mainly applicable to keyword search scenarios. For keywords in patent texts, in addition to the keywords given in the patent items themselves, keywords can also be extracted from the patent texts. This process can be completed using the pre-trained language model that is popular in the field of natural language processing. The present invention uses the RoBERTa model, which is trained on Chinese corpus and is more suitable for Chinese-related scenarios. Keyword extraction can be regarded as a sequence labeling problem. A BiLSTM layer is added after the RoBERTa model to process the sequence output by the RoBERTa model and determine which positions in the sequence are keywords.

[0035] Since patent texts are usually standardized, with sections such as patent title, background technology, abstract, and invention content, the original input format of the RoBERTa model was adjusted by utilizing this information, using this structured information as a supplement to the context. Furthermore, considering that the same keyword can have different meanings in different fields, the Chinese meaning corresponding to the patent classification number is also incorporated into the context when supplementing the context. The final text input format used is T1, T1 = ([CLS], TITLE, [SEP], ABSTRACT, [SEP], IPC_TEXT, [SEP], MAIN_TEXT), where [CLS] is a placeholder marking the beginning of the text, [SEP] is a delimiter, and TITLE, ABSTRACT, IPC_TEXT, and MAIN_TEXT are the positions of the expanded text sequence in the patent.

[0036] We use patent text that has been annotated with keyword positions and process it into the above input format. We then fine-tune the RoBERTa+Bi-LSTM model to obtain a keyword extraction model suitable for patent text scenarios.

[0037] After the keyword extraction model of the above-mentioned RoBERTa+BiLSTM structure is trained, the patent text is input to extract the keywords in the patent. The keywords need to be represented in the form of word vectors. This step can be obtained directly from the RoBERTa model. Figure 2 As shown, the present invention selects and concatenates the hidden vectors of the last four layers of the RoBERTa model to serve as word vector representations for the keywords at corresponding positions in the sequence. The dimension of each layer's hidden vector is 768, resulting in a word vector dimension of 768 x 4 = 3072 for each keyword, a very high dimensionality. This model is used to extract keywords from all patent texts in the patent database and save them as word vectors, which serve as keyword features for the patents.

[0038] like Figure 3 As shown, during the search, for one or more keywords input by the user, the present invention constructs one or more keywords into a query text in the format of ([CLS], IPC_TEXT, [SEP], KEYWORD_1, [SEP], KEYWORD_2, ..., [SEP], KEYWORD_N) and inputs it into the RoBERTa model. At this time, there is no need to use the subsequent BiLSTM layer. The IPC_TEXT is an optional part. If the user specifies the patent classification field of the query, the IPC is added to the context and aligned with the training process of the above model to provide more complete context information. Afterwards, the same method as in the previous paragraph is used to extract the latent vectors of the last 4 layers of the RoBERTa model as the word vector representation of the search keyword.

[0039] The word vectors obtained from the above pre-trained model have the characteristics of being context-dependent. For example, the word "apple" has two different representations in "apples on the fruit stall" and "Apple Company". And to a certain extent, the addition of word vectors also shows properties that conform to semantic expression. For example, the word vector of "crown" + "male" is close to the word vector representation of "king". Based on these properties, the present invention performs a weighted summation on the vectors converted from multiple keywords provided by the user. By default, the weights are equal. Users can adjust the weights of each keyword to adjust their search preferences.

[0040] Then, in the patent database, we will first preliminarily filter out patents containing the keywords provided by the user (if there are multiple keywords, only one is sufficient), such as Figure 3 As shown in the figure, the sum of multiple keyword vectors of the candidate patent is used as the feature vector of the patent. Then, the word vector of the search keyword is compared with the feature vector of the candidate patent, the cosine distance is calculated, and the K patents with the closest distance are selected and returned as the search results.

[0041] After obtaining the K patents returned by the keyword search, visualization processing is finally performed. Since the word vector and patent feature vector dimensions of the search keyword are both 3072, the dimension is too high and dimensionality reduction is required. The present invention adopts the Barnes-Hut t-SNE algorithm to reduce the above high-dimensional word vector to 2 dimensions. For samples with high-dimensional features, the Barnes-Hut t-SNE algorithm can retain the local properties of samples in the high-dimensional space after dimensionality reduction, that is, samples with similar distances will be closer after dimensionality reduction, and samples with greater distances will be farther after dimensionality reduction, which reduces the computational complexity and improves the computational speed, but is only limited to obtaining 2-dimensional or 3-dimensional representations, which just meets the visualization requirements. After reducing the search keyword vector and the candidate patent feature vector to 2 dimensions, a scatter plot is used for visualization. In the final scatter plot, as shown Figure 4As shown, intuitively, the closer the patents are to the search keyword points, the higher the degree of similarity; and the distance between the points of each patent also intuitively represents the degree of similarity between them; and the returned patent results may show a certain cluster, and some patents are closer to a certain topic than other patents, providing richer information for the searcher.

[0042] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A visual patent search method based on a pre-trained language model, characterized in that: The search steps include: S1. Using patent text annotated with keywords, train a RoBERTa+Bi-LSTM model to extract keywords from the patent text. S2: Input the patent text in the patent database into the RoBERTa+Bi-LSTM model trained in S1 according to the specified format, extract multiple keywords and their word vectors, and add the multiple word vectors together to form a high-dimensional vector representation of the patent text; S3: Input multiple keywords to be searched into the RoBERTa model in S1 according to the specified format to obtain the word vectors of the keywords. The word vectors of multiple keywords are added together to form a high-dimensional vector representation of the query text. S4, sequentially calculating the cosine similarity between the query text vector in S3 and the high-dimensional vectors of all patent texts in the patent database processed by S2; selecting patent texts with cosine similarity less than a set threshold as search candidate results; S5. Input the query text vector in S3 and the text vector of the search candidate patent obtained in S4 into the Barnes-Hut t-SNE model for dimension reduction, and reduce the dimension to 2. S6. The query text vector and candidate patent text vector after dimensionality reduction are presented in the form of points in a two-dimensional plane to form a scatter plot. The similarity relationship between the search results is visualized by the distance between the points in the scatter plot.

2. A visual patent search method based on a pre-trained language model according to claim 1, characterized in that: Before inputting the patent text in the patent database into the RoBERTa model, the patent text needs to be processed and the input format of the patent text needs to be converted into T1, T1=([CLS],TITLE, [SEP],ABSTRACT,[SEP],IPC_TEXT,[SEP],MAIN_TEXT), where [CLS] is a placeholder that indicates the beginning of the text; [SEP] is a separator; TITLE indicates the position of the patent name after the text sequence in the patent text is expanded; ABSTRACT indicates the position of the patent specification abstract after the text sequence in the patent text is expanded; IPC_TEXT indicates the position of the patent IPC classification number after the text sequence in the patent text is expanded; MAIN_TEXT indicates the position of the patent invention content after the text sequence in the patent text is expanded.

3. A visual patent search method based on a pre-trained language model according to claim 2, characterized in that: There are one or more keywords to be searched, and each keyword to be searched is arranged in sequence to form a keyword sequence to be searched, and the keyword sequence to be searched is input into the RoBERTa model; before inputting the keyword sequence to be searched into the RoBERTa model, it is necessary to perform data processing on the keywords to be searched, and convert the input format of the keywords to be searched into T2, T2=([CLS],IPC_TEXT,[SEP],KEYWORD_1,[SEP],KEYWORD_2,...,[SEP],KEYWORD_N), where IPC_TEXT represents the position of the patent IPC classification number after the keyword sequence to be searched is expanded; KEYWORD_1 represents the position of the first keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_2 represents the position of the second keyword to be searched after the keyword sequence to be searched is expanded; KEYWORD_N represents the position of the Nth keyword to be searched after the keyword sequence to be searched is expanded.

4. A visual patent search method based on a pre-trained language model according to claim 3, characterized in that: When using the RoBERTa model to perform corresponding splicing on the keywords in the patent text or the keywords to be retrieved in the keyword sequence to be retrieved, the vectors output by the last four hidden layers in the RoBERTa model are selected for splicing to obtain the corresponding spliced ​​vectors.

5. A visual patent search method based on a pre-trained language model according to claim 4, characterized in that: The Barnes-Hut t-SNE algorithm is used for dimensionality reduction.

6. A visual patent search method based on a pre-trained language model according to claim 5, characterized in that: The word vectors of each keyword in the sequence state vector and the word vectors of each keyword to be retrieved have a dimension of 3072. After dimensionality reduction processing using the Barnes-Hut t-SNE algorithm, the corresponding dimension is reduced from 3072 to 2.

7. The visual patent search method based on a pre-trained language model according to claim 3, characterized in that: The IPC_TEXT in the input format of the keyword to be searched can be retained or deleted according to the search requirements; when the IPC classification number needs to be limited, it is retained, otherwise it is deleted.

Images