Technical literature multi-dimensional analysis method based on semantic understanding

Abstract

The invention discloses a semantic understanding-based technical literature multi-dimensional analysis method, which comprises the following steps of: performing text vectorization processing on text data by utilizing a Sension-BERT model to generate dense vector representation of a text; vector dimensionality reduction is carried out by adopting UMAP, and redundant features are removed; performing unsupervised clustering analysis by using HDBSCAN to generate a clustering result; extracting a subject term from a clustering result by adopting a TF-ICF method; and carrying out multi-dimensional analysis on the clustering result. Compared with the prior art, a comprehensive and efficient patent analysis tool is provided through an advanced semantic understanding patent clustering method, so that deeper interpretation and analysis of technical literatures are realized while the cost benefit is ensured.

Description

Technical Field

[0001] The present invention relates to a technical document analysis method, and in particular to a technical document multi-dimensional analysis method based on semantic understanding. Background Art

[0002] The current technology of technical literature analysis, the widely used technology mainly relies on traditional text processing technology and algorithms to parse and evaluate patent documents. Although these methods are effective to a certain extent, they have significant limitations. First, the common word2vec model can partially capture the relationship between words when converting patent text into word vectors, but it performs poorly when dealing with polysemous words, which affects the understanding and mining of the deep semantics of the text. Secondly, the current analysis methods usually only focus on a single dimension and fail to build a comprehensive analysis framework, making it difficult to comprehensively evaluate the value and potential of technical literature. This single analysis perspective not only ignores the inherent complexity and richness of technical literature texts, but also fails to fully demonstrate the relationship between patents and their role in technological progress.

[0003] A more serious problem is that many advanced analysis techniques based on natural language processing (NLP) need to rely on large-scale annotated data sets for effective training to achieve good analysis results. However, building such a data set is time-consuming and laborious, which not only increases the difficulty of analysis, but also limits the expansion of analysis dimensions. Therefore, developing a technical literature analysis method that can not only deeply understand the text semantics, but also conduct comprehensive multi-dimensional analysis without relying on large-scale data sets has become a key challenge to improve the quality and efficiency of technical literature analysis. The solutions currently on the market have not yet completely overcome these problems, resulting in the depth and breadth of technical literature analysis being limited, affecting the accuracy and practicality of the analysis results. Summary of the invention

[0004] The purpose of this invention is to provide a multi-dimensional analysis method of technical documents based on semantic understanding. It aims to provide a comprehensive and efficient patent analysis tool through advanced patent clustering methods of semantic understanding, so as to achieve a deeper interpretation and analysis of technical documents while ensuring cost-effectiveness.

[0005] To achieve the above object, the present invention is implemented according to the following technical solutions:

[0006] The present invention comprises the following steps:

[0007] S1: Obtain text data of technical documents, use the Sentence-BERT model to perform text vectorization on the text data, and generate a dense vector representation of the text;

[0008] S2: Use UMAP to reduce the dimension of vectors and remove redundant features;

[0009] S3: Use HDBSCAN to perform unsupervised cluster analysis and generate clustering results;

[0010] S4: TF-ICF method was used to extract keywords from clustering results;

[0011] S5: Perform multi-dimensional analysis on the clustering results.

[0012] The beneficial effects of the present invention are:

[0013] The present invention is a multi-dimensional analysis method for technical documents based on semantic understanding. Compared with the prior art, the present invention achieves deep semantic understanding of patent texts and comprehensive multi-dimensional analysis through the following key technical points:

[0014] (1) Application of pre-trained models: The present invention uses advanced pre-trained models, such as BERT or Sentence-Transformers, which have been pre-trained on large-scale corpora and can better capture the deep semantic information in the text. By mapping the patent abstract text to a vector representation in a high-dimensional space, the present method can more accurately express the meaning of the text, especially for the processing of polysemous words, significantly improving the accuracy of the semantic understanding of patent documents.

[0015] (2) Unsupervised text clustering: After converting the patent abstract text into vectors, the present invention uses an efficient clustering algorithm to analyze these vectors and automatically identify the professional and technical topics in the patent text. This method not only reduces the reliance on manually constructed data sets, but also can discover implicit patterns and groups in patent documents, thereby revealing the technical connections between patents and providing users with a more accurate division of technical fields.

[0016] (3) Multi-dimensional keyword identification and analysis: In addition to traditional text content analysis, the present invention also introduces additional analysis dimensions, such as "efficacy" and "information technology". By identifying and clustering keywords in these dimensions, it can further reveal the application and impact of patents in different fields. This method broadens the analysis perspective, makes the analysis results more comprehensive, provides users with more diversified insights, and supports more complex decision-making processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of the clustering method of the present invention;

[0018] Figure 2 is the SBERT classification objective function;

[0019] Figure 3 It is a flow chart of multidimensional analysis method. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0021] like Figure 1 As shown: The present invention comprises the following steps:

[0022] S1: Obtain text data of technical documents, use the Sentence-BERT model to perform text vectorization on the text data, and generate a dense vector representation of the text;

[0023] S2: Use UMAP to reduce the dimension of vectors and remove redundant features;

[0024] S3: Use HDBSCAN to perform unsupervised cluster analysis and generate clustering results;

[0025] S4: TF-ICF method was used to extract keywords from clustering results;

[0026] S5: Perform multi-dimensional analysis on the clustering results.

[0027] The present invention uses the Sentence-BERT pre-training model to process the technical document abstract text to generate a dense vector representation of the text. Sentence-BERT is a BERT variant designed specifically for sentences. It can better capture the semantic similarity within sentences by pre-training on a large-scale corpus. In this method, Sentence-BERT is used to extract key semantic information in technical document abstracts and generate vector representations for each sentence. These vectors can accurately reflect the deep semantic structure of the sentence and provide a solid foundation for subsequent analysis.

[0028] SBERT uses a Siamese and Triplet Network to update weight parameters to ensure that the generated sentence vector has semantic information. By using a pre-trained model, the SBERT model can obtain a paragraph vector that is semantically meaningful. SBERT adds a Pooling operation to the last layer output of BERT / RoBERTa, mainly using the Mean-pooling strategy to calculate the average value of each Token output vector to represent the sentence vector, thereby generating a fixed-size sentence embedding vector.

[0029] like Figure 2As shown in the figure, SBERT follows the structure of the twin network and uses the same BERT to process the encoder of the input sentence, allowing BERT to better capture the relationship between sentences and obtain sentence vectors with semantic features. When SBERT processes text classification, it inputs sentence a and sentence b, and after BERT and Pooling operations, it can obtain the sentence vector S a and S b , the sentence vector S a and S b and the difference vector S between them a -S b Spliced ​​together to form a new feature vector, and then multiplied by the trainable weight matrix Wt, that is:

[0030] v = softmax(W t (s a ,s b |s a -s b |))

[0031] Among them, W t ∈R 3d*t , d is the dimension of the sentence vector, t is the number of classification labels, and v is a probability distribution vector.

[0032] When two sentences S1 and S2 in the technical literature text are input into SEBRT, the vector representation is:

[0033]

[0034] Among them, n represents the number of hidden layer units inside BERT.

[0035] UMAP Dimensionality Reduction:

[0036] The Sentence-BERT language model first uses BERT for text pre-training. Since the Chinese model of BERT usually uses a length limit of 512 characters, the pre-trained technical documents will become a vector matrix of N*512 (N is the number of technical documents). As N increases, a high-dimensional data set will be formed. In order to remove redundant features and improve the clustering effect of text, it is necessary to perform data dimensionality reduction operations on the vector matrix. To this end, the present invention proposes a method of using UMAP (Uniform Manifold Approximation and Projection) for dimensionality reduction.

[0037] UMAP is an effective dimensionality reduction algorithm that can project high-dimensional Sentence-BERT vectors into low-dimensional space while maintaining the topological structure of the data. Its theoretical basis is Riemannian geometry and algebraic topology. It mainly uses local manifold approximation and local fuzzy simplex set representation to construct the topological representation of high-dimensional data. That is, for high-dimensional data, given the low-dimensional representation of some data, a similar process can be used to construct an equivalent low-dimensional topological representation. At present, UMAP is the best method for text vector dimensionality reduction. Using UMAP in the process of data dimensionality reduction can not only reduce computational complexity and memory usage, but also retain the characteristics of the original data to the greatest extent.

[0038] UMAP dimensionality reduction simplifies the data structure, facilitating subsequent visualization and analysis, and also prepares input data for clustering algorithms.

[0039] The vectors after UMAP dimensionality reduction are then input into the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm for clustering. HDBSCAN is a clustering algorithm that does not require pre-set parameters. It can automatically identify natural clusters in the data and handle noise and outliers. The biggest difference from the traditional DBSCAN is that HDBSCAN can handle clustering problems of clusters of different densities and is more robust in parameter selection. The HDBSCAN algorithm introduces the idea of ​​hierarchical clustering and restricts the minimum subtree of the minimum spanning tree pruning to control the generated clusters not to be too small. In addition, the algorithm is less sensitive to parameters and does not need to set thresholds by itself. It only needs to define the minimum number of clusters.

[0040] HDBSCAN can evaluate the membership of each sample, and the membership value range is [0, 1]. If the membership value is 0, it means that the sample point is a noise point and does not belong to any cluster; if the membership value is 1, it means that the sample point is a center point, and the cluster center point attributes can represent the typical characteristics of the cluster. The algorithm calculation steps are as follows:

[0041] (1) Spatial transformation of data points: The traditional single-link clustering method clusters data points based on the minimum distance between them, which is easily affected by noise. HDBSCAN uses the mutual reachable distance instead of the minimum distance between data points, which can strengthen the close connection between data points in high-density areas and weaken the connection between data points in low-density areas. a and b represent the mutual reachable distance between two data points, and the calculation formula is:

[0042]

[0043] in, Indicates that the range of point a is the domain of K, and d(a,b) refers to the distance between points a and b. Represents the core distance of data point a, that is, the distance between a and its kth nearest neighboring data point.

[0044] When calculating the distance, the Euclidean distance is calculated as:

[0045]

[0046] Where a i ,b i are the coordinates of two points a and b in n-dimensional space.

[0047] In order to improve the efficiency of calculating the core distance, local sensitive hashing is used to optimize the search efficiency. The local sensitive hashing formula is:

[0048]

[0049] Where a is the vector representation of the data point, x is a random vector, y is a random offset, and w is the width of the hash bucket. Indicates rounding down.

[0050] (2) Construction of minimum spanning tree. The HDBSCAN algorithm regards the data as a weighted graph, where the data points are vertices and the weight of the edge between any two points is equal to the mutual reachable distance between these points. The Prim algorithm is used to construct the minimum spanning tree in the weighted graph.

[0051] (3) Establishment of cluster hierarchy. The algorithm uses mutual reach distance as weight, traverses and reorders the edges of the minimum spanning tree, and classifies each edge into a new cluster.

[0052] (4) Cluster tree compression. When the algorithm compresses and segments the coarse hierarchical structure, it compares the number of samples in the newly segmented cluster with the number of samples in the smallest cluster and removes the smaller one. The size of the smallest cluster is a parameter of HDBSCAN and is adjusted according to the number of technical literature samples.

[0053] (5) Cluster extraction. HDBSCAN defines a stability-based cluster selection method to identify clusters with higher stability in the data set. The stability σ c To evaluate the stability of clustering. c The calculation formula is

[0054]

[0055] Where x is the data point, c is the cluster, and λ is the reciprocal of the distance. birth To represent the λ value when the cluster is formed, λx represents the lambda value of the cluster separated from the parent cluster.

[0056] In the present invention, the HDBSCAN algorithm is used to identify professional technical topics in technical literature texts, thereby forming meaningful categories that represent potential technical fields and trends in technical literature.

[0057] Improved TF-IDF to extract keywords:

[0058] The clusters obtained after clustering are subjected to keyword extraction. The present invention adopts the TF-ICF algorithm which improves TF-IDF (Term Frequency-Inverse Document Fre-quency). TF-IDF is a statistical method that can judge the importance of a word according to the frequency of its appearance in a corpus. The TF-IDF algorithm believes that if a word appears frequently in an article and rarely appears in other articles, then the word has a good ability to distinguish categories. The formula of TF-IDF is TF*IDF, where TF represents the frequency of a word appearing in a document. The higher the frequency, the more times the word appears; IDF represents the inverse document frequency of a word. The fewer documents containing the word, the better the word can reflect the theme of the document. The specific calculation formulas of TF and IDF are as follows:

[0059]

[0060] where f k,Di is the frequency of keyword k appearing in document Di, and |Di| is the total number of all words in the i-th document.

[0061]

[0062] Where N D is the total number of documents, I indicates whether keyword k appears in Di documents, and the more common a word is, the lower its IDF value is.

[0063] Based on the TF-IDF algorithm, the present invention regards all documents in a topic cluster as a single document C, so the inverse document frequency is replaced by the inverse class frequency, and the importance score algorithm TF-ICF of the words in a topic cluster can be obtained:

[0064]

[0065] Where c is a single cluster document formed by merging all documents belonging to the same topic cluster, and f k,c is the frequency of keyword k in cluster c, |c| is the total number of all words in cluster c, A C is the average number of words in all clusters c containing the keyword, ∑ c∈C f k,c is the frequency of keyword k in all clusters c. Finally, in order to avoid outputting negative values, add one to the logarithmic operation result.

[0066] In this way, the TF-ICF process simulates the importance of words in clusters rather than individual documents, thereby generating a topic word distribution for each cluster. Therefore, this method can be used to mine the topic words in each cluster, thereby describing and characterizing different topics.

[0067] Multi-dimensional analysis method of technical literature:

[0068] In addition to the traditional cluster analysis of the text content of technical documents, the present invention also introduces additional analysis dimensions, such as "effectiveness", "information technology", etc. By identifying and clustering the keywords of these dimensions, it can further reveal the application and impact of technical documents in different fields.

[0069] Its core technical processes include Figure 3 As shown;

[0070] 1. Data preparation: High-quality technical literature data is the premise for ensuring the orderly and correct development of research work. Therefore, the present invention first constructs a search-based screening of technical literature data, and improves the data quality through data preprocessing methods to obtain technical literature data that can be analyzed.

[0071] 2. Professional and technical subject clustering: Using the technical literature clustering method based on semantic understanding, the abstract text of the technical literature is clustered and analyzed to obtain the professional and technical subject clustering results.

[0072] 3. Other dimension theme clustering:

[0073] (1) Use the Chinese word segmentation database Jieba to segment the abstract text of technical literature, count the word frequencies, and sort them in descending order by word frequency.

[0074] (2) Empirically pre-select representative keywords from other dimensions. For example, from the dimension of "information technology", select information technology words such as "computer, machine learning, neural network, algorithm"; then use Sentence-BERT to segment technical documents and vectorize keywords. Then use the cosine similarity algorithm to calculate the similarity of text vectors and find words with semantic relationship with information technology words from the technical document segmentation. The cosine similarity algorithm is:

[0075]

[0076] where a i ,b i are two text vectors in n-dimensional space, and Represent the modulus of vectors a and b respectively.

[0077] This step needs to be repeated several times to evaluate the appropriate initial representative words and cosine similarity threshold.

[0078] (3) After identifying the keywords of the technical literature text under this dimension, the keywords are clustered using a technical literature clustering method based on semantic understanding to obtain the theme under this dimension and the keywords contained in the theme.

[0079] (4) A rule-based algorithm is used to search for keywords of this dimension from the technical literature text. If a technical literature contains keywords of a topic, the support for the technical literature belonging to the topic increases. Finally, the topic with the highest support is selected as the topic of the technical literature under this dimension.

[0080] (5) Repeat steps (2)-(4) to obtain more topic information of other dimensions to support multi-dimensional analysis.

[0081] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.

Claims

1. A multi-dimensional analysis method of technical documents based on semantic understanding, characterized in that: The following steps are involved: S1: Obtain text data of technical documents, use the Sentence-BERT model to perform text vectorization on the text data, and generate a dense vector representation of the text; S2: Use UMAP to reduce the dimension of vectors and remove redundant features; S3: Use HDBSCAN to perform unsupervised cluster analysis and generate clustering results; S4: TF-ICF method was used to extract keywords from clustering results; S5: Perform multi-dimensional analysis on the clustering results.

2. The multi-dimensional analysis method of technical documents based on semantic understanding according to claim 1 is characterized by: In step S1, the Sentence-BERT model adopts a twin network structure. The encoder of the input sentence is processed by the same BERT. When SBERT processes text classification, it inputs sentence a and sentence b. After BERT and Pooling operations, the sentence vector S can be obtained. a and S b , the sentence vector S a and S b and the difference vector S between them a -S b Spliced ​​together to form a new feature vector, and then multiplied by the trainable weight matrix Wt, that is: v=softmax(W t (s a ,s b |s a -s b |)) Among them, W t v 3d*t , d is the dimension of the sentence vector, t is the number of classification labels, and v is a probability distribution vector; When two sentences S1 and S2 in the text data are input into SEBRT, the vector is represented as: Among them, n represents the number of hidden layer units inside BERT.

3. The multi-dimensional analysis method of technical documents based on semantic understanding according to claim 2 is characterized by: In step S2, UMAP performs vector dimensionality reduction by using local manifold approximation and local fuzzy simplex set representation to construct a topological representation of high-dimensional data, that is, given a low-dimensional representation of some data for high-dimensional data, a similar process is used to construct an equivalent low-dimensional topological representation.

4. The multi-dimensional analysis method of technical documents based on semantic understanding according to claim 3 is characterized by: The step S3 comprises the following steps: S31: Spatial transformation of data points: a and b represent the mutual distance between two data points. The calculation formula is: in, Indicates that the range of point a is the domain of K, and d(a,b) refers to the distance between points a and b. Represents the core distance of data point a, that is, the distance between a and its kth nearest data point; When calculating the distance, the Euclidean distance is calculated as: Where a i ,b i are the coordinates of points a and b in n-dimensional space; Local sensitive hashing is used to optimize search efficiency. The local sensitive hashing formula is: Where a is the vector representation of the data point, x is a random vector, y is a random offset, and w is the width of the hash bucket. Indicates rounding down; S32: Construction of minimum spanning tree: HDBSCAN algorithm regards data as a weighted graph, where data points are vertices and the weight of the edge between any two points is equal to the mutual reachable distance between these points; Prim's algorithm is used to construct the minimum spanning tree in the weighted graph; S33: Establishment of cluster hierarchy: Using mutual reach distance as weight, traverse and reorder the edges of the minimum spanning tree, and classify each edge into a new cluster; S34: Compression of clustering tree: When compressing and segmenting the coarse hierarchical structure, the number of samples of the newly segmented cluster is compared with the number of samples of the smallest cluster, and the smaller one is eliminated. The size of the smallest cluster is a parameter of HDBSCAN and is adjusted according to the number of technical literature samples; S35: Cluster extraction: Identify clusters with higher stability in the data set and use stability σ c To evaluate the stability of clustering; σ c The calculation formula is: Where x is the data point, c is the cluster, and λ is the reciprocal of the distance. birth To represent the λ value when the cluster is formed, λ x represents the lambda value of the cluster separated from the parent cluster.

5. The multi-dimensional analysis method of technical documents based on semantic understanding according to claim 4 is characterized by: The TF-ICF method formula in step S4 is TF*ICF, where TF represents the frequency of a word appearing in a document, and ICF represents the inverse cluster frequency of a word; the specific calculation formula of TF-ICF is as follows: Where c is a single cluster document formed by merging all documents belonging to the same topic cluster, and f k,c is the frequency of keyword k in cluster c, |c| is the total number of all words in cluster c, A C is the average number of words in all clusters c containing the keyword, ∑ c∈C f k,c is the frequency of keyword k in all clusters c; the logarithmic operation result is added by one.

6. The multi-dimensional analysis method of technical documents based on semantic understanding according to claim 5 is characterized by: The multi-dimensional analysis in step S5 includes professional and technical theme clustering and other dimensional theme clustering; firstly, a search-based screening of technical literature data is constructed, and the data quality is improved by a data preprocessing method to obtain technical literature data available for analysis; the professional and technical theme clustering adopts a clustering method based on semantic understanding to perform cluster analysis on the abstract text of the technical literature to obtain professional and technical theme clustering results; The other dimension topic clustering uses the Chinese word segmentation library jieba to segment the abstract text of the technical literature, count the word frequency, and sort it in descending order by word frequency. Then, Sentence-BERT is used to vectorize the technical literature word segmentation and keywords, and the cosine similarity algorithm is used to calculate the similarity of the text vectors. Words with semantic relationship with information technology words are found from the technical literature word segmentation; after identifying the keywords of the technical literature text in this dimension, a clustering method based on semantic understanding is used to cluster the keywords to obtain the theme under this dimension and the keywords contained in this theme; a rule algorithm is used to find the keywords of this dimension from the technical literature text. If the technical literature contains the keywords of a theme, the support of the technical literature belonging to this theme increases. Finally, the theme with the highest support is selected as the theme of the technical literature in this dimension.

Images