A text classification method based on document similarity

Abstract

The application discloses a text classification method based on document similarity, and belongs to the technical field of document processing. The method updates the word set of a document sentence through iterative optimization of a word segmentation mechanism and generates a first sentence vector, and generates a first similarity vector in combination with a sentence weight and the first sentence vector; a plurality of theme words are extracted from the word set according to an entity weight and a theme feature vector is generated, a knowledge supplement vector is extracted from a knowledge base based on the context vector of each theme word and the entity weight, a theme enhancement vector is generated in combination with the theme feature vector and the knowledge supplement vector, and a second similarity vector is generated based on the theme enhancement vector; after the first similarity vector and the second similarity vector are compared and optimized in terms of difference, a document similarity vector is obtained and a text category is output. Through iterative word segmentation optimization, theme-guided knowledge enhancement and a similarity comparison mechanism, the application can effectively improve the reliability and accuracy of text classification.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, and in particular to a text classification method based on document similarity. Background Technology

[0002] News document classification is an important application of natural language processing, widely used in fields such as public opinion analysis, information recommendation, and intelligence monitoring. Traditional methods employ bag-of-words models, TF-IDF, or deep learning models such as TextCNN and BERT for end-to-end classification. For example, Chinese patent application CN202310687090.5 discloses a news text classification method based on feature fusion and dual classification. This method preprocesses and segments the collected news text data, constructs a feature fusion model using BERT and TF-IDF, and then combines it with FastText to construct a news text classification model based on feature fusion and dual classification. After training, this model is used for news text classification, which can improve the efficiency and accuracy of news text classification. However, existing technologies lack effective utilization of the structural information of news documents, which can easily lead to the burying of key information in news documents, thus affecting the accuracy of text classification. In addition, existing word segmentation tools have limited accuracy in segmenting compound words and professional terms in the news field, and the resulting semantic feature bias may further impair the accuracy of text classification. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a text classification method based on document similarity. This method optimizes the word segmentation results of document sentences by constructing a news-specific dictionary and a language model. It sets up a two-branch similarity vector based on enhanced text sentence and topic knowledge, obtains a document similarity vector through difference comparison, and outputs the text category. This invention is particularly effective in the news domain, which has high document complexity and many ambiguous terms.

[0004] The technical solution of this invention is implemented as follows: A text classification method based on document similarity includes the following steps: Step 1: Segment the document into sentences. Each sentence is vectorized to obtain the original sentence vector. The original sentence vector is then processed by word segmentation to obtain a word set. A word vector sequence is generated based on the word set. The word vector sequence is then aggregated to generate the first sentence vector. Step 2: Calculate the semantic similarity between the original sentence vector and the first sentence vector. If the semantic similarity of any sentence is less than the target value, update the word set by iteratively optimizing the word segmentation mechanism, and regenerate the word vector sequence and the first sentence vector. Step 3: Calculate the sentence weight based on the position data of each sentence, and generate the first similarity vector by combining the sentence weights of all sentences and the first sentence vector; Step 4: After identifying the entity words in all word sets, generate an entity word set. Extract multiple topic words from the entity word set according to the entity weights. Combine the context vectors of each topic word with the entity weights to extract the corresponding knowledge attribute vectors from the knowledge base. Step 5: Aggregate the word vectors of all topic words to generate topic feature vectors, aggregate all knowledge attribute vectors to generate knowledge supplement vectors, combine topic feature vectors and knowledge supplement vectors to generate topic enhancement vectors, and then generate a second similarity vector based on topic enhancement vectors; Step 6: Calculate the combined difference between the first similarity vector and the second similarity vector. If the combined difference is less than the preset threshold, generate a document similarity vector and output the text category to end the task. Otherwise, adjust the configuration parameters of the topic enhancement vector and return to step 4.

[0005] In this invention, in step 1, the document is a news document. The word segmentation tool jieba is used to segment each sentence to obtain a word set for each sentence. Each word in the word set is mapped to a word vector. All word vectors are combined to generate a word vector sequence. The word vector sequence is aggregated through a self-attention pooling mechanism to obtain the first sentence vector.

[0006] In this invention, step 2, the specific steps of updating the word set through iterative optimization of the word segmentation mechanism, include: Step 201: Use a character set language model pre-trained based on news corpus to sequentially calculate the perplexity of each word in the word set; Step 202: Extract the word with the highest perplexity, merge the word with its neighboring words, perform maximum positive matching on the merged words based on the preset news professional dictionary, segment the words according to the matching results, and generate at least one candidate word segmentation set; Step 203: Calculate the average perplexity of each word in each candidate word segmentation set, select the candidate word segmentation set with the lowest average perplexity, update the word set and regenerate the first sentence vector, and calculate the semantic similarity between the first sentence vector and the original sentence vector; Step 204: Repeat steps 201 to 203 until the semantic similarity is greater than or equal to the target value.

[0007] In this invention, in step 3, the location data includes whether the statement is a title, the first sentence of a paragraph, or the last sentence of a paragraph, and the title number or paragraph number of the statement. Location parameters, including a title factor f, are generated based on the location data. title Paragraph factor f para and statement factor f sent The location importance z is calculated based on the location parameters. m Then, based on the importance of position z m Calculate statement weight β m .

[0008] In this invention, in step 3, the first statement vector of each statement is input into the classifier after semantic space alignment to obtain the statement similarity vector of each statement. The first similarity vector is generated by combining the statement weights and statement similarity vectors of all statements. The classifier contains multiple category models, each category model corresponds to a text category and contains at least one category semantic vector. The semantic space alignment refers to mapping the first statement vector to the semantic space of the category semantic vector.

[0009] In this invention, in step 4, the entity weight is calculated based on the positional importance of all statements in which the entity word appears. The entity words with the highest entity weight in the entity word set are taken as the topic words. The corresponding context text fragments are extracted based on the position of the topic words in different statements. All context text fragments of the topic word are concatenated and vectorized to obtain the context vector.

[0010] In this invention, in step 4, the knowledge base is a general knowledge graph. The topic words are linked to the corresponding entity nodes in the knowledge graph through named entity recognition. The extraction depth and the number of extracted attributes are determined according to the entity weight. Local subgraphs are extracted from the knowledge graph with the entity nodes as the center. Each knowledge information in the local subgraph is encoded as an attribute feature vector. The cosine similarity between each attribute feature vector and the context vector is calculated. The attribute feature vector with the highest cosine similarity is extracted as the knowledge attribute vector.

[0011] In this invention, in step 5, the entity weights of all topic terms are extracted, and the average value α of all entity weights is calculated. avg Maximum value α max and entropy α H Generate a reference vector u, and then use the topic feature vector v. topic Knowledge supplement vector v know The reference vector u is concatenated and then activated by sigmoid to obtain the gated weight vector g. m Based on the gated weight vector g m The topic feature vector v topic and knowledge supplement vector v know Perform element-wise weighted fusion to obtain the topic enhancement vector v tk Then, the topic enhancement vector is input into the classifier, and its cosine similarity with the category semantic vector of each category model is calculated to generate a second similarity vector.

[0012] In this invention, in step 6, the numerical difference c1 and the ranking difference c2 of the first similarity vector S1 and the second similarity vector S2 are calculated, where c1 = D. JS (P||Q), c2=(1-τ) / 2, D JS() represents the JS divergence function, P and Q are the probability distributions of the first similarity vector S1 and the second similarity vector S2, respectively, and τ is the rank correlation coefficient. The comprehensive difference c is then calculated by combining the numerical difference c1 and the ranking difference c2. s .

[0013] In this invention, in step 6, a document similarity vector is generated by combining the first similarity vector and the second similarity vector. The configuration parameters of the topic enhancement vector include the number of topic words, extraction depth, and the number of extracted attributes.

[0014] The document similarity-based text classification method of this invention has the following beneficial effects: This invention filters low-quality word segments by comparing the semantic similarity of the original sentence vector and the first sentence vector, and iteratively optimizes the word set using a language model and a news-specific dictionary to reduce the word segmentation error rate and improve the interpretability and accuracy of the semantic feature representations of the first sentence vector and topic feature vector generated subsequently based on the word set. This is particularly suitable for news documents with high domain randomness. Furthermore, multiple topic words are selected from the entity word set based on entity weights to generate topic feature vectors. Knowledge supplement vectors are extracted from the knowledge base based on the context vectors and entity weights of each topic word. Then, a gating fusion mechanism based on entity weights is introduced to adaptively adjust the fusion ratio of topic feature vectors and knowledge supplement vectors to achieve topic-specific knowledge enhancement combined with document structure information. Finally, a dual-branch similarity vector based on topic and text is generated based on the topic enhancement vector and the first sentence vector. Through a similarity comparison optimization mechanism based on difference, the configuration parameters of the topic enhancement vector are dynamically adjusted to generate a document similarity vector and output the text category, which can effectively improve the accuracy and efficiency of text classification. Attached Figure Description

[0015] Figure 1 This is a flowchart of the document similarity-based text classification method of the present invention; Figure 2 A schematic diagram illustrating the generation of knowledge attribute vectors for this invention; Figure 3 This is a schematic diagram illustrating a preferred method of generating sentence similarity vectors using a classifier according to the present invention. Figure 4 This is a schematic diagram illustrating the document similarity vector generated by this invention. Detailed Implementation

[0016] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0017] Knowledge graphs are knowledge bases that exist in the form of graph structures. They use entities as nodes and relationships as edges, forming a structured semantic network that can provide rich background information for text classification. Existing technologies, when introducing external knowledge graphs for text classification, typically extract all attribute features of entities and concatenate them, leading to the introduction of a large number of irrelevant knowledge attributes, which can easily cause semantic interference in text classification. Furthermore, knowledge fusion methods based on simple concatenation lack consideration for the original semantics, potentially impairing the accuracy of text classification. To address these issues, this invention proposes a text classification method based on document similarity. This method employs an iterative optimization mechanism based on a news-specific dictionary and language model to correct the word segmentation results of documents. It extracts topic words from a word set based on entity weights and generates topic feature vectors. Based on the context vectors of the topic words and entity weights, it extracts the most relevant knowledge attributes from the knowledge graph to generate knowledge supplement vectors. Then, through a gating fusion mechanism incorporating entity weights, it fuses the topic feature vectors and knowledge supplement vectors to obtain a topic enhancement vector, achieving precise knowledge enhancement for topic words to ensure the accuracy of subsequent topic-based text classification. Furthermore, by inputting the topic enhancement vector and text sentence vector into the classifier, a dual-branch similarity vector based on topic and text is obtained. After optimization by difference comparison, a document similarity vector is generated and the text category is output. This can improve the accuracy and efficiency of text classification, and is especially suitable for news fields with high comprehensiveness and a large number of specialized terms. Example 1

[0018] like Figures 1 to 4 As shown, the present invention provides a text classification method based on document similarity, which includes the following steps.

[0019] Step 1: The document is split into sentences. Each sentence is vectorized to obtain an original sentence vector, which is then segmented into words to obtain a word set. A word vector sequence is generated based on the word set, and the word vector sequence is aggregated to generate the first sentence vector. Preferably, in this embodiment, the document is a news document. The news document is first preprocessed, including removing noise tags, special symbols, redundant spaces, and text normalization. The noise tags include HTML tags and XML tags, special symbols include special characters and emoticons, and redundant spaces include multiple consecutive spaces and tabs. The text normalization unifies the Chinese, English, numbers, and punctuation marks in the document into a standard format, thereby obtaining the original text data of the document. Then, the original text data is split into sentences based on punctuation rules. A pre-trained semantic representation model, such as Sentence-BERT, is used to map each sentence into a fixed-dimensional original sentence vector. Further, the word segmentation tool jieba is used to segment each sentence to obtain a word set for each sentence. Then, based on the BERT vector model, each word in the word set is mapped into a word vector, and all word vectors are combined to generate a word vector sequence. This invention does not limit the vector model used to generate word vectors. In other embodiments, vector models such as Word2Vec and FastText can also be used. Finally, in order to highlight the semantics of important words and suppress the influence of noisy words, the attention weight of each word vector in the word vector sequence is calculated through a self-attention pooling mechanism. Based on the attention weights, all word vectors are weighted and summed to obtain the first sentence vector.

[0020] Step 2: Calculate the semantic similarity between the original sentence vector and the first sentence vector. If the semantic similarity of any sentence is less than the target value, update the word set through iterative optimization of the word segmentation mechanism, and regenerate the word vector sequence and the first sentence vector. Specifically, let E be the original sentence vector and the first sentence vector of sentence m. m0 E m1 Let m = 1, 2, ..., M, where M is the number of sentences in the document. In this invention, cosine similarity is used to calculate the semantic similarity S between the original sentence vector and the first sentence vector. m,0 Then S m,0 =E m0 ·E m1 / ||E m0 ||·||E m1 ||,E m0 ·E m1 Denotes the vector dot product, ||E m0 ||、||E m1 || represents vector E m0 E m1 The modulus length. A target value S is preset. exp If S m,0 <S expThis indicates that the current word segmentation result has low semantic consistency with the original sentence, and the word segmentation may have introduced noise or erroneous boundaries (such as compound words being cut off or incomplete entity recognition). It needs to be corrected through iterative optimization of the word segmentation mechanism. The specific steps are described in Example 2. If S m,0 ≥S exp This indicates that the current word segmentation quality is high, effectively preserving the semantic information of the original sentence, and can be directly used for subsequent processing without optimization. The target value S... exp The value range is usually between 0.7 and 0.9. The larger the target value, the higher the word segmentation quality. In this embodiment, the target value is set to 0.85.

[0021] Step 3: Calculate statement weights based on the positional data of each statement, and generate a first similarity vector by combining the statement weights of all statements with the first statement vector. The positional data includes whether the statement is a title, the first sentence of a paragraph, or the last sentence of a paragraph, and the title number or paragraph number in which the statement is located. Specifically, positional parameters are generated based on the positional data, positional importance is calculated based on the positional parameters, and then statement weights are calculated. The positional parameters include title factors, paragraph factors, and statement factors. The first statement vector of each statement is semantically aligned and then input into a classifier to obtain the statement similarity vector of each statement. The classifier contains multiple category models, each corresponding to a text category and containing at least one category semantic vector. The semantic space alignment refers to mapping the first statement vector to the semantic space of the category semantic vector to ensure that both are in the same measurable semantic space, thereby enabling the unified calculation of the semantic matching degree between different statements and each text category through cosine similarity. Subsequently, the similarity between different sentences and the same category model is extracted from all sentence similarity vectors. The sentence similarities are then weighted and summed to obtain the document similarity between the document and the category model. Combining the document similarities from all category models yields the first similarity vector based on plain text classification. The specific method for generating the first similarity vector is described in Example 3.

[0022] Step 4: After identifying entity words in all word sets, an entity word set is generated. Multiple topic words are extracted from the entity word set based on entity weights. The corresponding knowledge attribute vectors are then extracted from the knowledge base by combining the context vectors of each topic word with its entity weight. Specifically, entity weights are calculated based on the positional importance of the entity words in all statements. The entity words with the highest entity weights in the entity word set are selected as topic words. Contextual text fragments corresponding to the topic words are extracted based on their positions in different statements. All contextual text fragments of the topic word are concatenated and vectorized to obtain the context vector. The knowledge base is a general knowledge graph. Topic words are linked to corresponding entity nodes in the knowledge graph using named entity recognition. The extraction depth and number of extracted attributes are determined based on entity weights. Local subgraphs are extracted from the knowledge graph centered on entity nodes. Each knowledge information in the local subgraph is encoded as an attribute feature vector. The cosine similarity between each attribute feature vector and the context vector is calculated. The attribute feature vector with the highest cosine similarity is extracted as the knowledge attribute vector. The specific method for generating knowledge attribute vectors is described in Example 4.

[0023] Step 5: Aggregate the word vectors of all topic words to generate a topic feature vector, aggregate all knowledge attribute vectors to generate a knowledge supplement vector, combine the topic feature vector and the knowledge supplement vector to generate a topic enhancement vector, and then generate a second similarity vector based on the topic enhancement vector. Specifically, in this embodiment, the topic feature vector is generated by aggregating the word vectors of all topic words through average pooling, and the knowledge supplement vector is generated by aggregating all knowledge attribute vectors through maximum attention pooling. Then, the entity weights of all topic words are extracted, and the average, maximum, and entropy of all entity weights are calculated to generate a reference vector. The topic feature vector, the knowledge supplement vector, and the reference vector are concatenated, and after sigmoid activation, a gated weight vector is obtained. Based on the gated weight vector, the topic feature vector and the knowledge supplement vector are fused element-wise to obtain the topic enhancement vector. The topic enhancement vector is then input into the classifier, and its cosine similarity with the category semantic vectors of each category model is calculated to generate a second similarity vector, as described in Embodiment 4.

[0024] Step 6: Calculate the comprehensive difference between the first similarity vector and the second similarity vector. If the comprehensive difference is less than a preset threshold, generate a document similarity vector and output the text category to end the task; otherwise, adjust the configuration parameters of the topic enhancement vector and return to Step 4. Specifically, the numerical difference is obtained by calculating the JS divergence of the first and second similarity vectors, and then the ranking difference is calculated based on the ranking cross-correlation. The comprehensive difference is calculated by combining the numerical difference and the ranking difference. The comprehensive difference reflects the inconsistency between the text classification result and the topic enhancement classification result in terms of numerical distribution and category ranking. The smaller the value, the more reliable the topic enhancement classification result. When the comprehensive difference is less than the preset threshold, a document similarity vector is generated by combining the first and second similarity vectors, and the final classification result of the document is output based on the document similarity vector. The specific method for generating the document similarity vector is described in Example 5. Example 2

[0025] This embodiment further discloses a preferred method for updating the word set by iteratively optimizing the word segmentation mechanism in step 2. The present invention uses a language model and a news professional dictionary to iteratively optimize the word set in order to reduce the word segmentation error rate and improve the interpretability and accuracy of the semantic feature representation of the first sentence vector and topic feature vector generated based on the word set. The method includes the following steps.

[0026] Step 201: Calculate the perplexity of each word in the word set sequentially using a character set language model pre-trained based on news corpora. Assume that statement m has L characters. Segment each character of the statement separately to obtain a character set O, let O = [o1, o2, ..., o...]. L The character set is input into the character set language model to obtain the conditional probability p of each character. i Therefore, the perplexity r of the character is calculated. i r i =-log2p i Let i = 1, 2, ..., L. Extract each word from the word set sequentially, calculate the sum of the perplexity of all characters contained in the word, and thus obtain the perplexity of the word. The larger the value, the less accurate the word segmentation in the current sentence.

[0027] Step 202: Extract the word with the highest perplexity, merge this word with its adjacent words, perform maximum positive matching on the merged words based on a pre-defined news-specific dictionary, and segment the words according to the matching results to generate at least one candidate word segmentation set. The news-specific dictionary includes professional terminology, neologisms, and compound word terms in the news field. Specifically, extract the word with the highest perplexity, w, from the word set. max Combine it with the words on its left and right to obtain a new string w. newThen, the string is subjected to maximum forward matching using the aforementioned news dictionary. That is, starting from the beginning of the string, the longest word in the news dictionary is matched, and then the remaining part is matched, eventually resulting in at least one candidate word set.

[0028] Step 203: Calculate the average perplexity of each word in each candidate word segmentation set, select the candidate word segmentation set with the lowest average perplexity, replace the merged regions in the word set with the candidate word segmentation set, and update the word set accordingly. Subsequently, remap the updated word set to a sequence of word vectors, generate the first sentence vector through self-attention pooling, and calculate its semantic similarity with the original sentence vector again.

[0029] Step 204: Repeat steps 201 to 203 until the semantic similarity is greater than or equal to the target value, and output the final word set and the corresponding first sentence vector. Alternatively, a maximum number of iterations can be preset. If the maximum number of iterations is reached but the semantic similarity still does not meet the target, the word set with the highest semantic similarity during the iteration process is taken as the final result. Example 3

[0030] like Figure 4 As shown, this embodiment further discloses a preferred method for generating the first similarity vector.

[0031] Calculate statement weights. Extract the positional data of statement m, and calculate the positional importance z of the statement based on the positional data. m , z m =z0+λ1f title +λ2f para +λ3f sent Where z0 is the base value, z0=1. title This is the heading factor; it takes a value of 2 when the statement is a heading, and 0 otherwise. para This is a paragraph factor; it takes a value of 1.5 when the statement is in the first paragraph, and 0 otherwise. sent The sentence factor is λ1, which is 1 when the sentence is the first sentence of a paragraph, and 0 otherwise. λ1, λ2, and λ3 are the weight coefficients of the title factor, paragraph factor, and sentence factor, respectively. In this embodiment, λ1, λ2, and λ3 are all set to 1. Based on positional importance z... m Calculate the statement weight β of statement m m , M represents the number of statements in the document.

[0032] Generate a sentence similarity vector. For example... Figure 3 As shown, the first statement vector of each statement is semantically aligned and then input into the classifier. The statement similarity between the first statement vector and the category semantic vectors of each category model in the classifier is calculated, generating the statement similarity vector for the corresponding statement. Let the aligned first statement vector be E.m1 ', calculate its relationship with the category semantic vector L k Sentence similarity S mk S mk =(E m1 '·L k ) / (||E m1 '||·||L k ||). The similarity S between the combined statement m and the statements in the K category models. mk Given k=1,2,…,K, generate the statement similarity vector SS for this statement. m SS m =[S m1 ,S m2 ,…,S mk ,…,S mK ].

[0033] Generate the first similarity vector. For the category model k, extract the corresponding sentence similarity S from the sentence similarity vectors of each sentence. mk Then, combined with the statement weight β m We calculate the document similarity D between the document and category model k by weighting the similarity of all statements. k , Combining the document similarity scores from all category models yields the first document similarity vector S1 based on text classification, where S1 = [D1, D2, ..., D...]. k ,…,D K ]. Example 4

[0034] like Figure 4 As shown, this embodiment further discloses a preferred method for generating a second similarity vector.

[0035] Generate an entity word set and entity weights. Iterate through the word set of each sentence in the document and extract entity words from each sentence using a named entity recognition model (such as BERT-NER). For each entity word, record its sentence index and the start and end positions of the characters within that sentence, forming a triple position index (entity word, sentence index, start and end positions). Summarize the entity words extracted from all sentences, merging the position indices of the same entity word with multiple records into a position index list for that entity word, thus obtaining a unique entity word set. Then, based on the position index list of each entity word, obtain the position importance of the sentence to which each occurrence belongs, and sum the position importance of all occurrences to obtain the entity weight of that entity word. In this invention, this summation method allows entity words appearing in key positions such as titles and the first sentence of the first paragraph to receive higher weights, thereby reflecting their thematic representativeness in the document.

[0036] Generate topic terms and context vectors. Entity terms are sorted by entity weight from highest to lowest, and the top N entity terms are selected as topic terms, representing the most thematically representative key entities in the document. For each topic term, its position in the corresponding sentence is located using a positional index. Using that position as the center, the text is expanded forward and backward by at least one window length to extract the context fragments of that topic term. Then, the context fragments of that topic term in all appearing sentences are concatenated sequentially, and vectorized using a semantic representation model to obtain the context vector.

[0037] Knowledge attribute vectors for topic words are extracted from the knowledge base based on entity weights and context vectors, such as Figure 2 As shown. The knowledge base is a general knowledge graph, such as Wikidata, Baidu Knowledge Graph, CN-Dbpedia, etc. Keywords are linked to corresponding entity nodes in the knowledge graph using named entity recognition, and the extraction depth P is determined based on the entity weight α. l And the number of attributes extracted, P k A local subgraph corresponding to a topic word is extracted from the knowledge graph, centered on an entity node. l =ceil(α / α max ·P l,m ), P k =ceil(α / α max ·P k,m ). ceil() is the floor function, α max For the maximum entity weight, P l,m and P k,m These are the preset maximum extraction depth and maximum number of attributes, respectively. In this invention, P l,m =4, P k,m =20. For example, when P l =2, P k When the value is 10, it indicates that the system explores two hops outward from the entity node and selects the 10 most relevant attributes from the entity node and its neighboring nodes. Then, each piece of knowledge information in the local subgraph is encoded as an attribute feature vector. The cosine similarity between each attribute feature vector and the context vector is calculated, and the attribute feature vector with the highest cosine similarity is taken as the knowledge attribute vector A. n , n=1,2,…,N.

[0038] Generate topic feature vectors and knowledge supplement vectors. Extract word vectors w for each topic term. t,n The topic feature vector v is generated by aggregating the word vectors of all topic words through average pooling. topic , All knowledge attribute vectors A are aggregated through maximum attention pooling. n Generate knowledge supplement vector v know , j=1,2,…,D, where D is the dimension of the knowledge attribute vector. know,j A n,j These represent the knowledge supplement vectors v. know and knowledge attribute vector A n The vector value in the j-th dimension.

[0039] A topic enhancement vector is generated by combining topic feature vectors and knowledge supplementation vectors. In this invention, a gating fusion mechanism using entity weights is introduced to enhance the topic feature vector v. topic and knowledge supplement vector v know The fusion yields the topic enhancement vector v. tk Specifically, the entity weights of all keywords are extracted, and the average value α of all entity weights is calculated. avg Maximum value α max and entropy α H Generate reference vector u=(α) avg ,α max ,α H ). , , Among them, α n p represents the entity weight of the nth keyword. n For entity weight α n The normalized proportion, Then, the topic feature vector v topic Knowledge supplement vector v know The reference vector u is concatenated and input into a single-layer feedforward neural network, which is then activated by sigmoid to obtain the gated weight vector g. m g m =sigmoid(a1·[v topic ;v know ;u]+b1). Where a1 and b1 are the learnable parameters of a single-layer feedforward neural network. Finally, based on the gated weight vector g m For the topic feature vector v topic and knowledge supplement vector v know Perform element-wise weighted summation to obtain the topic enhancement vector v. tk v tk =g m ⊙v topic +(1-g m )⊙v know , ⊙ indicates element-wise multiplication.

[0040] A second similarity vector is generated. The topic enhancement vector is input into the classifier, and the cosine similarity between the topic enhancement vector and the category semantic vector of each category model is calculated, thus obtaining the second similarity vector of the document based on topic enhancement classification. In this invention, the second similarity vector is used to measure the category tendency of the document at the topic level after incorporating external knowledge, complementing the first similarity vector based on text classification, and providing a basis for subsequent comprehensive evaluation of text category by combining document semantics and topic information. Example 5

[0041] like Figure 4 As shown, this embodiment further discloses a preferred method for generating document similarity vectors and determining text categories. The present invention optimizes the text classification results of documents through a dual-path similarity comparison mechanism of text classification and topic enhancement classification.

[0042] Calculate the numerical difference. Normalize the first similarity vector S1 and the second similarity vector S2 using softmax to obtain their corresponding probability distributions P and Q, i.e., P = softmax(S1), Q = softmax(S2). Then calculate the numerical difference c1 between S1 and S2, where c1 = D. JS (P||Q), D JS () is the JS divergence function. c1∈[0,1], the smaller c1 is, the closer the numerical probability distributions of the first similarity vector and the second similarity vector are.

[0043] Calculate the ranking dissimilarity. Sort the similarity values ​​of the K category models in the first similarity vector S1 in ascending order, thus obtaining the order column S of S1. x Using the same method, the ordered column S of the second similarity vector S2 can be obtained. y Let S be the value of S. x =[x1,x2,…,x K ], S y =[y1,y2,…,y K ], calculate S x S y Average rank x avg y avg Then the rank correlation coefficients of S1 and S2 τ∈[-1,1]. Then, based on the rank correlation coefficient τ, the ranking difference c2 is calculated, c2=(1-τ) / 2, c2∈[0,1]. The smaller c2 is, the higher the ranking consistency between the first similarity vector and the second similarity vector.

[0044] The overall difference score is calculated by combining numerical difference score and ranking difference score. A preset balance factor μ is used to comprehensively reflect the consistency of the probability value distribution and category preference order of the first similarity vector S1 and the second similarity vector S2, and the overall difference score c is calculated accordingly.s Then c s =μc1+(1-μ)c2. The value of μ ranges from (0,1). In this invention, μ is set to 0.6.

[0045] If the overall difference is less than a preset threshold, it indicates that the plain text classification result is highly consistent with the topic enhancement classification result. That is, the topic enhancement branch provides reliable knowledge supplementation while maintaining consistency with the plain text branch. At this time, the first similarity vector S1 and the second similarity vector S2 are weighted and fused to obtain the document similarity vector S. doc Then S doc =ωS1+(1-ω)S2, where ω is the weight coefficient, ω=0.3. Then, the text category with the highest similarity in the document similarity vector is output as the classification result of the document.

[0046] If the overall difference is greater than or equal to the preset threshold, it indicates that the text classification result and the topic enhancement classification result are inconsistent in numerical distribution or category ranking, and there is a significant discrepancy. The current topic enhancement vector may have introduced inappropriate knowledge noise or omitted key topic information. It is necessary to adjust the configuration parameters of the topic enhancement vector, such as the number of topic words, extraction depth and the number of extracted attributes, and then return to step 4 to regenerate the topic enhancement vector and the second similarity vector.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text classification method based on document similarity, characterized in that, Includes the following steps: Step 1: Split the document into sentences. Each sentence is vectorized to obtain the original sentence vector. The original sentence vector is then processed by word segmentation to obtain a word set. A word vector sequence is generated based on the word set. The word vector sequence is then aggregated to generate the first sentence vector. Step 2: Calculate the semantic similarity between the original sentence vector and the first sentence vector. If the semantic similarity of any sentence is less than the target value, update the word set by iteratively optimizing the word segmentation mechanism, and regenerate the word vector sequence and the first sentence vector. Step 3: Calculate the sentence weight based on the position data of each sentence, and generate the first similarity vector by combining the sentence weights of all sentences and the first sentence vector; Step 4: After identifying the entity words in all word sets, generate an entity word set. Extract multiple topic words from the entity word set according to the entity weights. Combine the context vectors of each topic word with the entity weights to extract the corresponding knowledge attribute vectors from the knowledge base. Step 5: Aggregate the word vectors of all topic words to generate topic feature vectors, aggregate all knowledge attribute vectors to generate knowledge supplement vectors, combine topic feature vectors and knowledge supplement vectors to generate topic enhancement vectors, and then generate a second similarity vector based on topic enhancement vectors; Step 6: Calculate the combined difference between the first similarity vector and the second similarity vector. If the combined difference is less than a preset threshold, generate a document similarity vector and output the text category to end the task; otherwise, adjust the configuration parameters of the topic enhancement vector and return to Step 4. In step 3, the location data includes whether the statement is a title, the first sentence of a paragraph, or the last sentence of a paragraph, and the title number or paragraph number of the statement. Location parameters, including the title factor f, are generated based on the location data. title Paragraph factor f para and statement factor f sent The location importance z is calculated based on the location parameters. m Then, based on the importance of position z m Calculate statement weight β m , The first statement vector of each statement is semantically aligned and then input into the classifier to obtain the statement similarity vector of each statement. The first similarity vector is generated by combining the statement weights and statement similarity vectors of all statements. The classifier contains multiple category models, each category model corresponding to a text category and containing at least one category semantic vector. The semantic space alignment refers to mapping the first statement vector to the semantic space of the category semantic vector. In step 4, entity weights are calculated based on the positional importance of the entity words in all statements. The entity words with the highest weights in the entity word set are selected as topic words. Contextual text fragments corresponding to the topic words are extracted based on their positions in different statements. All contextual text fragments of the topic word are concatenated and vectorized to obtain a context vector. In step 5, the topic enhancement vector is input into the classifier, and its cosine similarity with the category semantic vectors of each category model is calculated to generate a second similarity vector.

2. The text classification method based on document similarity according to claim 1, characterized in that, In step 1, the document is a news document. The word segmentation tool jieba is used to segment each sentence to obtain the word set of each sentence. Each word in the word set is mapped to a word vector. All word vectors are combined to generate a word vector sequence. The word vector sequence is aggregated through a self-attention pooling mechanism to obtain the first sentence vector.

3. The text classification method based on document similarity according to claim 1, characterized in that, In step 2, the specific steps for updating the word set through iterative optimization of the word segmentation mechanism include: Step 201: Use a character set language model pre-trained based on news corpus to sequentially calculate the perplexity of each word in the word set; Step 202: Extract the word with the highest perplexity, merge the word with its neighboring words, perform maximum positive matching on the merged words based on the preset news professional dictionary, segment the words according to the matching results, and generate at least one candidate word segmentation set; Step 203: Calculate the average perplexity of each word in each candidate word segmentation set, select the candidate word segmentation set with the lowest average perplexity, update the word set and regenerate the first sentence vector, and calculate the semantic similarity between the first sentence vector and the original sentence vector; Step 204: Repeat steps 201 to 203 until the semantic similarity is greater than or equal to the target value.

4. The text classification method based on document similarity according to claim 1, characterized in that, In step 4, the knowledge base is a general knowledge graph. The topic words are linked to the corresponding entity nodes in the knowledge graph through named entity recognition. The extraction depth and the number of extracted attributes are determined according to the entity weight. Local subgraphs are extracted from the knowledge graph with the entity nodes as the center. Each knowledge information in the local subgraph is encoded into an attribute feature vector. The cosine similarity between each attribute feature vector and the context vector is calculated. The attribute feature vector with the highest cosine similarity is extracted as the knowledge attribute vector.

5. The text classification method based on document similarity according to claim 1, characterized in that, In step 5, the entity weights of all topic terms are extracted, and the average value α of all entity weights is calculated. avg Maximum value α max and entropy α H Generate a reference vector u, and then use the topic feature vector v. topic Knowledge supplement vector v know The reference vector u is concatenated and then activated by sigmoid to obtain the gated weight vector g. m Based on the gated weight vector g m The topic feature vector v topic and knowledge supplement vector v know Perform element-wise weighted fusion to obtain the topic enhancement vector v tk .

6. The text classification method based on document similarity according to claim 1, characterized in that, In step 6, the numerical difference c1 and the ranking difference c2 of the first similarity vector S1 and the second similarity vector S2 are calculated, where c1 = D. JS (P||Q), c2=(1-τ) / 2, D JS () represents the JS divergence function, P and Q are the probability distributions of the first similarity vector S1 and the second similarity vector S2, respectively, and τ is the rank correlation coefficient. The comprehensive difference c is then calculated by combining the numerical difference c1 and the ranking difference c2. s .

7. The text classification method based on document similarity according to claim 4, characterized in that, In step 6, a document similarity vector is generated by combining the first similarity vector and the second similarity vector. The configuration parameters of the topic enhancement vector include the number of topic words, extraction depth, and number of extracted attributes.

Images