A text analysis method and system combining knowledge base and knowledge graph

Abstract

The present application proposes a text analysis method and system that combines a knowledge base with a knowledge graph, wherein the method includes: for each unregistered word, searching for similar fields related to the unregistered word in at least two word segmentation segments of a word segmentation sequence, replacing the similar fields in the word segmentation sequence with the unregistered word to obtain an optimized word segmentation sequence; converting the optimized word segmentation sequence into an optimized word segmentation vector, calculating a first similarity based on the optimized word segmentation vector, and generating a summary text and a list of opinions based on the similarity. This method not only needs to be able to automatically identify unregistered words and determine their most likely meaning based on their context, but also needs to be able to dynamically adjust their interpretation in different types of text to ensure the consistency and accuracy of the text meaning.

Description

Technical Field

[0001] The present application relates to the field of text analysis technology, and in particular to a text analysis method and system that combines a knowledge base with a knowledge graph. Background Art

[0002] In the field of natural language processing (NLP), word segmentation models are widely used in text analysis tasks. However, traditional word segmentation models face significant challenges when dealing with unregistered vocabulary. Unregistered vocabulary refers to new words or proper nouns that are not included in pre-trained word segmentation models or existing knowledge bases. These words often lack clear semantic definitions, resulting in different meanings when they appear in different types of text, which affects the overall understanding and accuracy of the text. For example, in the automotive field, the rapid development of electric vehicles and related technologies generates a large amount of text information every day. This information may come from various sources such as news reports, research reports, user reviews, and industry analysis. For workers to fully understand a specific topic, it is unrealistic to directly read all relevant documents. Therefore, generating summaries and opinion lists has become an effective way to help workers quickly obtain key information. Common summary generation methods include the TextRank algorithm based on graph ranking. This method treats text as a graph structure, where each node represents a sentence and the edge weights are determined by the similarity between sentences. The PageRank algorithm calculates the importance score of each sentence, and the sentences with the highest scores are ultimately selected to form the summary. At the same time, the opinion list usually combines the TF-IDF method to extract high-frequency keywords, and uses named entity recognition (NER) technology to identify important entities in the document. After screening, it forms a collection of opinions reflecting the core topics.

[0003] Furthermore, the presence of unregistered words can lead to errors in text analysis. Specifically, when a word segmentation model encounters an unregistered word, it may incorrectly split it into multiple known words or simply ignore it, making the resulting summary text and opinion list less comprehensive and accurate. Summary of the Invention

[0004] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the first purpose of this application is to propose a text analysis method that combines a knowledge base with a knowledge graph to improve the accuracy of text analysis.

[0006] The second purpose of this application is to propose a text analysis system that combines a knowledge base with a knowledge graph.

[0007] To achieve the above objectives, the first embodiment of the present application proposes a text analysis method that combines a knowledge base with a knowledge graph, including:

[0008] Acquire and preprocess a text sample to obtain the preprocessed text sample;

[0009] Inputting the preprocessed text sample into a pre-trained word segmentation model to obtain a word segmentation sequence with word segmentation labels, determining a probability value of each word segment in the word segmentation sequence, and determining candidate unregistered words based on the probability value;

[0010] Based on the candidate unregistered words, determining unregistered words and adding them to the knowledge base;

[0011] For each of the unregistered words, searching for similar fields related to the unregistered words in at least two of the word segmentation segments of the word segmentation sequence, and replacing the similar fields in the word segmentation sequence with the unregistered words to obtain an optimized word segmentation sequence;

[0012] The optimized word segmentation sequence is converted into an optimized word segmentation vector, a first similarity is determined based on the optimized word segmentation vector, and a summary text and a viewpoint list are generated according to the first similarity.

[0013] In some implementations, the method further includes:

[0014] Determining the probability value of each of the word segmentation segments in the word segmentation sequence based on the emission probability and the transition probability determined in the pre-trained word segmentation model training process;

[0015] Obtaining a preset probability threshold;

[0016] The text information in the word segment with the probability value less than the probability threshold is used as the candidate unregistered vocabulary.

[0017] In some implementations, determining an unregistered word based on the candidate unregistered word and adding it to the knowledge base includes:

[0018] Determining a preliminary type of the candidate unregistered vocabulary;

[0019] Determining an optimized type of the candidate unregistered vocabulary based on the preliminary type;

[0020] Determining a vocabulary domain of the candidate unregistered vocabulary based on the optimization type;

[0021] Based on the vocabulary domain, the meaning of the candidate unregistered vocabulary is determined, and the candidate unregistered vocabulary is defined as the unregistered vocabulary and added to the knowledge base.

[0022] In some implementations, determining the preliminary type of the candidate unregistered vocabulary word includes:

[0023] Extracting text containing the candidate unregistered vocabulary from the knowledge base, determining co-occurring vocabulary pairs based on the candidate unregistered vocabulary and the N words before and after the candidate unregistered vocabulary in the text, and determining the point mutual information and local co-occurrence entropy of the co-occurring vocabulary pairs;

[0024] An adjustment function is constructed based on the point mutual information and the local co-occurrence entropy of the co-occurring word pairs to determine a first adjustment coefficient and a second adjustment coefficient; a comprehensive score of the co-occurring word pairs is determined based on the point mutual information, the local co-occurrence entropy, the first adjustment coefficient and the second adjustment coefficient of the co-occurring word pairs; the comprehensive scores of the co-occurring word pairs are arranged from high to low, and the co-occurring words of the co-occurring word pairs in the top 2 / 3 comprehensive scores are extracted and integrated into a co-occurring word set; and a preliminary type of candidate unregistered words is determined based on the co-occurring word set.

[0025] In some implementations, determining an optimized type of the candidate unregistered vocabulary based on the preliminary type includes:

[0026] Converting the co-occurring words in the co-occurring word set into word vectors, and determining the context vectors of the candidate unregistered words based on the word vectors and the comprehensive score;

[0027] Construct a representative vocabulary set, generate a representative vocabulary embedding vector for each representative vocabulary in the representative vocabulary set, aggregate the representative vocabulary embedding vectors in the representative vocabulary set to obtain an aggregated vector; determine the second similarity between the context vector and each representative vocabulary embedding vector in the aggregated vector, and use the vocabulary type corresponding to the representative vocabulary embedding vector with the highest second similarity as the optimized type of the candidate unregistered vocabulary.

[0028] In some implementations, determining the vocabulary domain of the candidate unregistered vocabulary based on the optimization type includes:

[0029] Generate a candidate unregistered vocabulary embedding vector for the candidate unregistered vocabulary;

[0030] Determine the semantic similarity between the candidate unregistered word embedding vector and each representative word embedding vector in the aggregate vector; sort the semantic similarities from high to low, and select the word types corresponding to the first x semantic similarities as the candidate type set, where x represents the number of selected word types and x>2;

[0031] Constructing a co-occurrence network based on a text library, comprising a plurality of nodes and edges; wherein the nodes comprise representative vocabulary nodes and candidate unregistered vocabulary nodes, and the co-occurrence probabilities between adjacent nodes are used as the edges;

[0032] Determining a set of adjacent nodes of the candidate unregistered vocabulary node and the representative vocabulary node in a co-occurrence network, determining a shared adjacent node of the adjacent node set, and determining a co-occurrence probability of the shared adjacent node with the candidate unregistered vocabulary node and / or the representative vocabulary node;

[0033] Determining the number of shared adjacent nodes;

[0034] The topological similarity between the unregistered vocabulary node and the representative vocabulary node is determined based on the co-occurrence probability and the number of shared adjacent nodes between the shared adjacent node and the candidate unregistered vocabulary node and / or the representative vocabulary node in the co-occurrence network; the candidate type corresponding to the representative vocabulary node in the topological similarity is used as the vocabulary domain of the unregistered vocabulary.

[0035] In some implementations, the vocabulary types corresponding to the first x semantic similarities are selected as the candidate type set, including:

[0036] Determining a sum of semantic similarities based on the semantic similarities of each of the representative vocabulary embedding vectors;

[0037] Obtaining a preset target ratio, and determining a target value according to the target ratio and the sum of the semantic similarities;

[0038] Accumulating the semantic similarities to obtain an accumulated semantic similarity, and determining a rate of change of the accumulated semantic similarity;

[0039] The minimum cumulative number of semantic similarities that first meets the condition that the cumulative semantic similarity change rate is less than the target value is found, the minimum cumulative number is used as the number of selected vocabulary types, and the selected vocabulary types are combined into a candidate type set.

[0040] In some implementations, the target ratio is 0.7-0.9.

[0041] In some implementations, determining the meaning of the candidate unregistered vocabulary based on the vocabulary domain, defining the candidate unregistered vocabulary as an unregistered vocabulary, and adding the candidate unregistered vocabulary to the knowledge base includes:

[0042] Searching for domain texts containing unregistered vocabulary from the vocabulary domain of the knowledge base, and identifying entities and attributes of the domain texts;

[0043] Constructing a co-occurrence matrix based on the unregistered vocabulary, the entity, and the attribute, storing the unregistered vocabulary and the entity / the attribute as domain co-occurrence vocabulary pairs in the co-occurrence matrix, and determining the point mutual information and the local co-occurrence entropy of the domain co-occurrence vocabulary pairs;

[0044] Determining a domain comprehensive score based on the point mutual information and the local co-occurrence entropy of the domain co-occurrence word pairs, and defining the entity and the attribute with the largest domain comprehensive score as a core entity and a key attribute;

[0045] The meaning of the candidate unregistered word is determined based on the core entity and the key attribute, the candidate unregistered word is determined as an unregistered word, and the meaning is added to the knowledge base.

[0046] To achieve the above objectives, the second embodiment of the present application proposes a text analysis system that combines a knowledge base with a knowledge graph, including:

[0047] Text sample processing module: used for acquiring and preprocessing text samples to obtain the preprocessed text samples;

[0048] Word segmentation module: connected to the text sample processing module, used to input the pre-processed text sample into the pre-trained word segmentation model, obtain a word segmentation sequence with word segmentation labels, and determine the probability value of each word segment in the word segmentation sequence, and determine candidate unregistered words based on the probability value; determine unregistered words based on the candidate unregistered words and add them to the knowledge base;

[0049] A word segmentation sequence optimization module is connected to the word segmentation module and is used to search for similar fields related to each unregistered word in at least two word segmentation segments in the word segmentation sequence, and replace the similar fields in the word segmentation sequence with the unregistered word to obtain an optimized word segmentation sequence;

[0050] Text analysis module: connected to the word segmentation sequence optimization module, used to convert the optimized word segmentation sequence into an optimized word segmentation vector, determine a first similarity based on the optimized word segmentation vector, and generate a summary text and a viewpoint list according to the first similarity.

[0051] This application improves the accuracy and consistency of text analysis by identifying and optimizing the processing of unregistered words. This method not only needs to be able to automatically identify unregistered words and determine their most likely meaning based on their context, but also needs to be able to dynamically adjust their interpretation in different types of text to ensure the consistency and accuracy of the text meaning. By integrating the advantages of knowledge bases and knowledge graphs, this method can enhance the understanding of unregistered words while ensuring word segmentation accuracy, thereby improving the performance of the entire text analysis.

[0052] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0054] Figure 1 A flowchart of a text analysis method combining a knowledge base and a knowledge graph provided in an embodiment of the present application;

[0055] Figure 2 This is a chart comparing the prediction accuracy of the present embodiment and the prior art in a text analysis method combining a knowledge base and a knowledge graph provided in an embodiment of the present application;

[0056] Figure 3 This is a structural diagram of a text analysis system that combines a knowledge base and a knowledge graph, as provided in an embodiment of the present application;

[0057] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0059] The following describes the text analysis method and system that combines the knowledge base and knowledge graph in an embodiment of the present application with reference to the accompanying drawings.

[0060] Figure 1 A flowchart of a text analysis method combining a knowledge base and a knowledge graph provided in an embodiment of the present application.

[0061] It should be noted that the executor of the text analysis method combining the knowledge base and the knowledge graph in the embodiment of the present application is the text analysis system combining the knowledge base and the knowledge graph in the embodiment of the present application. The text analysis system combining the knowledge base and the knowledge graph can be configured in an electronic device so that the electronic device can perform text analysis functions.

[0062] like Figure 1 As shown in FIG, the text analysis method combining the knowledge base with the knowledge graph includes the following steps:

[0063] Step 1: Acquire and preprocess a text sample to obtain the preprocessed text sample.

[0064] Use a scraping tool to scrape user comments or social media posts from websites, download publicly available datasets, and convert locally stored Word documents or other file formats into plain text (e.g., txt).

[0065] Data cleaning: Remove HTML tags. Remove special characters and numbers. Convert all letters to lowercase for unified processing.

[0066] Data deduplication: Hash the text and check whether the hash value is repeated.

[0067] After the above preprocessing, the preprocessed text sample is obtained.

[0068] Step 2: Input the preprocessed text sample into the pre-trained word segmentation model to obtain a word segmentation sequence with word segmentation labels, and determine the probability value of each word segmentation fragment in the word segmentation sequence, and determine the candidate unregistered words based on the probability value; based on the candidate unregistered words, determine the unregistered words and add them to the knowledge base.

[0069] This embodiment also includes: determining the probability value of each word segmentation fragment in the word segmentation sequence through the emission probability and transition probability determined by the pre-trained word segmentation model training process; obtaining a preset probability threshold, and taking the text information in the word segmentation fragment whose probability value is less than the probability threshold as a candidate unregistered vocabulary.

[0070] In this embodiment, the BERT model is selected as the pre-trained word segmentation model. The BERT model has strong context understanding capabilities and has been widely used in various natural language processing tasks.

[0071] This embodiment discloses a text analysis method that combines a knowledge base with a knowledge graph, particularly suitable for the automotive sector. To better apply the pre-trained BERT model to the automotive sector, specifically targeting common but critical terms such as engine efficiency and fuel economy, the model is fine-tuned for this domain. First, a high-quality automotive-specific corpus is constructed, including relevant text collected from automotive forums, technical documentation, product manuals, and social media. This data covers a wide range of automotive-related topics, ensuring that the model is exposed to a variety of expressions and specialized terminology. Next, some text is manually annotated to clarify the segmentation boundaries of each word or phrase. For example, "fuel economy" should be treated as a whole rather than being split into "fuel / economy." Based on this annotated data, the BERT model is fine-tuned by adding a sequence annotation layer, allowing it to learn how to correctly identify and segment specific automotive terms. During the fine-tuning process, special attention is paid to specialized terms that are easily mis-segmented, and the model's accuracy is improved through continuous adjustment and optimization. After fine-tuning, the model's performance is evaluated using an independent test set to ensure that it can accurately identify key terms such as engine efficiency in new text. If it is found that some terms still have recognition problems, the training data will be further expanded and refined, and the model's learning of the term will be strengthened in a targeted manner to achieve the best word segmentation effect.

[0072] Furthermore, when fine-tuning the BERT model, in addition to focusing on improving its ability to recognize specific terms, attention must also be paid to the model's ability to understand complex sentence structures and polysemous words. For example, in the sentence "This car model offers excellent fuel economy and is suitable for long-distance driving," fuel economy, as a key metric for evaluating vehicle fuel efficiency, must be accurately recognized as a complete concept. Furthermore, considering that the term "economy" can have different meanings in different contexts (such as the principle of economy), the fine-tuning process must also include diverse examples to help the model distinguish these nuances. Through meticulous model fine-tuning and continuous evaluation and iteration, the result is a powerful word segmentation tool that can both accurately recognize specific automotive terms and flexibly handle diverse linguistic expressions, significantly improving the accuracy and efficiency of text analysis and information extraction. This process not only enhances the model's adaptability to specific domains but also lays a solid foundation for subsequent knowledge graph construction, semantic understanding, and intelligent retrieval.

[0073] First, use an NLP tool to segment the preprocessed text sample into multiple sentences. When the preprocessed text sample, segmented into sentences, is fed into the pretrained BERT model, the model outputs the segmentation label for each character or word (B-SEG indicates the beginning of a word, I-SEG indicates the middle of a word, and O indicates that it does not belong to any word). The model also outputs a probability value for each segmentation segment, which reflects the model's confidence in each segmentation label.

[0074] For example, the preprocessed text sample: super fast charging / really / very / fast / charging / speed / beyond / imagination, after being input into the BERT model, the model outputs the following results: [(super, B-SEG, 0.95), (fast charging, I-SEG, 0.70), (really, B-SEG, 0.85), (very, O, 0.70), (fast, B-SEG, 0.65), (charging, B-SEG, 0.92), (speed, B-SEG, 0.88), (beyond, B-SEG, 0.80), (imagination, B-SEG, 0.85)].

[0075] The emission probability refers to the probability of observing a specific word given a certain word segmentation label. For example, in the above example, the emission probability P(super|B-SEG) represents the probability of observing super when the word segmentation label is B-SEG. The transition probability refers to the probability of transitioning from one word segmentation label to another. For example, P(B-SEG→I-SEG) represents the probability of transitioning from the B-SEG state to the I-SEG state. In the BERT model, the emission probability and transition probability are learned through the parameters within the model. Specifically, the BERT model uses the Transformer encoder to capture contextual information and outputs the probability distribution of the word segmentation label at each position through the softmax layer. Therefore, the emission probability and transition probability are actually part of the internal parameters of the model and do not need to be calculated separately.

[0076] This example extracted 10,000 texts from a corpus in the automotive field and performed word segmentation prediction. The probabilities of all word segments were calculated, resulting in the following distribution: word segment probabilities between 0.9 and 1.0 occurred 6,000 times, accounting for 60% of the total; word segment probabilities between 0.8 and 0.9 occurred 2,000 times, accounting for 20% of the total; word segment probabilities between 0.7 and 0.8 occurred 1,000 times, accounting for 10% of the total; word segment probabilities between 0.6 and 0.7 occurred 600 times, accounting for 6% of the total; and word segment probabilities less than 0.6 occurred 400 times, accounting for 4% of the total. Words with probabilities less than 0.7 are mostly rare words, abbreviations, or compound structures, which may not yet be included in the knowledge base. Therefore, choosing 0.7 as the probability threshold is based on a natural division of the actual probability distribution and is statistically reasonable. The text information in the word segment with a probability value less than the probability threshold is taken as a candidate unregistered word.

[0077] Determine unregistered words based on candidate unregistered words and add them to the knowledge base, including:

[0078] Step 21: Determine the preliminary type of the candidate unregistered vocabulary.

[0079] Step 211: extract the text containing the candidate unregistered word from the knowledge base, determine the co-occurring word pairs based on the candidate unregistered word and the N words before and after the candidate unregistered word in the text, and determine the point mutual information and local co-occurrence entropy of the co-occurring word pairs.

[0080] Pointwise mutual information (PMI) measures whether the probability of two words co-occurring is higher than random chance. A high PMI value for two words indicates a strong semantic connection between them. For example, a high PMI value for "super-fast charging" and "battery management system" indicates a close semantic connection between the two words, further confirming that super-fast charging may be a technology related to battery management systems.

[0081] , represents the point mutual information of the i-th co-occurring word pair, log represents the logarithmic function, Representative candidate unregistered words The probability of co-occurrence with the k-th word, Representative candidate unregistered words The probability of a single occurrence, Represents the probability of the kth word appearing alone.

[0082] Local co-occurrence entropy measures the diversity of a word's occurrences across different contexts. If a word has low entropy and primarily appears in a specific context, it indicates a high degree of semantic concentration within that specific domain. For example, the low local co-occurrence entropy for "preventing overcharging" indicates that it primarily appears in contexts related to charging safety. Therefore, it is considered a key attribute in the meaning of "super-fast charging."

[0083] , represents the local co-occurrence entropy of the i-th co-occurring word pair, Representative candidate unregistered words The conditional probability of appearing in the k-th lexical context.

[0084] Step 212: construct an adjustment function based on the point mutual information and the local co-occurrence entropy of the co-occurring word pairs, and determine a first adjustment coefficient and a second adjustment coefficient; determine a comprehensive score of the co-occurring word pairs based on the point mutual information, the local co-occurrence entropy, the first adjustment coefficient, and the second adjustment coefficient; arrange the comprehensive scores of the co-occurring word pairs from high to low, extract the co-occurring words of the co-occurring word pairs in the top 2 / 3 comprehensive scores, and integrate them into a co-occurring word set; determine a preliminary type of candidate unregistered words based on the co-occurring word set.

[0085] ;

[0086] ;

[0087] Where, is the first adjustment coefficient, is the second adjustment coefficient, represents the point mutual information of the i-th co-occurring word pair, represents the local co-occurrence entropy of the i-th co-occurring word pair, and e represents the exponential function.

[0088] The method proposed in this embodiment aims to dynamically adjust the adjustment coefficient of word pairs by combining point mutual information and local co-occurrence entropy to improve the accuracy and robustness of unregistered word recognition. This method not only uses high point mutual information values ​​to identify word pairs with strong correlation, but also considers low local co-occurrence entropy values ​​to screen out words with high context consistency, which serves as an important basis for judging whether a word belongs to a specific field. Specifically, a high point mutual information value indicates that the probability of two words co-occurring in a text is much higher than the probability of appearing independently, which means that there is a strong semantic connection between the two words; while a low local co-occurrence entropy value indicates that the word tends to appear in a relatively fixed context, reflecting its specificity in a specific field.

[0089] Furthermore, for word pairs with high point-wise mutual information and high local co-occurrence entropy, despite having a strong semantic connection, these words appear in a variety of contexts, resulting in weak domain specificity. Conversely, if a word pair has low point-wise mutual information but also low local co-occurrence entropy, this means that although the direct semantic connection is not strong, these words often appear in the same limited range, demonstrating a certain degree of domain specificity. To address these different statistical characteristics, this method designs a corresponding adjustment coefficient calculation formula, flexibly weighting the influence of point-wise mutual information and local co-occurrence entropy according to specific circumstances, ensuring that words with clear meanings and close relevance to specific topics can be accurately identified.

[0090] , is the comprehensive score of the i-th co-occurring word pair, and max is the maximum value operation. The comprehensive score is calculated by combining the maximum value of the point mutual information and local co-occurrence entropy adjusted by the first and second adjustment coefficients, and the point mutual information and local co-occurrence entropy of the current co-occurring word pair.

[0091] By calculating the comprehensive score of co-occurring word pairs, sorting the scores from high to low, and extracting the top two-thirds of co-occurring word pairs, these pairs are then integrated into a co-occurring word set, effectively determining the preliminary types of candidate unregistered words. The core of this method is to leverage the semantic relevance and contextual consistency between words in a statistical language model to infer the potential categories of unregistered words. Pointwise mutual information measures the strength of co-occurrence between two words, reflecting their semantic relevance; while local co-occurrence entropy captures the diversity of a word's distribution across different contexts. Combining these two metrics and dynamically weighting them using the first and second adjustment coefficients allows for more precise screening of word pairs that are highly relevant and semantically consistent with candidate unregistered words.

[0092] After integrating these high-scoring co-occurring words into a set, cluster analysis can be performed based on the lexical features in this set to summarize the preliminary types of candidate unregistered words. First, the following features are extracted: co-occurring word pairs, point mutual information, local co-occurrence entropy, and comprehensive score. After completing the feature extraction, the cluster analysis stage begins. Selecting a suitable clustering algorithm is key. Considering that text data usually has a high dimension and there may be non-spherical distributed data points, hierarchical clustering is preferred. The co-occurring word pairs are mapped to a multidimensional space according to their comprehensive scores, and then the selected clustering algorithm is applied to group them. Each cluster represents a group of semantically close words. For example, after performing a cluster analysis, if a cluster is found to contain words such as battery management system, fast charging station, and cruising range, it can be inferred that super fast charging belongs to the broad category of electric vehicle charging technology.

[0093] The last step is to interpret the clustering results and infer the preliminary type of unregistered words based on them. By analyzing the core words in each cluster, that is, those with high comprehensive scores, we can understand the theme or field of each cluster. Specifically, look at the core words of each cluster, combine professional knowledge or domain knowledge base, and assign a preliminary type to the candidate unregistered words. This process not only relies on the semantic information of the words themselves, but also makes full use of the contextual relationship in actual use. For example, the preliminary type of super fast charging may be determined as new energy vehicle technology, because most of the words closely related to it belong to this field. This cluster analysis-based method provides an accurate and reliable classification scheme that can effectively handle the problem of unregistered word recognition, especially in complex contexts, showing higher accuracy and stronger robustness.

[0094] Step 22: Determine the optimized type of the candidate unregistered vocabulary based on the preliminary type.

[0095] Step 221 : convert the co-occurring words in the co-occurring word set into word vectors, and determine the context vectors of the candidate unregistered words based on the word vectors and the comprehensive scores.

[0096] First, word vectors are generated using the Word2Vec model. A window size is defined, which determines the number of preceding and following words considered when calculating the context of a word. For example, if the window size is 2, then for each word in a sentence, the two preceding and two following words are considered. Then, the training process begins using the Skip-gram architecture: for each word sequence in the automotive corpus, the model attempts to predict all the context words within the window surrounding the current word. Specifically, for each center word, the model adjusts its vector representation to maximize the probability of predicting its context words. By repeatedly iterating over the entire automotive corpus, the model gradually optimizes these vectors, placing semantically similar words or those that frequently co-occur close together in the vector space. Ultimately, after training, a model is obtained that maps every word in the automotive corpus into a fixed-dimensional vector space. These vectors effectively capture the semantic relationships between words, providing a foundation for subsequent tasks such as unregistered word recognition. Any word appearing in the automotive corpus can be converted into a fixed-dimensional word vector. For example, words closely related to electric vehicles, charging stations, battery management, and super-fast charging can all be queried through the model to obtain corresponding word vectors. The co-occurring words in the co-occurring word set are passed as input to the model, and the model outputs the word vector. These vectors not only capture the meaning of the words themselves, but also imply their relationship with other related words, providing strong support for subsequent construction of context vectors and identification of unregistered words. Finally, the context vector of the candidate unregistered word is obtained by multiplying the word vector of the generated co-occurring word and the comprehensive score of the co-occurring word pair corresponding to the co-occurring word.

[0097] The comprehensive score is introduced as a weight, taking into account the different importance of different co-occurring words to the candidate unregistered words. The comprehensive score combines point mutual information and local co-occurrence entropy, and is dynamically weighted by the first adjustment coefficient and the second adjustment coefficient. This strategy not only emphasizes co-occurring words that have a strong correlation with unregistered words (high PMI value), but also considers those words that appear frequently in specific fields and have high specificity (low LCE value). For example, in the process of identifying super-fast charging, although mobile phones may also appear in discussions about battery technology, their importance is far less than that of electric vehicles or charging stations, because the latter are more closely related to super-fast charging and are more concentrated in the automotive field. In this way, the method can more accurately reflect the actual contribution of each co-occurring word to understanding unregistered words.

[0098] Step 222: construct a representative vocabulary set for each known vocabulary type, generate a representative vocabulary embedding vector for each representative vocabulary in each representative vocabulary set, aggregate the representative vocabulary embedding vectors in all the representative vocabulary sets to obtain an aggregated vector; determine the second similarity between the context vector and each representative vocabulary embedding vector in the aggregated vector, and use the vocabulary type corresponding to the representative vocabulary embedding vector with the highest second similarity as the optimized type of the candidate unregistered vocabulary.

[0099] Representative vocabulary is a core term or typical vocabulary of the current type, with high frequency and clear semantics in the training corpus. Methods for constructing a representative vocabulary include, but are not limited to: extracting from existing dictionaries (such as encyclopedias and industry glossaries); and using statistical methods such as TF-IDF to extract high-frequency, high-weighted words from text in the corresponding field.

[0100] For example, a representative vocabulary set of electric vehicle technology is: {battery management system, fast charging, power motor, charging station, cruising range}.

[0101] The vectors of all representative words are generated by the previously trained Word2Vec model.

[0102] An aggregate vector is the result of summarizing the embedding vectors of multiple representative words of a certain type. Aggregation methods include, but are not limited to, averaging, weighted averaging, and PCA dimensionality reduction. In this embodiment, the averaging method is preferred for aggregation. That is, the average of all representative word embedding vectors of the same type is taken by dimension to obtain the aggregate vector.

[0103] The second similarity between the context vector and each representative word embedding vector in the aggregate vector is determined by cosine similarity, and the word type corresponding to the representative word embedding vector with the highest second similarity is used as the optimized type of the candidate unregistered word, mainly by quantitatively analyzing the degree of proximity between the candidate unregistered word and different known types in the semantic space, so as to achieve accurate judgment of the type to which it belongs. It not only improves the accuracy of classification, but also makes the optimized type more specific, detailed and semantically targeted than the preliminary type. For example, during the calculation process, it was found that the semantic vectors of super fast charging are closest to those of fast charging, charging station and other words. Therefore, the optimized type of super fast charging can be defined as electric vehicle charging technology. This step not only takes into account the direct correlation between words, but also deeply analyzes their contextual environment, making the classification more accurate.

[0104] The context vector is generated based on the weighted co-occurring words and their comprehensive scores of the candidate unregistered words, reflecting the actual usage context and semantic tendency of the word in the text. The aggregate vector is the class center obtained by averaging the embedding vectors of multiple representative words under a certain vocabulary type, representing the typical semantic characteristics of that type. Therefore, when the cosine similarity between the context vector of the candidate unregistered word and the aggregate vector of a certain category is the highest, it means that the semantic environment of the word is closest to the category, and its semantic expression is most consistent with the core concept of the category. This matching method based on vector space distance is more mathematically rigorous and semantically interpretable than traditional rule matching or simple keyword comparison.

[0105] Refinement is a further refinement of the initial classification. For example, in the initial recognition stage, only super-fast charging can be determined to belong to the broader category of new energy technology. However, by introducing a set of representative words and calculating semantic similarity, it is possible to further identify its more specific subcategories, such as electric vehicle charging technology.

[0106] Step 23: Determine the vocabulary domain of the candidate unregistered vocabulary based on the optimization type.

[0107] Step 231: Generate a candidate unregistered vocabulary embedding vector for the candidate unregistered vocabulary.

[0108] Aggregate all representative vocabulary embedding vectors in each representative vocabulary set.

[0109] Step 232: Determine the semantic similarity between the candidate unregistered word embedding vector and each representative word embedding vector in the aggregate vector; sort the semantic similarities from high to low, and select the word types corresponding to the first x semantic similarities as the candidate type set, where x represents the number of selected word types, and x>2.

[0110] The semantic similarity between the embedding vector of the candidate unregistered word and the embedding vector of each representative word in the aggregate vector is determined by the cosine similarity method.

[0111] In order to determine the final candidate type set, this embodiment determines the total semantic similarity based on the semantic similarity of each representative word embedding vector.

[0112] A preset target ratio is obtained, and a target value is determined by multiplying the target ratio by the sum of the semantic similarities.

[0113] Accumulating the semantic similarities to obtain an accumulated semantic similarity, and determining a rate of change of the accumulated semantic similarity;

[0114] The minimum cumulative number of semantic similarities that first meets the condition that the cumulative semantic similarity change rate is less than the target value is found, the minimum cumulative number is used as the number of selected vocabulary types, and the selected vocabulary types are combined into a candidate type set.

[0115] The target ratio a is preferably between 0.7 and 0.9, which dynamically determines the number of the x most relevant types to retain. For example, if you need to recall all potentially relevant types as much as possible, you can set a to a lower value (such as 0.7), while if you pursue high precision, you can increase a to a higher value (such as 0.9).

[0116] By first satisfying the condition that the cumulative semantic similarity change rate is less than the target value, the system can accurately identify the most semantically relevant types that contribute most to understanding unregistered vocabulary, without incorporating excessive amounts of related or irrelevant information. This not only optimizes the quality of the candidate type set but also enhances the system's flexibility and robustness, enabling it to adaptively adjust its selection criteria in different application scenarios, thereby more accurately capturing key information in the text.

[0117] Exemplarily, through the above process, the following candidate type set is obtained: fast charging technology, DC fast charging device interface standard, and battery management system.

[0118] Step 233 : constructing a co-occurrence network based on the text library, including a number of nodes and edges. The nodes include representative vocabulary nodes and candidate unregistered vocabulary nodes, and the co-occurrence probabilities between adjacent nodes are used as the edges.

[0119] Exemplarily, this embodiment extracts the following text from the text library:

[0120] Super fast charging can fully charge an electric vehicle to 80% in 5 minutes;

[0121] This model supports 800V high-voltage platform and super-fast charging technology;

[0122] The fast charging station uses a new battery management system;

[0123] The charging pile is equipped with the latest super fast charging interface.

[0124] Set a window size (N words before and after) and count the co-occurrence frequencies between all words. Candidate unregistered word nodes: Super Fast Charging. Representative word nodes: Electric Vehicle Technology: Electric Vehicle, Battery Management System, Fast Charging, Range, Charging Pile, High-Voltage Platform. Charging Equipment: Charging Gun, Charging Cable, Charging Head, Power Bank.

[0125] Nodes: include all the above words. Edges: if two words appear in the same window, they are connected and the weight of the edge is marked as the co-occurrence probability (i.e., the number of co-occurrences / total number of texts).

[0126] For example, the terms "super-fast charging" and "electric vehicle" appear 10 times in 1,000 documents, with an edge weight of 10 / 1,000 = 0.01. "Super-fast charging" and "battery management system" appear 7 times, with an edge weight of 0.007. "Super-fast charging" and "charging pile" appear 9 times, with an edge weight of 0.009. "Super-fast charging" and "charging gun" appear 2 times, with an edge weight of 0.002. This constructs a co-occurrence network graph containing multiple nodes and edges.

[0127] Step 234, determining the adjacent node set of the candidate unregistered vocabulary node and the representative vocabulary node in the co-occurrence network, determining the shared adjacent nodes of the adjacent node set, and determining the co-occurrence probability of the shared adjacent nodes with the candidate unregistered vocabulary node and / or the representative vocabulary node.

[0128] For example, for the node "Super Fast Charging," the adjacent node set is: {Electric Vehicle, Battery Management System, Fast Charging, Range, Charging Pile, High-Voltage Platform}. For the representative vocabulary node "Electric Vehicle," the adjacent node set is: {Super Fast Charging, Battery Management System, Range, Power Motor, Charging Pile}. For the representative vocabulary node "Charging Pile," the adjacent node set is: {Super Fast Charging, Electric Vehicle, Charging Gun, Charging Cable, Charging Head}. For the representative vocabulary node "Charging Gun," the adjacent node set is: {Charging Pile, Charging Cable, Charging Head, Power Bank}.

[0129] The shared adjacent nodes between Superchargers and electric vehicles are: {Battery Management System, Range, Charging Pile}. The shared adjacent nodes between Superchargers and charging piles are: {Electric Vehicle, Charging Gun, Charging Cable, Charging Head}. The shared adjacent nodes between Superchargers and charging guns are: {Charging Pile}.

[0130] The co-occurrence probability of the shared neighboring node and the candidate unregistered vocabulary node and / or the representative vocabulary node is obtained from the co-occurrence network.

[0131] 1. Shared adjacent nodes between super-fast charging and electric vehicles: {battery management system, range, charging station}. The co-occurrence probability of the shared adjacent nodes with the candidate unregistered vocabulary node (super-fast charging) and / or the representative vocabulary node (electric vehicle) is shown in the following table:

[0132]

[0133] 2. Shared adjacent nodes between super-fast charging and charging piles: {electric vehicle, charging gun, charging cable, charging head}. The co-occurrence probability of the shared adjacent nodes with the candidate unregistered vocabulary node (super-fast charging) and / or the representative vocabulary node (charging pile) is shown in the following table:

[0134]

[0135] 3. The shared adjacent node between super-fast charging and charging gun is {charging pile}. The co-occurrence probability of the shared adjacent node and the candidate unregistered vocabulary node (super-fast charging) and / or the representative vocabulary node (charging gun) is shown in the following table:

[0136]

[0137] Step 235: Determine the number of shared adjacent nodes.

[0138] For each pair of candidate unregistered vocabulary nodes and representative vocabulary nodes, count the number of shared adjacent nodes: Supercharger and electric car: 3 shared nodes. Supercharger and charging pile: 4 shared nodes. Supercharger and charging gun: 1 shared node.

[0139] Step 236, determining the topological similarity between the unregistered vocabulary node and the representative vocabulary node based on the co-occurrence probability and the number of shared adjacent nodes between the shared adjacent node and the candidate unregistered vocabulary node and / or representative vocabulary node in the co-occurrence network; and taking the candidate type corresponding to the representative vocabulary node in the topological similarity as the vocabulary domain of the unregistered vocabulary.

[0140] In this embodiment, the candidate type corresponding to the representative vocabulary set in which the representative vocabulary node with the maximum topological similarity is located is used as the vocabulary domain of the unregistered vocabulary. If the candidate type corresponding to the representative vocabulary set in which the representative vocabulary node with the maximum topological similarity is located does not exist, the candidate type corresponding to the representative vocabulary set in which the representative vocabulary node with the second highest topological similarity is located is used as the vocabulary domain of the unregistered vocabulary.

[0141] ;

[0142] Where, represents the topological similarity, represents the set of adjacent nodes of the candidate unregistered vocabulary node u, represents the set of adjacent nodes of the representative vocabulary node v, represents the set of shared adjacent nodes, w represents the shared adjacent nodes, represents the number of shared adjacent nodes in the set of shared adjacent nodes, min represents the minimum operation, W(u, y) represents the co-occurrence probability of the candidate unregistered vocabulary node u and the shared adjacent node y, and W(v, y) represents the co-occurrence probability of the representative vocabulary node v and the shared adjacent node y.

[0143] This embodiment utilizes graph theory concepts to improve the accuracy of unregistered vocabulary recognition. The core of this method is to construct a co-occurrence network, representing the words in the text as nodes in the network, and using the co-occurrence relationship between words as the edges connecting these nodes. This method not only considers the direct connection between candidate unregistered vocabulary nodes and representative vocabulary nodes, but also reveals the indirect connection between the two in the semantic space by analyzing common adjacent nodes (i.e., shared adjacent nodes). The theoretical basis of this design lies in complex networks and graph theory, in which the strength of the relationship between nodes can be quantified by the co-occurrence probability, and the number of shared adjacent nodes reflects the closeness of two nodes in the network. In this way, a deeper understanding of the contextual relevance and semantic similarity between words can be achieved. From a technical perspective, the accuracy of unregistered vocabulary classification is significantly improved. Compared with traditional word frequency statistics or simple vector matching strategies, topological similarity calculation based on shared adjacent nodes can better capture the complex relationships between words and is particularly effective in identifying polysemous words or emerging terms.

[0144] Step 24: Determine the meaning of the candidate unregistered vocabulary based on the vocabulary domain, define it as an unregistered vocabulary, and add it to the knowledge base.

[0145] Step 241 : Search the domain text containing unregistered vocabulary from the vocabulary domain of the knowledge base, and identify entities and attributes in the domain text.

[0146] A knowledge base is a structured database that stores domain-specific terms, concepts, and their relationships. In this context, the knowledge base encompasses not only terms and their definitions, but also the semantic relationships between terms. A knowledge graph is a graphical representation of a knowledge base, representing entities and their relationships through nodes and edges. It can enhance the functionality of a knowledge base and provide richer semantic information.

[0147] The lexical domain of this embodiment specifically refers to each specific domain in the candidate type, which constitutes the specific semantic space to which the currently unregistered vocabulary may belong. A knowledge base search is performed on each candidate type to find out whether there is any relevant description or context about super fast charging. If there is text mentioning super fast charging in a candidate type, it is used as the domain text. Based on the domain text, entities and attributes are identified from the knowledge graph.

[0148] For example, the following text is retrieved in the candidate types: Super fast charging is a charging technology that can provide a large amount of electricity to electric vehicles in a very short time. It relies on a DC fast charging device interface standard that supports high current and high voltage, ensuring that electric vehicles of different brands and models can be safely and efficiently charged quickly using public or private charging piles.

[0149] Entities: Electric vehicles, DC fast-charging device interface standards, and public or private charging stations. Attributes: High current, high voltage, safety, and efficiency.

[0150] Step 242: construct a co-occurrence matrix based on the unregistered words, entities and attributes. The unregistered words and entities / attributes are stored in the co-occurrence matrix as domain co-occurrence word pairs, and the point mutual information and local co-occurrence entropy of the domain co-occurrence word pairs are determined.

[0151] Step 243 : Determine the domain comprehensive score based on the point mutual information and local co-occurrence entropy of the domain co-occurrence word pairs, and define the entity and attribute with the largest domain comprehensive score as the core entity and key attribute.

[0152] The point mutual information and local co-occurrence entropy of domain co-occurrence word pairs are added together to obtain the domain comprehensive score.

[0153] Step 244: Determine the meaning of the candidate unregistered word based on the core entity and the key attribute, determine the candidate unregistered word as an unregistered word, and add the meaning to the knowledge base.

[0154] For example, the definition of "super-fast charging" was defined as a charging technology that provides a large amount of energy to electric vehicles in a short period of time. This technology emphasizes not only efficient charging speeds (achieved through high current and high voltage) but also compatibility across different makes and models of electric vehicles and charging system safety. Finally, this definition was formally added to the knowledge base, and related terminology was updated.

[0155] Step 3: for each unregistered word, search for similar fields related to the unregistered word in at least two word segments in the word segmentation sequence, and replace the similar fields in the word segmentation sequence with the unregistered word to obtain an optimized word segmentation sequence.

[0156] When the word segmentation model is unclear about the meaning of unregistered words, it is easy to split them into two segmentation segments, such as "super" and "fast charging." In this case, "super" and "fast charging" can be defined as similar fields and replaced with unregistered words to obtain an optimized segmentation sequence.

[0157] Step 4: Convert the optimized word segmentation sequence into an optimized word segmentation vector, calculate a first similarity based on the optimized word segmentation vector, and generate a summary text and a viewpoint list according to the first similarity.

[0158] The optimized word segmentation vectors are also generated directly using the Word2Vec model. The first similarity is also calculated using cosine similarity.

[0159] To generate a summary, this embodiment uses the TextRank algorithm, a graph-based ranking algorithm. Specifically, the text is treated as a graph, where each graph node represents a sentence, and the weights of the edges are determined by the similarity between sentences. The PageRank algorithm calculates the importance score of each sentence, and the sentences with the highest scores are ultimately selected as the summary content. This approach considers not only the importance of individual sentences but also their position within the overall document structure and their relationship to other sentences, ensuring that the summary accurately reflects the document's main points. Furthermore, when generating the point list, the TF-IDF method is combined to extract high-frequency keywords, and named entity recognition (NER) technology is used to identify important entities in the document. These keywords and entities reflect the core topics or specific objects discussed in the text, and are filtered to form the final point list.

[0160] For example, for an article about super-fast charging technology: Super-fast charging is a charging technology that can provide electric vehicles with a large amount of power in a very short time. It relies on a DC fast charging device interface standard that supports high current and high voltage, ensuring that electric vehicles of different brands and models can be safely and efficiently charged quickly using public or private charging stations. After the above processing flow, the following results are obtained:

[0161] Abstract: Supercharging is a charging technology that can provide electric vehicles with a large amount of power in a very short time. It relies on a DC fast charging device interface standard that supports high current and high voltage.

[0162] List of viewpoints: [super fast charging, electric vehicles, DC fast charging equipment interface standard, high current, high voltage].

[0163] By combining the knowledge base with the knowledge graph, we optimize the processing of unregistered words and generate summaries and opinion lists based on their determined meanings. Figure 2 This is a comparison chart of the prediction accuracy of this embodiment and the prior art in a text analysis method combining a knowledge base and a knowledge graph provided by an embodiment of the present application; wherein, category 1 represents the sample of this application, category 2 represents the prior art sample, series 1, series 2 and series 3 represent the sample points selected by this application and the prior art respectively, the percentage of the horizontal axis represents the prediction accuracy, and the numerical values ​​on the right side of series 1, series 2 and series 3 correspond to the prediction accuracy of the sample points. Specifically, as Figure 2As shown, for the existing samples, the accuracy rates of Series 1, Series 2 and Series 3 are 83%, 87% and 85% respectively, while the corresponding accuracy rates of the samples of this application are 96%, 93% and 90% respectively. By comparison, it can be found that the method of the present invention has shown a higher accuracy rate at all sample points, with an average improvement of 5%-13%. This significant improvement shows that the method of combining the knowledge base and the knowledge graph to optimize the processing of unregistered vocabulary has effectively enhanced the ability to understand unregistered vocabulary, thereby significantly improving the overall performance of text analysis. Therefore, based on this improvement, the generated summary and opinion list will be more accurate and reliable.

[0164] In order to implement the above embodiments, the present application also proposes a text analysis system that combines a knowledge base with a knowledge graph. Figure 3 This is a structural diagram of a text analysis system that combines a knowledge base and a knowledge graph provided in an embodiment of the present application. Figure 3 As shown, the text analysis system combining the knowledge base with the knowledge graph may include: a text sample processing module 401, a word segmentation module 402, a word segmentation sequence optimization module 403, a text analysis module 404 and an effect evaluation module 405.

[0165] The text sample processing module 401 is used to obtain and pre-process text samples to obtain the pre-processed text samples;

[0166] The word segmentation module 402 is connected to the text sample processing module and is used to input the pre-processed text sample into the pre-trained word segmentation model to obtain a word segmentation sequence with word segmentation labels, determine the probability value of each word segment in the word segmentation sequence, determine candidate unregistered words based on the probability value, and determine unregistered words based on the candidate unregistered words and add them to the knowledge base;

[0167] The word segmentation sequence optimization module 403 is connected to the word segmentation module and is configured to search for similar fields related to each unregistered word in at least two of the word segmentation segments in the word segmentation sequence, and replace the similar fields in the word segmentation sequence with the unregistered word to obtain an optimized word segmentation sequence.

[0168] Text analysis module 404: connected to the word segmentation sequence optimization module, used to convert the optimized word segmentation sequence into an optimized word segmentation vector, determine a first similarity based on the optimized word segmentation vector, and generate a summary text and a viewpoint list based on the first similarity

[0169] It should be noted that the above explanation of the embodiment of the text analysis method combining the knowledge base and the knowledge graph is also applicable to the text analysis system combining the knowledge base and the knowledge graph of this embodiment, and will not be repeated here.

[0170] In order to implement the above embodiment, the present application also proposes an electronic device. Figure 4 , Figure 4 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 4 As shown, the electronic device 500 includes: a processor 501, and a memory 502 communicatively connected to the processor 501; the memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory to implement the method provided in the aforementioned embodiment.

[0171] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0172] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0173] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0174] It is important to note that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold beyond these legitimate uses. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes the relevant user information before using the feature. Furthermore, any necessary steps must be taken to safeguard and secure access to such personal information and ensure that others with access to personal information comply with its privacy policy and procedures.

[0175] This application contemplates providing implementations that allow users to selectively block the use or access of personal information data. Specifically, this disclosure contemplates providing hardware and / or software to prevent or block access to such personal information data. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0176] In the descriptions of the foregoing embodiments, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0177] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0178] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0179] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0180] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0181] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0182] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0183] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A text analysis method combining a knowledge base and a knowledge graph, characterized in that: The following steps are involved: Acquire and preprocess a text sample to obtain the preprocessed text sample; Inputting the preprocessed text sample into a pre-trained word segmentation model to obtain a word segmentation sequence with word segmentation labels, determining a probability value of each word segment in the word segmentation sequence, and determining candidate unregistered words based on the probability value; Based on the candidate unregistered words, determining unregistered words and adding them to the knowledge base; For each of the unregistered words, searching for similar fields related to the unregistered words in at least two of the word segmentation segments of the word segmentation sequence, and replacing the similar fields in the word segmentation sequence with the unregistered words to obtain an optimized word segmentation sequence; The optimized word segmentation sequence is converted into an optimized word segmentation vector, a first similarity is determined based on the optimized word segmentation vector, and a summary text and a viewpoint list are generated according to the first similarity.

2. The method according to claim 1, characterized in that Also includes: Determining the probability value of each of the word segmentation segments in the word segmentation sequence based on the emission probability and the transition probability determined in the pre-trained word segmentation model training process; Obtaining a preset probability threshold; The text information in the word segment with the probability value less than the probability threshold is used as the candidate unregistered vocabulary.

3. The method according to claim 2, characterized in that Based on the candidate unregistered words, determining unregistered words and adding them to the knowledge base includes: Determining a preliminary type of the candidate unregistered vocabulary; Determining an optimized type of the candidate unregistered vocabulary based on the preliminary type; Determining a vocabulary domain of the candidate unregistered vocabulary based on the optimization type; Based on the vocabulary domain, the meaning of the candidate unregistered vocabulary is determined, and the candidate unregistered vocabulary is defined as the unregistered vocabulary and added to the knowledge base.

4. The method according to claim 3, characterized in that Determining the preliminary type of the candidate unregistered vocabulary includes: Extracting text containing the candidate unregistered vocabulary from the knowledge base, determining co-occurring vocabulary pairs based on the candidate unregistered vocabulary and the N words before and after the candidate unregistered vocabulary in the text, and determining the point mutual information and local co-occurrence entropy of the co-occurring vocabulary pairs; An adjustment function is constructed based on the point mutual information and the local co-occurrence entropy of the co-occurring word pairs to determine a first adjustment coefficient and a second adjustment coefficient; a comprehensive score of the co-occurring word pairs is determined based on the point mutual information, the local co-occurrence entropy, the first adjustment coefficient and the second adjustment coefficient of the co-occurring word pairs; the comprehensive scores of the co-occurring word pairs are arranged from high to low, and the co-occurring words of the co-occurring word pairs in the top 2 / 3 comprehensive scores are extracted and integrated into a co-occurring word set; and a preliminary type of candidate unregistered words is determined based on the co-occurring word set.

5. The method according to claim 4, characterized in that Determining an optimized type of the candidate unregistered vocabulary based on the preliminary type includes: Converting the co-occurring words in the co-occurring word set into word vectors, and determining the context vectors of the candidate unregistered words based on the word vectors and the comprehensive score; Construct a representative vocabulary set, generate a representative vocabulary embedding vector for each representative vocabulary in the representative vocabulary set, aggregate the representative vocabulary embedding vectors in the representative vocabulary set to obtain an aggregated vector; determine the second similarity between the context vector and each representative vocabulary embedding vector in the aggregated vector, and use the vocabulary type corresponding to the representative vocabulary embedding vector with the highest second similarity as the optimized type of the candidate unregistered vocabulary.

6. The method according to claim 5, characterized in that Determining the vocabulary domain of the candidate unregistered vocabulary based on the optimization type includes: Generate a candidate unregistered vocabulary embedding vector for the candidate unregistered vocabulary; Determine the semantic similarity between the candidate unregistered word embedding vector and each representative word embedding vector in the aggregate vector; sort the semantic similarities from high to low, and select the word types corresponding to the first x semantic similarities as the candidate type set, where x represents the number of selected word types and x>2; Constructing a co-occurrence network based on a text library, comprising a plurality of nodes and edges; wherein the nodes comprise representative vocabulary nodes and candidate unregistered vocabulary nodes, and the co-occurrence probabilities between adjacent nodes are used as the edges; Determining a set of adjacent nodes of the candidate unregistered vocabulary node and the representative vocabulary node in a co-occurrence network, determining a shared adjacent node of the adjacent node set, and determining a co-occurrence probability of the shared adjacent node with the candidate unregistered vocabulary node and / or the representative vocabulary node; Determining the number of shared adjacent nodes; The topological similarity between the unregistered vocabulary node and the representative vocabulary node is determined based on the co-occurrence probability and the number of shared adjacent nodes between the shared adjacent node and the candidate unregistered vocabulary node and / or the representative vocabulary node in the co-occurrence network; the candidate type corresponding to the representative vocabulary node in the topological similarity is used as the vocabulary domain of the unregistered vocabulary.

7. The method according to claim 6, characterized in that The vocabulary types corresponding to the first x semantic similarities are selected as the candidate type set, including: Determining a sum of semantic similarities based on the semantic similarities of each of the representative vocabulary embedding vectors; Obtaining a preset target ratio, and determining a target value according to the target ratio and the sum of the semantic similarities; Accumulating the semantic similarities to obtain an accumulated semantic similarity, and determining a rate of change of the accumulated semantic similarity; The minimum cumulative number of semantic similarities that first meets the condition that the cumulative semantic similarity change rate is less than the target value is found, the minimum cumulative number is used as the number of selected vocabulary types, and the selected vocabulary types are combined into a candidate type set.

8. The method according to claim 7, characterized in that The target ratio is 0.7-0.

9.

9. The method according to claim 7, characterized in that Based on the vocabulary domain, the meaning of the candidate unregistered vocabulary is determined, and the candidate unregistered vocabulary is defined as an unregistered vocabulary, and added to the knowledge base, including: Searching for domain texts containing unregistered vocabulary from the vocabulary domain of the knowledge base, and identifying entities and attributes of the domain texts; Constructing a co-occurrence matrix based on the unregistered vocabulary, the entity, and the attribute, storing the unregistered vocabulary and the entity / the attribute as domain co-occurrence vocabulary pairs in the co-occurrence matrix, and determining the point mutual information and the local co-occurrence entropy of the domain co-occurrence vocabulary pairs; Determining a domain comprehensive score based on the point mutual information and the local co-occurrence entropy of the domain co-occurrence word pairs, and defining the entity and the attribute with the largest domain comprehensive score as a core entity and a key attribute; The meaning of the candidate unregistered word is determined based on the core entity and the key attribute, the candidate unregistered word is determined as an unregistered word, and the meaning is added to the knowledge base.

10. A text analysis system combining a knowledge base and a knowledge graph, used to execute the text analysis method combining a knowledge base and a knowledge graph as described in any one of claims 1 to 9, characterized in that: Includes the following modules: Text sample processing module: used for acquiring and preprocessing text samples to obtain the preprocessed text samples; Word segmentation module: connected to the text sample processing module, used to input the pre-processed text sample into the pre-trained word segmentation model, obtain a word segmentation sequence with word segmentation labels, and determine the probability value of each word segment in the word segmentation sequence, and determine candidate unregistered words based on the probability value; determine unregistered words based on the candidate unregistered words and add them to the knowledge base; A word segmentation sequence optimization module is connected to the word segmentation module and is used to search for similar fields related to each unregistered word in at least two word segmentation segments in the word segmentation sequence, and replace the similar fields in the word segmentation sequence with the unregistered word to obtain an optimized word segmentation sequence; Text analysis module: connected to the word segmentation sequence optimization module, used to convert the optimized word segmentation sequence into an optimized word segmentation vector, determine a first similarity based on the optimized word segmentation vector, and generate a summary text and a viewpoint list according to the first similarity.

Images