Knowledge association algorithm based on semantic similarity and co-occurrence word matching degree
By combining the jieba word segmentation library and the TF-IDF algorithm with cosine similarity calculation, the problem of insufficient generalization ability in cross-domain text data processing is solved, achieving more efficient and accurate text association and improving the effect of information retrieval and text classification.
Patent Information
- Application Number
- CN202511263972.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-05
- Publication Date
- 2025-12-30
AI Technical Summary
Existing technologies lack generalization ability in cross-domain or cross-topic text data processing, making it difficult to effectively utilize semantic similarity and co-occurrence word matching, thus limiting the accuracy and efficiency of information retrieval and text classification.
Keyword extraction is performed using the jieba word segmentation library. Combined with the TF-IDF algorithm and cosine similarity calculation, and taking into account semantic similarity and co-occurrence word matching, the accuracy and generalization ability of text association are improved through data preprocessing, semantic feature encoding and knowledge association query.
It improves the processing effect of cross-domain or cross-topic text data, enhances the accuracy and efficiency of information retrieval and text classification, and provides more accurate and comprehensive information services.
Smart Images
Figure CN121234937A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing technology, specifically relating to a knowledge association method based on semantic similarity and co-occurrence word matching. Background Technology
[0002] Semantic similarity is an indicator that measures the degree of semantic similarity between two texts. Compared with traditional literal matching-based similarity calculations, it can understand the meaning of the text more deeply, thus more accurately judging the similarity between texts. Co-occurrence word matching is an indicator that measures the frequency of co-occurrence of words in a text. Compared with traditional single word matching, it can consider the association between words and contextual information, providing richer text similarity information.
[0003] Semantic similarity and co-occurrence word matching can be applied in various fields such as information retrieval, text classification, and natural language processing. For example, in information retrieval, semantic similarity can improve the accuracy and relevance of search results, enabling users to find the information they need more quickly. In text classification, co-occurrence word matching can help identify the topic and category of a text, improving the accuracy and efficiency of classification.
[0004] In conclusion, semantic similarity and co-occurrence word matching, as important indicators for measuring text similarity, have broad application prospects.
[0005] However, due to limitations in application resources such as computational resources and the difficulty of implementing high-precision algorithms, semantic similarity and co-occurrence word matching have not been widely applied in all fields. To fully utilize the advantages of these two metrics while circumventing application resource constraints, researchers have proposed a knowledge association method, aiming to improve the computational efficiency and accuracy of semantic similarity and co-occurrence word matching by optimizing algorithms and models.
[0006] Based on their implementation methods, existing knowledge association methods can be broadly categorized into two types: 1) Methods based on traditional machine learning: These methods utilize manually designed feature engineering techniques to extract and classify text features, thereby achieving associations between texts. For example, text classification algorithms based on Support Vector Machines (SVMs) can extract and classify text features, thus achieving knowledge associations between texts. 2) Methods based on deep learning: These methods utilize deep learning networks to represent and learn text, thereby achieving more accurate text associations. For example, text matching models based on Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs) can perform deep representation and learning of text, achieving more accurate knowledge associations. Although these methods have achieved certain results in improving the accuracy and efficiency of knowledge association, their reliance on prior feature representations specific to a particular domain and the lack of cross-domain adaptive learning mechanisms result in poor generalization ability when facing cross-domain and cross-topic text data. Summary of the Invention
[0007] To overcome the limitations of existing technologies, this invention proposes an innovative knowledge association method. Based on semantic similarity and co-occurrence word matching, this method aims to achieve more efficient and accurate information retrieval through refined processing of massive amounts of text data. Specifically, this invention employs an advanced knowledge association algorithm that not only compares semantic similarity but also incorporates co-occurrence word matching calculations, thereby enabling a more comprehensive assessment of the relevance between texts. In terms of text semantic calculation, this invention utilizes the jieba word segmentation library to extract keywords from the text and selects the classic TF-IDF algorithm to effectively extract text features to measure their importance. Simultaneously, to more accurately calculate the semantic relevance between texts, this invention also employs cosine similarity as a metric. Furthermore, this invention pays special attention to the co-occurrence of words in the text, using it as an important indicator of the degree of text relevance. In summary, by comprehensively considering semantic similarity, co-occurrence word matching, and word co-occurrence, this invention can more comprehensively and accurately measure the degree of relevance between texts, thereby providing users with more precise and comprehensive information services.
[0008] The technical solution adopted by this invention to solve its technical problem is as follows: A knowledge association algorithm based on semantic similarity and co-occurrence word matching includes the following steps: 1. Data preprocessing; Extract the required knowledge data from the database, and clean and format this data to obtain the knowledge text to be queried. Z ={ Z 1 , Z 2 ,...,Z n}; 2. Semantic feature encoding Keyword extraction from cleaned knowledge text was performed using the jieba word segmentation library. Z k =extract_tags( Z The extracted keywords are used to construct the semantic feature representation of the text. The TF-IDF algorithm is used to calculate the feature of each keyword in the knowledge text, forming a high-dimensional semantic feature vector. V Z =TF-IDF( Z k ); 3. Knowledge search related queries The user enters a query statement Q The query vector is obtained by performing the operations in steps 1 and 2. V Q Then, by calculating the query vector V Q With each semantic feature vector V Z Using semantic similarity to find the most relevant knowledge T Z =Similarity max ( V Q , V Z ); 4. Knowledge-related queries Based on the word segmentation results of the user's query, the knowledge entries are segmented and deduplicated to obtain a candidate set of co-occurring words, which includes a list of keywords from the query. K Q and a list of keywords for knowledge entries K Z ;in, K Q = extract_tags( Q ), K Z = extract_tags( Z ) ; Calculate the keyword list of knowledge items K Z List of keywords in the user's query K Q The number of keywords appearing in the CCP's database is used to obtain the frequency of co-occurring keywords; The most relevant knowledge items are identified by ranking the co-occurring keywords based on their frequency. T Z =Normalization Score max ( K Q , K Z ), returns the ID of the associated knowledge entry.
[0009] Furthermore, in step 1, the knowledge ID and knowledge content are retrieved from the database table. The text in the knowledge content is segmented, stop words are removed, and necessary text normalization is performed to obtain the knowledge text to be queried. Z ={ Z 1 , Z 2 ,..., Z n}; Furthermore, text standardization refers to the uniformity of capitalization and the handling of special characters.
[0010] Furthermore, in step 2, the jieba word segmentation tool was used for precise word segmentation.
[0011] Furthermore, in step 3, the cosine similarity method is used to calculate semantic similarity.
[0012] The beneficial effects of this invention are as follows: Traditional text processing methods rely heavily on specific corpora and training datasets, often struggling to handle cross-domain or cross-topic text data, resulting in insufficient model generalization ability. This invention preprocesses the text and utilizes the jieba word segmentation library for keyword extraction, effectively filtering redundant words and retaining core semantic units. It also combines the classic TF-IDF algorithm to highlight high-frequency and highly discriminative feature words, thus more accurately measuring the importance of text features. Furthermore, to more accurately calculate the semantic relevance between texts, this invention employs cosine similarity as a metric, quantifying the matching degree of semantic distributions through a vector space model, avoiding the biases of traditional similarity calculations. In addition, this invention pays special attention to the co-occurrence of words in the text, using it as an important indicator of textual association, compensating for surface semantic association information that may be missed by single semantic analysis. These methodological designs ensure that this invention can uncover shared knowledge across domains or topics, improving the model's generalization ability and thus enhancing text processing effectiveness and user experience. Attached Figure Description
[0013] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0014] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0015] The present invention will be further described below with reference to the embodiments, which include, but are not limited to, the following embodiments.
[0016] This invention provides a knowledge association method based on semantic similarity and co-occurrence word matching, such as... Figure 1 As shown, the specific process is as follows: 1. Data Preprocessing In the data preprocessing stage, the required knowledge data is first extracted from the database and then cleaned and formatted. Specifically, knowledge IDs and knowledge content are retrieved from the database tables and processed according to specific field identifiers (such as whether it is key knowledge). Furthermore, the text within the knowledge content undergoes word segmentation, stop word removal, and necessary text normalization, such as case consistency and special character handling, to obtain the knowledge text to be queried. Z ={ Z 1 , Z 2 ,..., Z n}
[0017] In this invention, the jieba word segmentation tool is used for accurate word segmentation. Word segmentation is the foundation of Chinese text processing, which divides continuous text into independent lexical units.
[0018] 2. Semantic feature encoding In the semantic feature encoding stage, the jieba word segmentation library is used to extract keywords from the cleaned knowledge text. Z k =extract_tags( Z The extracted keywords are used to construct the semantic feature representation of the text. The TF-IDF algorithm is used to calculate the feature of each keyword in the knowledge text, forming a high-dimensional semantic feature vector. V Z =TF-IDF( Z k ).
[0019] 3. Knowledge search related queries During the knowledge search related query stage, the user enters a query statement. Q After performing the same operations as in steps 1 and 2, the query vector is obtained. V Q Then, by calculating the query vector V Q With each semantic feature vectorV Z Using semantic similarity to find the most relevant knowledge T Z =Similarity max ( V Q , V Z Semantic similarity is calculated using the cosine similarity method.
[0020] 4. Knowledge-related queries Co-occurrence word matching degree is calculated based on the number and frequency of keywords co-occurring in the query statement and knowledge entries.
[0021] Based on the word segmentation results of the user's query, the knowledge entries are segmented and deduplicated to obtain a candidate set of co-occurring words, which includes a list of keywords from the query. K Q and a list of keywords for knowledge entries K Z ;in, K Q = extract_tags( Q ), K Z = extract_tags( Z ) ; Calculate the keyword list of knowledge items K Z List of keywords in the user's query K Q The number of keywords appearing in the CCP's database is used to obtain the frequency of co-occurring keywords; The most relevant knowledge items are identified by ranking the co-occurring keywords based on their frequency. T Z =Normalization Score max ( K Q , K Z It returns the associated knowledge entry ID in JSON format to the user for further processing and use. Example 1: Knowledge association process based on user query "Hotel has complete facilities, is close to the subway station, and has good service".
[0022] This embodiment uses a user query for "hotel with complete facilities, close to the subway station, and good service" as an example to detail the specific implementation process of the knowledge association method based on semantic similarity and co-occurrence word matching described in this invention. Assume the knowledge base contains the following three knowledge entries: Knowledge 1: ID = 101, Content = "The hotel is fully equipped, the service staff have high quality, it takes about 8 minutes to walk to the subway station, and the breakfast variety is rich." Knowledge 2: ID = 102, Content = "The hotel rooms are newly equipped, provide free parking, 5 kilometers away from the high-speed railway station, and there is no subway station nearby." Knowledge 3: ID = 103, Content = "The hotel service is enthusiastic, there are many subway stations nearby, it can be reached in a 5-minute walk, and the facilities are perfect." Step 1: Data preprocessing Knowledge data loading and cleaning: Retrieve the knowledge ID and knowledge content from the database table knowledge_data to obtain the set of knowledge to be processed: Z ={ Z 1, Z 2, Z 3}, where Z 1 corresponds to Knowledge 1, where Z 2 corresponds to Knowledge 2, where Z 3 corresponds to Knowledge 3.
[0023] Text tokenization and normalization: Use the jieba tokenization library to tokenize each knowledge content, remove stop words (such as "of, about, many"), and obtain the token sequence: Z 1: Tokenization result: ['Hotel facilities', 'Complete', 'Service', 'Staff quality', 'High', 'Walk', 'Subway station', 'About', 'Minute', 'Breakfast', 'Variety', 'Rich'] Z 2: Tokenization result: ['Hotel', 'Room', 'Facilities', 'New', 'Provide', 'Free', 'Parking', 'High-speed railway station', 'Kilometer', 'Surroundings', 'Subway station'] Z 3: Tokenization result: ['Hotel', 'Service', 'Enthusiastic', 'Surroundings', 'Subway station', 'Many', 'Walk', 'Minute', 'Reach', 'Facilities', 'Perfect'] Step 2: Semantic feature encoding Keyword extraction: Use the extract_tags of the jieba tokenization library to extract the keywords (Top5) of each knowledge: Z k1 = extract_tags( Z 1) = ['Hotel facilities', 'Staff quality', 'Subway station', 'Breakfast', 'Walk'] Zk2 = extract_tags( Z 2) = ['Subway station', 'High-speed rail', 'Parking', 'Surrounding area', 'Hotel'] Z k3 = extract_tags( Z 3) = ['Subway station', 'Walking distance', 'Surrounding area', 'Hotel', 'Facilities'] TF-IDF Vector Construction: The TF-IDF Vectorizer model is used to calculate keyword features and construct a vocabulary. V = ['Personnel quality', 'Parking', 'Surrounding area', 'Subway station', 'Breakfast', 'Walking distance', 'Facilities', 'Hotel', 'Hotel facilities', 'High-speed rail'], and form a semantic feature vector (which needs to be normalized): V Z1 = TF-IDF( Z k1 = [0.5, 0, 0, 0.3, 0.5, 0.38, 0, 0, 0.5, 0] V Z2 = TF-IDF( Z k2 = [0, 0.53, 0.41, 0.32, 0, 0, 0, 0.41, 0, 0.53] V Z3 = TF-IDF( Z k3 = [0, 0, 0.43, 0.34, 0, 0.43, 0.57, 0.43, 0, 0] Step 3: Knowledge Search Related Queries User query preprocessing: User query Q =“The hotel has complete facilities, is close to the subway station, and offers good service”, after processing in steps 1-2: Word segmentation results: Q =['Hotel facilities', 'Fully equipped', 'Subway station', 'Close to', 'Services'] Keyword extraction: Q k = extract_tags( Q = ['Hotel facilities', 'Subway station', 'Full range', 'Services'] Query vector: VQ = TF-IDF( Q k = [0, 0, 0, 0.51, 0, 0, 0, 0, 0.86, 0] Semantic similarity calculation: using cosine similarity. V Q Similarity with each knowledge vector: Similarity( V Q , V Z1 ) = ( V Q · V Z1 ) / (|| V Q ||·|| V Z1 ||) = 0.586 Similarity( V Q , V Z2 ) = 0.160 Similarity( V Q , V Z2 ) = 0.171 Top search results: Sorted by similarity, taking top_search = 2, we get the search-related knowledge IDs: search_id = "101, 103" Step 4: Knowledge-related query Co-occurrence word extraction: Deduplication is performed on user queries and knowledge entries to obtain a candidate set of co-occurrence words. ●Q Co-occurrence word candidate results: ['hotel facilities', 'complete', 'subway station', 'near', 'services'] ●Z 1: Co-occurrence candidate words: ['hotel facilities', 'complete', 'service', 'staff quality', 'high', 'walking', 'subway station', 'about', 'minutes', 'breakfast', 'variety', 'abundant'] ●Z 2: Co-occurrence word candidate results: ['hotel', 'room', 'facilities', 'new', 'offers', 'free', 'parking', 'high-speed rail', 'station', 'km', 'surrounding', 'subway station'] ●Z3: Co-occurrence candidate words: ['hotel', 'service', 'enthusiasm', 'surrounding area', 'subway station', 'numerous', 'walking', 'minutes', 'reach', 'facilities', 'complete'] Query Keyword List K Q = extract_tags ( Q = ['Hotel facilities', 'Subway station', 'Full range', 'Services'] Knowledge 1 Keyword List K Z1 = extract_tags ( Z 1) = ['Hotel facilities', 'Staff quality', 'Subway station', 'Breakfast', 'Walking distance', 'Complete', 'Minutes', 'Variety', 'Abundant', 'Service'] Knowledge 2 Keyword List K Z2 = extract_tags ( Z 2) = ['Subway station', 'High-speed rail', 'Parking', 'Nearby', 'Hotel', 'Free', 'Rooms', 'Facilities', 'km', 'Available'] Knowledge 3 Keyword List K Z3 = extract_tags ( Z 3) = ['Subway station', 'Walking distance', 'Surrounding area', 'Hotel', 'Facilities', 'Minutes', 'Enthusiastic', 'Well-equipped', 'Numerous', 'Services'] Co-occurrence word matching degree calculation: Knowledge 1: Co-occurring words are K Q ∩ K Z1 = ['Hotel facilities', 'Complete', 'Services', 'Subway station'], the total co-occurrence frequency is 4, the total number of keywords in Knowledge 1 is 10, the normalized score = 4 / 10 = 0.4 Knowledge 2: Co-occurring words are K Q ∩ K Z2 =['Subway Station'], the total co-occurrence frequency is 1, the total number of keywords in Knowledge 2 is 10, and the normalized score is 1 / 10 = 0.1 Knowledge 3: Co-occurring words are K Q ∩K Z3 =['Service', 'Subway Station'], the total co-occurrence frequency is 2, the total number of keywords in Knowledge 3 is 10, and the normalized score is 3 / 10 = 0.3 Top Connection Results: Matching scores of co-occurring words sorted by normalized score [0.4, 0.1, 0.3]. Taking top_connection=2, the resulting connection knowledge IDs are: connection_id = “101, 103”. The content described herein represents specific implementation methods, intended to facilitate understanding and practice of this application by those skilled in the art. Modifications to these embodiments by those skilled in the art will be readily apparent, and the general principles set forth herein can be applied in other embodiments without departing from the spirit and scope of this application. Therefore, this application should not be limited to the embodiments presented herein, but should encompass the broadest scope consistent with the principles and innovative features proposed herein.
Claims
1. A knowledge association method based on semantic similarity and co-occurrence word matching degree, characterized in that, Including the following steps: 1、Data preprocessing Extract the required knowledge data from the database, and clean and format these data to obtain the knowledge text to be queried Z = {Z1, Z2,..., Z n}; 2、Semantic feature coding jieba library for cleaning the knowledge text for keyword extraction Z k = extract_tags(Z), the extracted keywords are used to construct the semantic feature representation of the text, and the TF-IDF algorithm is used to calculate the features of each keyword in the knowledge text, forming a high-dimensional semantic feature vector V Z = TF-IDF(Z k ); 3、Knowledge search related query The user inputs a query statement Q, and performs the operations of the first and second steps to obtain a query vector V Q The query vector V Q is then compared with each semantic feature vector V Z to find the most relevant knowledge T Z = Similarity max (V Q , V Z ); 4、Knowledge association query According to the word segmentation result of the user query statement, the knowledge entries are segmented and deduplicated to obtain a co-occurrence word candidate set, the co-occurrence word candidate set including a keyword list K of the query statement and a keyword list K of the knowledge entries Q . Z ; wherein K Q = extract_tags(Q), K Z = extract_tags(Z); Calculate the keyword list K of knowledge items Z Keyword list K in the user query statement Q The number of keywords appearing in the CCP's database is used to obtain the frequency of co-occurring keywords; Sort by the frequency of co-occurrence keywords to find the most relevant knowledge entry T Z = NormalizationScore max (K Q , K Z ), return the associated knowledge entry ID.
2. The knowledge association method based on semantic similarity and co-occurrence word matching degree according to claim 1, characterized in that, In the first step, the knowledge ID and knowledge content are retrieved from the database table, the text in the knowledge content is segmented, stop words are removed, and necessary text normalization is performed to obtain the knowledge text to be queried Z = {Z1, Z2,..., ZN}. n} 3. The knowledge association method based on semantic similarity and co-occurrence word matching degree according to claim 2, characterized in that, Text normalization refers to the uniformity of capitalization and special character processing.
4. The knowledge association method based on semantic similarity and co-occurrence word matching degree according to claim 3, characterized in that, In step 1, the jieba segmentation tool is used for accurate segmentation.
5. The knowledge association method based on semantic similarity and co-occurrence word matching degree according to claim 4, characterized in that, In step 3, the cosine similarity method is used to calculate semantic similarity.