Method for disambiguating knowledge points in text knowledge base
By combining traditional methods and deep learning knowledge point disambiguation methods in text knowledge bases, using community detection and deep learning to calculate similarity scores, the problem of insufficient semantic understanding is solved, the accuracy and efficiency of disambiguation is improved, and the modern Information processing requirements.
Patent Information
- Application Number
- CN202411031412.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
AI Technical Summary
The existing technology has problems such as insufficient semantic understanding and inaccurate ambiguity in knowledge point disambiguation. In addition, deep learning-based methods require large training data and high computing resource consumption, making it difficult to meet the needs of modern information processing.
A knowledge point disambiguation method in the text knowledge base that combines traditional methods and deep learning is adopted to obtain a collection of candidate knowledge points through community detection and deep learning, and the similarity score is calculated from multiple dimensions such as semantics and context to determine the best disambiguation result.
It improves the accuracy, reliability, completeness and efficiency of knowledge points disambiguation, avoids the problem of disambiguation caused by insufficient semantic information understanding, and meets the needs of modern information processing.
Smart Images

Figure CN119005314A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of text disambiguation, and in particular relates to a method for disambiguating knowledge points in a text knowledge base. Background Art
[0002] With the rapid development of the Internet and digital technology, people can easily obtain a large amount of text information, including but not limited to news articles, academic papers, social media content, etc. However, this text information is usually highly ambiguous and complex, making it a challenging task to accurately extract knowledge points from it.
[0003] In the field of natural language processing (NLP), knowledge point disambiguation is a long-standing problem. For example, when "apple" is mentioned in an article, it may refer to the fruit, or it may refer to a technology company, or even to a certain brand of laptop. Therefore, in order to accurately understand and utilize text information, it is necessary to disambiguate knowledge points, that is, to determine the exact meaning of the knowledge point in a specific context.
[0004] Traditional knowledge point disambiguation methods are mainly based on shallow features such as word frequency and part-of-speech tagging, and have problems such as insufficient understanding of context and inaccurate ambiguity judgment. With the development of deep learning and semantic representation technology, neural network-based methods have begun to attract attention. They can use rich semantic information to disambiguate knowledge points, but there are still challenges such as large training data requirements and high computing resource consumption.
[0005] Therefore, it is necessary to propose a method for knowledge point disambiguation in a text knowledge base that combines traditional methods and deep learning technology to improve disambiguation accuracy and efficiency and meet the needs of modern information processing. This method should be able to make full use of the rich semantic information in the text knowledge base and accurately disambiguate knowledge points in combination with the context, providing reliable knowledge services for various NLP applications. Summary of the invention
[0006] The purpose of the present invention is to provide a method for disambiguating knowledge points in a text knowledge base, by obtaining knowledge points to be disambiguated, using community detection and deep learning to obtain a set of candidate knowledge points, calculating similarity scores from multiple dimensions such as semantics and context, and determining the best disambiguation result. Compared with traditional methods, this method avoids the problem of inaccurate disambiguation caused by insufficient semantic understanding, and improves accuracy, reliability, completeness and efficiency.
[0007] The technical solution adopted by the present invention is as follows:
[0008] A method for disambiguating knowledge points in a text knowledge base comprises the following steps:
[0009] Obtain text content, extract knowledge points to be disambiguated from the text content through the knowledge base, and preprocess the text content;
[0010] Perform global statistics on the knowledge points to be disambiguated, convert the knowledge points to be disambiguated into vectors to be verified and store them in the vector database;
[0011] Generate several candidate knowledge points according to the knowledge points to be disambiguated;
[0012] Calculate the similarity between the knowledge point to be disambiguated and the candidate knowledge point through the disambiguation model, and obtain the comprehensive similarity score of each candidate knowledge point;
[0013] The disambiguation result is determined based on the similarity score and the disambiguation result is responded to the front end.
[0014] In a preferred embodiment, the preprocessing of the text content comprises the following steps:
[0015] Perform data cleaning on the text content and standardize the data format to obtain cleaned data;
[0016] Perform word segmentation on the cleaned data to split the text content into several vocabulary units;
[0017] Tag the vocabulary units with parts of speech and identify the part of speech of each vocabulary unit;
[0018] According to the results of part-of-speech tagging, the vocabulary units are filtered and screened, stop words are removed, and preprocessed data is obtained.
[0019] In a preferred solution, converting the knowledge points to be disambiguated into vectors to be verified and storing them in a vector database comprises the following steps:
[0020] Construct mapping rules;
[0021] Extract key features from preprocessed data, the key features including: entity features and concept features;
[0022] The key features are quantitatively mapped according to the mapping rules, and the mapping results are output as vector data to be disambiguated and stored in the vector database.
[0023] In a preferred embodiment, generating a plurality of candidate knowledge points according to the knowledge points to be disambiguated comprises the following steps:
[0024] Extract the relationship between key features based on the preprocessed data, and construct a relationship network consisting of multiple network nodes and edges, where each network node represents a key feature and each edge represents the relationship between adjacent key features;
[0025] Identify communities in a relationship network, where each community consists of multiple closely connected nodes, and multiple nodes in the same community have similar characteristics or attributes;
[0026] Cluster the key features with similar semantic relationships in the knowledge base into the same community to obtain the first set;
[0027] The elements in the first candidate set are analyzed and screened according to the context of the knowledge point to be disambiguated to obtain a candidate knowledge point set.
[0028] In a preferred solution, calculating the similarity between the knowledge point to be disambiguated and the candidate knowledge point through a disambiguation model and obtaining a comprehensive similarity score for each candidate knowledge point includes the following steps:
[0029] Constructing a disambiguation model, which at least includes: a semantic calculation model, a context calculation model, and a cosine similarity calculation model;
[0030] Construct multiple scoring tables, where the multiple scoring tables correspond to the multiple calculation models one by one, each scoring table is set with multiple scoring intervals, and each scoring interval corresponds to a score;
[0031] Input each element in the candidate knowledge point set into multiple calculation models in turn, and obtain multiple scores for each candidate knowledge point through the multiple calculation models;
[0032] Input multiple scores of each candidate knowledge point into the similarity calculation formula to calculate the comprehensive similarity score of each candidate fulcrum. The similarity calculation formula is: F = f 1 +f 2 +f 3 , where F represents the comprehensive similarity score, f 1 represents the semantic score, f 2 represents the context score, f 3 Represents the cosine similarity calculation model score.
[0033] In a preferred solution, the step of responding the disambiguation result to the front end comprises the following steps:
[0034] Mark the candidate knowledge point with the highest comprehensive similarity score as the disambiguation result, and format the disambiguation result to meet the front-end display requirements;
[0035] Encapsulate it together with the original text content into response data;
[0036] Send the response data to the front end.
[0037] In a preferred solution, the step before inputting each element in the candidate set into multiple computing models in sequence also includes: converting the candidate knowledge point into candidate vector data through the knowledge base.
[0038] A knowledge point disambiguation system in a text knowledge base, applicable to any of the above-mentioned methods for disambiguating knowledge points in a text knowledge base, comprising:
[0039] An acquisition module, which is used to acquire text content and extract knowledge points to be disambiguated;
[0040] A processing module, the processing module is used to pre-process the text content and convert the knowledge points to be disambiguated into vector data;
[0041] An evaluation module, the evaluation module is used to calculate the similarity scores of candidate knowledge points;
[0042] A feedback collection module, which is used to output the disambiguation result and collect user feedback, and transmit the feedback to the processing module;
[0043] A storage module is used to store data and related information.
[0044] The technical effects achieved by the present invention are:
[0045] The present invention obtains knowledge points to be disambiguated in text content, obtains a set of candidate knowledge points based on community detection technology and deep learning methods, and determines the best disambiguation result by calculating the comprehensive similarity score of each candidate knowledge point from multiple dimensions such as semantics and context. Compared with traditional disambiguation methods, the present invention avoids the situation where insufficient understanding of semantic information makes it difficult to accurately capture the meaning of knowledge points, resulting in inaccurate disambiguation results, and improves the accuracy, reliability, completeness and disambiguation efficiency of knowledge point disambiguation. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of the knowledge point disambiguation method of the present invention;
[0047] Figure 2 It is a schematic diagram of the framework of the knowledge point disambiguation system of the present invention. DETAILED DESCRIPTION
[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0050] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" that appears in different places in this specification does not refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.
[0051] Please see attached Figure 1 As shown, the first embodiment of the present invention provides a method for disambiguating knowledge points in a text knowledge base, including the following steps:
[0052] S1: Obtain text content, extract knowledge points to be disambiguated from the text content through the knowledge base, and preprocess the text content;
[0053] S2: Perform global statistics on the knowledge points to be disambiguated, convert the knowledge points to be disambiguated into vectors to be verified and store them in the vector database;
[0054] S3: Generate several candidate knowledge points based on the knowledge points to be disambiguated;
[0055] S4: Calculate the similarity between the knowledge point to be disambiguated and the candidate knowledge point through the disambiguation model, and obtain the comprehensive similarity score of each candidate knowledge point;
[0056] S5: Determine the disambiguation result based on the similarity score, respond to the disambiguation result to the front end, and obtain user feedback.
[0057] In this embodiment, in the field of natural language processing, knowledge point disambiguation has always been a key issue that needs to be solved urgently. Although traditional disambiguation methods rely on shallow features such as word frequency and part-of-speech tagging to a certain extent, they are powerless in terms of in-depth understanding of context and accuracy of ambiguous judgment. With the rapid development of deep learning and semantic representation technology, neural network-based methods use rich semantic information to significantly improve the effect of knowledge point disambiguation, but they need to face challenges such as huge training data requirements and high computing resource consumption. In this embodiment, through a pre-built knowledge base, the required text content is first obtained, and the knowledge base is used to accurately extract the knowledge points to be disambiguated in the text. On this basis, necessary preprocessing work is performed on the text, such as noise removal, word segmentation, etc. Subsequently, a global statistical analysis is performed on the extracted knowledge points to be disambiguated so as to convert them into vector forms to be verified. These vectors will be stored in a vector database in an orderly manner for subsequent query and application. On this basis, according to these knowledge points to be disambiguated, the system will generate several potential candidate knowledge points to ensure that each candidate knowledge point can provide a possibility for the subsequent disambiguation process, thereby improving the accuracy and efficiency of disambiguation. After that, the pre-built disambiguation model is used to accurately calculate the similarity between the knowledge points to be disambiguated and the candidate knowledge points. According to the calculation results, the system will give a comprehensive similarity score for each candidate knowledge point. This score will serve as an important basis for determining the disambiguation result. According to the specific value of the similarity score, the system will accurately determine the disambiguation result. Finally, the result will be responded to the front end in a standard format for users to review and use. In this way, the system can efficiently process the knowledge points to be disambiguated in the text and provide users with accurate knowledge points and information. At the same time, with the help of the pre-built knowledge base and disambiguation model, the system has strong adaptability and scalability, and can cope with various complex disambiguation scenarios, thereby improving the user experience and satisfaction, and helping them to obtain more accurate and comprehensive knowledge in the process of learning and application.
[0058] In a preferred embodiment, the preprocessing of the text content comprises the following steps:
[0059] Perform data cleaning on the text content, remove irrelevant information and noise, and standardize the data format to obtain clean data;
[0060] Perform word segmentation on the cleaned data to split the text content into several vocabulary units;
[0061] Tag the vocabulary units with parts of speech and identify the part of speech of each vocabulary unit;
[0062] According to the results of part-of-speech tagging, the vocabulary units are filtered and screened, stop words are removed, and preprocessed data is obtained.
[0063] In this embodiment, at the initial stage of the data processing flow, the text content is first subjected to data cleaning to accurately identify and remove redundant information and noise existing in the text content (such as HTML tags, special symbols, etc., correcting spelling mistakes and other redundant information and irrelevant characters), and at the same time, the format of the data is normalized (such as standardizing character encoding, unifying English uppercase and lowercase letters, unifying traditional and simplified Chinese characters, etc.) to ensure that the obtained data is clean and conforms to a unified standard, ensuring the accuracy and effectiveness of subsequent data processing work. After the data cleaning work is completed, more detailed word segmentation processing is performed on the data (word segmentation is the process of splitting continuous text content into individual lexical units), decomposing the text content into smaller, easier-to-understand and process units for subsequent data processing and data analysis. Then, the part-of-speech of each lexical unit is identified to identify the part-of-speech of each lexical unit (such as nouns, verbs, adjectives, etc.) so that each lexical unit can be accurately classified, ensuring the accuracy and integrity of data processing. After part-of-speech tagging, based on the tagging results, the lexical units are filtered and screened to identify and remove stop words (stop words refer to words that appear frequently in the text but do not contribute substantially to the expression of meaning, such as "de", "shi", "zai", etc.), improving the purity, processing efficiency, accuracy, reliability and integrity of the data.
[0064] In a preferred embodiment, the conversion of the knowledge points to be disambiguated into vectors to be verified and storing them in the vector database includes the following steps:
[0065] Construct mapping rules;
[0066] Extract key features from the preprocessed data, and the key features include: entity features and concept features;
[0067] Quantitatively map the key features according to the mapping rules, and output the mapping results as vectors of data to be disambiguated and store them in the vector database.
[0068] It should be noted that the mapping rules can be based on linguistic or statistical principles, such as term frequency-inverse document frequency (TF-IDF), Word2Vec, BERT, etc. These technologies can help us capture the semantic information in the text and transform it into vector form.
[0069] In this embodiment, mapping rules are pre-constructed, and the following key features are extracted from the pre-processed data: entity features (such as names of people, places, and organizations, etc.) and concept features (such as topics, categories, relationships, etc.). The semantic information in the text is understood through the entity features and concept features, and the key features in the text are converted into vector data to be disambiguated through the mapping rules (a vector is a mathematical structure that represents the characteristics of data by its length and direction). In text processing, each word or phrase can be mapped to a unique vector (for example, a frequently occurring entity may get a longer vector, while a rare entity may get a shorter vector). These vectors can be compared and operated in a high-dimensional space. In this way, we can convert the semantic information in the text into a numerical form that can be understood by computers. After the vector data is converted, the computer can process and understand the data more efficiently (a vector is a mathematical structure that represents the characteristics of the data by its length and direction). In text processing, each word or phrase can be mapped to a unique vector, which can be compared and operated in a high-dimensional space. In this way, the semantic information in the text is converted into a numerical form that the computer can understand and stored in the database, so as to facilitate the subsequent rapid search for other knowledge points similar to the knowledge point to be disambiguated, so as to perform accurate disambiguation operations. In the process of knowledge point disambiguation, the conversion of the cleaned data into vector data not only improves the efficiency and accuracy of data processing, but also enhances the generalization and scalability of the model. Through this conversion, it can ensure that the machine learning model can better understand and utilize the hidden information in the data, thereby improving the overall performance.
[0070] In a preferred embodiment, the step of generating a plurality of candidate knowledge points according to the knowledge points to be disambiguated comprises the following steps:
[0071] Extract the relationship between key features based on the preprocessed data, and construct a relationship network consisting of multiple network nodes and edges, where each network node represents a key feature and each edge represents the relationship between adjacent key features;
[0072] Identify communities in a relationship network using community detection algorithms and deep learning methods, where each community consists of multiple closely connected nodes, and multiple nodes in the same community have similar characteristics or attributes;
[0073] Cluster the key features with similar semantic relationships in the knowledge base into the same community to obtain the first set;
[0074] The elements in the first candidate set are analyzed and screened according to the context of the knowledge point to be disambiguated to obtain a candidate knowledge point set.
[0075] It should be noted that the use of community detection technology and deep learning methods can not only more accurately identify knowledge points in the text and cluster them into corresponding topic groups to effectively reduce the ambiguity of knowledge points and improve the accuracy of disambiguation, but also effectively reduce the need for manual intervention, improve the degree of automation of knowledge point disambiguation, quickly process large-scale text data, and generate accurate disambiguation results, thereby improving the efficiency of disambiguation. At the same time, it also has strong versatility, applicability and good complex text processing capabilities, and can effectively extract a set of high-quality candidate knowledge points related to the knowledge points to be disambiguated from the huge text knowledge base based on contextual information.
[0076] In this implementation, based on the preprocessed data, the relationship between key features is obtained by mining the text content and extracting features, and a relationship network containing multiple network nodes and edges is constructed, wherein each network node clearly represents a key feature, and each edge represents the specific relationship between adjacent key features. By applying advanced community detection algorithms and deep learning techniques, various communities in the relationship network are identified, and based on the information in the knowledge base, key features with similar semantic relationships are clustered into the same community to ensure the accuracy and rationality of the clustering results, and a set of key features that are potentially associated with the knowledge points to be disambiguated is obtained and recorded as the first set. Based on the specific context of the knowledge points to be disambiguated, the elements in the first set are deeply analyzed and screened to obtain a set of candidate knowledge points. In this implementation, the specific context and background of the knowledge points to be disambiguated are combined to ensure the pertinence and accuracy of the set of candidate knowledge points.
[0077] In a preferred embodiment, the similarity between the knowledge point to be disambiguated and the candidate knowledge point is calculated by using a disambiguation model to obtain a comprehensive similarity score for each candidate knowledge point, including the following steps:
[0078] Constructing a disambiguation model, which at least includes: a semantic calculation model, a context calculation model, and a cosine similarity calculation model;
[0079] Construct multiple scoring tables, where the multiple scoring tables correspond to the multiple calculation models one by one, each scoring table is set with multiple scoring intervals, and each scoring interval corresponds to a score;
[0080] Input each element in the candidate knowledge point set into multiple calculation models in turn, and obtain multiple scores for each candidate knowledge point through the multiple calculation models, wherein the multiple scores include semantic scores, context scores, and cosine similarity scores;
[0081] Input multiple scores of each candidate knowledge point into the similarity calculation formula to calculate the comprehensive similarity score of each candidate fulcrum. The similarity calculation formula is: F = f 1 +f 2 +f3 , where F represents the comprehensive similarity score, f 1 represents the semantic score, f 2 represents the context score, f 3 Represents the cosine similarity calculation model score.
[0082] In this implementation, in order to achieve accurate information understanding and processing, a disambiguation model is constructed that includes a semantic calculation model, a context calculation model, and a cosine similarity calculation model. The semantic calculation model is responsible for parsing and calculating the deep meaning of the text, and maps words or phrases to a high-dimensional vector space through deep learning technology to capture their semantic information. The context calculation model focuses on the relationship between the knowledge points to be disambiguated and the candidate knowledge points in the text context, and uses context information to enhance the accuracy of disambiguation. The cosine similarity calculation model measures the similarity between two vectors by calculating the cosine value of the angle between them. The closer the cosine value is to 1, the closer the directions of the two vectors are. , that is, the higher the similarity, the higher the score. By inputting each element in the candidate knowledge point set into the semantic calculation model, the context calculation model and the cosine similarity calculation model in turn, and calculating the semantic score, context score and cosine similarity score of each candidate knowledge point, these scores comprehensively reflect the similarity between the candidate knowledge point and the knowledge point to be disambiguated in semantics, context and vector space. Substitute the above multiple scores into the formula to calculate the comprehensive similarity score of each candidate knowledge point. Through the comprehensive similarity score, a quantitative indicator of the similarity between the knowledge point to be disambiguated and each candidate knowledge point is established, which helps us find the most relevant knowledge point in the required information and improves the accuracy of knowledge point similarity calculation.
[0083] It should be noted that when constructing the scoring table and similarity calculation formula, we can adjust and optimize it according to the specific text processing needs and scenarios. For example, if there are a large number of professional terms or domain knowledge in the text, we can increase the consideration of professional terms and domain knowledge to improve the accuracy of the semantic calculation model; if there are a large number of synonyms or near-synonyms in the text, we can increase the consideration of context and use context information to distinguish the subtle differences between these synonyms or near-synonyms; if there are a large number of short texts or fragmented information in the text, we can increase the consideration of cosine similarity and use the position relationship in the vector space to capture the relationship between these short texts or fragmented information. Through this multi-model, multi-scoring disambiguation method, we can more comprehensively consider the various information in the text and improve the accuracy of disambiguation. At the same time, we can also flexibly configure and adjust according to the specific text processing needs and scenarios to meet different application needs.
[0084] In a preferred embodiment, the step of responding the disambiguation result to the front end includes the following steps:
[0085] Mark the candidate knowledge point with the highest comprehensive similarity score as the disambiguation result, and format the disambiguation result to meet the front-end display requirements;
[0086] Encapsulate it together with the original text content into response data;
[0087] Send the response data to the front end.
[0088] In this implementation, the candidate knowledge point with the highest comprehensive similarity score is determined as the final disambiguation result, and then the disambiguation result is formatted to ensure that it can be correctly displayed on the front-end interface to meet the user's reading needs (formatting may include adjustments to fonts, colors, sizes, alignments, etc. to ensure that the disambiguation results are clear, eye-catching and easy to understand on the page), and then the processed disambiguation results are packaged together with the original text content into response data. Through packaging, it is ensured that the disambiguation results and the original text content will not be lost or disordered during transmission, and the integrity and consistency of the data are maintained for display on the front end. In this way, we can not only achieve accurate disambiguation of knowledge points, but also display the disambiguation results to users in an intuitive and easy-to-understand manner, thereby improving the user experience. At the same time, we can also continuously optimize our disambiguation models and algorithms based on user feedback and needs to improve the accuracy and efficiency of disambiguation.
[0089] In a preferred embodiment, the step before inputting each element in the candidate set into multiple calculation models in sequence also includes: converting the candidate knowledge point into candidate vector data through the knowledge base.
[0090] In this embodiment, by converting candidate knowledge points into candidate vector data and converting them into numerical form that can be understood by a computer, after being converted into vector data, the computer can process and understand these data more efficiently.
[0091] See also Figure 2 As shown, a knowledge point disambiguation system in a text knowledge base is applicable to any of the above-mentioned methods for disambiguating knowledge points in a text knowledge base, comprising:
[0092] An acquisition module, which is used to acquire text content and extract knowledge points to be disambiguated;
[0093] A processing module, the processing module is used to pre-process the text content and convert the knowledge points to be disambiguated into vector data;
[0094] An evaluation module, the evaluation module is used to calculate the similarity scores of candidate knowledge points;
[0095] A feedback collection module, which is used to output the disambiguation result and collect user feedback, and transmit the feedback to the processing module;
[0096] A storage module is used to store data and related information.
[0097] In this embodiment, user feedback on the disambiguation results is collected through a feedback collection module, and the disambiguation model is optimized accordingly to improve the disambiguation accuracy. User feedback can be collected through the front-end interface, including user satisfaction evaluation of the disambiguation results, suggested improvement points and other information. The collected user feedback will be used as part of the disambiguation model training to continuously iterate and improve the disambiguation algorithm, thereby providing a more accurate disambiguation service that is closer to user needs.
[0098] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.
Claims
1. A method for disambiguating knowledge points in a text knowledge base, characterized by: The following steps are involved: Obtain text content, extract knowledge points to be disambiguated from the text content through the knowledge base, and preprocess the text content; Perform global statistics on the knowledge points to be disambiguated, convert the knowledge points to be disambiguated into vectors to be verified and store them in the vector database; Generate several candidate knowledge points according to the knowledge points to be disambiguated; Calculate the similarity between the knowledge point to be disambiguated and the candidate knowledge point through the disambiguation model, and obtain the comprehensive similarity score of each candidate knowledge point; The disambiguation result is determined based on the similarity score and the disambiguation result is responded to the front end.
2. A method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The preprocessing of the text content comprises the following steps: Perform data cleaning on the text content and standardize the data format to obtain cleaned data; Perform word segmentation on the cleaned data to split the text content into several vocabulary units; Tag the vocabulary units with parts of speech and identify the part of speech of each vocabulary unit; According to the results of part-of-speech tagging, the vocabulary units are filtered and screened, stop words are removed, and preprocessed data is obtained.
3. A method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The step of converting the knowledge points to be disambiguated into vectors to be verified and storing them in a vector database comprises the following steps: Construct mapping rules; Extract key features from preprocessed data, the key features including: entity features and concept features; The key features are quantitatively mapped according to the mapping rules, and the mapping results are output as vector data to be disambiguated and stored in the vector database.
4. A method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The step of generating a plurality of candidate knowledge points according to the knowledge points to be disambiguated comprises the following steps: Extract the relationship between key features based on the preprocessed data, and construct a relationship network consisting of multiple network nodes and edges, where each network node represents a key feature and each edge represents the relationship between adjacent key features; Identify communities in a relationship network, where each community consists of multiple closely connected nodes, and multiple nodes in the same community have similar characteristics or attributes; Cluster the key features with similar semantic relationships in the knowledge base into the same community to obtain the first set; The elements in the first candidate set are analyzed and screened according to the context of the knowledge point to be disambiguated to obtain a candidate knowledge point set.
5. The method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The similarity between the knowledge point to be disambiguated and the candidate knowledge point is calculated by the disambiguation model to obtain a comprehensive similarity score for each candidate knowledge point, including the following steps: Constructing a disambiguation model, which at least includes: a semantic calculation model, a context calculation model, and a cosine similarity calculation model; Construct multiple scoring tables, each of which corresponds to multiple calculation models. Each scoring table is provided with multiple scoring intervals, and each scoring interval corresponds to a score. Input each element in the candidate knowledge point set into multiple calculation models in turn, and obtain multiple scores for each candidate knowledge point through the multiple calculation models; The multiple scores of each candidate knowledge point are input into the similarity calculation formula to calculate the comprehensive similarity score of each candidate fulcrum. The similarity calculation formula is: F=f1+f2+f3, where F represents the comprehensive similarity score, f1 represents the semantic score, f2 represents the context score, and f3 represents the cosine similarity calculation model score.
6. The method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The step of responding the disambiguation result to the front end includes the following steps: Mark the candidate knowledge point with the highest comprehensive similarity score as the disambiguation result, and format the disambiguation result to meet the front-end display requirements; Encapsulate it together with the original text content into response data; Send the response data to the front end.
7. The method for disambiguating knowledge points in a text knowledge base according to claim 1, characterized in that: The step before inputting each element in the candidate set into multiple computing models in sequence also includes: converting the candidate knowledge points into candidate vector data through the knowledge base.
8. A knowledge point disambiguation system in a text knowledge base, applicable to a knowledge point disambiguation method in a text knowledge base according to any one of claims 1 to 7, comprising: An acquisition module, which is used to acquire text content and extract knowledge points to be disambiguated; A processing module, the processing module is used to pre-process the text content and convert the knowledge points to be disambiguated into vector data; An evaluation module, the evaluation module is used to calculate the similarity scores of candidate knowledge points; A feedback collection module, which is used to output the disambiguation result and collect user feedback, and transmit the feedback to the processing module; A storage module is used to store data and related information.
Citation Information
Cited By
Bill and call bill database interaction method, system, equipment or product based on question library enhanced verification
CN120353819A
Bill and call record database interaction method, system, device or product based on question library enhanced verification
CN120353819B
