Hotspot problem-based knowledge base incremental updating method and device, equipment and medium

Abstract

The application discloses a kind of knowledge base incremental updating method, device, equipment and medium based on hot spot problem, to solve the problem of RAG knowledge base update lag in relevant technology.Therein method includes: rewriting processing to the question input by user;If AI model judges according to the rewritten question and the knowledge in knowledge base cannot answer question, then store the rewritten question;When meeting preset trigger condition, the rewritten question stored is divided into question cluster using hierarchical clustering and K-means clustering, and target question cluster meeting preset screening condition is selected, the question in target question cluster is summarized using AI model, and the hot spot question corresponding to each target question cluster is obtained;Through web crawler, obtain the web page content related to each hot spot question, based on the semantic correlation degree of hot spot question and each web page content related, extract the text content with highest semantic correlation degree as the related knowledge of hot spot question;Related knowledge is stored to knowledge base.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method, apparatus, device, and medium for incremental updating of a knowledge base based on hot topics. Background Technology

[0002] In today's information-saturated era, with data growing exponentially, how to quickly and accurately obtain the information needed has become a pressing issue. Traditional search technologies, primarily based on keyword matching, struggle to accurately understand user query intent, often resulting in imprecise search results riddled with irrelevant information, increasing the time cost for users to filter and summarize information.

[0003] With the development of artificial intelligence technology, especially the advancements in natural language processing and deep learning, generative AI models have gradually emerged. These models can generate natural language text based on input, bringing new ideas to information retrieval. However, purely generative AI models suffer from the "illusion" problem in practical applications, meaning that the generated content may not match the facts of the real world. In scenarios requiring highly reliable information, such as medical diagnosis and legal interpretation, this "illusion" can have serious consequences.

[0004] To overcome the shortcomings of traditional retrieval techniques and purely generative AI models, Retrieval Augmented Generation (RAG) technology has emerged. It combines information retrieval with generative AI, aiming to guide the generation process by retrieving relevant information, thereby improving the accuracy and reliability of the generated content.

[0005] The existing RAG system is limited by a static knowledge base architecture, which is slow to update. When users query time-sensitive content (such as government information, financial information, etc.), the static knowledge base cannot capture the latest information in real time, causing the system output to deviate from the actual situation and affecting the user experience. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and medium for incremental updating of a knowledge base based on hot topics, in order to solve the problem of lagging updates to the RAG knowledge base in related technologies.

[0007] In a first aspect, the present invention provides a method for incremental updating of a knowledge base based on hot topics, including:

[0008] The user-input question is rewritten so that the rewritten question contains the complete information needed for the AI ​​model to understand the question.

[0009] If the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored.

[0010] When the preset triggering conditions are met, hierarchical clustering and K-means clustering are used to divide the stored rewritten questions into question clusters, and target question clusters that meet the preset screening conditions are selected. The AI ​​model is used to summarize the questions in the target question clusters to obtain the hot issues corresponding to each target question cluster.

[0011] By crawling web pages, web content related to various hot topics is obtained. Based on the semantic relevance between the hot topics and the related web content, at least one text content with the highest semantic relevance is extracted as the relevant knowledge of the hot topics.

[0012] Store relevant knowledge about various hot topics in a knowledge base.

[0013] Optionally, the user-input question may be rewritten, including:

[0014] In a single-turn dialogue scenario, the question is corrected for typos and semantically standardized to eliminate redundant information and unclear expressions in the question and generate a semantically complete query statement.

[0015] In multi-turn dialogue scenarios, based on historical information input by the user in the dialogue, the question is eliminated by referencing and information is completed to generate a semantically complete query statement.

[0016] The query statement is then used as the rewritten question.

[0017] Optionally, if the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored, including:

[0018] The rewritten question is used to search the knowledge base and obtain search results.

[0019] The AI ​​model is used to analyze key information that matches the search results with the rewritten question;

[0020] If the key information is missing from the search results, or if the key information is incomplete, inaccurate, or not relevant enough to the rewritten question, then the question cannot be answered, and the rewritten question is stored.

[0021] Optionally, hierarchical clustering and K-means clustering are used to divide the rewritten stored questions into question clusters, and target question clusters that meet preset screening criteria are selected, including:

[0022] The rewritten problem is vectorized to obtain a set of problem vectors;

[0023] The problem vector set is divided into a first preset number of initial clusters using hierarchical clustering.

[0024] Using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final problem clusters.

[0025] From the final divided problem clusters, candidate clusters with a number of problems within the cluster greater than or equal to a second preset threshold are selected, and from the candidate clusters, a target problem cluster with the largest number of problems is selected, which is a third preset threshold.

[0026] Optionally, using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final partitioned problem clusters, including:

[0027] For each initial cluster, the mean vector of the problem vectors in the initial cluster is calculated as the cluster center of the initial cluster;

[0028] Based on the cluster centers of the initial clusters, each problem vector is assigned to the cluster corresponding to the nearest cluster center;

[0029] Based on the allocation results, the mean vector of the problem vector in each cluster is recalculated to update the cluster center of each cluster;

[0030] Repeat the above allocation and cluster center update operations until the preset maximum number of iterations is reached to obtain the final problem clusters.

[0031] Optionally, web crawlers can be used to obtain web page content related to various trending topics, including:

[0032] Each hot topic is segmented into words and stop words are filtered to obtain a word set for each hot topic.

[0033] For each hot topic word set, the TF-IDF algorithm is used to calculate the weight of each word in the word set, and the fourth preset number of words with the highest weight are selected as the keyword group of the hot topic.

[0034] Use a web crawler to crawl web pages related to each hot topic based on keyword groups.

[0035] Optionally, based on the semantic relevance between the hot topic and the content of related web pages, at least one text content with the highest semantic relevance is extracted as relevant knowledge of the hot topic, including:

[0036] For each relevant webpage related to the aforementioned hot topic, a DOM tree transformation is performed to obtain the corresponding DOM tree for the webpage;

[0037] Traverse the text nodes of the DOM tree, and calculate the comprehensive score of each node based on the tag type, text density, DOM depth and negative attributes. Select the fifth preset number of nodes with the highest comprehensive score and concatenate them as the core text content to obtain the core text content of each relevant webpage of the hot topic.

[0038] The BM25 algorithm is used to calculate the relevance score between the keyword group of the hot topic and the core text content of each related webpage. The sixth preset number of core text contents with the highest relevance score are selected as the relevant knowledge of the hot topic.

[0039] Secondly, the present invention provides a knowledge base incremental update device based on hot topic issues, comprising:

[0040] The question rewriting module is used to rewrite the questions input by the user so that the rewritten question contains the complete information needed by the AI ​​model to understand the question;

[0041] The question interception module is used to store the rewritten question if the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base.

[0042] The hot topic mining module is used to divide the stored rewritten questions into question clusters using hierarchical clustering and K-means clustering when preset trigger conditions are met, select target question clusters that meet preset screening conditions, and use an AI model to summarize the questions in the target question clusters to obtain the hot topics corresponding to each target question cluster.

[0043] The knowledge base incremental update module is used to obtain web page content related to various hot issues through web crawlers, extract at least one text content with the highest semantic relevance as the relevant knowledge of the hot issues based on the semantic relevance between the hot issues and the relevant web page content, and store the relevant knowledge of various hot issues into the knowledge base.

[0044] Optionally, when rewriting the user-input question, the question rewriting module is specifically used for:

[0045] In a single-turn dialogue scenario, the question is corrected for typos and semantically standardized to eliminate redundant information and unclear expressions in the question and generate a semantically complete query statement.

[0046] In multi-turn dialogue scenarios, based on historical information input by the user in the dialogue, the question is eliminated by referencing and information is completed to generate a semantically complete query statement.

[0047] The query statement is then used as the rewritten question.

[0048] Optionally, when the question interception module determines, through an AI model, that the question cannot be answered based on the rewritten question and the knowledge in the knowledge base, and then stores the rewritten question, it is specifically used for:

[0049] The rewritten question is used to search the knowledge base and obtain search results.

[0050] The AI ​​model is used to analyze key information that matches the search results with the rewritten question;

[0051] If the key information is missing from the search results, or if the key information is incomplete, inaccurate, or not relevant enough to the rewritten question, then the question cannot be answered, and the rewritten question is stored.

[0052] Optionally, when the hot topic mining module uses hierarchical clustering and K-means clustering to divide the stored rewritten questions into question clusters and selects target question clusters that meet preset screening conditions, it is specifically used for:

[0053] The rewritten problem is vectorized to obtain a set of problem vectors;

[0054] The problem vector set is divided into a first preset number of initial clusters using hierarchical clustering.

[0055] Using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final problem clusters.

[0056] From the final divided problem clusters, candidate clusters with a number of problems within the cluster greater than or equal to a second preset threshold are selected, and from the candidate clusters, a target problem cluster with the largest number of problems is selected, which is a third preset threshold.

[0057] Optionally, when the hotspot problem mining module uses the cluster centers of the initial clusters as initial seed points and employs the K-means algorithm to perform clustering optimization on the problem vector set to obtain the final divided problem clusters, it is specifically used for:

[0058] For each initial cluster, the mean vector of the problem vectors in the initial cluster is calculated as the cluster center of the initial cluster;

[0059] Based on the cluster centers of the initial clusters, each problem vector is assigned to the cluster corresponding to the nearest cluster center;

[0060] Based on the allocation results, the mean vector of the problem vector in each cluster is recalculated to update the cluster center of each cluster;

[0061] Repeat the above allocation and cluster center update operations until the preset maximum number of iterations is reached to obtain the final problem clusters.

[0062] Optionally, when the knowledge base incremental update module retrieves web page content related to various hot topics through web crawlers, it is specifically used for:

[0063] Each hot topic is segmented into words and stop words are filtered to obtain a word set for each hot topic.

[0064] For each hot topic word set, the TF-IDF algorithm is used to calculate the weight of each word in the word set, and the fourth preset number of words with the highest weight are selected as the keyword group of the hot topic.

[0065] Use a web crawler to crawl web pages related to each hot topic based on keyword groups.

[0066] Optionally, the knowledge base incremental update module extracts at least one text content with the highest semantic relevance based on the semantic relevance between the hot topic and the related web page content, as relevant knowledge for the hot topic, specifically used for:

[0067] For each relevant webpage related to the aforementioned hot topic, a DOM tree transformation is performed to obtain the corresponding DOM tree for the webpage;

[0068] Traverse the text nodes of the DOM tree, and calculate the comprehensive score of each node based on the tag type, text density, DOM depth and negative attributes. Select the fifth preset number of nodes with the highest comprehensive score and concatenate them as the core text content to obtain the core text content of each relevant webpage of the hot topic.

[0069] The BM25 algorithm is used to calculate the relevance score between the keyword group of the hot topic and the core text content of each related webpage. The sixth preset number of core text contents with the highest relevance score are selected as the relevant knowledge of the hot topic.

[0070] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any of the first aspects.

[0071] Fourthly, the present invention provides a computer-readable storage medium having program instructions stored thereon, which, when executed, implement the method of any one of the first aspects.

[0072] Fifthly, the present invention provides a computer program product stored in a storage medium, which can implement the method of the first aspect described above when the program product is run.

[0073] The technical solution provided by this invention has the following advantages compared with existing technologies: By rewriting the user-input questions to include the complete information required for AI model understanding, when the AI ​​model determines that the knowledge base cannot answer the rewritten questions, the system stores them. Upon meeting preset trigger conditions, it uses a combination of hierarchical clustering and K-means clustering to efficiently and semantically consistently divide the stored questions into clusters, thereby selecting high-frequency, highly representative target question clusters and summarizing them to generate hot topics, ensuring the clustering quality and stability of hot topic identification. Subsequently, web crawlers are used to obtain web page content related to each hot topic, and based on the semantic relevance between hot topics and web page content, the text with the highest relevance is extracted as relevant knowledge. This makes the knowledge finally stored in the knowledge base more relevant to user questions, closely aligning with the actual needs of question answering. This achieves targeted and incremental updates of the knowledge base to meet user question-and-answer needs, significantly improving the timeliness, coverage, and user experience of the question-and-answer system. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 A flowchart of a knowledge base incremental update method based on hot topics provided in an embodiment of the present invention;

[0077] Figure 2 This is a schematic diagram of the structure of a knowledge base incremental update device based on hot topic issues provided in an embodiment of the present invention;

[0078] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0079] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0080] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0081] Figure 1 A flowchart of a knowledge base incremental update method based on hot topic issues is provided in this embodiment of the invention. This method can be executed by a knowledge base incremental update device based on hot topic issues. This device can be implemented in software and / or hardware and can be configured in an electronic device, such as a server or terminal. The terminal specifically includes mobile phones, computers, in-vehicle systems, etc. The method includes the following steps:

[0082] S101. Rewrite the question input by the user so that the rewritten question contains the complete information required for the AI ​​model to understand the question.

[0083] User-input questions may contain typos, ambiguous intent, and multi-turn dialogues. To more accurately identify hot questions not present in the knowledge base, the original user-input questions need to be rewritten into semantically complete, independent, and understandable queries. This provides a reliable foundation for accurately determining whether the knowledge base covers the question and for high-quality clustering of missed questions. AI models refer to machine learning models capable of semantic understanding, reasoning, or generation based on natural language input, such as Large Language Models (LLMs), question-answering models, and text generation models.

[0084] Problem rewriting can be divided into single-round rewriting and multi-round rewriting. In the case of single-round rewriting, the main task is to solve the problem of typos. In the case of multi-round rewriting, the main task is to complete the problem and eliminate the reference based on the context.

[0085] In one implementation, rewriting the user-inputted question includes: in a single-turn dialogue scenario, correcting typos and standardizing the semantics of the question to eliminate redundant information and unclear expressions in the question, generating a semantically complete query statement; in a multi-turn dialogue scenario, eliminating referentials and completing information based on historical information input in the user dialogue to generate a semantically complete query statement; and using the query statement as the rewritten question.

[0086] In single-turn dialogue scenarios, the question undergoes typo correction and semantic normalization to eliminate redundant information and unclear expressions, generating a semantically complete query statement. In multi-turn dialogue scenarios, based on contextual information from the user's dialogue history, the current question undergoes referential resolution (e.g., replacing pronouns like "it" and "this" with specific entities) and information completion (e.g., supplementing omitted subjects or objects), generating a query statement that can be understood independently without relying on history. This query statement is then used as the rewritten question. Rewriting can be achieved using a Large Language Model (LLM), for example, by using a pre-defined Prompt template to guide the model to output rewritten results in standardized JSON format, ensuring output consistency and system parsability. The following is an example of a rewritten prompt template from an embodiment of this invention.

[0087] The single-round rewrite prompt template is as follows, where {history} contains user questions:

[0088] You are an AI problem rewriting assistant. Your job is to rewrite problems.

[0089] You are tasked with rewriting a question based on a given dialogue:

[0090] - If a user's question contains typos, is unclear, or is redundant, it needs to be rewritten into an accurate, complete, and clear statement.

[0091] - Your output is in strict JSON format.

[0092] ## Output:

[0093] JSON format: {{"user": the rewritten statement that can be understood without the preceding text}}

[0094] ## Input:

[0095] {history}

[0096] The following is a prompt template that has been rewritten multiple times:

[0097] You are an AI problem rewriting assistant. Your job is to rewrite problems.

[0098] You are tasked with rewriting a question based on a given dialogue:

[0099] - Based on the input dialogue between the user and the bot, rewrite the user's last statement into a sentence that can be understood without the preceding context.

[0100] - Your output is in strict JSON format.

[0101] ## Output:

[0102] JSON format: {{"user": the rewritten statement that can be understood without the preceding text}}

[0103] ## Require:

[0104] - If the user's last post was unrelated to the preceding text, no rewriting is needed.

[0105] - If a user's last post is related to the preceding text, but can be understood without context, then no rewriting is necessary.

[0106] - If the user's last post is related to the preceding text and requires context to understand, rewrite it.

[0107] ## Input:

[0108] {history}

[0109] By distinguishing between single-turn and multi-turn dialogue scenarios and employing strategies such as misspelling correction, semantic normalization, and context completion and referential elimination in multi-turn dialogues, this invention can generate semantically complete and context-independent query statements, thereby effectively improving the accuracy of subsequent question clustering and the reliability of hotspot identification.

[0110] S102. If the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored.

[0111] To identify and capture user question needs that are not covered by the current knowledge base, a large language model is used to perform semantic-level judgment on the search results. The large language model can receive the rewritten question and the search results of the question in the knowledge base as input through preset structured prompts, and output the judgment result of whether it is answerable and the reason. This can more accurately distinguish between "answerable" and "unanswerable" user questions, avoid misjudging some relevant but insufficient results as hits, and thus improve the quality of the missing question data.

[0112] In one implementation, if the AI ​​model determines that the question cannot be answered based on the rewritten question and the knowledge in the knowledge base, then storing the rewritten question includes: searching the knowledge base based on the rewritten question to obtain search results; using the AI ​​model to analyze key information that matches the search results with the rewritten question; if the key information is missing in the search results, or if the key information is incomplete, inaccurate, or has insufficient relevance to the rewritten question, then it is determined that the question cannot be answered, and the rewritten question is stored.

[0113] The rewritten query is used to retrieve information from the knowledge base. A large language model (MLM) is then used to determine whether the retrieved results can directly answer the user's question. If the MLM determines that the retrieved results are sufficient to answer the user's question, it directly outputs the answer. If it cannot answer the question, it returns a fixed answer and sets the `hit` field to `false`. This design reduces the number of calls to the MLM and improves system efficiency. Specifically, the MLM can analyze whether the retrieved results contain key information matching the question. If the retrieved results lack key information, or if relevant information exists but is incomplete, factually incorrect, logically broken, or insufficiently relevant to the question, then the knowledge base is deemed unable to effectively answer the question.

[0114] For questions where the `hit` field is false in the previous step, these questions did not hit the knowledge base. The question, the time of the question, and other fields need to be stored in the "Missing Questions" table in the PostgreSQL database for future use in identifying trending questions.

[0115] The prompt for determining whether a user's question is answerable using a large language model is as follows:

[0116] You now need to process user questions and corresponding search results, and complete the following tasks:

[0117] Carefully analyze the user's question and the search results provided to determine whether the search results contain sufficient, accurate, and complete information to directly answer the question.

[0118] If the information in the search results is sufficient to cover the core requirements of the question (including specific details, logical relationships, data support, etc.), a clear and accurate answer will be generated based on the search results, without the need for additional information. The answer will be placed in the "answer" field of the output JSON object, and "hit" will be set to true, indicating that the search results are sufficient to answer the user's question.

[0119] - If the search results contain missing information (such as key data not mentioned, broken logical chains, or unexplained core concepts), incorrect information, or low relevance to the question, making it impossible to fully answer the user's question, then there is no need to generate an answer. Instead, in the output JSON object, set "answer" to "Sorry, I cannot answer your question with my current knowledge" and set "hit" to false, indicating that the search results are insufficient to answer the user's question.

[0120] - Your output is in strict JSON format.

[0121] ## Input:

[0122] - User issue: {query}

[0123] - Search results: {context}

[0124] ## Output:

[0125] JSON format: {{"answer": the answer to the question, "hit": whether the search results can answer the user's question}}

[0126] This invention uses an AI model to perform semantic-level relevance judgment between search results and questions, which can more accurately identify questions that are not covered by the knowledge base, avoid misjudging partially relevant but insufficient results as hits, thereby improving the quality of data on hit questions and providing more reliable input for hotspot mining.

[0127] S103. When the preset triggering conditions are met, hierarchical clustering and K-means clustering are used to divide the rewritten stored questions into question clusters, and target question clusters that meet the preset screening conditions are selected. The AI ​​model is used to summarize the questions in the target question clusters to obtain the hot issues corresponding to each target question cluster.

[0128] Preset trigger conditions can be set to execute at a predetermined time based on business needs (such as every 6 hours / 12 hours / 24 hours), or they can be set to a predetermined threshold for the number of issues to execute quantitatively based on the number of issues stored.

[0129] The purpose of this step is to automatically identify semantic topics of high user interest, i.e., hot topics, from a large number of scattered missed questions, providing a basis for targeted crawling. Considering that single clustering algorithms (such as pure K-means) are sensitive to initial centers and struggle to determine the optimal number of clusters, while pure hierarchical clustering has high computational complexity and lacks iterative optimization capabilities, this embodiment of the invention adopts a combined strategy of "hierarchical clustering + K-means". First, hierarchical clustering adaptively generates structurally sound initial clusters and cluster centers, which are then used as seed points for global optimization in K-means, balancing clustering quality and computational efficiency, thereby more accurately aggregating semantically similar questions.

[0130] In one implementation, hierarchical clustering and K-means clustering are used to divide the stored rewritten questions into question clusters, and target question clusters that meet preset screening conditions are selected. The process includes: vectorizing the rewritten questions to obtain a set of question vectors; dividing the set of question vectors into a first preset number of initial clusters using hierarchical clustering; using the cluster centers of the initial clusters as initial seed points, performing clustering optimization on the set of question vectors using the K-means algorithm to obtain the final divided question clusters; selecting candidate clusters from the final divided question clusters whose number of questions within the cluster is greater than or equal to a second preset threshold, and selecting a third preset number of target question clusters from the candidate clusters with the largest number of questions.

[0131] In one implementation, using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final partitioned problem clusters. This includes: for each initial cluster, calculating the mean vector of the problem vectors in the initial cluster as the cluster center of the initial cluster; assigning each problem vector to the cluster corresponding to the nearest cluster center based on the cluster centers of the initial clusters; recalculating the mean vector of the problem vectors in each cluster according to the assignment results to update the cluster centers of each cluster; repeating the above assignment and cluster center update operations until a preset maximum number of iterations is reached to obtain the final partitioned problem clusters.

[0132] First, the rewritten questions are vectorized (e.g., by generating sentence vectors using an embedding model) to obtain a set of question vectors. Then, hierarchical clustering is used to divide this set into initial clusters of a first preset number (e.g., K=50). Next, using the cluster centers (i.e., the mean vectors within each cluster) as initial seed points, the K-means algorithm is used to optimize the clustering of the entire question vector set, resulting in the final question clusters. Finally, candidate clusters are selected from the final clusters whose number of questions is greater than or equal to a second preset threshold (e.g., Ψ=10, indicating at least 10 users have asked similar questions), and from these, a third preset number (e.g., TopK) is selected based on the highest number of questions. h =5) target problem cluster.

[0133] Specifically, in the K-means optimization process, the mean vector of the question vectors for each initial cluster is calculated as the initial cluster center. Based on these centers, all question vectors are assigned to the clusters corresponding to the nearest cluster centers. The mean vectors of each cluster are then recalculated based on the new assignment results to update the cluster centers. This assignment and update operation is repeated until a preset maximum number of iterations (e.g., Φ=100) is reached, thereby obtaining a stable and compact final cluster partition. Subsequently, the original question text from each target question cluster is input into the AI ​​model, which generates concise and generalized natural language descriptions, which serve as the hotspot questions corresponding to that cluster.

[0134] In summary, S103 can be completed through the following specific steps:

[0135] S1031. Retrieve the most recent (6 hours / 12 hours / 24 hours) missed issues from the PostgreSQL database.

[0136] S1032. Using an embedding model to address the problem Convert to vector Finally, all problem vectors are combined into a set. , where n is the number of problem vectors.

[0137] S1033. Use a combination of hierarchical clustering and K-means clustering to perform semantic clustering on the problem. Details are as follows:

[0138] 10331. Treat each problem vector as an independent initial cluster. The initial cluster set is obtained. .

[0139] 10332, Regarding the first Wheel cluster set Calculate the distance between all cluster pairs, where and The distance can be expressed as: ,in It is a cluster The number of vectors in the middle, It is a vector and The cosine distance is then used to merge the clusters with the smallest distance. and New clusters are obtained Generate a new cluster set The significance of the above formula lies in the fact that, for each cluster pair... and ,Pick Each data point in and Each data point in Calculate the pairwise distances between them, then average them to obtain the cluster pairs. and The distance between them.

[0140] 10333. Repeat the previous step until the number of clusters in the cluster set decreases to the preset value K, thus obtaining the initial cluster set.

[0141] 10334. For each initial cluster The mean vector of all data points within a cluster is calculated as the initial cluster centers for K-means. .

[0142] 10335, Output using hierarchical clustering Using the initial seed point, perform iterative K-means optimization. For the ... Round of iterations, processing each vector Assign to the cluster corresponding to the nearest cluster center: ,in Indicates the first In round of iteration The cluster label to which it belongs ultimately generates the cluster partition for that round. .

[0143] 10336. Based on the new cluster partitioning, recalculate the mean vector of all data points within each cluster as the new cluster centers. Then, continue iterating using the new cluster centers until the maximum number of iterations Φ is reached. This yields the final cluster partitioning. .

[0144] S1034. Cluster partitioning obtained from clustering In the selection process, the TopK clusters with elements greater than Ψ are chosen. h Cluster Then, the vectors within each cluster are mapped back to the corresponding question text. Finally, a large model is used to summarize and generalize the questions within each cluster to obtain the TopK. h A hot topic.

[0145] This invention first uses hierarchical clustering to generate a structurally sound initial cluster, and then uses the cluster centers as seed points for K-means global optimization. This reduces the sensitivity to the initial centers while ensuring clustering quality, effectively overcoming the shortcomings of single clustering algorithms that are prone to getting trapped in local optima or have high computational complexity. This allows for more accurate aggregation of semantically similar user problems.

[0146] S104. Obtain web page content related to various hot topics through web crawlers, and extract at least one text content with the highest semantic relevance as relevant knowledge of the hot topic based on the semantic relevance between the hot topic and the relevant web page content.

[0147] After identifying trending topics, for each topic, the TF-IDF algorithm is used to extract keywords. Then, a web crawler is used to retrieve relevant web pages based on these keywords, and the pages are cleaned using algorithms based on DOM trees and BM25. Finally, the crawled knowledge is stored in a knowledge base. To selectively acquire high-quality, highly relevant new knowledge from the open web to fill gaps in the knowledge base, this invention employs a three-level screening mechanism: keyword-driven crawling, initial screening using DOM tree structure, and semantic refinement using the BM25 algorithm. This significantly improves the relevance and usability of the acquired knowledge.

[0148] In one implementation, web crawlers are used to obtain web page content related to various hot topics, including: segmenting each hot topic into words and filtering stop words to obtain a word set for each hot topic; for each word set of a hot topic, the TF-IDF algorithm is used to calculate the weight of each word in the word set, and the fourth preset number of words with the highest weight are selected as the keyword group of the hot topic; and web crawlers are used to crawl web pages related to the hot topic based on the keyword group of each hot topic.

[0149] For each hot topic, Chinese word segmentation is performed and stop words are filtered to obtain a word set; the TF-IDF algorithm is used to calculate the weight score of each word, which reflects its importance in the question and its rarity in the entire corpus, and the words with the highest weight in the fourth preset number (e.g., TopX=5) are selected to form a keyword group; then, a web crawler is called to use the keyword group as the query condition to crawl relevant web pages from search engines or specified sites.

[0150] This invention extracts keyword groups of hot topics based on the TF-IDF algorithm as query conditions for web crawlers, which can focus on the core semantic words in the questions, effectively improve the relevance and recall of web crawling, and reduce the crawling overhead of irrelevant pages.

[0151] In one implementation, based on the semantic relevance between the hot topic and the content of each related webpage, at least one text content with the highest semantic relevance is extracted as the relevant knowledge of the hot topic. This includes: performing DOM tree transformation on each related webpage of the hot topic to obtain the DOM tree corresponding to the webpage; traversing the text nodes of the DOM tree and calculating a comprehensive node score based on the tag type, text density, DOM depth, and negative attributes of each node; selecting a fifth preset number of nodes with the highest comprehensive node score and concatenating them as core text content to obtain the core text content of each related webpage of the hot topic; using the BM25 algorithm to calculate the relevance score between the keyword group of the hot topic and the core text content of each related webpage; and selecting a sixth preset number of core text content with the highest relevance score as the relevant knowledge of the hot topic.

[0152] Parse the webpage's HTML into a DOM tree, then traverse all text nodes based on tag type (e.g., ...). 、 <h1>High priority <nav> 、 <footer>The comprehensive score of each node is calculated based on its priority, text density (number of plain text characters / total node length), DOM depth (the deeper the node, the more likely it is to be the main text), and negative attributes (such as deducting points if the class or id contains "ads" or "banner"). The fifth preset number of nodes with the highest scores (such as λ=3) are selected and concatenated according to their original order in the DOM tree to form the core text content of the webpage.

[0153] Building upon this foundation, to select the most relevant content from the core text of multiple web pages, the BM25 algorithm is used to calculate the relevance score between the keyword groups of the hot topic and each core text. BM25 is a probabilistic ranking function widely used in information retrieval, effectively balancing the influence of word frequency, document length, and inverse document frequency. Finally, the core text content with the highest relevance score (the sixth preset number, e.g., σ=2) is selected as the relevant knowledge for the hot topic, ensuring that the content entered into the database is both comprehensive and accurate.

[0154] In summary, S104 can be completed through the following specific steps:

[0155] S1041, Regarding the question The jieba library was used for word segmentation, and meaningless stop words were filtered out. Finally, the TF-IDF algorithm was used to calculate the score of each word and sort them. The top X words were selected as the keyword group for this problem. .

[0156] S1042. Use a web crawler to retrieve relevant data from the internet based on the keywords obtained in the previous step. Then, use a web page cleaning algorithm based on DOM trees and BM25 to clean the retrieved web pages. The specific steps are as follows:

[0157] 10421. Regarding web pages Parse the HTML string of the webpage and convert it into a DOM tree. .

[0158] 10422. Traversing the DOM tree Then, the tag type, content, DOM depth, and other attributes of each node are extracted to obtain a set of potential text nodes. ,in This represents the number of nodes.

[0159] 10423. Calculate the overall score for each node. ,in Tag priority is scored, with different tags having different importance weights. The text density score is calculated as the ratio of the number of plain text characters within a node to the length of the node's complete HTML content (including tags, spaces, and special characters). Rate the depth of a node in the DOM tree. As a negative attribute weight, when the class name or ID of a node contains keywords such as "nav" or "ads", it indicates that the current node may be navigation bar or advertisement content, and points need to be deducted in order to eliminate redundant content.

[0160] 10424. Next, select the top λ nodes with the highest overall scores and reorder them according to their original positions in the DOM tree to maintain content coherence. Finally, extract the text content of each node and concatenate them to output the webpage. Core text content .

[0161] 10425. Regarding the question The web crawler will crawl many web pages, and after cleaning, it will obtain a lot of text content. In order to filter out the content that best matches the problem from these text contents, this invention uses the BM25 algorithm to calculate the problem. Keyword phrases With each text content The relevance scores are used to select the σ texts with the highest scores as the final question knowledge. The formula for calculating the relevance score is as follows:

[0162]

[0163] in, The number of keywords. Keywords In the text word frequency in `avgdl` represents the length of the text, and `avgdl` represents the average length of all text. This is the inverse document frequency of the keyword. and It is an adjustable parameter.

[0164] For example, when calculating the node comprehensive score in step 10423, the label priority score, depth score, and negative attribute weight score can be determined through custom rules: for example, <article>The most crucial text container, worth 10 points; <section>8-point text segmentation container; <h1>7-point first-level heading (core theme); 4. Paragraph tags (core text), etc. Positive bonus points: the importance of the tags themselves (e.g., <article>Compare More importantly, text information density (the higher the proportion of text, the more likely it is to be the main text), DOM depth (nodes with reasonable depth are more likely to be the main text); negative deductions: node class name / ID contains redundant identifiers such as advertisements and navigation (e.g., class="footer").

[0165] Depth score indicates how many tags are in the content of that node, which needs to be counted by traversing the DOM tree. For example, the depth of a node in the DOM tree (e.g., 1, 2, ...). <article>Below It could be 4).

[0166] Custom rules for negative attribute weighting scoring include: av / navigation 5 points for navigation-related; ads / ad / advert 8 points for advertising-related; side / sidebar 4 points for sidebar, etc.

[0167] This invention combines DOM tree structure features for initial webpage screening and uses the BM25 algorithm to perform semantic relevance ranking of core text and keyword groups. This effectively filters out noisy content such as advertisements and navigation bars while retaining highly relevant main text, significantly improving the quality of the knowledge entered into the database and the matching degree of user questions and answers.

[0168] S105. Store relevant knowledge about various hot issues in the knowledge base.

[0169] The selected text knowledge is stored in the RAG knowledge base database and converted into vector form for storage in the vector database, so that the RAG system can efficiently retrieve new knowledge when users ask questions.

[0170] This invention rewrites user-input questions to include the complete information needed for AI model understanding. When the AI ​​model determines that the knowledge base cannot answer the rewritten question, the system stores it. Upon meeting preset trigger conditions, it uses a combination of hierarchical clustering and K-means clustering to efficiently and semantically consistently divide the stored questions into clusters. This process selects high-frequency, highly representative target question clusters and summarizes them into hot topics, ensuring the clustering quality and stability of hot topic identification. Subsequently, web crawlers are used to obtain web page content related to each hot topic. Based on the semantic relevance between hot topics and web page content, the text with the highest relevance is extracted as relevant knowledge. This makes the knowledge ultimately stored in the knowledge base more relevant to user questions, closely aligning with the actual needs of question answering. This achieves targeted, incremental updates to the knowledge base to meet user question-and-answer needs, significantly improving the timeliness, coverage, and user experience of the question-and-answer system.

[0171] Figure 2 This is a schematic diagram of the structure of a knowledge base incremental update device based on hot topic issues provided in an embodiment of the present invention. The knowledge base incremental update device based on hot topic issues provided in this embodiment of the present invention can execute the processing flow provided in the embodiment of the knowledge base incremental update method based on hot topic issues, such as... Figure 2 As shown, the knowledge base incremental update device based on hot topics includes:

[0172] The question rewriting module 201 is used to rewrite the question input by the user so that the rewritten question contains the complete information required for the AI ​​model to understand the question;

[0173] The question interception module 202 is used to store the rewritten question if the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base.

[0174] The hot topic mining module 203 is used to divide the stored rewritten questions into question clusters using hierarchical clustering and K-means clustering when the preset triggering conditions are met, select target question clusters that meet the preset screening conditions, and use an AI model to summarize the questions in the target question clusters to obtain the hot topic questions corresponding to each target question cluster.

[0175] The knowledge base incremental update module 204 is used to obtain web page content related to various hot issues through web crawlers, extract at least one text content with the highest semantic relevance as the relevant knowledge of the hot issues based on the semantic relevance between the hot issues and the relevant web page content, and store the relevant knowledge of various hot issues into the knowledge base.

[0176] In some embodiments, when the question rewriting module 201 rewrites the user-input question, it is specifically used for:

[0177] In a single-turn dialogue scenario, the question is corrected for typos and semantically standardized to eliminate redundant information and unclear expressions in the question and generate a semantically complete query statement.

[0178] In multi-turn dialogue scenarios, based on historical information input by the user in the dialogue, the question is eliminated by referencing and information is completed to generate a semantically complete query statement.

[0179] The query statement is then used as the rewritten question.

[0180] In some embodiments, when the question interception module 202 determines, through an AI model, that the question cannot be answered based on the rewritten question and the knowledge in the knowledge base, and then stores the rewritten question, it is specifically used for:

[0181] The rewritten question is used to search the knowledge base and obtain search results.

[0182] The AI ​​model is used to analyze key information that matches the search results with the rewritten question;

[0183] If the key information is missing from the search results, or if the key information is incomplete, inaccurate, or not relevant enough to the rewritten question, then the question cannot be answered, and the rewritten question is stored.

[0184] In some embodiments, when the hot topic mining module 203 uses hierarchical clustering and K-means clustering to divide the stored rewritten questions into question clusters and selects target question clusters that meet preset screening conditions, it is specifically used for:

[0185] The rewritten problem is vectorized to obtain a set of problem vectors;

[0186] The problem vector set is divided into a first preset number of initial clusters using hierarchical clustering.

[0187] Using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final problem clusters.

[0188] From the final divided problem clusters, candidate clusters with a number of problems within the cluster greater than or equal to a second preset threshold are selected, and from the candidate clusters, a target problem cluster with the largest number of problems is selected, which is a third preset threshold.

[0189] In some embodiments, when the hotspot problem mining module 203 uses the cluster centers of the initial clusters as initial seed points and employs the K-means algorithm to perform clustering optimization on the problem vector set to obtain the final divided problem clusters, it is specifically used for:

[0190] For each initial cluster, the mean vector of the problem vectors in the initial cluster is calculated as the cluster center of the initial cluster;

[0191] Based on the cluster centers of the initial clusters, each problem vector is assigned to the cluster corresponding to the nearest cluster center;

[0192] Based on the allocation results, the mean vector of the problem vector in each cluster is recalculated to update the cluster center of each cluster;

[0193] Repeat the above allocation and cluster center update operations until the preset maximum number of iterations is reached to obtain the final problem clusters.

[0194] In some embodiments, when the knowledge base incremental update module 204 obtains web page content related to various hot topics through a web crawler, it is specifically used for:

[0195] Each hot topic is segmented into words and stop words are filtered to obtain a word set for each hot topic.

[0196] For each hot topic word set, the TF-IDF algorithm is used to calculate the weight of each word in the word set, and the fourth preset number of words with the highest weight are selected as the keyword group of the hot topic.

[0197] Use a web crawler to crawl web pages related to each hot topic based on keyword groups.

[0198] In some embodiments, the knowledge base incremental update module 204 extracts at least one text content with the highest semantic relevance as relevant knowledge of the hot topic based on the semantic relevance between the hot topic and the related web page content, specifically for:

[0199] For each relevant webpage related to the aforementioned hot topic, a DOM tree transformation is performed to obtain the corresponding DOM tree for the webpage;

[0200] Traverse the text nodes of the DOM tree, and calculate the comprehensive score of each node based on the tag type, text density, DOM depth and negative attributes. Select the fifth preset number of nodes with the highest comprehensive score and concatenate them as the core text content to obtain the core text content of each relevant webpage of the hot topic.

[0201] The BM25 algorithm is used to calculate the relevance score between the keyword group of the hot topic and the core text content of each related webpage. The sixth preset number of core text contents with the highest relevance score are selected as the relevant knowledge of the hot topic.

[0202] Figure 2 The knowledge base incremental update device based on hot topic issues in the illustrated embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0203] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. See below for details. Figure 3 It shows a structural schematic diagram suitable for implementing the electronic device 300 in the embodiments of the present invention. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0204] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303 to implement the hot-topic-based knowledge base incremental update method as described in the embodiments of the present invention. Various programs and data required for the operation of the electronic device 300 are also stored in the RAM 303. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0205] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0206] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the knowledge base incremental update method based on hot topic issues as described above. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of the embodiments of the present invention.

[0207] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0208] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0209] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0210] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to:

[0211] The user-input question is rewritten so that the rewritten question contains the complete information needed for the AI ​​model to understand the question.

[0212] If the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored.

[0213] When the preset triggering conditions are met, hierarchical clustering and K-means clustering are used to divide the stored rewritten questions into question clusters, and target question clusters that meet the preset screening conditions are selected. The AI ​​model is used to summarize the questions in the target question clusters to obtain the hot issues corresponding to each target question cluster.

[0214] By crawling web pages, web content related to various hot topics is obtained. Based on the semantic relevance between the hot topics and the related web content, at least one text content with the highest semantic relevance is extracted as the relevant knowledge of the hot topics.

[0215] Store relevant knowledge about various hot topics in a knowledge base.

[0216] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also perform other steps described in the above embodiments.

[0217] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Python, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0219] The units described in the embodiments of the present invention can be implemented in software or in hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0220] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0221] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0222] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0223] This invention also provides a computer program product, which is stored in a storage medium. When the program product is run, it can implement the methods of any of the above embodiments, and its execution mode and beneficial effects are similar, so they will not be described again here.

[0224] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0225] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0226] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0227] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0228] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. < / article> < / article> < / h1> < / section> < / article> < / footer> < / nav> < / h1>

Claims

1. A method for incremental updating of a knowledge base based on hot topics, characterized in that, include: The user-input question is rewritten so that the rewritten question contains the complete information needed for the AI ​​model to understand the question. If the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored. When the preset triggering conditions are met, hierarchical clustering and K-means clustering are used to divide the stored rewritten questions into question clusters, and target question clusters that meet the preset screening conditions are selected. The AI ​​model is used to summarize the questions in the target question clusters to obtain the hot issues corresponding to each target question cluster. By crawling web pages, web content related to various hot topics is obtained. Based on the semantic relevance between the hot topics and the related web content, at least one text content with the highest semantic relevance is extracted as the relevant knowledge of the hot topics. Store relevant knowledge about various hot topics in a knowledge base.

2. The knowledge base incremental update method based on hot topic issues according to claim 1, characterized in that, Rewrite the user input question, including: In a single-turn dialogue scenario, the question is corrected for typos and semantically standardized to eliminate redundant information and unclear expressions in the question and generate a semantically complete query statement. In multi-turn dialogue scenarios, based on historical information input by the user in the dialogue, the question is eliminated by referencing and information is completed to generate a semantically complete query statement. The query statement is then used as the rewritten question.

3. The knowledge base incremental update method based on hot topic issues according to claim 1, characterized in that, If the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base, then the rewritten question is stored, including: The rewritten question is used to search the knowledge base and obtain search results. The AI ​​model is used to analyze key information that matches the search results with the rewritten question; If the key information is missing from the search results, or if the key information is incomplete, inaccurate, or not relevant enough to the rewritten question, then the question cannot be answered, and the rewritten question is stored.

4. The knowledge base incremental update method based on hot topic issues according to claim 1, characterized in that, Hierarchical clustering and K-means clustering are used to divide the rewritten problem into problem clusters, and target problem clusters that meet preset screening criteria are selected, including: The rewritten problem is vectorized to obtain a set of problem vectors; The problem vector set is divided into a first preset number of initial clusters using hierarchical clustering. Using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set to obtain the final problem clusters. From the final divided problem clusters, candidate clusters with a number of problems within the cluster greater than or equal to a second preset threshold are selected, and from the candidate clusters, a target problem cluster with the largest number of problems is selected, which is a third preset threshold.

5. The knowledge base incremental update method based on hot topic issues according to claim 4, characterized in that, Using the cluster centers of the initial clusters as initial seed points, the K-means algorithm is used to optimize the clustering of the problem vector set, resulting in the final partitioned problem clusters, including: For each initial cluster, the mean vector of the problem vectors in the initial cluster is calculated as the cluster center of the initial cluster; Based on the cluster centers of the initial clusters, each problem vector is assigned to the cluster corresponding to the nearest cluster center; Based on the allocation results, the mean vector of the problem vector in each cluster is recalculated to update the cluster center of each cluster; Repeat the above allocation and cluster center update operations until the preset maximum number of iterations is reached to obtain the final problem clusters.

6. The knowledge base incremental update method based on hot topic issues according to claim 1, characterized in that, Use web crawlers to obtain web page content related to various trending topics, including: Each hot topic is segmented into words and stop words are filtered to obtain a word set for each hot topic. For each hot topic word set, the TF-IDF algorithm is used to calculate the weight of each word in the word set, and the fourth preset number of words with the highest weight are selected as the keyword group of the hot topic. Use a web crawler to crawl web pages related to each hot topic based on keyword groups.

7. The knowledge base incremental update method based on hot topic issues according to claim 6, characterized in that, Based on the semantic relevance of the hot topic and related web page content, at least one text content with the highest semantic relevance is extracted as relevant knowledge for the hot topic, including: For each relevant webpage related to the aforementioned hot topic, a DOM tree transformation is performed to obtain the corresponding DOM tree for the webpage; Traverse the text nodes of the DOM tree, and calculate the comprehensive score of each node based on the tag type, text density, DOM depth and negative attributes. Select the fifth preset number of nodes with the highest comprehensive score and concatenate them as the core text content to obtain the core text content of each relevant webpage of the hot topic. The BM25 algorithm is used to calculate the relevance score between the keyword group of the hot topic and the core text content of each related webpage. The sixth preset number of core text contents with the highest relevance score are selected as the relevant knowledge of the hot topic.

8. A knowledge base incremental update device based on hot topics, characterized in that, include: The question rewriting module is used to rewrite the questions input by the user so that the rewritten question contains the complete information needed by the AI ​​model to understand the question; The question interception module is used to store the rewritten question if the AI ​​model determines that it cannot answer the question based on the rewritten question and the knowledge in the knowledge base. The hot topic mining module is used to divide the stored rewritten questions into question clusters using hierarchical clustering and K-means clustering when preset trigger conditions are met, select target question clusters that meet preset screening conditions, and use an AI model to summarize the questions in the target question clusters to obtain the hot topics corresponding to each target question cluster. The knowledge base incremental update module is used to obtain web page content related to various hot issues through web crawlers, extract at least one text content with the highest semantic relevance as the relevant knowledge of the hot issues based on the semantic relevance between the hot issues and the relevant web page content, and store the relevant knowledge of various hot issues into the knowledge base.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed, implement the method as described in any one of claims 1-7.

Images