Retrieval method, system and equipment based on webpage search and knowledge graph and medium

Abstract

The invention provides a retrieval method, system and equipment based on webpage search and a knowledge graph and a medium, and belongs to the technical field of intelligent retrieval. The method comprises the following steps: acquiring an input problem, and performing problem analysis and field classification by utilizing a large language model to generate field features; the method comprises the following steps: retrieving related documents from the Internet according to an input problem by using multi-stage retrieval of webpage search and combining sparse retrieval and dense retrieval technologies, and constructing a candidate document pool as a first retrieval result; according to an input question, performing structured query of the knowledge graph, and extracting entity and relation information related to the question as a second retrieval result; according to the domain features, calculating a corresponding information change rate, and according to the information change rate, carrying out adaptive integration on the first retrieval result and the second retrieval result to generate a target answer; and evaluating the confidence coefficient of the target answer by using a large language model, verifying the target answer according to the confidence coefficient, and determining output content based on a verification result.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent retrieval technology, and more specifically, relates to a retrieval method, system, device and medium based on web page search and knowledge graph. Background Art

[0002] With the rapid development of Internet technology, information retrieval has become an important way for people to obtain knowledge and data. Traditional user query response systems mainly rely on keyword matching or fixed algorithm rules to return search results. These systems may perform well when processing simple queries, but their limitations are obvious when faced with complex, cross-domain queries.

[0003] First, traditional methods have obvious deficiencies in processing complex queries. Since they usually rely on simple keyword matching, it is difficult to accurately capture the semantic information and contextual relationships in the query. This results in the system often failing to return accurate and comprehensive results when processing complex queries involving multiple concepts, relationships, or fields. In addition, traditional methods lack a deep understanding and utilization of the characteristics of different fields, so they are unable to select effective information sources based on the characteristics of the field, further limiting their ability to process complex queries.

[0004] Secondly, traditional methods are powerless in the face of massive data. With the popularization of the Internet and the advent of the big data era, the amount of data has exploded. When dealing with such a large data set, traditional methods often find it difficult to efficiently filter out the most valuable information. This not only reduces the system's response speed, but may also cause users to miss key information, thus affecting the user experience.

[0005] In addition, traditional methods also have certain problems when generating response content. Since they lack an effective verification mechanism for the authenticity and accuracy of information, they are prone to generating response content containing false information. This is particularly prominent in the field of artificial intelligence generated content (AIGC), because AIGC systems usually rely on a large amount of data for learning and generation. If the data itself contains errors or false information, the generated response content may also be affected. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a retrieval method, system, device and medium based on web page search and knowledge graph. By combining the structured knowledge advantages of knowledge graph and the flexibility of web page search, multi-stage retrieval technology and self-evaluation mechanism are introduced, which significantly improves the response accuracy and reliability of large language models in complex queries.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, the present application provides a retrieval method based on web page search and knowledge graph, including: Get the input question, use the large language model to parse the question and classify the domain, determine the domain category of the question, and generate domain features; Using multi-stage retrieval of web search, combined with sparse retrieval and dense retrieval techniques, relevant documents are retrieved from the Internet according to the input question, and a candidate document pool is constructed as the first retrieval result; According to the input question, a structured query is performed on the knowledge graph to extract entity and relationship information related to the question as the second search result; According to the domain characteristics, the corresponding information change rate is calculated, and the first search result and the second search result are adaptively integrated according to the information change rate to generate a target answer; The large language model is used to evaluate the confidence of the target answer, the target answer is verified according to the confidence, and the output content is determined based on the verification result.

[0008] In an optional implementation, the step of obtaining an input question, performing question analysis and domain classification using a large language model, determining the domain category of the question, and generating domain features includes: Get the input question, extract the preset punctuation marks in the question, convert the uppercase letters in the question into lowercase letters, and perform word segmentation on the question; Input the processed question into the large language model, use the large language model to parse the semantic structure, keywords and context information of the question, and generate the parsing result; Based on the parsing results, use the classification algorithm or the built-in classification function of the large language model to classify the problem into a specific field, determine the field category of the problem, and generate field features.

[0009] In an optional embodiment, the multi-stage retrieval using web page search, combined with sparse retrieval and dense retrieval techniques, retrieves relevant documents from the Internet according to the input question, and constructs a candidate document pool as the first retrieval result, including: Retrieve candidate documents related to the question from the Internet using sparse retrieval technology; A dense retrieval method is adopted, and an embedding model is used to evaluate the semantic similarity of the candidate documents, and the most relevant documents are selected; the selected most relevant documents are further sorted using a re-ranking model, and a preset number of most relevant documents are screened out to construct a candidate document pool as the first retrieval result.

[0010] In an optional implementation, the step of retrieving candidate documents related to the question from the Internet using a sparse retrieval technique includes: Use HTML parsing library to parse web documents related to the question; Split the parsed document into multiple document chunks; For the input question, the BM25 algorithm is used to calculate the similarity with the document block, and the K document blocks with the highest scores are selected as K candidate documents. The specific calculation formula is as follows:

[0011] in, is a term in the query Q, It is a term The inverse document frequency of It is a term In the documentation The word frequency in is the video saturation parameter, b is the length normalization parameter, It is a document Length, is the average length of all documents in the corpus.

[0012] In an optional embodiment, the dense retrieval method is used to evaluate the semantic similarity of the candidate documents using an embedding model and select the most relevant documents; the most relevant documents are further sorted using a re-ranking model to screen out a preset number of most relevant documents and construct a candidate document pool as the first retrieval result, including: Perform dense embedding retrieval on the selected K candidate documents and use the embedding model to calculate the semantic similarity between the question and each candidate document; Select the top N candidate documents as the most relevant documents based on semantic similarity; Re-score the selected N most relevant documents using the bge-reranker-large model; According to the re-scoring results, the re-scoring is sorted in descending order, and the most relevant documents corresponding to the first M re-scoring are selected and stored in the candidate document pool as the first retrieval result.

[0013] In an optional implementation, the calculating of the corresponding information change rate according to the domain characteristics, and adaptively integrating the first search result and the second search result according to the information change rate to generate the target answer includes: According to the characteristics of the field, determine the status of the new knowledge base and the status of the old knowledge base in the field, and calculate the information change rate by the following formula :

[0014] in, is the state of the new knowledge base, is the state of the old knowledge base, T is the time interval; Determine the rate of change of information Whether it is greater than a preset judgment threshold; If so, generate a target answer based on the first search result and the key information in the second search result; If not, then based on the second search result, combined with the latest candidate document in the first search result, a target answer is generated.

[0015] In an optional implementation, the step of evaluating the confidence of the target answer using the large language model, verifying the target answer according to the confidence, and determining the output content based on the verification result includes: Use a large language model to evaluate the confidence of the target answer; Determine whether the confidence is greater than the confidence threshold; If so, output the corresponding target answer; If not, a prompt message indicating that the retrieval failed is output.

[0016] In a second aspect, the embodiment of the present application further provides a retrieval system based on web page search and knowledge graph, including: The query parsing and classification module is used to obtain input questions, use the large language model to perform question parsing and domain classification, determine the domain category of the question, and generate domain features; A web page retrieval module is used to use multi-stage retrieval of web page search, combining sparse retrieval and dense retrieval technology, to retrieve relevant documents from the Internet according to the input question, and to construct a candidate document pool as the first retrieval result; The knowledge graph query module is used to perform a structured query on the knowledge graph based on the input question, and extract entity and relationship information related to the question as the second search result; An adaptive integration module, used to calculate the corresponding information change rate according to the domain characteristics, and adaptively integrate the first search result and the second search result according to the information change rate to generate a target answer; The self-assessment module is used to evaluate the confidence of the target answer using a large language model, verify the target answer according to the confidence, and determine the output content based on the verification result.

[0017] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the retrieval method based on web page search and knowledge graph as described in any one of the above items are implemented.

[0018] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the retrieval method based on web page search and knowledge graph as described in any of the above items are implemented.

[0019] It can be seen from the above technical solutions that the present invention has the following advantages: In the retrieval method based on web search and knowledge graph provided in this application, a large language model is first used for field classification to assign the input query to a specific knowledge graph field. Subsequently, based on the domain-specific knowledge graph, a structured query is generated to retrieve accurate information. At the same time, combined with multi-stage web retrieval technology, sparse and dense retrieval methods are used to collect relevant information from a wide range of web pages to construct a rich set of candidate answers. In order to optimize the quality of candidate documents, this method also designs a self-evaluation mechanism to evaluate the confidence of the generated answers through a large model to ensure the accuracy of the final output. Finally, a comprehensive, rich and high-confidence response is generated by combining the results of the knowledge graph and web search. This method has the advantages of simplicity, efficiency, simple maintenance, and high accuracy. It can give high-quality responses to complex queries in different fields, significantly improving the accuracy and reliability of large language models (LLMs) in complex query responses.

[0020] This application implements an adaptive retrieval enhancement generation method that integrates web search and knowledge graph, integrates the wide coverage of web search and the structured advantages of knowledge graph, and ensures the accuracy and reliability of complex query responses through intelligent query parsing and multi-stage information retrieval. After receiving the user's complex query, this application first uses a large language model to classify questions and identify fields, and generates structured query requests that are highly relevant to the question. Subsequently, two major processes, web search and knowledge graph retrieval, are started in parallel: the web search process uses efficient multi-stage retrieval technology to filter and aggregate relevant information from massive web pages, while the knowledge graph process directly retrieves accurate entity and relationship information from a structured knowledge base. The two processes complement each other to form a comprehensive and in-depth set of candidate answers. Finally, combined with generative ranking technology and the deep understanding ability of large language models, the candidate answers are intelligently evaluated and integrated to generate comprehensive and accurate responses. This method not only improves the efficiency and accuracy of retrieval, but also significantly enhances the practicality and reliability of large language models in complex query tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 A schematic diagram of the flow of the retrieval method based on web page search and knowledge graph provided in this application.

[0023] Figure 2 A schematic diagram of the structure of the retrieval system based on web page search and knowledge graph provided for this application.

[0024] Figure 3 An architectural flow chart of the retrieval system based on web page search and knowledge graph provided for this application.

[0025] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0026] In the specific steps of the retrieval method based on web page search and knowledge graph, which will be described in detail below, various embodiments of the present disclosure will be described more comprehensively. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0027] Hereinafter, the terms "include" or "may include" used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include", "have", and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or a combination of the foregoing, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components, or a combination of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or a combination of the foregoing.

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] See also Figure 1 The figure is a flowchart of a method for a retrieval method based on web page search and knowledge graph in a specific embodiment, the method comprising: S1: Get the input question, use the large language model to parse the question and classify the domain, determine the domain category of the question, and generate domain features.

[0030] In this step, we first obtain the input question, extract the preset punctuation marks in the question, convert the uppercase letters in the question into lowercase letters, and perform word segmentation on the question; then, we input the processed question into the large language model, and use the large language model to parse the semantic structure, keywords, and contextual information of the question to generate the parsing result; finally, based on the parsing result, we use the classification algorithm or the built-in classification function of the large language model to classify the question into a specific field, determine the field category of the question, and generate the field feature.

[0031] For example, the input question is first parsed using a large language model (LLM) and classified into specific fields (such as finance, music, movies, etc.). This process ensures that subsequent processing can be optimized for the characteristics of specific fields.

[0032] It should be noted that the large language model is a powerful artificial intelligence model with excellent text generation and comprehension capabilities. By training a large amount of text data, it can learn the patterns and regularities of language and generate coherent and reasonable answers or texts based on the input prompts or questions. Large language models have been widely used in fields such as machine translation, automatic summarization, dialogue systems, and text generation. Retrieval enhancement technology combines traditional information retrieval technology with advanced language generation models to enable large language models to access and utilize the latest and relevant external information to generate answers. Retrieval enhancement generation technology first uses the retrieval component to query a large number of documents, and then assists the language model in generating more accurate, comprehensive, and informative responses based on the retrieved information.

[0033] S2: Use multi-stage retrieval of web search, combining sparse retrieval and dense retrieval techniques, to retrieve relevant documents from the Internet according to the input question and build a candidate document pool as the first retrieval result.

[0034] In this step, first, sparse retrieval technology is used to retrieve candidate documents related to the question from the Internet; then a dense retrieval method is used to evaluate the semantic similarity of the candidate documents using an embedding model, and the most relevant documents are selected; finally, a re-ranking model is used to further rank the selected most relevant documents, and a preset number of the most relevant documents are screened out to construct a candidate document pool as the first retrieval result.

[0035] For example, this step implements multi-stage retrieval using web search. First, sparse retrieval techniques (such as the BM25 algorithm) are used to retrieve candidate documents related to the question from the Internet. Then, dense retrieval methods are used to evaluate the semantic similarity of candidate documents using an embedding model and select the most relevant documents. Finally, the selected documents are further ranked using a re-ranking model to ensure the accuracy of the answer.

[0036] S3: Based on the input question, perform a structured query on the knowledge graph and extract entity and relationship information related to the question as the second search result.

[0037] The purpose of this step is to perform structured query on the knowledge graph. For example, based on the query question input by the user, a structured query statement is generated, the structured query is executed, and the accurate and latest entity and relationship information related to the query is extracted as the second search result.

[0038] S4: According to the domain characteristics, the corresponding information change rate is calculated, and the first search result and the second search result are adaptively integrated according to the information change rate to generate a target answer.

[0039] In a specific implementation, the state of the new knowledge base and the state of the old knowledge base in the field are determined according to the field characteristics, and the information change rate is calculated by the following formula: :

[0040] in, is the state of the new knowledge base, is the state of the old knowledge base, T is the time interval; Determine the rate of change of information Whether it is greater than a preset judgment threshold; If so, generate a target answer based on the first search result and the key information in the second search result; If not, then based on the second search result, combined with the latest candidate document in the first search result, a target answer is generated.

[0041] It can be seen that this step realizes the adaptive integration of query results of web search and knowledge graph. According to the domain characteristics of the question and the rate of information change, the retrieval results that mainly rely on web search or knowledge graph are intelligently selected. For stable domains, more reliance is placed on knowledge graph; for rapidly changing domains, the latest web content is combined. Combining the retrieval results of web search and knowledge graph, the system generates comprehensive and accurate answers.

[0042] For example, based on the domain relevance score and information change rate, the results of web page retrieval and knowledge graph query are adaptively fused. For domains with a small information change rate (such as encyclopedias), the knowledge graph is mainly used. For domains with a large information change rate (such as news and trends), the latest web page retrieval results are combined. Finally, the LLM is used to generate the final answer based on the fused results.

[0043] S5: Use the large language model to evaluate the confidence of the target answer, verify the target answer based on the confidence, and determine the output content based on the verification result.

[0044] In a specific implementation, the confidence of the target answer is first evaluated using a large language model, and then it is determined whether the confidence is greater than a confidence threshold; if so, the corresponding target answer is output; if not, a prompt message indicating retrieval failure is output.

[0045] For example, during the answer generation process, the confidence (high, medium, low) of the generated answer is evaluated by LLM. According to the set confidence threshold, the high-confidence answers are screened out. If the confidence of LLM is lower than the confidence threshold, "I don't know" will be output to ensure the reliability of the answer. If the confidence of LLM is higher than the confidence threshold, the target answer is presented to the user.

[0046] In this embodiment, a retrieval method based on web page search and knowledge graph is disclosed, which creatively combines the advantages of web page search and knowledge graph, and combines with the generation ability of large language models to meet the challenges of complex queries. In this method, the questions input by the user are first accurately classified into different knowledge fields through an intelligent field classifier, and then the structured knowledge base of the knowledge graph is used to generate accurate structured queries for each field. At the same time, the web page search works in parallel, and through multi-stage retrieval technology, combined with sparse and dense retrieval methods, relevant documents are efficiently collected from massive web pages to build a comprehensive and accurate candidate document library. On this basis, an adaptive integration framework is introduced to intelligently combine web page-based and knowledge graph-based retrieval methods according to the characteristics of different fields and the rate of information change, and generate the final answer. This process not only ensures the accuracy and reliability of the answer, but also generates a detailed and structured long-form response through the accuracy of the knowledge graph and the extensiveness of the web page search, thereby significantly improving the response quality and user experience of complex queries.

[0047] In an embodiment of the present invention, based on step S2, a possible embodiment is given below to illustrate its specific implementation scheme in a non-limiting manner.

[0048] In step S2, the efficiency and accuracy of information retrieval are effectively balanced by combining sparse and dense retrieval techniques. The specific process is as follows: First, use the HTML parsing library to parse the webpage content related to the query. Split the parsed document into multiple paragraphs or tag-level blocks. For the user's query Q, use the BM25 algorithm to calculate the similarity with the document block, and select the K document blocks with the highest scores as candidate documents. The specific calculation formula is as follows:

[0049] in, is a term in the query Q, It is a term The inverse document frequency of It is a term In the documentation The word frequency in is the video saturation parameter, b is the length normalization parameter, It is a document Length, is the average length of all documents in the corpus.

[0050] The second stage is intensive retrieval. The K candidate documents selected in the first stage are subjected to intensive embedding retrieval, and the semantic similarity between the query Q and each candidate document is calculated using the embedding model. The top N most relevant documents are selected based on the semantic similarity.

[0051] The last step is re-ranking. The re-ranking model (bge-reranker-large model is used in this method) is used to re-score the N most relevant documents selected by the intensive search. Based on the re-scoring results, the top M most relevant documents are selected and stored in the candidate document pool as the first search result.

[0052] In order to facilitate understanding of the retrieval method based on web page search and knowledge graph disclosed in the present invention, the following takes the user input "Can I eat sea cucumbers if I have cardiovascular disease" as an example to specifically illustrate the execution process of the method: 1. First, the Large Language Model (LLM) is used to parse the input question "Can I eat sea cucumbers if I have cardiovascular disease?" LLM classifies the query into the medical and health field based on the keywords of the question, such as "cardiovascular disease" and "sea cucumber". This classification ensures that subsequent processing can be optimized for the characteristics of the medical and health field.

[0053] 2. First, we use sparse retrieval techniques such as the BM25 algorithm to retrieve candidate documents related to cardiovascular disease and sea cucumbers from the Internet. Then, we use a dense retrieval method based on an embedding model to evaluate the semantic similarity between the candidate documents and the query and select the most relevant documents. We use a re-ranking model to further rank the selected documents to ensure that the final selected documents have the highest accuracy and relevance in answering the question "Can I eat sea cucumbers if I have cardiovascular disease?"

[0054] 3. Generate structured query statements based on user queries, such as querying for items related to dietary recommendations for patients with cardiovascular disease. The system performs structured queries and extracts accurate and up-to-date medical information related to the query from the knowledge graph, such as the effects of sea cucumbers on patients with cardiovascular disease.

[0055] 4. Intelligently select retrieval results that mainly rely on web searches or knowledge graphs based on the characteristics and information change rate of the medical and health field. For queries in the medical field, more reliance may be placed on authoritative medical information in the knowledge graph, while combining the latest web content to generate comprehensive and accurate answers.

[0056] 5. During the answer generation process, the system requires LLM to evaluate the confidence of the generated answer. The system presents the answer to the user based on the confidence assessment of LLM. If the confidence assessment is "high", the system may output "I don't know" or prompt the user to consult a professional medical staff to ensure the reliability and safety of the answer.

[0057] like Figure 2 As shown, the following is an embodiment of the retrieval system based on web page search and knowledge graph provided by the embodiment of the present disclosure. The system and the retrieval method based on web page search and knowledge graph in the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the retrieval system based on web page search and knowledge graph, please refer to the embodiment of the retrieval method based on web page search and knowledge graph mentioned above.

[0058] A retrieval system based on web page search and knowledge graph, comprising: a query parsing and classification module 1, a web page retrieval module 2, a knowledge graph query module 3, an adaptive integration module 4 and a self-assessment module 5.

[0059] The query parsing and classification module 1 is used to obtain input questions, use a large language model to perform question parsing and domain classification, determine the domain category of the question, and generate domain features.

[0060] The web page retrieval module 2 is used to use multi-stage retrieval of web page search, combining sparse retrieval and dense retrieval technology, to retrieve relevant documents from the Internet according to the input question, and to construct a candidate document pool as the first retrieval result.

[0061] The knowledge graph query module 3 is used to perform a structured query on the knowledge graph based on the input question, and extract entity and relationship information related to the question as the second search result.

[0062] The adaptive integration module 4 is used to calculate the corresponding information change rate according to the domain characteristics, and adaptively integrate the first search result and the second search result according to the information change rate to generate a target answer.

[0063] The self-assessment module 5 is used to evaluate the confidence of the target answer using the large language model, verify the target answer according to the confidence, and determine the output content based on the verification result.

[0064] In a specific embodiment, Figure 3As shown, this embodiment discloses a retrieval system based on web page search and knowledge graph, which can be regarded as an adaptive retrieval enhancement generation system integrating web page search and knowledge graph from a functional perspective, including: query parsing and classification module 1, web page retrieval module 2, knowledge graph query module 3, adaptive integration module 4 and self-assessment module 5.

[0065] Among them, the query parsing and classification module 1 is responsible for parsing and classifying the complex queries input by the user into different fields, providing a basis for subsequent processing. The web page retrieval module 2 uses sparse and dense retrieval technology to efficiently retrieve relevant documents from the web page to ensure the comprehensiveness and accuracy of the information. The knowledge graph query module 3 retrieves accurate information from the knowledge graph through structured queries for specific fields. The adaptive integration module 4 intelligently combines the web page-based and knowledge graph-based retrieval methods according to the characteristics and information change rates of different fields to achieve the best retrieval effect, and combines the retrieved web page content and knowledge graph information to generate the final answer. The self-assessment module 5 evaluates the confidence of the answer and performs differentiated processing for answers with different confidence levels to improve the accuracy of questions and answers. These modules work together to form the retrieval system based on web page search and knowledge graph of the present invention, which significantly improves the accuracy and reliability of large language models in answering complex queries.

[0066] The retrieval system based on web search and knowledge graph provided in this embodiment significantly improves the accuracy and reliability of the retrieval enhancement generation system by integrating the extensiveness of web search and the structured advantages of knowledge graph. The system first uses a large language model to perform problem analysis and domain classification to determine the domain category of the problem. Subsequently, the multi-stage retrieval module based on web search, combined with sparse retrieval and dense retrieval technology, efficiently retrieves relevant documents from massive web pages and builds a candidate document pool. At the same time, the knowledge graph module extracts accurate and up-to-date entity and relationship information from the knowledge graph through structured queries. Finally, the adaptive integration module intelligently combines the retrieval results of web search and knowledge graph according to domain characteristics and information change rate to generate comprehensive and accurate responses. In addition, the system also introduces a self-assessment mechanism to evaluate the confidence of the generated answers, further reduce "illusion" content, and improve the overall response quality.

[0067] Figure 4 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0068] The retrieval method based on web page search and knowledge graph provided in the embodiment of the present application can be applied to electronic devices. It will be appreciated by those skilled in the art that the electronic device structure involved in the embodiment of the present invention does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown, or combine certain components, or arrange different components. In an embodiment of the present invention, the electronic device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0069] The electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging management module, a power management module, a battery, a wireless communication module, an audio module, a speaker, a microphone, a sensor module, buttons, a camera, a display, and a SIM card interface, etc.

[0070] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors.

[0071] The processor can be the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0072] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. The memory may store instructions or data that the processor has just used or is cyclically used. If the processor needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves system efficiency.

[0073] The external memory interface can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement data storage functions. For example, files such as music and videos can be saved in the external memory card.

[0074] The internal memory can be used to store computer executable program codes, which include instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. The internal memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0075] The wireless communication function of an electronic device can be realized through an antenna, a wireless communication module, a modem processor, and a baseband processor.

[0076] The wireless communication module can provide wireless communication solutions for electronic devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0077] Electronic devices can implement audio functions, etc. through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0078] Electronic devices can achieve shooting functions through ISP, camera, video codec, GPU, display and application processor.

[0079] Electronic devices can achieve display functions through GPU, display screen and application processor.

[0080] The GPU is a microprocessor for image processing that connects the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs that execute program instructions to generate or change display information.

[0081] The display screen is used to display images, videos, etc. The display screen includes a display panel.

[0082] The above-mentioned electronic device implements the retrieval method based on web search and knowledge graph of the present application by integrating the wide coverage of web search and the structured advantages of knowledge graph, and at the same time uses the large language model for in-depth problem analysis and field classification, which can accurately locate the field scope of user needs and generate targeted field features. Through a multi-stage retrieval strategy, combined with sparse and dense retrieval techniques, a high-quality candidate document pool is effectively constructed. In addition, structured queries are performed using knowledge graphs to further explore entity and relationship information closely related to the problem. By calculating the information change rate and adaptively integrating the first retrieval result and the second retrieval result, a more accurate target answer is generated. Finally, the confidence assessment of the large language model provides a strong guarantee for the reliability of the answer. This series of innovative measures has greatly improved the retrieval efficiency and answer quality, and achieved comprehensive optimization of the user experience. The storage medium provided in the present application stores a program product that can implement a retrieval method based on web page search and knowledge graph.

[0083] The retrieval method based on web search and knowledge graph includes: obtaining an input question, using a large language model to perform question analysis and domain classification, determining the domain category of the question, and generating domain features; using multi-stage retrieval of web search, combining sparse retrieval and dense retrieval technology, retrieving relevant documents from the Internet according to the input question, and constructing a candidate document pool as the first retrieval result; performing structured query of the knowledge graph according to the input question, extracting entity and relationship information related to the question as the second retrieval result; calculating the corresponding information change rate according to the domain features, and adaptively integrating the first retrieval result and the second retrieval result according to the information change rate to generate a target answer; using a large language model to evaluate the confidence of the target answer, verifying the target answer according to the confidence, and determining the output content based on the verification result.

[0084] In some possible implementations, the retrieval method based on web page search and knowledge graph disclosed herein can be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0085] The storage medium of the present disclosure can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0086] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A retrieval method based on web page search and knowledge graph, characterized in that: include: Get the input question, use the large language model to parse the question and classify the domain, determine the domain category of the question, and generate domain features; Using multi-stage retrieval of web search, combined with sparse retrieval and dense retrieval techniques, relevant documents are retrieved from the Internet according to the input question, and a candidate document pool is constructed as the first retrieval result; According to the input question, a structured query is performed on the knowledge graph to extract entity and relationship information related to the question as the second search result; According to the domain characteristics, the corresponding information change rate is calculated, and the first search result and the second search result are adaptively integrated according to the information change rate to generate a target answer; The large language model is used to evaluate the confidence of the target answer, the target answer is verified according to the confidence, and the output content is determined based on the verification result.

2. The retrieval method based on web page search and knowledge graph according to claim 1, characterized in that: The input question is obtained, and the large language model is used to perform question analysis and domain classification, determine the domain category of the question, and generate domain features, including: Get the input question, extract the preset punctuation marks in the question, convert the uppercase letters in the question into lowercase letters, and perform word segmentation on the question; Input the processed question into the large language model, use the large language model to parse the semantic structure, keywords and context information of the question, and generate the parsing result; Based on the parsing results, use the classification algorithm or the built-in classification function of the large language model to classify the problem into a specific field, determine the field category of the problem, and generate field features.

3. The retrieval method based on web page search and knowledge graph according to claim 2, characterized in that: The multi-stage retrieval using web page search, combining sparse retrieval and dense retrieval technology, retrieves relevant documents from the Internet according to the input question, and constructs a candidate document pool as the first retrieval result, including: Retrieve candidate documents related to the question from the Internet using sparse retrieval technology; A dense retrieval method is adopted, and an embedding model is used to evaluate the semantic similarity of the candidate documents, and the most relevant documents are selected; the selected most relevant documents are further sorted using a re-ranking model, and a preset number of most relevant documents are screened out to construct a candidate document pool as the first retrieval result.

4. The retrieval method based on web page search and knowledge graph according to claim 3 is characterized in that: The method of retrieving candidate documents related to the question from the Internet using sparse retrieval technology includes: Use HTML parsing library to parse web documents related to the question; Split the parsed document into multiple document chunks; For the input question, the BM25 algorithm is used to calculate the similarity with the document block, and the K document blocks with the highest scores are selected as K candidate documents. The specific calculation formula is as follows: in, is a term in the query Q, It is a term The inverse document frequency of It is a term In the documentation The word frequency in is the video saturation parameter, b is the length normalization parameter, It is a document Length, is the average length of all documents in the corpus.

5. The retrieval method based on web page search and knowledge graph according to claim 4 is characterized in that: The method adopts a dense retrieval method, uses an embedding model to evaluate the semantic similarity of the candidate documents, and selects the most relevant documents; uses a re-ranking model to further rank the selected most relevant documents, screens out a preset number of most relevant documents, and constructs a candidate document pool as the first retrieval result, including: Perform dense embedding retrieval on the selected K candidate documents and use the embedding model to calculate the semantic similarity between the question and each candidate document; Select the top N candidate documents as the most relevant documents based on semantic similarity; Re-score the selected N most relevant documents using the bge-reranker-large model; According to the re-scoring results, the re-scoring is sorted in descending order, and the most relevant documents corresponding to the first M re-scoring are selected and stored in the candidate document pool as the first retrieval result.

6. The retrieval method based on web page search and knowledge graph according to claim 5, characterized in that: The method of calculating the corresponding information change rate according to the domain characteristics, and adaptively integrating the first search result and the second search result according to the information change rate to generate a target answer includes: According to the characteristics of the field, determine the status of the new knowledge base and the status of the old knowledge base in the field, and calculate the information change rate by the following formula : in, is the state of the new knowledge base, is the state of the old knowledge base, T is the time interval; Determine the rate of change of information Whether it is greater than a preset judgment threshold; If so, generate a target answer based on the first search result and the key information in the second search result; If not, then based on the second search result, combined with the latest candidate document in the first search result, a target answer is generated.

7. The retrieval method based on web page search and knowledge graph according to claim 6, characterized in that: The method of evaluating the confidence of the target answer by using the large language model, verifying the target answer according to the confidence, and determining the output content based on the verification result includes: Use a large language model to evaluate the confidence of the target answer; Determine whether the confidence is greater than the confidence threshold; If so, output the corresponding target answer; If not, a prompt message indicating that the retrieval failed is output.

8. A retrieval system based on web page search and knowledge graph, characterized in that: The system adopts the retrieval method based on web page search and knowledge graph as described in any one of claims 1 to 7; The system comprises: The query parsing and classification module is used to obtain input questions, use the large language model to perform question parsing and domain classification, determine the domain category of the question, and generate domain features; A web page retrieval module is used to use multi-stage retrieval of web page search, combining sparse retrieval and dense retrieval technology, to retrieve relevant documents from the Internet according to the input question, and to construct a candidate document pool as the first retrieval result; The knowledge graph query module is used to perform a structured query on the knowledge graph based on the input question, and extract entity and relationship information related to the question as the second search result; An adaptive integration module, used to calculate the corresponding information change rate according to the domain characteristics, and adaptively integrate the first search result and the second search result according to the information change rate to generate a target answer; The self-assessment module is used to evaluate the confidence of the target answer using a large language model, verify the target answer according to the confidence, and determine the output content based on the verification result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the retrieval method based on web page search and knowledge graph as described in any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the retrieval method based on web page search and knowledge graph as described in any one of claims 1 to 7 are implemented.

Images