Retrieval result determination method and device, equipment and computer medium

Abstract

The invention discloses a retrieval result determination method and device, equipment and a computer medium. The method comprises the steps that at least one associated document associated with question information is determined according to the question information input by a user; determining a preliminary retrieval result corresponding to the question information based on the at least one associated document and the question information; and determining a target retrieval result corresponding to the question information according to the preliminary retrieval result. The effect of improving the accuracy of the retrieval result obtained when the corresponding retrieval result is fed back based on the question information of the user can be achieved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent question answering technology, and in particular, relates to a search result determination method, device, equipment and computer medium. Background Art

[0002] In the modern information society, knowledge base systems have become a key tool for enterprises and organizations to process and manage large amounts of information. These systems play a vital role in information storage and retrieval. However, most current knowledge base systems still have some shortcomings and challenges. For example, existing retrieval technologies often cannot accurately locate the documents corresponding to actual questions, resulting in retrieval results that are difficult to meet the actual needs of users. In addition, the current retrieval enhancement generation system (RAG, Retrieve Augment Generation) is relatively simple in processing user feedback and lacks intelligent feedback processing and optimization mechanisms. In related technologies, the accuracy of the corresponding retrieval results based on the user's question information feedback is low. Summary of the invention

[0003] The embodiment of the present application provides an implementation scheme different from the prior art to solve the technical problem in the related art that the accuracy of the corresponding search results is low when the corresponding search results are fed back based on the user's question information.

[0004] In a first aspect, the present application provides a method for determining a search result, comprising:

[0005] Determine at least one associated document associated with the question information according to the question information input by the user;

[0006] Determine a preliminary search result corresponding to the question information based on the at least one associated document and the question information;

[0007] A target search result corresponding to the question information is determined according to the preliminary search result.

[0008] In a second aspect, the present application provides a search result determination device, comprising:

[0009] An associated document determining unit, configured to determine at least one associated document associated with the question information according to the question information input by the user;

[0010] The retrieval result determination unit is further used to determine the preliminary retrieval result corresponding to the question information based on the at least one associated document and the question information; and determine the target retrieval result corresponding to the question information according to the preliminary retrieval result.

[0011] In a third aspect, the present application provides an electronic device, including:

[0012] Processor; and

[0013] A memory, configured to store executable instructions of the processor;

[0014] The processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0017] The present application provides a solution for determining at least one associated document associated with question information input by a user; determining a preliminary search result corresponding to the question information based on the at least one associated document and the question information; and determining a target search result corresponding to the question information based on the preliminary search result. The solution can determine the document associated with the question information based on the question information input by the user, thereby improving the accuracy of the search results when the corresponding search results are fed back based on the user's question information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1a A schematic diagram of the structure of a search result determination system provided in an embodiment of the present application;

[0020] Figure 1b A flowchart of a method for determining search results provided in an embodiment of the present application;

[0021] Figure 2a A flowchart of a method for determining search results provided in an embodiment of the present application;

[0022] Figure 2b A schematic diagram of a process of generating and storing keywords in a knowledge base provided by an embodiment of the present application;

[0023] Figure 2cA schematic diagram of a process for determining associated documents provided in an embodiment of the present application;

[0024] Figure 2d A schematic diagram of a data processing flow after determining an initial search result provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of the structure of a search result determination device provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0028] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0029] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0030] The knowledge base system is a system that uses the stored knowledge to interpret input data, generate operational hypotheses and verify them. It combines the research results of database systems and artificial intelligence, and realizes the effective storage and intelligent application of knowledge through the research of formal language and natural language processing.

[0031] RAG (Retrieval-Augmented Generation) is an artificial intelligence technology that combines information retrieval technology with language generation models. The core of the RAG technical framework is that when a large language model (LLM) faces the task of answering questions or creating text, it will first search and filter out materials closely related to the task in a large-scale document library, and then accurately guide the subsequent answer generation or text construction process based on these materials, aiming to improve the accuracy and reliability of the model output in this way.

[0032] In the modern information society, knowledge base systems have become a key tool for enterprises and organizations to process and manage large amounts of information. These systems play a vital role in information storage and retrieval. However, most current knowledge base systems still have some shortcomings and challenges. For example, existing retrieval technologies often cannot accurately locate the documents corresponding to actual questions, resulting in retrieval results that are difficult to meet the actual needs of users. In addition, the current retrieval enhancement generation system (RAG, Retrieve Augment Generation) is relatively simple in processing user feedback and lacks intelligent feedback processing and optimization mechanisms. In related technologies, the accuracy of the corresponding retrieval results based on the user's question information feedback is low.

[0033] In order to solve these problems, this application proposes a knowledge retrieval method based on keyword matching and multi-stage self-optimization feedback. By introducing a keyword relevance judgment mechanism and a multi-stage self-optimization feedback mechanism based on a large model, this method aims to improve the retrieval accuracy of the knowledge base and ensure that the generated answers gradually approach the optimal state, thereby improving the answer quality and the intelligence level of the system.

[0034] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0035] Figure 1aThe present application provides a structural diagram of a search result determination system, which may include a terminal 10 and a server 20. Optionally, the terminal 10 may be used to: in response to a user-triggered operation for displaying a search query system platform, display a page of the search query system platform, and after detecting that the user inputs question information and uploads a document to be analyzed, obtain the question information input by the user and multiple documents to be analyzed; and transmit the question information and multiple documents to be analyzed to the server 20, so that the server 20 performs the following steps: determining at least one associated document associated with the question information according to the question information input by the user and the multiple documents to be analyzed; determining a preliminary search result corresponding to the question information based on the at least one associated document and the question information; and determining a target search result corresponding to the question information according to the preliminary search result. After the server 20 determines the target search result, it may send the target search result to the terminal 10, so that the terminal 10 displays the target search result through the page of the search query system platform. Optionally, after the server determines the aforementioned at least one associated document, it may also send the at least one associated document to the terminal 10, so that the terminal 10 displays the at least one associated document through the page of the search query system platform.

[0036] Optionally, the aforementioned determining a target search result corresponding to the question information based on the preliminary search result includes: obtaining first evaluation information of a user for the preliminary search result; and determining the target search result corresponding to the question information based on the first evaluation information.

[0037] Figure 1b A flow chart of a method for determining retrieval results provided in the present application. In some optional embodiments, a user may first input question information into a search query system platform, and then the terminal 10 may send the question information to a server 20, so that the server 20 outputs associated documents associated with the question information based on keyword similarity, and then analyzes the question information and the associated documents based on a large language model, and performs multi-stage self-optimization to obtain a final target retrieval result.

[0038] In other optional embodiments, the user can input question information on the page of the search query system platform displayed by the terminal 10, and the terminal 10 can output related documents associated with the question information based on keyword similarity, and then analyze the question information and related documents based on the large language model, and perform multi-stage self-optimization to obtain the final target retrieval result.

[0039] Optionally, the aforementioned search query system can be a RAG system. When the user enters question information in RAG, the system first performs word segmentation on the user's question information and matches it with the keywords in the knowledge data (i.e., the document to be analyzed) entered by the user. If the keyword is matched, the corresponding document is directly output; if the keyword is not matched, the system will use the question and the document to perform vector similarity matching and output the document with the highest similarity. These documents are processed by the large language model (LLM) and provided to the user. Through the user's multi-stage scoring feedback and prompt word optimization, the answers generated by the large model can more accurately meet user needs, thereby improving the quality of the answers.

[0040] The solution of this application is mainly divided into the following two main processes:

[0041] Keyword-based relevance judgment mechanism: By constructing and judging keywords, it ensures that users can accurately find relevant documents when searching for questions, thereby improving the accuracy and efficiency of retrieval. The use of keywords enables the system to quickly match users' search needs and reduce the interference of irrelevant results. After uploading knowledge data, that is, documents to be analyzed, users can choose to generate keywords manually or by a large language model (LLM). When the user enters question information, the system uses these keywords to quickly locate relevant documents, thereby greatly improving the accuracy and efficiency of retrieval.

[0042] Multi-stage self-optimization feedback mechanism based on large models: After the user retrieves the question information and obtains a preliminary answer, the system provides scoring options (1 to 5 points), and the user scores based on the satisfaction with the answer. Each score corresponds to a different feedback level. The system generates corresponding prompt words based on the score to guide the large model to regenerate the answer. The user gives a second score and feedback to the newly generated answer. The system further optimizes the answer based on the secondary feedback, and finally decides whether to update the knowledge base through the ultimate feedback. The knowledge base can be used when the user searches for question information later to determine the information retrieval results of the user's input question. This multi-stage self-optimization feedback mechanism ensures that the generated answers gradually approach the optimal state through continuous feedback and optimization, thereby improving the quality of the answers and the intelligence level of the system.

[0043] The execution principle and interaction process of each module in the solution of the present application can be found in the description of the following method embodiments.

[0044] Figure 2a A flowchart of a search result determination method provided for an exemplary embodiment of the present application is provided. The execution subject of the method can be executed by the aforementioned terminal 10 or the aforementioned server 20, or can be executed by the aforementioned terminal 10 and the server 20 in cooperation with each other. The present application does not limit this. The method at least includes the following steps S201-S203:

[0045] S201, determining at least one associated document associated with the question information according to the question information input by the user;

[0046] In some optional embodiments of the present application, the solution of the present application may be applicable to a scenario of a search query system platform, and a user may input question information through a question information input page of the search query system platform.

[0047] Specifically, users can upload documents to be analyzed to the search query system platform.

[0048] The aforementioned at least one associated document associated with the question information refers to at least one document in which the corresponding keywords in the multiple documents to be analyzed contain subwords obtained after word segmentation of the question information.

[0049] Optionally, the user may input the aforementioned question information and multiple documents to be analyzed on the same page of the search query system platform.

[0050] Optionally, the user may input the aforementioned question information and the plurality of documents to be analyzed on different pages of the search query system platform. Optionally, the plurality of documents to be analyzed may be uploaded earlier than the input of the question information.

[0051] In some optional embodiments of the present application, in the aforementioned step S201, determining at least one associated document associated with the question information according to the question information input by the user includes the following steps S2011-S2012:

[0052] S2011, segmenting the question information input by the user to obtain multiple subwords;

[0053] Optionally, the word segmentation technology may be jieba word segmentation, and fine-grained word segmentation may better match keywords.

[0054] Optionally, when the question information is segmented, the granularity of the segmentation can be finer. For example, after segmenting "I came to ABCD University", we get: I / came to / AB / CD / ABCD / D University / University.

[0055] S2012: Determine at least one associated document associated with the question information based on the multiple sub-words and the multiple documents to be analyzed.

[0056] In some optional embodiments of the present application, in S2012, determining at least one associated document associated with the question information according to the multiple subwords and the multiple documents to be analyzed includes the following steps S221-S222:

[0057] S221, for each sub-word in the plurality of sub-words, determining from the plurality of documents to be analyzed a document to be analyzed in which a keyword matching the sub-word exists in a corresponding keyword, obtaining a document group to be analyzed corresponding to the sub-word, and further obtaining a plurality of document groups to be analyzed corresponding to the plurality of sub-words;

[0058] Optionally, the specific matching technology mainly adopts full-mode word segmentation matching.

[0059] Optionally, the keyword and the subword are the same, and it is regarded that the keyword and the subword match.

[0060] Optionally, for each sub-word, it is possible to determine the documents to be analyzed that include the current sub-word in the corresponding keywords. For each sub-word, the number of documents to be analyzed that include the current sub-word in the corresponding keywords can be one or more. The one or more documents to be analyzed constitute the document group to be analyzed corresponding to the current sub-word.

[0061] Optionally, for some sub-words, the number of documents to be analyzed that include the current sub-word in the corresponding keyword may also be zero.

[0062] Each document to be analyzed corresponds to one or more keywords.

[0063] In some optional embodiments, the user may also upload only one document to be analyzed.

[0064] In some optional embodiments, regarding the method for determining the keywords corresponding to the document to be analyzed, the method further includes: for each document to be analyzed, when detecting that a user inputs an input instruction for the keywords corresponding to the document to be analyzed, setting the keywords corresponding to the document to be analyzed based on the input instruction. The input instruction may include the keywords of the document to be analyzed.

[0065] In other optional embodiments, regarding the method of determining the keywords corresponding to the document to be analyzed, the above method also includes: for each document to be analyzed, in response to a user's instruction to generate keywords corresponding to the document to be analyzed using a keyword generation model, the document to be analyzed is input into the keyword generation model to obtain the keywords corresponding to the document to be analyzed.

[0066] Optionally, when the user uses the keyword generation model to generate keywords corresponding to the document to be analyzed, the keywords corresponding to the document to be analyzed can be repeatedly generated according to relevant instructions of the user until the user is satisfied.

[0067] Further, see Figure 2b As shown, Figure 2b A schematic diagram of the process of generating and storing keywords provided in this application into a knowledge base.

[0068] S222: Determine at least one associated document associated with the question information based on the documents to be analyzed included in the plurality of document groups to be analyzed.

[0069] Optionally, the number of documents to be analyzed included in each group of documents to be analyzed may be 0 or may not be 0.

[0070] Specifically, determining at least one associated document associated with the problem information based on the documents to be analyzed included in the multiple document groups to be analyzed includes: when the number of documents to be analyzed included in the multiple document groups to be analyzed is not 0, taking all the documents to be analyzed included in the multiple document groups to be analyzed as at least one associated document associated with the problem information.

[0071] Optionally, in all the documents to be analyzed included in the multiple document groups to be analyzed, the same document to be analyzed may appear once or multiple times. The more times it appears, the more keywords in the keywords corresponding to the document to be analyzed match the subwords.

[0072] In some optional embodiments of the present application, determining at least one associated document associated with the question information based on the documents to be analyzed included in the plurality of document groups to be analyzed includes:

[0073] When the number of the documents to be analyzed included in the plurality of document groups to be analyzed is 0, for each document to be analyzed in the plurality of documents to be analyzed, performing vector similarity matching between the document to be analyzed and the question information to obtain a similarity value between the document to be analyzed and the question information;

[0074] Among the multiple documents to be analyzed, a document to be analyzed that has the highest similarity value with the question information is used as at least one associated document associated with the question information.

[0075] The higher the similarity between the document to be analyzed and the question information, the higher the similarity value between the document to be analyzed and the question information.

[0076] Optionally, cosine similarity may be used to calculate the similarity value between the document to be analyzed and the question information.

[0077] In some other optional embodiments of the present application, determining at least one associated document associated with the question information based on the documents to be analyzed included in the plurality of document groups to be analyzed includes: when the number of the documents to be analyzed included in the plurality of document groups to be analyzed is 0, for each document to be analyzed in the plurality of documents to be analyzed, performing vector similarity matching between the document to be analyzed and the question information to obtain a similarity value between the document to be analyzed and the question information;

[0078] Among the multiple documents to be analyzed, a document to be analyzed which has the highest similarity value with the question information and a similarity value greater than a preset similarity threshold is used as at least one associated document associated with the question information.

[0079] Optionally, if among multiple documents to be analyzed, the number of documents to be analyzed whose similarity value with the question information is greater than a preset similarity threshold is 0, then null is returned.

[0080] Further, see Figure 2c , Figure 2c A schematic diagram of the process of determining associated documents provided for this application.

[0081] In some optional embodiments of the present application, the aforementioned at least one associated document may also be displayed in one of the pages of the search query system platform.

[0082] S202, determining a preliminary search result corresponding to the question information based on the at least one associated document and the question information;

[0083] In some optional embodiments of the present application, in step S202, determining the preliminary search result corresponding to the question information based on the at least one associated document and the question information includes the following steps S021-S022:

[0084] S021. Using the associated document with the largest number of occurrences among the at least one associated document as a target document;

[0085] When there are multiple associated documents with the largest number of occurrences, all of the multiple associated documents may be used as the target document, or any one of the multiple associated documents may be used as the target document.

[0086] S022. Determine a preliminary search result corresponding to the question information based on the target document and the question information.

[0087] In some optional embodiments of the present application, in S022, a preliminary search result corresponding to the question information is determined based on the target document and the question information, including: inputting the target document and the question information into a first preset model to obtain a preliminary search result corresponding to the question information.

[0088] Optionally, the aforementioned first preset model is a large language model.

[0089] The first preset model is used to generate a preliminary search result corresponding to the question information based on the input question information and the target document, wherein the preliminary search result refers to an answer result corresponding to the question information.

[0090] S203: Determine a target search result corresponding to the question information according to the preliminary search result.

[0091] Optionally, in the aforementioned step S203, determining a target search result corresponding to the question information according to the preliminary search result includes: using the preliminary search result as the target search result corresponding to the question information.

[0092] Optionally, in step S203, determining a target search result corresponding to the question information according to the preliminary search result includes the following steps S301-S302:

[0093] S301, obtaining first evaluation information of the user for the preliminary search result;

[0094] The first evaluation information is used to indicate the user's satisfaction with the preliminary search results.

[0095] In some optional embodiments of the present application, the first evaluation information may be rating information.

[0096] Different scores represent different meanings. The score range is 1 to 5. The higher the score, the higher the user's satisfaction with the initial search results. The details are as follows:

[0097] 1 point: The generated answer is seriously off-topic from the user's question.

[0098] 2 points: The generated answer does not match the user's needs.

[0099] 3 points: The generated answer basically meets the user's needs.

[0100] 4 points: The generated answer is good, but there is still room for improvement.

[0101] 5 points: The generated answer is perfect.

[0102] In some optional embodiments of the present application, after the preliminary search results are obtained and displayed, if the detection exceeds the preset time, the user does not rate, or does not ask the next question, the question process is terminated to reduce interference to the user and improve the user experience.

[0103] S302: Determine a target search result corresponding to the question information according to the first evaluation information.

[0104] In some optional embodiments of the present application, in step S302, determining the target search result corresponding to the question information according to the first evaluation information includes the following steps S3021-SS3022:

[0105] S3021. When the first evaluation information indicates that the score is less than a first preset score, determine first prompt information corresponding to the first evaluation information according to the first evaluation information; optionally, the first preset score is 4.

[0106] Optionally, the aforementioned determining the first prompt information corresponding to the first evaluation information based on the first evaluation information includes: determining the first prompt information based on the meaning of the first evaluation information, the question information, the preliminary retrieval result, the document name of the target document, the user's historical query information, and task information for indicating regeneration of the retrieval result corresponding to the question information.

[0107] In some optional embodiments of the present application, the first prompt information is determined based on the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and the task information used to indicate the regeneration of the retrieval results corresponding to the question information, including: using the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and the task information used to indicate the regeneration of the retrieval results corresponding to the question information as the first prompt information.

[0108] In some other optional embodiments of the present application, determining the first prompt information corresponding to the first evaluation information according to the first evaluation information includes: inputting the first evaluation information into a second preset model to obtain the first prompt information corresponding to the first evaluation information, and the second preset model may also be a large language model.

[0109] Optionally, the first prompt information includes: the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and task information for indicating the regeneration of the search results corresponding to the question information.

[0110] Optionally, the meaning of the first evaluation information may be set in advance based on the score indicated by the first evaluation information.

[0111] Optionally, the meaning of the first evaluation information is used to indicate the user's satisfaction with the preliminary search results.

[0112] Optionally, when the first evaluation information indicates that the score is 1, the first prompt information is: "The answer you gave seriously deviates from the actual needs of the user. Please reconsider answering the user's question from other directions. This is the user's question: {question} and the last answer: {last_answer}. Please regenerate the answer based on the document: {documents} and the chat history: {history}."

[0113] Optionally, when the first evaluation information indicates that the score is 2 points, the first prompt information is: "The answer you gave is not consistent with the user's requirement. Please think about the question carefully. This is the user's question: {question} and the last answer given: {last_answer}. Please regenerate the answer based on the document: {documents} and the chat history: {history}."

[0114] Optionally, when the first evaluation information indicates that the score is 3 points, the first prompt information is: "Your answer basically meets the user's needs, but it is not good enough, complete and accurate enough. This is the user's question: {question} and the last answer given: {last_answer}. Please regenerate the answer based on the document: {documents} and the chat history: {history}."

[0115] S3022. Determine a target search result corresponding to the question information according to the first prompt information.

[0116] In some optional embodiments of the present application, in the aforementioned step S3022, determining the target retrieval result corresponding to the question information based on the first prompt information includes: inputting the first prompt information into a third preset model to obtain the target retrieval result corresponding to the question information.

[0117] Among them, the third preset model can be a large language model, which is used to regenerate the search result corresponding to the question information, that is, the target search result, according to the input first prompt information.

[0118] In some optional embodiments of the present application, in step S302, determining the target retrieval result corresponding to the question information based on the first evaluation information also includes: when the first evaluation information indicates that the score is not less than a first preset score, taking the preliminary retrieval result as the target retrieval result.

[0119] Optionally, when the first evaluation information indicates that the score is not less than a first preset score and is the highest score used for scoring, the question information and the target search results are updated to the user's personal knowledge base to ensure the timeliness and accuracy of the knowledge base.

[0120] In some optional embodiments of the present application, after the target retrieval results are obtained and displayed, if the detection exceeds the preset time, the user does not score, or does not ask the next question, the current question process is terminated to reduce interference to the user and improve the user experience.

[0121] In some optional embodiments of the present application, in step S302, determining the target search result corresponding to the question information according to the first evaluation information includes the following steps S041-S044:

[0122] S041. When the score indicated by the first evaluation information is less than a first preset score, determining first prompt information corresponding to the first evaluation information according to the first evaluation information;

[0123] S042, determining the to-be-determined search result corresponding to the question information according to the first prompt information;

[0124] The search result to be determined is the search result generated for the second time for the question information.

[0125] S043, obtaining second evaluation information of the user for the search result to be determined;

[0126] S044. Determine a target search result corresponding to the question information according to the second evaluation information.

[0127] The second evaluation information is used to indicate the user's satisfaction with the search result to be determined.

[0128] Optionally, the aforementioned determining the first prompt information corresponding to the first evaluation information based on the first evaluation information includes: determining the first prompt information based on the meaning of the first evaluation information, the question information, the preliminary retrieval result, the document name of the target document, the user's historical query information, and task information for indicating regeneration of the retrieval result corresponding to the question information.

[0129] In some optional embodiments of the present application, the first prompt information is determined based on the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and the task information used to indicate the regeneration of the retrieval results corresponding to the question information, including: using the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and the task information used to indicate the regeneration of the retrieval results corresponding to the question information as the first prompt information.

[0130] In some other optional embodiments of the present application, determining the first prompt information corresponding to the first evaluation information according to the first evaluation information includes: inputting the first evaluation information into a second preset model to obtain the first prompt information corresponding to the first evaluation information, and the second preset model may also be a large language model.

[0131] Optionally, the first prompt information includes: the meaning of the first evaluation information, the question information, the preliminary search results, the document name of the target document, the user's historical query information, and task information for indicating the regeneration of the search results corresponding to the question information.

[0132] In some optional embodiments of the present application, in the aforementioned step S042, determining the to-be-determined retrieval result corresponding to the question information according to the first prompt information includes: inputting the first prompt information into a fourth preset model to obtain the to-be-determined retrieval result corresponding to the question information.

[0133] Among them, the fourth preset model can be a large language model, which is used to regenerate the search result corresponding to the question information according to the input first prompt information, that is, the search result to be determined.

[0134] In some optional embodiments of the present application, in step S044, determining the target search result corresponding to the question information according to the second evaluation information includes the following steps S41-S42:

[0135] S41, when the second evaluation information indicates that the current score has decreased relative to the previous score, or the second evaluation information indicates that the current score has increased relative to the previous score but the current score is less than the first preset score, or the current score is the same as the previous score, determining second prompt information corresponding to the second evaluation information according to the second evaluation information;

[0136] S42: Determine a target search result corresponding to the question information according to the second prompt information.

[0137] Optionally, the aforementioned determining the second prompt information corresponding to the second evaluation information based on the second evaluation information includes: determining the second prompt information based on the meaning of the second evaluation information, the question information, the retrieval result to be determined, the previous score, the preliminary retrieval result, the document name of the target document, the user's historical query information, the change of the second evaluation information relative to the first evaluation information, and task information for indicating the regeneration of the retrieval result corresponding to the question information.

[0138] In some optional embodiments of the present application, the aforementioned steps determine the second prompt information based on the meaning of the second evaluation information, the question information, the retrieval result to be determined, the previous score, the preliminary retrieval result, the document name of the target document, the user's historical query information, the change of the second evaluation information relative to the first evaluation information, and the task information used to indicate the regeneration of the retrieval result corresponding to the question information, including: using the meaning of the second evaluation information, the question information, the retrieval result to be determined, the previous score, the preliminary retrieval result, the document name of the target document, the user's historical query information, the change of the second evaluation information relative to the first evaluation information, and the task information used to indicate the regeneration of the retrieval result corresponding to the question information as the second prompt information.

[0139] Optionally, the second prompt information may also include a prompt indicating that the higher the score, the better.

[0140] In some other optional embodiments of the present application, determining second prompt information corresponding to the second evaluation information according to the second evaluation information includes: inputting the second evaluation information into a fifth preset model to obtain second prompt information.

[0141] Among them, the fifth preset model can be a large language model, which is used to obtain the second prompt information according to the input second evaluation information.

[0142] Optionally, the second prompt information includes: the meaning of the second evaluation information, the question information, the retrieval results to be determined, the previous score, the preliminary retrieval results, the document name of the target document, the user's historical query information, the change of the second evaluation information relative to the first evaluation information, and task information for indicating the regeneration of the retrieval results corresponding to the question information.

[0143] In some optional embodiments of the present application, in step S044, determining the target search result corresponding to the question information according to the second evaluation information further includes:

[0144] When the second evaluation information indicates that the current score is improved relative to the previous score but the current score is not less than the first preset score, the to-be-determined search result is used as the target search result.

[0145] In some optional embodiments of the present application, in step S044, determining the target search result corresponding to the question information according to the second evaluation information further includes:

[0146] When the second evaluation information indicates that the current score is improved relative to the previous score, the current score is not less than the first preset score and is the highest score used for scoring, the to-be-determined retrieval result is used as the target retrieval result, and the target retrieval result is updated to the knowledge base.

[0147] In some optional embodiments of the present application, after the search results to be determined are obtained and displayed, if the detection exceeds the preset time, the user has not scored, or has not asked the next question, the question process will be ended to reduce interference to the user and improve the user experience.

[0148] Optionally, when the second evaluation information indicates that the current score has dropped relative to the previous score, the second prompt information may be: "The user has scored the answer you gave, from 1 to 5 points, the higher the better. The question is: {question}. The current score given to you is: {now_score}, and the answer is: {answer}. The last score was: {old_score}, and the previous answer was: {old_answer}. The score has dropped. Please regenerate your answer based on the document: {documents} combined with the chat history: {history} and the trend of the score decline.".

[0149] Optionally, when the current score is the same as the previous score, the second prompt information may be: the prompt word is "The user has scored the answer you gave, 1 to 5 points, the higher the better. The question is: {question}. The current score given to you is: {now_score}, and the answer is: {answer}. The last score was: {old_score}, and the previous answer was: {old_answer}. The score has not changed, and the answer does not meet the user's needs. Please consider changing direction and regenerating the answer based on the document: {documents} and the chat history: {history}.".

[0150] Optionally, when the second evaluation information indicates that the current score is improved relative to the previous score but the current score is less than the first preset score, the second prompt information is: "The user has scored the answer you gave, 1 to 5 points, the higher the better. The question is: {question}. The current score given to you is: {now_score}, and the answer is: {answer}. The last score was: {old_score}, and the previous answer was: {old_answer}. The score has increased, which means that the answer you gave is close to the user's real answer. Please follow the documents: {documents} and the chat history: {history}, continue the above ideas, and regenerate the answer.".

[0151] In some optional embodiments of the present application, after the target retrieval results are obtained and displayed, if the detection exceeds the preset time, the user does not score, or does not ask the next question, the current question process is terminated to reduce interference to the user and improve the user experience.

[0152] In some optional embodiments of the present application, the method further includes the following S1-S2:

[0153] S1. Obtaining third evaluation information of the user on the target search result;

[0154] S2. When the third evaluation information indicates that the current score is the highest, the target search result is updated to the knowledge base. When the user scores other scores, the current question process ends.

[0155] Further, in the solution of the present application, the data processing flow after determining the initial search results can also be found in Figure 2d shown.

[0156] The scheme of the present application proposes a knowledge retrieval method based on keyword matching and multi-stage self-optimization feedback. Through the relevance judgment of keywords and the multi-stage self-optimization feedback mechanism based on the large model, the retrieval accuracy and dynamic update capability of the knowledge base are improved. When searching for questions, users can accurately find relevant documents, thereby improving the accuracy and efficiency of retrieval. At the same time, the knowledge base can be continuously optimized and updated according to user feedback to maintain the timeliness and relevance of information. Through interaction with users, users are allowed to rate multiple feedbacks on the answers generated by the system, and a scoring mechanism of 1 to 5 points is used to achieve a refined evaluation and optimization of the quality of the answers. The system triggers the corresponding prompt words according to the user's rating to guide the large model to regenerate the answer. The user's secondary rating and feedback further optimize the answer quality, and finally realize an intelligent feedback optimization mechanism. This optimization strategy based on prompt words can effectively improve the quality and relevance of the generated answers, ensure that the system gradually approaches the optimal answer, thereby improving the answer quality, user satisfaction and system intelligence level, and can also improve the response quality and accuracy of the knowledge base system.

[0157] Secondly, the solution of the present application improves the accuracy and efficiency of document retrieval through user-generated or automatically generated keywords. Through the accurate judgment of keywords, the solution of the present application enables the system to locate relevant documents more quickly, reducing the time and energy of users searching in massive amounts of information. The use of keywords makes the retrieval process more efficient and accurate. By introducing multi-stage scoring feedback and large model prompt word optimization mechanism, the system can continuously adjust and improve the generated answers based on user feedback, thereby improving the accuracy and relevance of the answers. Based on the prompt word optimization strategy of the large model, the system can intelligently process user feedback, guide the large model to generate better answers through prompt words, and improve the intelligence level of the system.

[0158] The present application provides a solution for determining at least one associated document associated with question information input by a user; determining a preliminary search result corresponding to the question information based on the at least one associated document and the question information; and determining a target search result corresponding to the question information based on the preliminary search result. The solution can determine the document associated with the question information based on the question information input by the user, thereby improving the accuracy of the search results when the corresponding search results are fed back based on the user's question information.

[0159] Figure 3A schematic diagram of a search result determination device provided for an exemplary embodiment of the present application; wherein the device comprises:

[0160] The associated document determining unit 31 is used to determine at least one associated document associated with the question information according to the question information input by the user;

[0161] The retrieval result determination unit 32 is further configured to determine a preliminary retrieval result corresponding to the question information based on the at least one associated document and the question information; and determine a target retrieval result corresponding to the question information according to the preliminary retrieval result.

[0162] In some optional embodiments of the present application, when the aforementioned device is used to determine the target search result corresponding to the question information according to the preliminary search result, it is specifically used to:

[0163] Obtaining first evaluation information of the user for the preliminary search result;

[0164] A target search result corresponding to the question information is determined according to the first evaluation information.

[0165] In some optional embodiments of the present application, when the aforementioned apparatus is used to determine at least one associated document associated with the question information according to the question information input by the user, it is specifically used to:

[0166] Segmenting the question information input by the user to obtain multiple subwords;

[0167] At least one associated document associated with the question information is determined according to the multiple sub-words and the multiple documents to be analyzed.

[0168] In some optional embodiments of the present application, when the aforementioned device is used to determine at least one associated document associated with the question information according to the multiple subwords and the multiple documents to be analyzed, it is specifically used to:

[0169] For each sub-word in the plurality of sub-words, determining from the plurality of documents to be analyzed a document to be analyzed in which a keyword matching the sub-word exists in a corresponding keyword, obtaining a document group to be analyzed corresponding to the sub-word, and further obtaining a plurality of document groups to be analyzed corresponding to the plurality of sub-words;

[0170] At least one associated document associated with the question information is determined based on the documents to be analyzed included in the plurality of document groups to be analyzed.

[0171] In some optional embodiments of the present application, when the aforementioned apparatus is used to determine the preliminary search result corresponding to the question information based on the at least one associated document and the question information, it is specifically used to:

[0172] The associated document with the largest number of occurrences among the at least one associated document is used as a target document;

[0173] A preliminary search result corresponding to the question information is determined based on the target document and the question information.

[0174] In some optional embodiments of the present application, when the aforementioned device is used to determine the target search result corresponding to the question information according to the first evaluation information, it is specifically used to:

[0175] When the score indicated by the first evaluation information is less than a first preset score, determining first prompt information corresponding to the first evaluation information according to the first evaluation information;

[0176] Determine, according to the first prompt information, a to-be-determined search result corresponding to the question information;

[0177] Obtaining second evaluation information of the user for the to-be-determined search result;

[0178] A target search result corresponding to the question information is determined according to the second evaluation information.

[0179] In some optional embodiments of the present application, when the aforementioned device is used to determine the target search result corresponding to the question information according to the second evaluation information, it is specifically used to:

[0180] When the second evaluation information indicates that the current score has decreased relative to the previous score, or the second evaluation information indicates that the current score has increased relative to the previous score but the current score is less than the first preset score, or the current score is the same as the previous score, determining the second prompt information corresponding to the second evaluation information according to the second evaluation information;

[0181] A target search result corresponding to the question information is determined according to the second prompt information.

[0182] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0183] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0184] Figure 4 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include:

[0185] The memory 401 and the processor 402, the memory 401 is used to store the computer program and transmit the program code to the processor 402. In other words, the processor 402 can call and run the computer program from the memory 401 to implement the method in the embodiment of the present application.

[0186] For example, the processor 402 may be configured to execute the above method embodiments according to instructions in the computer program.

[0187] In some embodiments of the present application, the processor 402 may include but is not limited to:

[0188] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0189] In some embodiments of the present application, the memory 401 includes but is not limited to:

[0190] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0191] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 401 and executed by the processor 402 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0192] like Figure 4 As shown, the electronic device may also include:

[0193] The transceiver 403 may be connected to the processor 402 or the memory 401 .

[0194] The processor 402 may control the transceiver 403 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 403 may include a transmitter and a receiver. The transceiver 403 may further include an antenna, and the number of antennas may be one or more.

[0195] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0196] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0197] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0198] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0199] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0200] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0201] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining a search result, characterized in that: include: Determine at least one associated document associated with the question information according to the question information input by the user; Determine a preliminary search result corresponding to the question information based on the at least one associated document and the question information; A target search result corresponding to the question information is determined according to the preliminary search result.

2. The method according to claim 1, characterized in that Determining the target search result corresponding to the question information according to the preliminary search result includes: Obtaining first evaluation information of the user for the preliminary search result; A target search result corresponding to the question information is determined according to the first evaluation information.

3. The method according to claim 1, characterized in that The step of determining at least one associated document associated with the question information according to the question information input by the user includes: Segmenting the question information input by the user to obtain multiple subwords; At least one associated document associated with the question information is determined according to the multiple sub-words and the multiple documents to be analyzed.

4. The method according to claim 3, characterized in that: The step of determining at least one associated document associated with the question information based on the multiple subwords and the multiple documents to be analyzed includes: For each sub-word in the plurality of sub-words, determining from the plurality of documents to be analyzed a document to be analyzed in which a keyword matching the sub-word exists in the corresponding keyword, obtaining a document group to be analyzed corresponding to the sub-word, and further obtaining a plurality of document groups to be analyzed corresponding to the plurality of sub-words; At least one associated document associated with the question information is determined based on the documents to be analyzed included in the plurality of document groups to be analyzed.

5. The method according to claim 1, characterized in that The determining, based on the at least one associated document and the question information, a preliminary search result corresponding to the question information comprises: The associated document with the largest number of occurrences among the at least one associated document is used as the target document; A preliminary search result corresponding to the question information is determined based on the target document and the question information.

6. The method according to claim 2, characterized in that The determining, according to the first evaluation information, a target search result corresponding to the question information includes: When the score indicated by the first evaluation information is less than a first preset score, determining first prompt information corresponding to the first evaluation information according to the first evaluation information; Determine the to-be-determined search result corresponding to the question information according to the first prompt information; Obtaining second evaluation information of the user for the to-be-determined search result; A target search result corresponding to the question information is determined according to the second evaluation information.

7. The method according to claim 6, characterized in that The determining, according to the second evaluation information, a target search result corresponding to the question information includes: When the second evaluation information indicates that the current score has decreased relative to the previous score, or the second evaluation information indicates that the current score has increased relative to the previous score but the current score is less than the first preset score, or the current score is the same as the previous score, determining the second prompt information corresponding to the second evaluation information according to the second evaluation information; A target search result corresponding to the question information is determined according to the second prompt information.

8. A search result determination device, characterized in that: include: An associated document determining unit, configured to determine at least one associated document associated with the question information according to the question information input by the user; The retrieval result determination unit is further used to determine the preliminary retrieval result corresponding to the question information based on the at least one associated document and the question information; and determine the target retrieval result corresponding to the question information according to the preliminary retrieval result.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images