Answer generation method and device, equipment, storage medium and product

Abstract

The application discloses a response generation method and device, equipment, a storage medium and a product, relates to the technical field of natural language processing, and discloses the following steps: acquiring a question corresponding to a dialogue instruction in response to the dialogue instruction; taking the question as a search word, recalling a plurality of target search results, the target search result being a search result with a relevant degree of the question meeting a relevant condition; inserting noise into the plurality of target search results, the noise being an answer with a relevant degree of the question not meeting the relevant condition; and generating an answer corresponding to the question based on the question and the plurality of target search results with the inserted noise through a large language model. The method can ensure the accuracy of the generated answer and improve the quality of the answer.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, apparatus, device, storage medium and product for generating answers. Background Technology

[0002] With the rapid development of the internet and the dramatic increase in information volume, users' demand for quickly and accurately obtaining the information they need is growing, thus giving rise to question-and-answer systems. A question-and-answer system is a natural language processing technology designed to automatically answer questions posed by users in natural language. Unlike traditional search engines, question-and-answer systems can directly provide users with answers to their questions, rather than just relevant web page links.

[0003] In related technologies, question-answering systems analyze user questions, search for relevant information in a large number of documents, and attempt to extract the answer that best meets the user's needs. In some cases, the models used by question-answering systems may not be sensitive enough to the differences between multiple input data, resulting in low accuracy of the answers given based on the input data.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, device, storage medium, and product for generating answers, which can ensure the accuracy of the generated answers and improve the quality of the answers.

[0006] To achieve the above objectives, this application proposes a method for generating answers, the method comprising:

[0007] In response to a dialogue command, obtain the question corresponding to the dialogue command;

[0008] Using the question as a search term, multiple target search results are retrieved, where the target search results are those that are relevant to the question and meet the relevant criteria.

[0009] Noise is inserted into the multiple target search results, where the noise refers to answers whose relevance to the question does not meet the relevance criteria.

[0010] Using a large language model, an answer to the question is generated based on the question and multiple target search results after the noise is inserted.

[0011] Optionally, the step of using the question as a search term to retrieve multiple target search results includes:

[0012] Using the question as a search term, multiple original search results were retrieved;

[0013] The first number of original search results ranked first among the multiple original search results are determined as the multiple target search results; or, the first number of original search results with the highest relevance to the question among the multiple original search results are determined as the multiple target search results.

[0014] Optionally, before inserting noise into the plurality of target search results, the method further includes at least one of the following:

[0015] The original search result with the lowest relevance to the question among the multiple original search results is taken as the noise.

[0016] Among the multiple original search results, any original search result whose relevance to the question is lower than a preset threshold is taken as the noise.

[0017] The last original search result among the multiple original search results is taken as the noise.

[0018] The noise is defined as any one of the last two original search results among the multiple original search results.

[0019] Optionally, inserting noise into the plurality of target search results includes:

[0020] Determine the ranking of the multiple target search results;

[0021] The noise is inserted into the target sorting position in the multiple target search results.

[0022] Optionally, determining the order of the multiple target search results includes:

[0023] The search term is subjected to intent recognition to obtain the target intent corresponding to the search term;

[0024] Based on the target intent, the target sorting method corresponding to the target intent is determined from the correspondence between intent and sorting method;

[0025] The multiple target search results are sorted according to the target sorting method.

[0026] Optionally, in the correspondence, the sorting method corresponding to the target intent is reverse sorting; sorting the multiple target search results according to the target sorting method includes:

[0027] Determine the original ranking of the multiple target search results;

[0028] Based on the original ranking of the multiple target search results, the multiple target search results are reverse-ranked.

[0029] Optionally, the step of performing intent recognition on the search term to obtain the target intent corresponding to the search term includes:

[0030] Multi-level intent recognition is performed on the search terms, and the highest-level target intent is determined from the identified multiple levels of intent, where the higher-level intent is a sub-intent of the lower-level intent.

[0031] Optionally, the step of performing multi-level intent recognition on the search term, and determining the highest-level target intent from the identified multiple levels of intent, includes:

[0032] The search terms are subjected to multi-level intent recognition by multiple intent recognition models, and the highest level intent is determined from the multiple levels of intent recognized by each intent recognition model.

[0033] Select the most frequently occurring intent from among the identified top-level intents as the target intent.

[0034] Optionally, the step of performing multi-level intent recognition on the search term, and determining the highest-level target intent from the identified multiple levels of intent, includes:

[0035] Based on the search term, the intent identified by each intent recognition model is obtained sequentially according to the order of multiple intent recognition models until the intent identified by the last intent recognition model is obtained. The intent identified by the last intent recognition model is taken as the highest level target intent. Except for the first intent recognition model, each of the other intent recognition models performs intent recognition based on the intent identified by the previous intent recognition model. The resulting intent is a sub-intent of the intent identified by the previous intent recognition model.

[0036] Optionally, the step of using the question as a search term to retrieve multiple original search results includes:

[0037] Expand at least one new search term based on the search term;

[0038] Information was searched for each search term before and after expansion, resulting in multiple original search results.

[0039] Optionally, expanding at least one new search term based on the search term includes at least one of the following:

[0040] The search term is split into multiple new search terms;

[0041] The search terms are rewritten to obtain at least one new rewritten search term;

[0042] The search term is split, and at least one new search term is reconstructed based on the splitting results;

[0043] The search terms are corrected to obtain new, corrected search terms.

[0044] Furthermore, to achieve the above objectives, this application also proposes a response generation apparatus, the apparatus comprising:

[0045] The question acquisition module is used to acquire the question corresponding to the dialogue command in response to the dialogue command;

[0046] The information retrieval module is used to retrieve multiple target search results by using the question as a search term. The target search results are search results that meet the relevant conditions in terms of relevance to the question.

[0047] A noise insertion module is used to insert noise into the multiple target search results, wherein the noise refers to answers whose relevance to the question does not meet the relevance criteria.

[0048] The answer generation module is used to generate an answer to the question based on the question and multiple target search results after inserting the noise, using a large language model.

[0049] Optionally, the information retrieval module includes:

[0050] The information recall unit is used to recall multiple original search results by using the question as a search term.

[0051] An information determination unit is configured to determine the first number of original search results ranked first among the plurality of original search results as the plurality of target search results, or to determine the first number of original search results that are most relevant to the question among the plurality of original search results as the plurality of target search results.

[0052] Optionally, the apparatus further includes a noise determination module, the noise determination module being configured to perform at least one of the following:

[0053] The original search result with the lowest relevance to the question among the multiple original search results is taken as the noise.

[0054] Among the multiple original search results, any original search result whose relevance to the question is lower than a preset threshold is taken as the noise.

[0055] The last original search result among the multiple original search results is taken as the noise.

[0056] The noise is defined as any one of the last two original search results among the multiple original search results.

[0057] Optionally, the noise insertion module includes:

[0058] A sorting determination unit is used to determine the sorting order of the multiple target search results;

[0059] A noise insertion unit is used to insert the noise into the target sorting position in the multiple target search results.

[0060] Optionally, the sorting determination unit includes:

[0061] An intent recognition subunit is used to perform intent recognition on the search term to obtain the target intent corresponding to the search term;

[0062] The intent determination subunit is used to determine the target sorting method corresponding to the target intent from the correspondence between intent and sorting method based on the target intent;

[0063] The result sorting subunit is used to sort the multiple target search results according to the target sorting method.

[0064] Optionally, in the correspondence, the sorting method corresponding to the target intent is reverse sorting;

[0065] The result sorting subunit is used to determine the original sorting result of the multiple target search results; and to reverse sort the multiple target search results based on the original sorting result.

[0066] Optionally, the intent recognition subunit is used to perform multi-level intent recognition on the search term, and determine the highest-level target intent from the identified multiple levels of intent, wherein the higher-level intent is a sub-intent of the lower-level intent.

[0067] Optionally, the intent recognition subunit is configured to perform multi-level intent recognition on the search term using multiple intent recognition models, determine the highest-level intent from the multiple levels of intents identified by each intent recognition model, and select the most frequently occurring intent from the determined multiple highest-level intents as the target intent.

[0068] Optionally, the intent recognition subunit is used to sequentially obtain the intent recognized by each intent recognition model according to the order of multiple intent recognition models based on the search term, until the intent recognized by the last intent recognition model is obtained, and the intent recognized by the last intent recognition model is taken as the highest level target intent; wherein, except for the first intent recognition model, each of the other intent recognition models performs intent recognition based on the intent recognized by the previous intent recognition model, and the obtained intent is a sub-intent of the intent recognized by the previous intent recognition model.

[0069] Optionally, the information recall unit includes:

[0070] The search term expansion subunit is used to expand at least one new search term based on the search term;

[0071] The information search subunit is used to perform information searches based on each search term before and after expansion, and obtain multiple original search results.

[0072] Optionally, the search term expansion subunit is configured to perform at least one of the following:

[0073] The search term is split into multiple new search terms;

[0074] The search terms are rewritten to obtain at least one new rewritten search term;

[0075] The search term is split, and at least one new search term is reconstructed based on the splitting results;

[0076] The search terms are corrected to obtain new, corrected search terms.

[0077] In addition, to achieve the above objectives, this application also proposes an answer generation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the answer generation method as described above.

[0078] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the answer generation method described above.

[0079] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the answer generation method described above.

[0080] One or more technical solutions proposed in this application have at least the following technical effects:

[0081] The answer generation scheme provided in this application responds to a dialogue command, retrieves the question corresponding to the command, uses the question as a search term, and recalls multiple target search results. Considering that these multiple target search results are all results whose relevance to the question meets the relevant criteria, generating an answer based solely on these multiple target search results using a large language model might not be sensitive enough to the differences between them, thus affecting the answer quality. Therefore, noise is inserted into the multiple target search results. Since the noise represents answers whose relevance to the question does not meet the relevant criteria, the question and the noise-injected multiple target search results are fed to the large language model. The noise serves as a reference, helping the large language model determine which of the multiple target search results is better. This makes the large language model more sensitive to the differences between the multiple target search results, thereby filtering out content that better matches the question from the multiple target search results to generate a more accurate answer and improve the answer quality. Attached Figure Description

[0082] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0083] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of an implementation environment for the response generation method of this application;

[0085] Figure 2 This is a flowchart illustrating the first embodiment of the response generation method for this application.

[0086] Figure 3 This is a flowchart illustrating the second embodiment of the response generation method for this application.

[0087] Figure 4 This is a flowchart illustrating the third embodiment of the response generation method for this application.

[0088] Figure 5 This is a schematic diagram of the module structure of the answer generation device in the embodiments of this application;

[0089] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the answer generation method in this application embodiment.

[0090] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0091] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0092] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0093] Figure 1 This is a schematic diagram illustrating an implementation environment provided by an embodiment of this disclosure. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network. For example, the terminal 101 is installed with a target application provided by the server 102, and the terminal 101 can perform functions such as data transmission and message interaction through the target application.

[0094] For example, terminal 101 can be a computer, mobile phone, tablet computer, or other terminal. For example, the target application can be a target application within the operating system of terminal 101, or a target application provided by a third party. For example, the target application can be a search application, a short video application, a shopping application, a navigation application, etc. For example, server 102 can be the backend server corresponding to the target application. Accordingly, server 102 can be a search application server, a short video application server, a shopping application server, a navigation application server, etc.

[0095] In this application, terminal 101 responds to a dialogue command and retrieves the question corresponding to the dialogue command. It then sends the question to server 102. Server 102 receives the question, uses it as a search term, and retrieves multiple target search results. These target search results are those whose relevance to the question meets the relevance criteria. Noise is then inserted into these multiple target search results; the noise refers to answers whose relevance to the question does not meet the relevance criteria. Next, using a large language model, an answer corresponding to the question is generated based on the question and the multiple target search results after the noise is inserted. This answer is then sent to terminal 101. Terminal 101 receives and displays the answer.

[0096] Alternatively, the above-mentioned answer generation process can also be completed by terminal 101 alone. Alternatively, terminal 101 can complete it through an installed target application, such as a search application. This application embodiment does not impose any limitations on this.

[0097] The answer generation method provided in this application is applicable to various scenarios. For example, in e-commerce, users can ask questions about product information they need, such as specifications and usage methods, within shopping applications. The terminal then uses the method provided to quickly find the product information the user needs and present it as an answer, thereby improving the shopping experience and conversion rate. In education, the terminal can use the method to answer students' questions, assisting their learning and acting as an intelligent tutor. In tourism, users can ask questions about attractions, flights, and accommodations. The terminal uses the method provided to answer users' questions, assisting them in planning their trips.

[0098] Figure 2 This is a flowchart illustrating the first embodiment of the method for generating responses to this application. (Refer to...) Figure 2 Taking the terminal as the executing entity as an example, the answer generation method includes the following steps S10 to S40:

[0099] Step S10: In response to the dialogue command, obtain the question corresponding to the dialogue command.

[0100] Dialogue instructions: Requests or commands issued by users to the system via voice, text, or other means. These can be direct questions, such as "What's the weather like in Beijing today?", or indirect expressions of needs, such as "I want to know the latest movie schedules." Regardless of the specific content of the dialogue instruction, this application refers to the content expressed in dialogue instructions collectively as "questions."

[0101] For example, the terminal displays a dialog interface through which the user inputs any dialogue content. In response to the user's input, the terminal generates a dialogue command, using the user's input as the corresponding question.

[0102] Step S20: Using the question as a search term, retrieve multiple target search results. The target search results are search results that meet the relevant conditions in terms of relevance to the question.

[0103] Search terms are keywords or phrases used to find information in databases, knowledge bases, or the internet. This application uses the question as a search term and leverages it to retrieve multiple target search results, such as document fragments, webpage links, and database records. These multiple target search results serve as the information source for subsequent model-generated answers.

[0104] Optionally, the terminal can retrieve target search results from various information sources based on search terms. Exemplary information sources include: pre-built knowledge bases, web page content, domain-specific knowledge graphs, community-generated content such as user-contributed content from social media platforms and forums, and third-party data services such as weather forecasts and stock quotes.

[0105] Optionally, the terminal retrieves multiple original search results based on the search terms, and then filters out the target search results from the multiple original search results that meet the relevant conditions regarding their relevance to the question.

[0106] Relevance measures how well a search result matches the question. Relevance criteria are a pre-defined set of standards or rules used to determine whether a search result is relevant enough to be selected as a target search result. In other words, relevance criteria are used to filter target search results. Relevance criteria can include various restrictions such as time limits, sorting limits, quantity limits, and relevance level limits, thereby filtering target search results that meet multiple restrictions within the relevance criteria.

[0107] For example, the relevant conditions include a relevance threshold. The terminal identifies the original search results from multiple original search results whose relevance to the question is greater than the threshold as target search results. Alternatively, the relevant conditions include a first quantity. The terminal identifies the first quantity of original search results that appear at the top of the list as multiple target search results. Since the first-ranked original search results are generally more relevant to the question, this method can filter out target search results with a high degree of matching to the question. Alternatively, the terminal identifies the first quantity of original search results with the highest relevance to the question as multiple target search results. That is, the multiple original search results are first sorted in descending order of relevance to the question, and then the first quantity of original search results that appear at the top are selected as target search results.

[0108] Step S30: Insert noise into multiple target search results. Noise refers to answers that do not meet the relevance criteria to the question.

[0109] If the relevance between the noise and the question does not meet the relevant conditions, it indicates that the noise and the question have a low degree of matching. Subsequently, the noise is inserted into multiple target search results, and the large language model can easily identify the noise from the multiple target search results after the noise is inserted.

[0110] Optionally, the terminal randomly inserts noise into multiple search results; that is, it randomly determines the insertion position of the noise and inserts it into the multiple target search results according to that position. Alternatively, the terminal first determines the order of the multiple target search results and inserts the noise into the target order position within the multiple target search results. The target order position can be determined according to the application scenario; for example, the target order position may be between the last and second-to-last target search results. This embodiment of the application does not impose such limitations.

[0111] Step S40: Using a large language model, generate the answer to the question based on the question and multiple target search results after inserting noise.

[0112] For example, the question and multiple target search results with added noise are input into a large language model. The large language model can then generate an answer to the question based on the question and the multiple target search results with added noise, and output the answer. The terminal can then display the answer in the dialogue interface for the user to view.

[0113] The answer generation scheme provided in this application responds to a dialogue command, retrieves the question corresponding to the command, uses the question as a search term, and recalls multiple target search results. Considering that these multiple target search results are all results whose relevance to the question meets the relevant criteria, generating an answer based solely on these multiple target search results using a large language model might not be sensitive enough to the differences between them, thus affecting the answer quality. Therefore, noise is inserted into the multiple target search results. Since the noise represents answers whose relevance to the question does not meet the relevant criteria, the question and the noise-injected multiple target search results are fed to the large language model. The noise serves as a reference, helping the large language model determine which of the multiple target search results is better. This makes the large language model more sensitive to the differences between the multiple target search results, thereby filtering out content that better matches the question from the multiple target search results to generate a more accurate answer and improve the answer quality.

[0114] Based on the first embodiment described above, a second embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description and will not be repeated hereafter. (Refer to...) Figure 3 In the second embodiment, step S20 includes steps S201 to S202.

[0115] Step S201: Use the question as a search term to retrieve multiple original search results.

[0116] Optionally, this step can be implemented as follows: expand at least one new search term based on the search term; perform information searches based on each search term before and after expansion to obtain multiple original search results.

[0117] The implementation of expanding at least one new search term based on a search term includes at least one of the following:

[0118] The first method involves splitting the search term into multiple new search terms. Splitting refers to breaking down a complex search term into several simpler, independent search terms. For example, if the original search term is "Amusement Park A ticket price," the resulting new search terms could include "Amusement Park A" and "ticket price."

[0119] The second method involves rewriting the search terms to obtain at least one new rewritten search term. Rewriting refers to using methods such as synonym substitution to change the original search term into a search term with the same meaning but a different expression. For example, the original search term is "Where is the best Chinese restaurant?", and the rewritten search term is "Highly rated Chinese restaurants nearby".

[0120] The third method involves splitting the search terms and reconstructing at least one new search term based on the splitting results. Reconstruction refers to reorganizing the split search terms to obtain search terms that have the same meaning as the original search terms but are expressed differently. For example, if the original search term is "weather forecast for region B tomorrow", the new search term reconstructed based on the splitting results would be "weather forecast for region B tomorrow".

[0121] The fourth method involves correcting the search terms to obtain new, corrected search terms. Correction refers to fixing spelling, grammatical, and other errors in the original search terms to arrive at new ones. For example, if the original search term is "Will it rain tomorrow?", the corrected search term would be "Will it rain tomorrow?".

[0122] In this embodiment, new search terms are expanded by rewriting, splitting, reconstructing, and correcting the search terms. Information is then searched based on each search term before and after expansion, resulting in multiple original search results. This approach enables the search system to better understand the user's intent and obtain more relevant original search results. Furthermore, it increases the number of recalled original search results, providing more data support for the large language model to generate answers.

[0123] Step S202: Determine the first number of original search results ranked first among the multiple original search results as multiple target search results, or determine the first number of original search results with the highest relevance to the question among the multiple original search results as multiple target search results.

[0124] Here, the first quantity is the preset number of target search results. These first quantity of target search results will subsequently serve as information sources, and the large language model will refer to these multiple target search results to generate the answer to the question. For example, if the terminal uses the question as a search term and retrieves 100 original search results, and the first quantity is 10, then the terminal will select 10 target search results from the 100 original search results.

[0125] It's important to note that the retrieved original search results are ranked based on factors such as the relevance of the search results to the question, the frequency of keywords from the question appearing in the search result summary, the authority of the search results, and the timeliness of the search results. Therefore, generally, original search results with higher relevance to the question will be ranked higher. However, it cannot be guaranteed that the original search results ranked higher are necessarily more relevant than those ranked lower. Thus, regardless of whether the terminal determines the top-ranked original search results or the top-ranked original search results with the highest relevance to the question as the target search results, it ensures that the determined target search results are strongly related to the question, meaning they have a high degree of matching, which is beneficial for the model to generate high-quality answers based on the target search results.

[0126] Before inserting noise into multiple target search results, the noise must first be acquired. Optionally, the acquisition of noise can be achieved in at least one of the following ways:

[0127] First, the original search result with the lowest relevance to the question among multiple original search results is identified as noise. This ensures that the identified noise has a low relevance to the question, making it easier for the subsequent model to identify the noise from the multiple target search results after the noise has been inserted.

[0128] Second, any original search result whose relevance to the question is below a preset threshold is considered noise. The preset threshold is a pre-defined relevance threshold, and its value is set relatively small. Therefore, any original search result whose relevance to the question is below the preset threshold has a relatively low relevance to the question. Treating any original search result with a relevance below the preset threshold as noise makes it easier for the subsequent model to identify the noise from the multiple target search results after the noise has been inserted.

[0129] Third, the last original search result among multiple original search results is treated as noise. Since the last original search result is generally less relevant to the question, treating it as noise makes it easier for the subsequent model to identify the noise from the multiple target search results after the noise is inserted.

[0130] Fourth, any one of the last two original search results from the multiple original search results is considered noise. The second number can be set as needed. For example, if there are 100 original search results and 5 second results, the terminal selects any one of the last 5 original search results from the 100 original search results as the target search result. Since the number of original search results is large and the second number is small, any one of the last two original search results is relatively unrelated to the question. Therefore, considering any one of the last two original search results as noise makes it easier for the subsequent model to identify the noise from the multiple target search results after the noise is inserted.

[0131] Based on the first embodiment of this application described above, a third embodiment of this application is proposed. Contents that are the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. See also... Figure 4 In the third embodiment, step S30 includes steps S301 and S302.

[0132] Step S301: Determine the order of multiple target search results.

[0133] Optionally, the ranking of multiple target search results remains consistent with their ranking within the multiple original search results. That is, an original search result that ranks higher in the original search results will also rank higher in the multiple target search results. For example, if there are 100 original search results retrieved, and the top 10 are identified as target search results, then the ranking of these 10 target search results will remain unchanged.

[0134] Optionally, multiple target search results can be reordered based on the intent of the search terms. Accordingly, the terminal performs intent recognition on the search terms to obtain the target intent corresponding to the search terms; based on the target intent, the target sorting method corresponding to the target intent is determined from the correspondence between intent and sorting method; and the multiple target search results are sorted according to the target sorting method.

[0135] Sorting methods refer to the arrangement of multiple target search results according to certain criteria. The choice of sorting method depends on the understanding of user intent. The correspondence between intent and sorting method can be customized as needed. For example, if the intent is a news search, the sorting method would be based on timeliness, meaning recently published or upcoming events would be ranked first. For example, if the intent is an academic information search, the sorting method would be based on website authority, meaning content from websites with higher authority would be ranked first. For example, if the intent is a product information search, the sorting method would be based on relevance, meaning search results more relevant to the intent would be ranked first.

[0136] The embodiments of this application select the sorting method of matching results according to different types of intent, which can ensure that the target search results that are more in line with the user's needs are ranked first. In this way, when the large language model generates an answer based on the sorted target search results, it is easier for the model to filter important information and generate an answer that is more in line with the user's needs.

[0137] Optionally, in the correspondence between intent and sorting method, the sorting method corresponding to the target intent is reverse sorting. Accordingly, sorting multiple target search results according to the target sorting method includes: determining the original sorting results of the multiple target search results; and performing reverse sorting on the multiple target search results based on the original sorting results.

[0138] The original ranking of the target search results is consistent with the ranking of multiple target search results within multiple original search results. For example, if there are 100 original search results, and the top 10 are identified as target search results, their original ranking remains unchanged. However, if the target intent corresponds to a reverse ranking, these 10 target search results are reverse-ranked. After reversal, the target search result that was originally ranked 1st becomes ranked 10th, the target search result that was originally ranked 2nd becomes ranked 9th, and so on.

[0139] For example, the ranking method for the target intent in knowledge-based question-and-answer categories is reverse ranking. Tests have shown that reversing the ranking of search results corresponding to the search terms for the target intent in knowledge-based question-and-answer categories improves the answer quality of the large language model by more than 5%.

[0140] For example, for any intent, the original and reverse ranking results of the target search results corresponding to the search terms for that intent are obtained. Using a large language model, answers to the question are generated based on both the original and reverse ranking results. The quality of the two answers is compared; if the answer corresponding to the reverse ranking result is of higher quality, the terminal sets the ranking method for that intent to reverse. This method can identify multiple intents that require reverse ranking. Therefore, when answering questions belonging to this type of intent, the quality of the generated answer can be improved by reversing the target search results.

[0141] Optionally, the terminal performs intent recognition on the search terms to obtain the target intent corresponding to the search terms, including: performing multi-level intent recognition on the search terms, determining the highest-level target intent from the identified multiple levels of intent, wherein the higher-level intent is a sub-intent of the lower-level intent.

[0142] Intent recognition refers to inferring a user's specific needs or purposes by analyzing their search terms. This goes beyond simple keyword matching; it involves a deeper understanding of the meaning behind the user's query. In many cases, a user's expression reflects not just a single intent, but a complex and interconnected series of intents. The goal of multi-level intent recognition is to delve into these multi-layered user intents to provide more accurate information to the user.

[0143] The level of intent recognition can be set as needed. For example, a three-level intent recognition process can be used for search terms: primary intent recognition, intermediate intent recognition, and advanced intent recognition. The intent identified by intermediate intent recognition is a sub-intent of the intent identified by primary intent recognition, and the intent identified by advanced intent recognition is a sub-intent of the intent identified by intermediate intent recognition. For instance, for the search term "write an article about A winning the championship," the result of primary intent recognition might be "writing," the result of intermediate intent recognition might be "writing a press release," and the result of advanced intent recognition might be "writing a press release about athlete A winning the championship." Similarly, for the search term "best Chinese restaurant in region A," the result of primary intent recognition might be "find restaurants," the result of intermediate intent recognition might be "find the best restaurants in region A," and the result of advanced intent recognition might be "find the highest-rated Chinese restaurant in region A."

[0144] It is understandable that, compared to lower-level intents, higher-level intents contain more details and express intent more accurately. Therefore, using the highest-level intent as the target intent can ensure the accuracy of the determined target intent, thereby ensuring the suitability of the subsequent ranking method of the target search results with the question.

[0145] In this embodiment of the application, multi-level intent recognition is performed on the search term, and the highest-level target intent is determined from the identified multiple levels of intent. This includes the following two implementation methods:

[0146] The first method involves the terminal using multiple intent recognition models to perform multi-level intent recognition on the search terms, determining the highest-level intent from the multiple levels of intents identified by each intent recognition model, and selecting the most frequently occurring intent from the determined highest-level intents as the target intent.

[0147] Multiple intent recognition models operate in parallel. Each model independently identifies the intent of the search term, yielding independent results. For example, five intent recognition models independently perform three levels of intent recognition on the search term, each obtaining the highest-level intent. The terminal then selects the most frequently occurring intent from the five highest-level intents as the target intent.

[0148] By employing multiple intent recognition models working in parallel to perform multi-level intent recognition on search terms, user intent can be captured from different angles and levels, improving the accuracy of intent recognition. Selecting the most frequently occurring intent from multiple highest-level intents as the target intent avoids the biases or limitations that may exist with a single model, ensuring the reliability of intent recognition results even when faced with complex search terms.

[0149] The second method involves the terminal acquiring the intent identified by each intent recognition model in sequence based on the search term, until the intent identified by the last intent recognition model is acquired. The intent identified by the last intent recognition model is then taken as the highest-level target intent. Except for the first intent recognition model, each of the other intent recognition models performs intent recognition based on the intent identified by the previous intent recognition model, and the resulting intent is a sub-intent of the intent identified by the previous intent recognition model.

[0150] The multiple intent recognition models operate sequentially. The output of the previous intent recognition model serves as the input to the next. The next intent recognition model refines the intent recognized by the previous model. In this way, through multiple intent recognition models, the intent can be progressively refined to obtain the accurate intent output by the last intent recognition model.

[0151] In this embodiment, by sequentially using multiple intent recognition models, a deeper understanding of the user's intent can be achieved, progressing from general to specific, thus constructing a hierarchical intent recognition system. Each model further refines the intent based on the previous model, helping to capture more specific and refined user needs. Furthermore, each model has the opportunity to correct any errors from the previous model, thereby improving the accuracy of the final intent recognition.

[0152] It should be noted that the multiple intent recognition models involved in this application may be the same or different. Furthermore, each intent recognition model can be of any type, such as convolutional neural networks, deep learning models, etc., and this application does not impose any limitations on this. Additionally, the data used to train each intent recognition model can be custom intent classification data. The intent classification data includes a large number of search terms and corresponding multiple levels of intent.

[0153] Step S302: Insert noise into the target ranking position in multiple target search results.

[0154] The target ranking position can be set as needed. For example, the target ranking position is between any two target search results in the latter half of the search results. For instance, if there are 10 target search results, the target ranking position is between any two of the last 5 target search results. Of course, the target ranking position can also be specific to a fixed position. For example, the target ranking position is between the 8th and 9th target search results. Testing has shown that inserting noise between any two target search results in the latter half of the search results results in higher quality answers generated by the large language model compared to inserting noise between any two target search results in the former half.

[0155] It's important to note that because questions are generally quite long and contain many keywords, when using the question as a search term to retrieve search results, results that are closer to the question may be ranked lower in the search results. Therefore, inserting noise into the later positions of multiple target search results may make the model's reasoning path clearer, thereby improving the quality of the model's answers.

[0156] Another point to note is that if only multiple target search results are input into a large language model, because these results are all strongly related to the question, the model is not sensitive enough to the differences between them. This leads to the model not carefully comparing the multiple search results and generating answers arbitrarily. For example, the model might randomly sort or randomly merge the multiple target search results. In contrast, once noise is inserted into the multiple target search results, the model can use the noise as a reference. To avoid these negative answers, it will more carefully distinguish the differences between the multiple target search results, thus selecting content that better matches the question to generate the answer, thereby improving the quality of the answer.

[0157] Another point to note is that the above examples are only for understanding this application and do not constitute a limitation on the answer generation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0158] This application also provides an answer generation device, please refer to... Figure 5 The answer generation device includes:

[0159] Question acquisition module 10 is used to acquire the question corresponding to the dialogue command in response to the dialogue command;

[0160] The information retrieval module 20 is used to retrieve multiple target search results by using the question as a search term. The target search results are search results that meet the relevant conditions in terms of relevance to the question.

[0161] The noise insertion module 30 is used to insert noise into multiple target search results. Noise refers to answers that do not meet the relevance criteria to the question.

[0162] The answer generation module 40 is used to generate answers to questions based on multiple target search results with added noise, using a large language model.

[0163] Optionally, the information recall module 20 includes:

[0164] The information recall unit is used to recall multiple original search results by using the question as a search term.

[0165] The information determination unit is used to determine the first number of original search results ranked first among multiple original search results as multiple target search results, or to determine the first number of original search results with the highest relevance to the question among multiple original search results as multiple target search results.

[0166] Optionally, the device further includes a noise determination module, which performs at least one of the following:

[0167] The original search result with the lowest relevance to the question among multiple original search results is considered as noise.

[0168] Any original search result whose relevance to the question is below a preset threshold will be treated as noise.

[0169] The last original search result among multiple original search results is considered as noise.

[0170] Any one of the last two original search results among multiple original search results is considered noise.

[0171] Optionally, the noise insertion module 30 includes:

[0172] The sorting determination unit is used to determine the order of multiple target search results;

[0173] The noise insertion unit is used to insert noise into the target ranking position in multiple target search results.

[0174] Optionally, the sorting determination unit includes:

[0175] The intent recognition subunit is used to identify the intent of search terms and obtain the target intent corresponding to the search terms.

[0176] The intent determination subunit is used to determine the target sorting method corresponding to the target intent based on the correspondence between intent and sorting method;

[0177] The result sorting subunit is used to sort multiple target search results according to the target sorting method.

[0178] Optionally, in the correspondence, the sorting method corresponding to the target intent is reverse sorting;

[0179] The result sorting subunit is used to determine the original sorting results of multiple target search results; based on the original sorting results of multiple target search results, the multiple target search results are sorted in reverse order.

[0180] Optionally, the intent recognition subunit is used to perform multi-level intent recognition on the search term, and to determine the highest-level target intent from the multiple levels of intents identified, wherein the higher-level intent is a sub-intent of the lower-level intent.

[0181] Optionally, the intent recognition subunit is used to perform multi-level intent recognition on the search term through multiple intent recognition models, determine the highest level intent from the multiple levels of intents identified by each intent recognition model, and select the most frequently occurring intent from the multiple determined highest level intents as the target intent.

[0182] Optionally, the intent recognition subunit is used to sequentially obtain the intent recognized by each intent recognition model based on the search term and in the order of multiple intent recognition models, until the intent recognized by the last intent recognition model is obtained, and the intent recognized by the last intent recognition model is taken as the highest level target intent; wherein, except for the first intent recognition model, each of the other intent recognition models performs intent recognition based on the intent recognized by the previous intent recognition model, and the obtained intent is a sub-intent of the intent recognized by the previous intent recognition model.

[0183] Optionally, the information recall unit includes:

[0184] The search term expansion subunit is used to expand at least one new search term based on the search term;

[0185] The information search subunit is used to perform information searches based on each search term before and after expansion, and obtain multiple original search results.

[0186] Optionally, the search term expansion subunit is used to perform at least one of the following:

[0187] The search term is split into multiple new search terms;

[0188] Rewrite the search terms to obtain at least one new rewritten search term;

[0189] The search term is split, and at least one new search term is reconstructed based on the splitting results;

[0190] The search terms are corrected to obtain new, corrected search terms.

[0191] The answer generation scheme provided in this application responds to a dialogue command, retrieves the question corresponding to the command, uses the question as a search term, and recalls multiple target search results. Considering that these multiple target search results are all results whose relevance to the question meets the relevant criteria, generating an answer based solely on these multiple target search results using a large language model might not be sensitive enough to the differences between them, thus affecting the answer quality. Therefore, noise is inserted into the multiple target search results. Since the noise represents answers whose relevance to the question does not meet the relevant criteria, the question and the noise-injected multiple target search results are fed to the large language model. The noise serves as a reference, helping the large language model determine which of the multiple target search results is better. This makes the large language model more sensitive to the differences between the multiple target search results, thereby filtering out content that better matches the question from the multiple target search results to generate a more accurate answer and improve the answer quality.

[0192] The answer generation apparatus provided in this application, employing the answer generation method in the above embodiments, can solve the technical problem in related technologies where the model is not sensitive enough to the differences between multiple input data, resulting in low accuracy of the answers generated based on the input data. Compared with the prior art, the beneficial effects of the answer generation apparatus provided in this application are the same as those of the answer generation method provided in the above embodiments, and other technical features in the answer generation apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0193] This application provides an answer generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the answer generation method in Embodiment 1 above.

[0194] The following is for reference. Figure 6 The diagram illustrates a structural schematic suitable for implementing the answer generation device of the embodiments of this application. The answer generation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The answer generation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0195] like Figure 6 As shown, the response generation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the response generation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the response generating device to communicate wirelessly or wiredly with other devices to exchange data. Although response generating devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0196] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0197] The answer generation device provided in this application, employing the answer generation method in the above embodiments, can solve the technical problem in related technologies where the model is not sensitive enough to the differences between multiple input data, resulting in low accuracy of the answers generated based on the input data. Compared with the prior art, the beneficial effects of the answer generation device provided in this application are the same as those of the answer generation method provided in the above embodiments, and other technical features in this answer generation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0198] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0199] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0200] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the answer generation method in the above embodiments.

[0201] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0202] The aforementioned computer-readable storage medium may be included in the response generation device; or it may exist independently and not assembled into the response generation device.

[0203] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the answer generation device, cause the answer generation device to: in response to a dialogue instruction, obtain the question corresponding to the dialogue instruction; use the question as a search term to retrieve multiple target search results, the target search results being search results whose relevance to the question meets the relevant criteria; insert noise into the multiple target search results, the noise referring to answers whose relevance to the question does not meet the relevant criteria; and generate an answer corresponding to the question based on the question and the multiple target search results with inserted noise using a large language model.

[0204] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0206] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0207] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described answer generation method. This addresses the technical problem in related technologies where models are not sensitive enough to differences between multiple input data, leading to low accuracy of answers generated based on the input data. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the answer generation method provided in the above embodiments, and will not be elaborated upon here.

[0208] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the answer generation method described above.

[0209] The computer program product provided in this application can solve the technical problem in related technologies where models are not sensitive enough to differences between multiple input data, resulting in low accuracy of answers generated based on input data. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the answer generation method provided in the above embodiments, and will not be repeated here.

[0210] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for generating answers, characterized in that, The method includes: In response to a dialogue command, obtain the question corresponding to the dialogue command; Using the question as a search term, multiple original search results were retrieved; Multiple target search results and noise are determined from the multiple original search results. The target search results are search results whose relevance to the question meets the relevant conditions, and the noise refers to search results whose relevance to the question does not meet the relevant conditions. Insert the noise into the multiple target search results; Using a large language model, an answer to the question is generated based on the question and multiple target search results after the noise is inserted. Inserting the noise into the multiple target search results includes: Determine the order of the multiple target search results; insert the noise into the target order position in the multiple target search results, wherein the target order position is between any two target search results in the latter half of the target search results; Determining the order of the multiple target search results includes: The search terms are subjected to intent recognition to obtain the target intent corresponding to the search terms; based on the target intent, the target sorting method corresponding to the target intent is determined from the correspondence between intent and sorting method; the multiple target search results are sorted according to the target sorting method. Among them, the sorting method for news search intent is based on timeliness; the sorting method for academic information search intent is based on website authority; the sorting method for product information search intent is based on relevance; and the sorting method for knowledge Q&A intent is reverse sorting.

2. The method as described in claim 1, characterized in that, Multiple target search results are determined from the multiple original search results, including: The first number of original search results ranked first among the multiple original search results are determined as the multiple target search results; or, the first number of original search results with the highest relevance to the question among the multiple original search results are determined as the multiple target search results.

3. The method as described in claim 2, characterized in that, Noise is identified from the plurality of original search results, including at least one of the following: The original search result with the lowest relevance to the question among the multiple original search results is taken as the noise. Among the multiple original search results, any original search result whose relevance to the question is lower than a preset threshold is taken as the noise. The last original search result among the multiple original search results is taken as the noise. The noise is defined as any one of the last two original search results among the multiple original search results.

4. The method as described in claim 1, characterized in that, In the correspondence, the sorting method corresponding to the target intent is reverse sorting; sorting the multiple target search results according to the target sorting method includes: Determine the original ranking of the multiple target search results; Based on the original ranking of the multiple target search results, the multiple target search results are reverse-ranked.

5. An answer generation device, characterized in that, The device includes: The question acquisition module is used to acquire the question corresponding to the dialogue command in response to the dialogue command; The information retrieval module is used to retrieve multiple original search results by using the question as a search term; and to determine multiple target search results from the multiple original search results, wherein the target search results are search results whose relevance to the question meets the relevant conditions. A noise determination module is used to determine noise from the multiple original search results, wherein the noise refers to search results whose relevance to the question does not meet the relevance conditions; A noise insertion module is used to insert the noise into the multiple target search results; The answer generation module is used to generate an answer to the question based on the question and multiple target search results after inserting the noise, using a large language model; The noise insertion module includes: A sorting determination unit is used to determine the sorting order of the multiple target search results; A noise insertion unit is used to insert the noise into the target sorting position in the plurality of target search results, wherein the target sorting position is between any two target search results in the latter half of the target search results; The sorting determination unit includes: An intent recognition subunit is used to perform intent recognition on the search term to obtain the target intent corresponding to the search term; The intent determination subunit is used to determine the target sorting method corresponding to the target intent from the correspondence between intent and sorting method based on the target intent; The result sorting subunit is used to sort the multiple target search results according to the target sorting method; Among them, the sorting method for news search intent is based on timeliness; the sorting method for academic information search intent is based on website authority; the sorting method for product information search intent is based on relevance; and the sorting method for knowledge Q&A intent is reverse sorting.

6. An answer generation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the answer generation method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the answer generation method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the answer generation method as described in any one of claims 1 to 4.

Images