Answer generation method and device, equipment, storage medium and product

Abstract

The invention discloses an answer generation method and device, equipment, a storage medium and a product, relates to the technical field of natural language processing, and discloses a method for obtaining a question corresponding to a dialogue instruction in response to the dialogue instruction; taking the question as a search word, and recalling a plurality of target search results, the target search results being search results of which the degree of correlation with the question meets a correlation condition; noise is inserted into the multiple target search results, wherein the noise refers to answers of which the correlation degrees with the questions do not meet the correlation conditions; and through a large language model, based on the question and the plurality of target search results after noise insertion, generating an answer corresponding to the question. The method can ensure the accuracy of the generated answer and improve the answer quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a question answering generation method and device, equipment, a storage medium and a product. BACKGROUND

[0002] With the rapid development of the Internet and the sharp increase in information, users' demand for quickly and accurately obtaining required information is growing, and therefore a question answering system emerges as the times require. The question answering system is a natural language processing technology, which aims to automatically answer questions in the form of natural language proposed by users. Unlike traditional search engines, the question answering system can directly provide answers to users' questions, rather than only relevant web page links.

[0003] In the related art, the question answering system searches for relevant information in a large number of documents by analyzing users' questions, and attempts to extract the most suitable answer for users' requirements from the relevant information. In some cases, the model used by the question answering system can not be sensitive enough to the differences between multiple input data, resulting in low accuracy of the answer given based on the input data.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a question answering generation method, device, equipment, storage medium and product, which can ensure the accuracy of the generated answer and improve the quality of the answer.

[0006] To achieve the above purpose, the present application provides a question answering generation method, which comprises:

[0007] In response to a dialogue instruction, a question corresponding to the dialogue instruction is obtained;

[0008] The question is used as a search term to recall a plurality of target search results, the target search result being a search result whose relevance to the question meets a relevance condition;

[0009] Noise is inserted into the plurality of target search results, the noise being an answer whose relevance to the question does not meet the relevance condition;

[0010] A large language model is used to generate an answer corresponding to the question based on the question and the plurality of target search results with the inserted noise.

[0011] Optionally, the step of using the question as a search term to recall a plurality of target search results comprises:

[0012] The question is used as a search term to recall a plurality of original search results;

[0013] The first number of original search results in the plurality of original search results are determined as the plurality of target search results, or the first number of original search results in the plurality of original search results with the highest degree of relevance to the question are determined as the plurality of target search results.

[0014] Optionally, before the inserting noise into the plurality of target search results, the method further comprises at least one of the following:

[0015] The original search result with the lowest degree of relevance to the question in the plurality of original search results is taken as the noise;

[0016] Any original search result in the plurality of original search results with a degree of relevance lower than a preset threshold is taken as the noise;

[0017] The last original search result in the plurality of original search results is taken as the noise;

[0018] Any original search result in the last second number of original search results in the plurality of original search results is taken as the noise.

[0019] Optionally, the inserting noise into the plurality of target search results comprises:

[0020] Determining the order of the plurality of target search results;

[0021] Inserting the noise into a target order position in the plurality of target search results.

[0022] Optionally, the determining the order of the plurality of target search results comprises:

[0023] Performing intent recognition on the search term to obtain a target intent corresponding to the search term;

[0024] Based on the target intent, determining a target order mode corresponding to the target intent from a correspondence relationship between intents and order modes;

[0025] Ordering the plurality of target search results according to the target order mode.

[0026] Optionally, in the correspondence relationship, the order mode corresponding to the target intent is reverse ordering; and the ordering the plurality of target search results according to the target order mode comprises:

[0027] Determining an original order result of the plurality of target search results;

[0028] Based on the original order result of the plurality of target search results, performing reverse ordering on the plurality of target search results.

[0029] Optionally, the intent recognition on the search term comprises:

[0030] The search term is subjected to multi-level intent recognition, and a highest-level target intent is determined from the multiple levels of recognized intents, wherein a high-level intent is a sub-intent of a low-level intent.

[0031] Optionally, the multi-level intent recognition on the search term comprises:

[0032] The search term is subjected to multi-level intent recognition by multiple intent recognition models, and a highest-level intent is determined from the multiple levels of intents recognized by each intent recognition model.

[0033] A highest-frequency intent is selected from the multiple highest-level intents as the target intent.

[0034] Optionally, the multi-level intent recognition on the search term comprises:

[0035] Based on the search term, intents recognized by each intent recognition model are sequentially obtained in the order of the multiple intent recognition models until the intent recognized by the last intent recognition model is obtained, and the intent recognized by the last intent recognition model is taken as the highest-level target intent; wherein, except for the first intent recognition model, each of the remaining intent recognition models performs intent recognition based on the intent recognized by the previous intent recognition model, and the obtained intent is a sub-intent of the intent recognized by the previous intent recognition model.

[0036] Optionally, the problem is taken as a search term to recall multiple original search results, comprising:

[0037] At least one new search term is expanded based on the search term.

[0038] Information search is performed based on each of the search terms before and after expansion to obtain multiple original search results.

[0039] Optionally, the expansion of at least one new search term based on the search term comprises at least one of the following:

[0040] The search term is split to obtain multiple new search terms after splitting.

[0041] The search term is rewritten to obtain at least one new search term after rewriting.

[0042] splitting the search term, reconstructing at least one new search term based on a result of the splitting;

[0043] correcting the search term to obtain a new search term after correction.

[0044] In addition, to achieve the above object, the application further provides an answer generation device, which comprises:

[0045] a question acquisition module configured to acquire a question corresponding to a dialogue instruction in response to the dialogue instruction;

[0046] an information recall module configured to recall a plurality of target search results as search terms, the target search results being search results whose relevance to the question meets a relevance condition;

[0047] a noise insertion module configured to insert noise into the plurality of target search results, the noise being an answer whose relevance to the question does not meet the relevance condition;

[0048] an answer generation module configured to generate an answer corresponding to the question by a large language model based on the question and the plurality of target search results after the noise is inserted.

[0049] Optionally, the information recall module comprises:

[0050] an information recall unit configured to recall a plurality of original search results as search terms;

[0051] an information determination unit configured to determine a first number of original search results in the plurality of original search results as the plurality of target search results according to a sequence, or determine the first number of original search results in the plurality of original search results as the plurality of target search results according to a relevance to the question.

[0052] Optionally, the device further comprises a noise determination module configured to perform at least one of the following:

[0053] determining an original search result in the plurality of original search results with the lowest relevance to the question as the noise;

[0054] determining any original search result in the plurality of original search results with a relevance lower than a preset threshold to the question as the noise;

[0055] determining an original search result in the plurality of original search results at the end as the noise;

[0056] Any original search result in the second number of original search results ranked at the last in the plurality of original search results is taken as the noise.

[0057] Optionally, the noise insertion module comprises:

[0058] The sorting determination unit is configured to determine the sorting of the plurality of target search results.

[0059] The noise insertion unit is configured to insert the noise into a target sorting position in the plurality of target search results.

[0060] Optionally, the sorting determination unit comprises:

[0061] The intent recognition subunit is configured to perform intent recognition on the search term to obtain a target intent corresponding to the search term.

[0062] The intent determination subunit is configured to determine, based on the target intent, a target sorting manner corresponding to the target intent from a correspondence between intents and sorting manners.

[0063] The result sorting subunit is configured to sort the plurality of target search results according to the target sorting manner.

[0064] Optionally, in the correspondence, the sorting manner corresponding to the target intent is reverse sorting.

[0065] The result sorting subunit is configured to determine an original sorting result of the plurality of target search results, and perform reverse sorting on the plurality of target search results based on the original sorting result of the plurality of target search results.

[0066] Optionally, the intent recognition subunit is configured to perform multi-level intent recognition on the search term, and determine a highest-level target intent from a plurality of levels of intents recognized, wherein an intent at a high level is a sub-intent of an intent at a low level.

[0067] Optionally, the intent recognition subunit is configured to perform multi-level intent recognition on the search term by a plurality of intent recognition models respectively, to determine a highest-level intent from a plurality of levels of intents recognized by each intent recognition model respectively, and to select, from the plurality of highest-level intents determined, an intent with the highest frequency of occurrence as the target intent.

[0068] Optionally, the intent recognition subunit is configured to sequentially obtain the intent identified by each intent recognition model in sequence of the plurality of intent recognition models based on the search term until the intent identified by the last intent recognition model is obtained, and take the intent identified by the last intent recognition model as the highest-level target intent; wherein each intent recognition model other than the first intent recognition model performs intent recognition based on the intent identified by the previous intent recognition model, and the obtained intent is a sub-intent of the intent identified by the previous intent recognition model.

[0069] Optionally, the information retrieval unit comprises:

[0070] a search term expansion subunit configured to expand at least one new search term based on the search term;

[0071] an information search subunit configured to perform information search based on each search term before and after expansion respectively to obtain a plurality of original search results.

[0072] Optionally, the search term expansion subunit is configured to perform at least one of the following:

[0073] split the search term to obtain a plurality of new search terms after splitting;

[0074] rewrite the search term to obtain at least one new search term after rewriting;

[0075] split the search term, and reconstruct at least one new search term based on the splitting result;

[0076] correct the search term to obtain a new search term after correction.

[0077] In addition, to achieve the above object, the present application further provides an answer generation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the answer generation method as described above.

[0078] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the answer generation method as described above.

[0079] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the answer generation method as described above.

[0080] The one or more technical solutions provided by the present application have at least the following technical effects:

[0081] The answer generation scheme provided in the application, in response to the dialogue instruction, obtains a question corresponding to the dialogue instruction, takes the question as a search term, and recalls a plurality of target search results. Considering that the plurality of target search results are search results whose relevance degrees with the question meet a relevance condition, an answer is generated by a large language model based on the plurality of target search results. The model may not be sensitive enough to the differences between the plurality of target search results, thereby affecting the quality of the answer. Therefore, noise is inserted into the plurality of target search results. Since the noise is an answer whose relevance degree with the question does not meet the relevance condition, the question and the plurality of target search results after the noise is inserted are given to the large language model. The noise as a reference can help the large language to determine which of the plurality of target search results is better, so that the large language model is more sensitive to the differences between the plurality of target search results, thereby filtering out content that matches the question more from the plurality of target search results to generate a more accurate answer and improve the quality of the answer. BRIEF DESCRIPTION OF DRAWINGS

[0082] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0083] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0084] Figure 1 A schematic diagram of an implementation environment of the answer generation method of the present application;

[0085] Figure 2 A flowchart provided by the first embodiment of the answer generation method of the present application;

[0086] Figure 3 A flowchart provided by the second embodiment of the answer generation method of the present application;

[0087] Figure 4 A flowchart provided by the third embodiment of the answer generation method of the present application;

[0088] Figure 5 A module structure diagram of the answer generation device of the embodiment of the present application;

[0089] Figure 6 A device structure diagram of the hardware running environment related to the answer generation method in the embodiment of the present application.

[0090] The objectives, functional features and advantages of the present application will be further illustrated in conjunction with the embodiments, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0091] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not used to limit the present application.

[0092] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0093] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure. Referring to Figure 1 , the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected through a wireless or wired network. Illustratively, the terminal 101 is installed with a target application provided by the server 102, and the terminal 101 can realize functions such as data transmission and message interaction through the target application.

[0094] Illustratively, the terminal 101 is a computer, a mobile phone, a tablet computer or other terminals. Illustratively, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is a search application, a short video application, a shopping application, a navigation application, etc. Illustratively, the server 102 is a background server corresponding to the target application. Correspondingly, the server 102 is a search application server, a short video application server, a shopping application server, a navigation application server, etc.

[0095] In the present application, the terminal 101 is used to acquire a question corresponding to a dialogue instruction in response to the dialogue instruction. The question is sent to the server 102. The server 102 receives the question, takes the question as a search term, recalls a plurality of target search results, and the target search results are search results whose relevance to the question meets a relevance condition. Then, noise is inserted into the plurality of target search results, and the noise refers to an answer whose relevance to the question does not meet the relevance condition. Next, through a large language model, an answer corresponding to the question is generated based on the question and the plurality of target search results with the inserted noise. Then, the answer is sent to the terminal 101. The terminal 101 receives and displays the answer.

[0096] Alternatively, the above-mentioned answer generation process can also be completed by the terminal 101 alone. Alternatively, the terminal 101 completes it through the installed target application, such as a search application. The present application embodiments do not limit this.

[0097] The answer generation method provided in the application is applicable to various scenarios. For example, an e-commerce scenario. In a shopping application, a user can ask questions about product information that he or she needs to know, such as product specifications, usage methods, and the like. The terminal quickly finds the product information required by the user by using the method provided in the application, and displays the product information to the user in the form of an answer, thereby improving the shopping experience and conversion rate. For another example, an education scenario. The terminal can use the method provided in the application to answer questions and solve problems for students, assist students in learning, and become an intelligent tutor for students. For another example, a tourism scenario. A user can ask questions about scenic spot information, flight information, accommodation information, and the like that he or she needs to know. The terminal answers the user by using the method provided in the application, and assists the user in making travel arrangements.

[0098] Figure 2 A flowchart of a first embodiment of the answer generation method of the application is shown in FIG. 1. Referring to FIG. 1, the answer generation method includes the following steps S10-S40: Figure 2 Taking a terminal as an execution subject, the answer generation method includes the following steps S10-S40:

[0099] Step S10, in response to a dialogue instruction, obtaining a question corresponding to the dialogue instruction.

[0100] The dialogue instruction is a request or command issued by a user to a system in the form of voice, text, or the like. It can be a direct question, such as “What is the weather in Beijing today?”, or an indirectly expressed demand, such as “I want to know the movie schedule recently”. Regardless of the specific content in the dialogue instruction, the content expressed in the dialogue instruction is collectively referred to as a question.

[0101] For example, the terminal displays a dialogue interface, and the user inputs any dialogue content through the dialogue interface. The terminal generates a dialogue instruction in response to the dialogue content input by the user, and takes the dialogue content input by the user as a question corresponding to the dialogue instruction.

[0102] Step S20, taking the question as a search term, recalling a plurality of target search results, the target search results being search results whose relevance to the question meets a relevance condition.

[0103] The search term is a keyword or phrase used to find information in a database, a knowledge base, or the Internet. The question is taken as a search term in the application, and a plurality of target search results, such as document fragments, web links, database records, and the like, are recalled by using the search term, which are used as information sources for generating answers by subsequent models.

[0104] Optionally, the terminal can recall target search results from various information sources based on the search term. Exemplarily, the information sources include: a pre-built knowledge base, web content, a knowledge graph of a specific field, community-generated content such as content contributed by users on social media platforms, forums, etc., and third-party provided data services such as weather forecast services, stock market services, etc.

[0105] Optionally, the terminal recalls a plurality of original search results based on the search term, and then filters target search results from the plurality of original search results according to a relevance condition.

[0106] The relevance degree is used to measure the matching degree of a search result with the question. The relevance condition is a series of standards or rules preset to determine whether a search result is relevant enough to be selected as a target search result. That is, the relevance condition is used to filter target search results. The relevance condition can include time limits, sorting limits, quantity limits, relevance degree limits, etc., to filter target search results that meet multiple limit conditions in the relevance condition.

[0107] Exemplarily, the relevance condition includes a relevance degree threshold, and the terminal determines the original search results with a relevance degree greater than the relevance degree threshold as target search results. Alternatively, the relevance condition includes a first quantity, and the terminal determines the original search results in the first quantity in the order of sorting as target search results. Since the original search results in the first quantity are generally more relevant to the question, this method can filter target search results with a higher matching degree with the question. Alternatively, the terminal determines the original search results with the highest relevance degree with the question in the first quantity as target search results. That is, the plurality of original search results are first sorted in descending order of relevance degree with the question, and then the first quantity of original search results in the order of sorting are selected as target search results.

[0108] In step S30, noise is inserted into the plurality of target search results, where the noise refers to an answer with a relevance degree with the question that does not meet the relevance condition.

[0109] The noise does not meet the relevance condition with the question, indicating that the noise has a lower matching degree with the question. Subsequently, the noise is inserted into the plurality of target search results, and the large language model can easily identify the noise from the plurality of target search results after the noise is inserted.

[0110] Optionally, the terminal inserts noise into the plurality of search results randomly, that is, a position of inserting the noise is determined randomly, and the noise is inserted into the plurality of target search results according to the position. Alternatively, the terminal determines the order of the plurality of target search results first, and inserts the noise into a target order position in the plurality of target search results. The target order position can be determined according to an application scenario, for example, the target order position is between the last target search result and the second last target search result, and the present application does not limit this.

[0111] In step S40, the large language model generates an answer corresponding to the question based on the question and the plurality of target search results with the inserted noise.

[0112] For example, the question and the plurality of target search results with the inserted noise are input into the large language model, and the large language model generates an answer corresponding to the question based on the question and the plurality of target search results with the inserted noise, thereby outputting the answer. Then, the terminal can display the answer in the dialogue interface for the user to view.

[0113] The answer generation scheme provided in the present application responds to the dialogue instruction, obtains a question corresponding to the dialogue instruction, takes the question as a search term, and recalls a plurality of target search results. Considering that the plurality of target search results are all search results whose relevance to the question meets a relevant condition, the large language model generates an answer based on the plurality of target search results, and the model may not be sensitive enough to the differences between the plurality of target search results, thereby affecting the quality of the answer. Therefore, noise is inserted into the plurality of target search results. Since the noise is an answer whose relevance to the question does not meet the relevant condition, the large language model is given the question and the plurality of target search results with the inserted noise. The noise as a reference can help the large language to determine which of the plurality of target search results is better, so that the large language model is more sensitive to the differences between the plurality of target search results, thereby filtering out more matching content from the plurality of target search results to generate a more accurate answer and improve the quality of the answer.

[0114] Based on the above first embodiment, the second embodiment of the present application is proposed. The same or similar contents as the first embodiment can be referred to the above description, and will not be described in detail hereinafter. Refer to Figure 3 In the second embodiment, the step S20 includes steps S201-S202.

[0115] In step S201, the question is taken as a search term, and a plurality of original search results are recalled.

[0116] Optionally, the implementation manner of the step is as follows: at least one new search term is expanded based on the search term; and information search is performed based on each search term before and after expansion, to obtain a plurality of original search results.

[0117] The implementation manner of expanding at least one new search term based on the search term includes at least one of the following:

[0118] Firstly, the search term is split to obtain a plurality of new search terms after splitting. Splitting refers to decomposing a complex search term into a plurality of simple and independent search terms. For example, the original search term is "A amusement park ticket price", and the plurality of new search terms obtained after splitting include "A amusement park" and "ticket price".

[0119] Secondly, the search term is rewritten to obtain at least one new search term after rewriting. Rewriting refers to replacing the original search term with a search term with the same meaning but different expression by using a synonym or the like. For example, the original search term is "Where is the best Chinese restaurant?", and the new search term after rewriting is "Nearby Chinese restaurant with high rating".

[0120] Thirdly, the search term is split, and at least one new search term is reconstructed based on the splitting result. Reconstruction refers to reorganizing the split search terms to obtain a search term with the same meaning but different expression as the original search term. For example, the original search term is "Tomorrow's weather forecast in B area", and the new search term reconstructed based on the splitting result is "Tomorrow's weather forecast in B area".

[0121] Fourthly, the search term is corrected to obtain a new search term after correction. Correction refers to correcting spelling errors, grammatical errors, and the like in the original search term to obtain a new search term. For example, the original search term is "Name tomorrow rain?", and the new search term after correction is "Will it rain tomorrow?".

[0122] In the embodiments of the present application, new search terms are expanded by rewriting, splitting, reconstructing, correcting, and the like of the search term, and information search is performed based on each search term before and after expansion to obtain a plurality of original search results. On the one hand, the search system can better understand the user's intention and obtain more relevant original search results. On the other hand, the number of recalled original search results can be increased to provide more data support for generating answers by a large language model.

[0123] In step S202, a first number of original search results ranked in the front among the plurality of original search results are determined as the plurality of target search results, or a first number of original search results with the highest relevance to the question among the plurality of original search results are determined as the plurality of target search results.

[0124] The first quantity is a preset quantity of target search results, and the first quantity of target search results are subsequently used as information sources, and the large language model generates answers corresponding to the question by referring to the plurality of target search results. For example, the terminal recalls 100 original search results by taking the question as a search word. The first quantity is 10, and the terminal selects 10 target search results from the 100 original search results.

[0125] It should be noted that the plurality of original search results recalled are sorted according to information such as a relevance of the search results to the question, a number of times that a keyword in the question appears in a summary of the search result, authority of the search result, timeliness of the search result, and the like. Therefore, generally, the original search results with a higher relevance to the question are sorted in front, but it cannot be guaranteed that the original search result sorted in front has a higher relevance to the question than the original search result sorted in back. Therefore, whether the terminal determines the first quantity of original search results sorted in front in the plurality of original search results as the plurality of target search results, or determines the first quantity of original search results with the highest relevance to the question in the plurality of original search results as the plurality of target search results, can guarantee that the determined plurality of target search results are strongly related to the question, that is, have a higher matching degree with the question, and are beneficial to the model to generate high-quality answers by referring to the target search results.

[0126] Before inserting noise into the plurality of target search results, the noise is acquired. Optionally, the implementation manner of acquiring the noise includes at least one of the following:

[0127] First, any original search result with a lowest relevance to the question in the plurality of original search results is taken as the noise. In this way, it can be guaranteed that the determined noise has a lower relevance to the question, and it is convenient for the subsequent model to distinguish the noise from the plurality of target search results after inserting the noise.

[0128] Second, any original search result with a relevance lower than a preset threshold in the plurality of original search results is taken as the noise. The preset threshold is a preset relevance threshold, and the preset threshold is set to be small in value. Therefore, any original search result with a relevance lower than the preset threshold has a relatively low relevance to the question, and it is convenient for the subsequent model to distinguish the noise from the plurality of target search results after inserting the noise.

[0129] Third, the original search result ranked last in the plurality of original search results is taken as the noise. Since the original search result ranked last generally has a lower relevance to the question, it is convenient for the subsequent model to distinguish the noise from the plurality of target search results after inserting the noise.

[0130] Fourthly, any one of the second number of original search results ranked at the last in the plurality of original search results is taken as noise. The second number can be set according to needs. For example, the number of original search results is 100, and the second number is 5. Then, the terminal selects any one of the 5 original search results ranked at the last in the 100 original search results as the target search result. Since the number of original search results is large and the second number is small, the relevance of any one of the second number of original search results ranked at the last to the question is relatively low. Therefore, any one of the second number of original search results ranked at the last is taken as noise, which facilitates the subsequent model to distinguish the noise from the plurality of target search results with the inserted noise.

[0131] Based on the above first embodiment of the present application, the third embodiment of the present application is proposed. The same or similar contents as the first embodiment can be referred to the above description, and will not be described in detail hereinafter. Refer to Figure 4 In the third embodiment, the step S30 includes the step S301 and the step S302.

[0132] The step S301 determines the ranking of the plurality of target search results.

[0133] Optionally, the ranking of the plurality of target search results is consistent with the ranking of the plurality of target search results in the plurality of original search results. That is, the original search result with a high ranking position in the original search results still has a high ranking position in the plurality of target search results. For example, there are 100 original search results recalled, and the top 10 original search results are determined as target search results. Then, the ranking of the 10 target search results remains unchanged.

[0134] Optionally, the plurality of target search results are reordered according to the intent of the search term. Correspondingly, the terminal performs intent recognition on the search term to obtain a target intent corresponding to the search term; based on the target intent, a target ranking mode corresponding to the target intent is determined from the correspondence between the intent and the ranking mode; and the plurality of target search results are ranked according to the target ranking mode.

[0135] The ranking mode refers to arranging the plurality of target search results according to certain standards, and the selection of the ranking mode depends on the understanding of the user intent. According to needs, the correspondence between the intent and the ranking mode can be customized. For example, the intent is news query, and the ranking mode is time-sensitive ranking, i.e., the events published recently or about to happen are ranked first. For example, the intent is academic information query, and the ranking mode is website authority-based ranking, i.e., the contents of the website with higher authority are ranked first. For example, the intent is product information query, and the ranking mode is relevance-based display, i.e., the search results with higher relevance to the intent are ranked first.

[0136] The embodiment of the application selects a matched result ranking mode according to different types of intents, which can ensure that the target search results that are more consistent with user needs are ranked in the front, so that when the subsequent large language model generates an answer based on the ranked multiple target search results, the model can filter important information, thereby generating an answer that is more consistent with user needs.

[0137] Optionally, in the correspondence between the intent and the ranking mode, the ranking mode corresponding to the target intent is reverse ranking. Correspondingly, ranking the multiple target search results according to the target ranking mode includes: determining the original ranking result of the multiple target search results; and performing reverse ranking on the multiple target search results based on the original ranking result of the multiple target search results.

[0138] The original ranking result of the target search results is consistent with the ranking of the multiple target search results in the multiple original search results. For example, there are 100 original search results recalled, and the top 10 original search results are determined as target search results, and the ranking of the 10 target search results remains unchanged. In this case, if the ranking mode corresponding to the target intent is reverse ranking, the 10 target search results are reverse ranked. After reverse ranking, the target search result originally ranked first is ranked tenth, the target search result originally ranked second is ranked ninth, and so on.

[0139] For example, the ranking mode corresponding to the target intent of the knowledge question and answer type is reverse ranking. After testing, the quality of the answer of the large language model is improved by more than 5% after reverse ranking of the target search results corresponding to the search words corresponding to the target intent of the knowledge answer type.

[0140] For example, for any intent, the original ranking result and the reverse ranking result of the target search results corresponding to the search words corresponding to the intent are obtained. Through the large language model, the answer corresponding to the question is generated based on the original ranking result and the reverse ranking result respectively. By comparing the quality of the two answers, if the quality of the answer corresponding to the reverse ranking result is higher, the terminal sets the ranking mode corresponding to the intent as reverse ranking. In this way, multiple intents corresponding to the reverse ranking mode can be determined. Then when answering questions belonging to this type of intent, the quality of the generated answer can be improved by reverse ranking the target search results.

[0141] Optionally, the terminal performs intent recognition on the search words to obtain the target intent corresponding to the search words, including: performing multi-level intent recognition on the search words, and determining a highest-level target intent from the multiple levels of intents recognized, wherein the high-level intent is a sub-intent of the low-level intent.

[0142] Intent recognition refers to inferring the specific needs or purposes of a user by analyzing the user's search words. It is not just a simple keyword matching, but a deep understanding of the meaning behind the user's query. In many cases, the user's expression reflects not only a single intent, but also a series of complex and interrelated intents. The goal of multi-level intent recognition is to deeply mine the user's multi-level intent to provide more accurate information to the user.

[0143] The number of levels of intent recognition can be set as needed. For example, three levels of intent recognition are performed on the search words, namely primary intent recognition, secondary intent recognition, and high-level intent recognition. The intent recognized by secondary intent recognition is a sub-intent of the intent recognized by primary intent recognition, and the intent recognized by high-level intent recognition is a sub-intent of the intent recognized by secondary intent recognition. For example, for the search word "write an article about A winning the championship", the result of primary intent recognition may be "writing", the result of secondary intent recognition may be "writing a news article", and the result of high-level intent recognition may be "writing a news article about athlete A winning the championship". For example, for the search word "best Chinese restaurant in A area", the result of primary intent recognition may be "finding a restaurant", the result of secondary intent recognition may be "finding the best restaurant in A area", and the result of high-level intent recognition may be "finding the highest-rated Chinese restaurant in A area".

[0144] It can be understood that, compared with low-level intents, the highest-level intent contains more details and the intent expression is more accurate. Therefore, taking the highest-level intent as the target intent can ensure the accuracy of the determined target intent, thereby ensuring the adaptability of the subsequent selected target search result sorting mode to the question.

[0145] In the embodiments of the present application, multi-level intent recognition is performed on the search words, and the highest-level target intent is determined from the multiple levels of intents recognized. There are two implementation ways as follows:

[0146] First, the terminal performs multi-level intent recognition on the search words through multiple intent recognition models, respectively determines the highest-level intent from the multiple levels of intents recognized by each intent recognition model, and selects the highest frequency as the target intent from the multiple highest-level intents determined.

[0147] The multiple intent recognition models work in parallel. Each intent recognition model independently performs intent recognition on the search words, and the obtained intent recognition results are independent. For example, five intent recognition models independently perform three-level intent recognition on the search words, and each obtains the highest-level intent. Then the terminal selects the highest frequency from the five highest-level intents as the target intent.

[0148] By parallel working of multiple intent recognition models, multi-level intent recognition is performed on the search word respectively, so as to capture the user's intent from different angles and levels and improve the accuracy of intent recognition. The highest frequency of multiple top-level intents is selected as the target intent, which can avoid the deviation or limitation of a single model, and ensure the reliability of the intent recognition result even for complex search words.

[0149] Secondly, the terminal sequentially obtains the intent recognized by each intent recognition model according to the order of the multiple intent recognition models based on the search word, until the intent recognized by the last intent recognition model is obtained, and the intent recognized by the last intent recognition model is taken as the target intent of the highest level.

[0150] The multiple intent recognition models are sequentially worked. The output of a previous intent recognition model is taken as the input of a next intent recognition model. The next intent recognition model is used to refine the intent recognized by the previous intent recognition model, so that the intent is gradually refined through the multiple intent recognition models, and the accurate intent output by the last intent recognition model is obtained.

[0151] In the embodiments of the present application, the user's intent can be gradually and deeply understood by sequentially using multiple intent recognition models, from general to specific, and a hierarchical intent recognition system is constructed. Each model further refines the intent on the basis of the previous model, which helps to capture more specific and more refined user requirements. And each model has the opportunity to correct the possible errors of the previous model, thereby improving the accuracy of the final intent recognition.

[0152] It should be noted that the multiple intent recognition models involved in the present application can be the same or different. And each intent recognition model can be any type of model, such as convolutional neural network, deep learning model, etc., which is not limited in the embodiments of the present application. In addition, the data used for training each intent recognition model can be self-defined intent classification data. The intent classification data includes a large number of search words and corresponding multiple levels of intents.

[0153] In step S302, the noise is inserted into the target ranking position in the multiple target search results.

[0154] The target sorting position can be set as needed. For example, the target sorting position is between any two target search results in the latter half of the sorted target search results. For example, there are 10 target search results, and the target sorting position is between any two target search results in the latter 5 target search results. Of course, the target sorting position can also be specific to a certain fixed position. For example, the target sorting position is between the 8th and 9th target search results. Tests show that inserting noise between any two target search results in the latter half of the sorted target search results can generate higher-quality answers from the large language model than inserting noise between any two target search results in the former half of the sorted target search results.

[0155] It should be noted that since the question is generally longer and contains more keywords, when the question is used as a search term to recall search results, the search results that are more relevant to the question may be ranked at the end of the multiple search results. Therefore, inserting noise into the latter position of the multiple target search results can make the model's reasoning path for the question clearer, thereby improving the quality of the model's answers.

[0156] Another point to note is that if only the multiple target search results are input into the large language model, since the multiple target search results are all strongly related to the question, the model is not sensitive enough to the differences between the multiple target search results, resulting in the model not carefully comparing the multiple target search results and randomly generating answers to the question. For example, the model can randomly sort and output the multiple target search results, or randomly fuse and output the multiple target search results. Unlike this, once noise is inserted into the multiple target search results, the model can use the noise as a reference to avoid bad answers such as noise, and will more carefully distinguish the differences between the multiple target search results, thereby filtering out more matching content from the multiple target search results to generate answers to the question, thereby improving the quality of the answers.

[0157] Another point to note is that the above examples are only for understanding the present application and do not constitute a limitation on the answer generation method of the present application. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0158] The present application also provides an answer generation device, please refer to Figure 5 The answer generation device comprises:

[0159] The question acquisition module 10 is configured to acquire a question corresponding to the dialogue instruction in response to the dialogue instruction.

[0160] The information recall module 20 is configured to use the question as a search term to recall multiple target search results, and the target search results are search results that meet a relevance condition in terms of relevance to the question.

[0161] a noise insertion module 30 configured to insert noise into the plurality of target search results, the noise being an answer that does not meet a relevance condition in terms of relevance to the question;

[0162] a response generation module 40 configured to generate, by using a large language model, an answer corresponding to the question based on the question and the plurality of target search results into which the noise is inserted.

[0163] Optionally, the information retrieval module 20 comprises:

[0164] an information retrieval unit configured to retrieve a plurality of original search results by taking the question as a search term;

[0165] an information determination unit configured to determine, as the plurality of target search results, a first number of original search results that are ranked first among the plurality of original search results, or determine, as the plurality of target search results, a first number of original search results that have the highest relevance to the question among the plurality of original search results.

[0166] Optionally, the apparatus further comprises a noise determination module configured to perform at least one of the following:

[0167] determining, as the noise, an original search result that has the lowest relevance to the question among the plurality of original search results;

[0168] determining, as the noise, any original search result that has a relevance lower than a preset threshold value to the question among the plurality of original search results;

[0169] determining, as the noise, an original search result that is ranked last among the plurality of original search results;

[0170] determining, as the noise, any original search result among a second number of original search results that are ranked last among the plurality of original search results.

[0171] Optionally, the noise insertion module 30 comprises:

[0172] a ranking determination unit configured to determine a ranking of the plurality of target search results;

[0173] a noise insertion unit configured to insert the noise into a target ranking position among the plurality of target search results.

[0174] Optionally, the ranking determination unit comprises:

[0175] an intent recognition subunit configured to perform intent recognition on the search term to obtain a target intent corresponding to the search term;

[0176] an intent determination subunit configured to determine, based on the target intent, a target ranking manner corresponding to the target intent from a correspondence between intents and ranking manners.

[0177] a result sorting subunit, configured to sort the plurality of target search results according to a target sorting manner.

[0178] Optionally, in the corresponding relationship, the sorting manner corresponding to the target intent is reverse sorting.

[0179] a result sorting subunit, configured to determine original sorting results of the plurality of target search results, and perform reverse sorting on the plurality of target search results based on the original sorting results of the plurality of target search results.

[0180] Optionally, the intent recognition subunit is configured to perform multi-level intent recognition on the search term, and determine a highest-level target intent from the plurality of levels of intents recognized.

[0181] Optionally, the intent recognition subunit is configured to perform multi-level intent recognition on the search term by using a plurality of intent recognition models respectively, determine a highest-level intent from the plurality of levels of intents recognized by each of the intent recognition models respectively, and select the target intent as the highest-level intent with the highest frequency of occurrence from the plurality of highest-level intents determined.

[0182] Optionally, the intent recognition subunit is configured to sequentially obtain an intent recognized by each of the intent recognition models in an order of the plurality of intent recognition models based on the search term until an intent recognized by a last intent recognition model is obtained, and take the intent recognized by the last intent recognition model as the highest-level target intent; wherein, each of the intent recognition models other than a first intent recognition model performs intent recognition based on an intent recognized by a previous intent recognition model, and the obtained intent is a sub-intent of the intent recognized by the previous intent recognition model.

[0183] Optionally, the information retrieval unit comprises:

[0184] a search term expansion subunit, configured to expand at least one new search term based on the search term;

[0185] an information search subunit, configured to perform information search based on each of the search term before expansion and the search term after expansion respectively to obtain a plurality of original search results.

[0186] Optionally, the search term expansion subunit is configured to perform at least one of the following:

[0187] split the search term to obtain a plurality of new search terms after splitting;

[0188] rewrite the search term to obtain at least one new search term after rewriting;

[0189] split the search term, and reconstruct at least one new search term based on a splitting result.

[0190] The search term is corrected to obtain a new search term after correction.

[0191] The answer generation scheme provided by the present application, in response to the dialogue instruction, obtains the question corresponding to the dialogue instruction, takes the question as the search term, and recalls a plurality of target search results. Considering that the plurality of target search results are search results whose relevance to the question meets the relevant condition, the model may not be sensitive enough to the differences between the plurality of target search results, thereby affecting the quality of the answer. Therefore, noise is inserted into the plurality of target search results. Since the noise is an answer whose relevance to the question does not meet the relevant condition, the question and the plurality of target search results after inserting the noise are given to the large language model. The noise as a reference can help the large language to determine which one of the plurality of target search results is better, so that the large language model is more sensitive to the differences between the plurality of target search results, thereby filtering out the content that matches the question more from the plurality of target search results to generate a more accurate answer and improve the quality of the answer.

[0192] The answer generation device provided by the present application adopts the answer generation method in the above-mentioned embodiments, and can solve the technical problem that the model in the related art is not sensitive enough to the differences between a plurality of input data, resulting in low accuracy of the answer generated based on the input data. Compared with the prior art, the answer generation device provided by the present application has the same beneficial effects as the answer generation method provided by the above-mentioned embodiments, and other technical features in the answer generation device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0193] The present application provides an answer generation device, which comprises at least one processor and a memory in communication connection with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the answer generation method in Embodiment I.

[0194] Reference will now be made to Figure 6 which shows a structural schematic diagram of an answer generation device suitable for implementing the embodiments of the present application. The answer generation device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like.Figure 6 The answer generation device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0195] like Figure 6 As shown, the answer generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the answer generation device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the answer generation device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows an answer generation device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have instead.

[0196] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0197] The answering generation device provided in the application adopts the answering generation method in the above embodiments, and can solve the technical problem that the model is not sensitive enough to the differences between multiple input data in the related art, resulting in low accuracy of the answer generated based on the input data. Compared with the prior art, the answering generation device provided in the application has the same beneficial effects as the answering generation method provided in the above embodiments, and other technical features in the answering generation device are the same as the features disclosed in the previous embodiment method, which will not be described here.

[0198] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0199] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0200] The present application provides a computer readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the answering generation method in the above embodiments.

[0201] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to: electrical wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0202] The computer-readable storage medium may be included in the answer generation device, or may exist independently without being incorporated into the answer generation device.

[0203] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the answer generation device, the answer generation device: responds to the dialogue instruction and obtains the question corresponding to the dialogue instruction; uses the question as the search term and recalls multiple target search results, where the target search results are search results whose relevance to the question meets the relevant conditions; inserts noise into the multiple target search results, where noise refers to answers whose relevance to the question does not meet the relevant conditions; and generates an answer corresponding to the question through a large language model based on the question and the multiple target search results after the noise is inserted.

[0204] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0205] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0206] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0207] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned answer generation method, and can solve the technical problem that the model is not sensitive enough to the differences between multiple input data in the related art, resulting in low accuracy of the answer generated based on the input data. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the answer generation method provided by the above-mentioned embodiments, which will not be repeated here.

[0208] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the answer generation method as described above.

[0209] The computer program product provided by the present application can solve the technical problem that the model is not sensitive enough to the differences between multiple input data in the related art, resulting in low accuracy of the answer generated based on the input data. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the answer generation method provided by the above-mentioned embodiments, which will not be repeated here.

[0210] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the contents of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for generating an answer, characterized in that: The method comprises: In response to a dialogue instruction, obtaining a question corresponding to the dialogue instruction; Using the question as a search term, recalling multiple target search results, wherein the target search results are search results whose relevance to the question meets relevant conditions; inserting noise into the plurality of target search results, wherein the noise refers to answers whose relevance to the question does not meet the relevance condition; An answer corresponding to the question is generated through a large language model based on the question and the multiple target search results after the noise is inserted.

2. The method according to claim 1, wherein The question is used as a search term to retrieve multiple target search results, including: Using the question as a search term, multiple original search results are retrieved; The first number of original search results ranked at the top among the multiple original search results are determined as the multiple target search results, or the first number of original search results with the highest relevance to the question among the multiple original search results are determined as the multiple target search results.

3. The method according to claim 2, wherein Before inserting noise into the plurality of target search results, the method further includes at least one of the following: Using the original search result with the lowest relevance to the question among the multiple original search results as the noise; any original search result among the plurality of original search results, the relevance of which to the question is lower than a preset threshold, is used as the noise; The original search result ranked last among the multiple original search results is used as the noise; Any original search result in the second number of original search results that are ranked last among the multiple original search results is used as the noise.

4. The method according to claim 1, wherein The inserting noise into the plurality of target search results includes: Determining the ranking of the plurality of target search results; The noise is inserted into a target ranking position in the plurality of target search results.

5. The method according to claim 4, wherein Determining the ranking of the plurality of target search results includes: Performing intent recognition on the search term to obtain a target intent corresponding to the search term; Based on the target intent, determining a target sorting method corresponding to the target intent from a correspondence between intent and sorting method; The multiple target search results are sorted according to the target sorting method.

6. The method according to claim 5, wherein In the corresponding relationship, the sorting method corresponding to the target intent is reverse sorting; and sorting the multiple target search results according to the target sorting method includes: Determining original ranking results of the plurality of target search results; Based on the original sorting results of the multiple target search results, the multiple target search results are sorted in reverse order.

7. An answer generation device, characterized in that: The device comprises: A question acquisition module, configured to acquire, in response to a dialogue instruction, a question corresponding to the dialogue instruction; An information recall module, configured to use the question as a search term and recall a plurality of target search results, wherein the target search results are search results whose relevance to the question meets relevant conditions; a noise insertion module, configured to insert noise into the plurality of target search results, wherein the noise refers to answers whose relevance to the question does not meet the relevance condition; The answer generation module is used to generate an answer corresponding to the question based on the question and the multiple target search results after the noise is inserted through a large language model.

8. An answer generation device, characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the answer generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the answer generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which implements the steps of the answer generation method according to any one of claims 1 to 6 when executed by a processor.

Images