Dialogue processing method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a dialogue processing method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of computers, in particular to the technical field of artificial intelligence such as deep learning, large models and natural language processing. According to the specific scheme, the method comprises the steps that firstly, a common question answering FAQ library is retrieved based on an obtained input statement, a query statement set is generated based on the input statement under the condition that the FAQ library does not contain an answer corresponding to the input statement, then a preset knowledge base is retrieved based on the query statement set, retrieval data are obtained, and the retrieval data are stored in a database; and then generating a first reply statement based on the input statement, the query statement set and the retrieval data, and finally outputting the first reply statement.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as deep learning, big models, and natural language processing, and specifically to a dialogue processing method, device, electronic device, and storage medium based on a big model. Background Art

[0002] In the prior art, a retrieval system usually first performs a search based on an acquired query statement, and then returns the search results to the user, and then the user obtains the desired answer from the returned search results. However, the answers generated in this way are not accurate and efficient. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] The first embodiment of the present disclosure proposes a dialog processing method based on a large model, including:

[0005] Based on the obtained input sentence, retrieve the FAQ database;

[0006] In the case that the FAQ library does not contain an answer corresponding to the input sentence, generating a query sentence set based on the input sentence;

[0007] Based on the query statement set, a preset knowledge base is searched to obtain search data;

[0008] Generate a first reply statement based on the input statement, the query statement set and the search data;

[0009] Output the first reply sentence.

[0010] The second aspect of the present disclosure provides a dialog processing device based on a large model, including:

[0011] A retrieval module, used for retrieving a FAQ database based on the acquired input sentence;

[0012] A first generating module, configured to generate a query statement set based on the input statement when the FAQ library does not contain an answer corresponding to the input statement;

[0013] An acquisition module, used to search a preset knowledge base based on the query statement set to acquire search data;

[0014] A second generating module, used for generating a first reply statement based on the input statement, the query statement set and the search data;

[0015] An output module is used to output the first reply statement.

[0016] The third aspect of the present disclosure proposes a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the large model-based dialogue processing method proposed in the first aspect of the present disclosure is implemented.

[0017] The fourth aspect of the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the large model-based dialogue processing method provided in the first aspect of the present disclosure is implemented.

[0018] The fifth aspect of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the large model-based dialogue processing method as provided in the first aspect of the present disclosure.

[0019] The large model-based dialogue processing method, device, electronic device, and storage medium provided by the present disclosure have the following beneficial effects:

[0020] In the disclosed embodiment, firstly, based on the acquired input sentence, the FAQ library of frequently asked questions is searched. If the FAQ library does not contain the answer corresponding to the input sentence, a query statement set is generated based on the input sentence. Then, based on the query statement set, a preset knowledge base is searched to obtain the search data. Then, based on the input sentence, the query statement set and the search data, a first reply statement is generated, and finally, the first reply statement is output. Thus, by firstly searching the FAQ library based on the input sentence, if the FAQ library does not contain the answer corresponding to the input sentence, a query statement set corresponding to the input sentence is obtained, and based on the query statement set, the knowledge base is searched to obtain the search data. Finally, based on the input sentence, the query statement set and the search data, the answer is obtained and output, thereby improving the accuracy of the generated answer and improving the efficiency and reliability of the dialogue processing method based on the large model.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0024] Figure 2A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0025] Figure 3 A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0026] Figure 4 A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0027] Figure 5 A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0028] Figure 6 A flowchart of a large model-based dialogue processing method provided by an embodiment of the present disclosure;

[0029] Figure 7 A flowchart of a conversation processing method based on a large model provided in an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of the structure of a large model-based dialogue processing device provided in an embodiment of the present disclosure;

[0031] Fig. 9 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] The present disclosure relates to artificial intelligence technology fields such as deep learning, large models, and natural language processing.

[0034] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0035] Deep Learning (DL) is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0036] The large model can also be called the Foundation Model. The model extracts knowledge through billions of corpora or images, learns and then produces a large model with billions of parameters.

[0037] Natural Language Processing (NLP) is an interdisciplinary subject in the fields of computer science, artificial intelligence and linguistics. It mainly studies how to enable computers to understand, process, generate and simulate human language, so as to achieve the ability to have natural conversations with humans.

[0038] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] The following describes a large model-based dialog processing method, device, electronic device, and storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0040] Figure 1 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0041] like Figure 1 As shown, the dialogue processing method based on the large model may include the following steps:

[0042] Step 101, based on the acquired input sentence, search the FAQ database for answers to frequently asked questions.

[0043] It should be noted that the input sentence can be a sentence directly input by the user, or it can also be an input sentence obtained after recognizing multimodal information (such as text, pictures, voice, etc.) input by the user. The present disclosure does not limit this.

[0044] Among them, FAQ is the abbreviation of Frequently Asked Questions (FAQ).

[0045] It should be noted that the FAQ library may include any number of frequently asked questions and their answers, and the content thereof may be preset, which is not limited in the present disclosure.

[0046] In the present disclosure, a preset knowledge base can be parsed, and questions in the knowledge base can be extracted using a large model. When extracting questions, the large model can adjust prompt information based on needs, update the question extraction direction and extract questions, and then use the large model to determine the answer corresponding to the question to obtain the corresponding question and answer (Question Answering, QA) pair. Thereafter, the QA pairs are evaluated through preset evaluation indicators in combination with the large model. The evaluation can be based on multiple dimensions such as the quality, factuality, and format of the answer. Finally, the QA pairs that pass the evaluation are stored in the FAQ library, thereby improving the quality of the QA pairs in the FAQ library. The present disclosure does not limit this.

[0047] In the present disclosure, after obtaining an input sentence, the FAQ library may be searched first to match the answer corresponding to the input sentence, thereby improving the efficiency and accuracy of obtaining the answer corresponding to the input sentence.

[0048] Step 102: if the FAQ database does not contain an answer corresponding to the input sentence, generate a query sentence set based on the input sentence.

[0049] It should be noted that the query statement set may be a set of query statements related to the input statement, which may include any number of query statements, and the present disclosure does not limit this.

[0050] In the present disclosure, after searching the FAQ library based on the acquired input statement, if the FAQ library does not contain the answer corresponding to the input statement, in order to obtain the answer corresponding to the input statement and ensure the reliability of the answer, a query statement set can be first generated based on the input statement.

[0051] It should be noted that after searching the FAQ database based on the acquired input sentence, if the FAQ database contains an answer corresponding to the input sentence, the answer can be directly output to the user, and the present disclosure does not limit this.

[0052] Step 103: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0053] Among them, the preset knowledge base may be a knowledge base containing multimodal data, which may include multimodal data such as text, images, tables, etc., and the data contained therein may be vectorized data, which is not limited in the present disclosure.

[0054] It should be noted that the knowledge base also includes meta-information, which can be used to characterize the attribute information of the data in the knowledge base, which can be obtained by extracting the attribute information of the data in the knowledge base through a large model. For example, the meta-information may include the location of the data in the document, the title, author, time, and other information of the document, so that data can be traced back when searching the knowledge base, thereby improving the reliability and comprehensiveness of the retrieved data obtained, which is not limited in this disclosure.

[0055] It should be noted that when searching the preset knowledge base to obtain search data, the search data can also be sorted based on the relevance between the search data and the query statement set, thereby providing conditions for improving the accuracy of the answer to the generated input statement.

[0056] It should be noted that the search data may be multimodal data, for example, the search data may be text data, image data, etc., and the present disclosure does not limit this.

[0057] In the present disclosure, after generating a query statement set based on an input statement, a preset knowledge base may be searched based on the query statement set to obtain search data, thereby providing a data basis for determining the answer corresponding to the input statement.

[0058] Step 104, generating a first reply statement based on the input statement, the query statement set and the search data.

[0059] Among them, the first reply sentence is the answer corresponding to the input sentence.

[0060] In the present disclosure, after obtaining the retrieval data, a first reply statement can be generated based on the input statement, the query statement set and the retrieval data, thereby improving the accuracy of the generated first reply statement.

[0061] It should be noted that when generating the first reply statement based on the input statement, the query statement set and the retrieval data, the first reply statement can be generated based on a preset large model, and there is no public limitation on this.

[0062] Step 105, output the first reply statement.

[0063] In the present disclosure, after the first reply statement is generated, the first reply statement can be output and fed back to the user.

[0064] In the disclosed embodiment, firstly, based on the acquired input sentence, the FAQ library of frequently asked questions is searched. If the FAQ library does not contain the answer corresponding to the input sentence, a query statement set is generated based on the input sentence. Then, based on the query statement set, a preset knowledge base is searched to obtain the search data. Then, based on the input sentence, the query statement set and the search data, a first reply statement is generated, and finally, the first reply statement is output. Thus, by firstly searching the FAQ library based on the input sentence, if the FAQ library does not contain the answer corresponding to the input sentence, a query statement set corresponding to the input sentence is obtained, and based on the query statement set, the knowledge base is searched to obtain the search data. Finally, based on the input sentence, the query statement set and the search data, the answer is obtained and output, thereby improving the accuracy of the generated answer and improving the efficiency and reliability of the dialogue processing method based on the large model.

[0065] Figure 2 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0066] like Figure 2 As shown, the dialogue processing method based on the large model may include the following steps:

[0067] Step 201, based on the acquired input sentence, search the FAQ database for answers to frequently asked questions.

[0068] The specific implementation form of step 201 can refer to the detailed description in other embodiments of the present disclosure, and will not be described in detail here.

[0069] Step 202: If the FAQ database does not contain an answer corresponding to the input statement, the input statement is processed by at least one of disassembling, expanding, and rewriting to generate a query statement set.

[0070] In the present disclosure, after searching the FAQ library based on an input statement, if the FAQ library does not contain an answer corresponding to the input statement, in order to obtain the answer corresponding to the input statement, the input statement can first be disassembled, expanded, and rewritten to generate a query statement set corresponding to the input statement, thereby improving the probability and reliability of obtaining the answer corresponding to the input statement based on the query statement set.

[0071] Optionally, when performing at least one of disassembling, expanding, and rewriting the input sentence, a preset large model may be used for processing, and the present disclosure does not limit this.

[0072] Optionally, when performing at least one of the processes of disassembling, expanding, and rewriting the input sentence, the input sentence can also be firstly analyzed for intent, the intent and keywords of the input sentence can be determined, and candidate words with similar semantics to the keywords can be obtained. Then, based on the intent of the input sentence, the current sentence generation template can be determined. Finally, based on the intent of the input sentence, the sentence generation template, and the candidate words, a query statement set can be generated, thereby improving the accuracy of the generated query statement set. The present disclosure does not limit this.

[0073] The intention of the input sentence may be any intention. For example, the intention of the input sentence may be information query, task execution, knowledge question answering, chatting, command control, etc., which is not limited in the present disclosure.

[0074] Among them, the candidate words with similar semantics to the keyword can be synonyms of the keyword, or can also be synonyms of the keyword, etc., which is not limited in the present disclosure.

[0075] The statement generation template may be a template for generating a query statement, which may be pre-set, and the present disclosure does not limit this.

[0076] It should be noted that the intention of the input sentence is different, and the determined sentence generation template may be different, and the present disclosure does not limit this.

[0077] Step 203: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0078] Step 204, generating a first reply statement based on the input statement, the query statement set and the search data.

[0079] Step 205, output the first reply statement.

[0080] The specific implementation forms of steps 203 to 205 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0081] In the disclosed embodiment, first, based on the acquired input statement, a FAQ database of frequently asked questions is searched. If the FAQ database does not contain an answer corresponding to the input statement, the input statement is subjected to at least one of disassembling, expanding, and rewriting to generate a query statement set. Then, based on the query statement set, a preset knowledge base is searched to obtain retrieval data. Then, based on the input statement, the query statement set, and the retrieval data, a first reply statement is generated, and finally the first reply statement is output. Thus, when the FAQ database is searched based on the input statement, if the FAQ database does not contain an answer corresponding to the input statement, the query statement set corresponding to the input statement is obtained by disassembling, expanding, and rewriting the input statement, and the retrieval data corresponding to the query statement set is obtained from the knowledge base. Based on the input statement, the query data set, and the retrieval data, an answer is generated, thereby improving the accuracy of the determined answer and improving the efficiency of dialogue processing while ensuring the comprehensiveness and reliability of the retrieval data.

[0082] Figure 3 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0083] like Figure 3 As shown, the dialogue processing method based on the large model may include the following steps:

[0084] Step 301, based on the acquired input sentence, search the FAQ database for answers to frequently asked questions.

[0085] Step 302: if the FAQ database does not contain an answer corresponding to the input sentence, generate a query sentence set based on the input sentence.

[0086] Step 303: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0087] Step 304, generating a first reply statement based on the input statement, the query statement set and the search data.

[0088] The specific implementation forms of steps 301 to 304 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0089] Step 305: input the input statement, the query statement set, the search data and the first reply statement into the evaluation model to obtain the evaluation result.

[0090] The evaluation model may be a large model used for evaluation, which may be pre-set, and the present disclosure does not limit this.

[0091] In the present disclosure, after generating the first reply statement, in order to determine the accuracy and reliability of the first reply statement, the input statement, the query statement set, the retrieval data and the first reply statement can be input into the evaluation model to obtain the evaluation results, thereby providing conditions for improving the accuracy and reliability of the determined first reply statement.

[0092] Step 306: When the evaluation result indicates that the first reply statement passes the evaluation, output the first reply statement.

[0093] In the present disclosure, when the evaluation result indicates that the first reply statement passes the evaluation, it can be determined that the accuracy of the first reply statement is relatively high, and it can be determined as the answer corresponding to the input statement. At this time, the first reply statement can be output, thereby improving the dialogue processing effect and reliability.

[0094] In the disclosed embodiment, firstly, based on the acquired input statement, a FAQ database of frequently asked questions is searched. If the FAQ database does not contain an answer corresponding to the input statement, a query statement set is generated based on the input statement, and based on the query statement set, a preset knowledge base is searched to obtain search data. Then, based on the input statement, the query statement set and the search data, a first reply statement is generated. After that, the input statement, the query statement set, the search data and the first reply statement are input into an evaluation model to obtain an evaluation result. If the evaluation result indicates that the first reply statement passes the evaluation, the first reply statement is output. Thus, after the FAQ database and the knowledge base are searched respectively using the input statement and the query statement set to generate the answer to the input statement, the answer is evaluated using the evaluation model, and if the evaluation passes, the answer is output, thereby improving the accuracy of the determined answer and improving the dialogue processing effect and efficiency.

[0095] Figure 4 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0096] like Figure 4 As shown, the dialogue processing method based on the large model may include the following steps:

[0097] Step 401, based on the acquired input sentence, search the FAQ database.

[0098] Step 402: if the FAQ database does not contain an answer corresponding to the input sentence, generate a query sentence set based on the input sentence.

[0099] Step 403: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0100] Step 404, generating a first reply statement based on the input statement, the query statement set and the search data.

[0101] Step 405, input the input statement, the query statement set, the search data and the first reply statement into the evaluation model to obtain the evaluation result.

[0102] The specific implementation forms of steps 401 to 405 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0103] Step 406, when the evaluation result indicates that the first reply statement has not passed the evaluation, return to perform the operation of generating the first reply statement based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement.

[0104] In the present disclosure, after obtaining the evaluation result, when the evaluation result indicates that the first reply statement has failed the evaluation, it can be determined that the accuracy of the currently generated first reply statement is low. At this time, the operation of generating the first reply statement can be returned based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement, thereby improving the accuracy of the obtained first reply statement.

[0105] Step 407, output a new first reply statement.

[0106] In the present disclosure, after obtaining a new first reply statement, the new first reply statement can be output.

[0107] The specific implementation of step 407 may refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0108] In the disclosed embodiment, firstly, based on the obtained input statement, a FAQ database of frequently asked questions is searched. If the FAQ database does not contain an answer corresponding to the input statement, a query statement set is generated based on the input statement, and based on the query statement set, a preset knowledge base is searched to obtain the search data. Then, based on the input statement, the query statement set and the search data, a first reply statement is generated. After that, the input statement, the query statement set, the search data and the first reply statement are input into the evaluation model to obtain the evaluation result. If the evaluation result indicates that the first reply statement has not passed the evaluation, the operation of generating the first reply statement is returned based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement, and finally the new first reply statement is output. Thus, after obtaining the answer corresponding to the input statement, the answer is evaluated. If the evaluation result indicates that the answer has not passed the evaluation, the operation of executing the answer is returned based on the evaluation result to obtain and output the new answer, thereby ensuring the accuracy of the generated answer and improving the effect and reliability of the dialogue processing.

[0109] Figure 5A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0110] like Figure 5 As shown, the dialogue processing method based on the large model may include the following steps:

[0111] Step 501, based on the acquired input sentence, search the FAQ database for answers to frequently asked questions.

[0112] Step 502: if the FAQ database does not contain an answer corresponding to the input sentence, generate a query sentence set based on the input sentence.

[0113] Step 503: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0114] Step 504, generating a first reply statement based on the input statement, the query statement set and the search data.

[0115] Step 505: input the input statement, the query statement set, the search data and the first reply statement into the evaluation model to obtain the evaluation result.

[0116] The specific implementation forms of steps 501 to 505 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0117] Step 506, when the evaluation result indicates that the retrieved data has not passed the evaluation, the operation of generating a query statement set is returned based on the evaluation result corresponding to the retrieved data until a new first reply statement is obtained.

[0118] In the present disclosure, when the evaluation result indicates that the retrieved data has not passed the evaluation, it can be determined that the accuracy of the retrieved data is low. At this time, the operation of generating a query statement set can be returned based on the evaluation result corresponding to the retrieved data to obtain a new query statement set, and then based on the new query statement set, the knowledge base is searched to obtain new retrieval data, and finally, based on the input data, the new query statement set and the new retrieval data, a new first reply statement is generated, so that based on the evaluation result, the reason why the first reply statement has not passed the evaluation can be quickly located, thereby improving the efficiency and accuracy of generating a new first reply statement.

[0119] Optionally, when the operation of generating a query statement set is performed based on the evaluation result corresponding to the retrieval data, the retrieval strategy can be first updated based on the evaluation result corresponding to the retrieval data, and then the operation of generating a query statement set is performed based on the updated retrieval strategy and the evaluation result, thereby providing conditions for improving the accuracy of the new first reply statement obtained. The present disclosure does not limit this.

[0120] Among them, the search strategy can be pre-set, which may include at least one of the following: the scale of the search data, the proportion of the search results corresponding to each query statement in the search data, the sorting method of each search result in the search data, etc., so as to ensure the accuracy and comprehensiveness of the new search data obtained, and the present disclosure does not limit this.

[0121] It should be noted that the sorting method of each search result in the search data can be any pre-set sorting method. For example, the more query statements the search data hits, the higher the ranking of the search data, or when the number of query statements the search data hits is the same, the greater the weight of the query statement hit by the search data, the higher the ranking of the search data, and the present disclosure does not limit this.

[0122] Step 507, determining the evaluation result corresponding to the new first reply statement.

[0123] In the present disclosure, after obtaining a new first reply statement, in order to determine the accuracy of the new first reply statement, the evaluation model can be first used to evaluate the new first reply statement to obtain the evaluation result corresponding to the new first reply statement, thereby providing conditions for determining the accuracy of the new first reply statement.

[0124] Step 508: if the new first reply statement still fails the evaluation, obtain reference information from the new search data.

[0125] The reference information may be information related to the input statement other than the first reply statement.

[0126] In the present disclosure, after determining the evaluation result corresponding to the new first reply statement, if the new first reply statement still fails the evaluation, it can be determined that the accuracy of the new first reply statement is low. At this time, the new first reply statement is not output, and reference information can be obtained from the new retrieval data.

[0127] Step 509, returning reference information and indication information, wherein the indication information is used to indicate the first reply statement corresponding to the reference information non-input statement.

[0128] In the present disclosure, after obtaining the reference information, the reference information and the indication information may be returned to provide feedback to the user, and the reference information may be provided for the user's reference.

[0129] In the disclosed embodiment, firstly, based on the obtained input statement, a FAQ database of frequently asked questions is searched. If the FAQ database does not contain an answer corresponding to the input statement, a query statement set is generated based on the input statement, and based on the query statement set, a preset knowledge base is searched to obtain the retrieval data. Then, based on the input statement, the query statement set and the retrieval data, a first reply statement is generated, and the input statement, the query statement set, the retrieval data and the first reply statement are input into an evaluation model to obtain the evaluation result. If the evaluation result indicates that the retrieval data has not passed the evaluation, the operation of generating the query statement set is returned based on the evaluation result corresponding to the retrieval data until a new first reply statement is obtained, and then the evaluation result corresponding to the new first reply statement is determined. If the new first reply statement still has not passed the evaluation, reference information is obtained from the new retrieval data. Finally, the reference information and the indication information are returned. Therefore, after obtaining the answer corresponding to the input statement, the answer is evaluated using the evaluation model. When the evaluation result indicates that the retrieval data has not passed the evaluation, the operation of generating a query statement set is returned based on the evaluation result corresponding to the retrieval data. A new answer is obtained based on the input statement, the new query statement set and the new retrieval data. When the new answer still has not passed the evaluation, reference information is obtained from the new retrieval data, and the reference information and indication information are returned, thereby improving the effect of dialogue processing and improving the accuracy and reliability of generated answers.

[0130] Figure 6 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0131] like Figure 6 As shown, the dialogue processing method based on the large model may include the following steps:

[0132] Step 601, based on the acquired input sentence, search the FAQ database.

[0133] Step 602: if the FAQ database does not contain an answer corresponding to the input sentence, generate a query sentence set based on the input sentence.

[0134] Step 603: Based on the query statement set, a preset knowledge base is searched to obtain search data.

[0135] Step 604, generating a first reply statement based on the input statement, the query statement set and the search data.

[0136] Step 605, output the first reply statement.

[0137] The specific implementation forms of steps 601 to 605 can refer to the detailed descriptions in other embodiments of the present disclosure and will not be described in detail here.

[0138] Step 606, obtaining feedback information corresponding to the first reply statement.

[0139] It should be noted that the feedback information corresponding to the first reply statement obtained may be positive feedback information, or may be negative feedback information, and the present disclosure does not limit this.

[0140] In the present disclosure, after outputting the first reply statement and feeding it back to the user, in order to determine whether the generated first reply statement can meet the user's needs, the user's feedback information corresponding to the first reply statement can be first obtained, thereby obtaining the user's attitude towards the first reply statement through the feedback information.

[0141] Step 607: Update the FAQ database based on the feedback information, the input sentence and the first reply sentence.

[0142] In the present disclosure, after obtaining feedback information corresponding to the first reply statement, the FAQ library can be updated based on the feedback information, the input statement and the first reply statement, thereby improving the accuracy and reliability of the FAQ library.

[0143] Optionally, when the feedback information is the first type of information and the FAQ library does not contain a question statement that matches the input statement, it can be determined that the FAQ library does not contain the input statement and the first reply statement, and the first reply statement is not obtained by searching the FAQ library. At this time, the input statement and the first reply statement can be associated and stored in the FAQ library to expand the QA pairs in the FAQ library and improve the comprehensiveness of the QA pairs in the FAQ library.

[0144] The first type of information may be positive feedback information. For example, the first type of information may be a user's like information, or may be a user's positive evaluation information, etc., which is not limited in the present disclosure.

[0145] Optionally, when the feedback information is the second type of information and the FAQ library contains a question statement that matches the input statement, it can be determined that the first reply statement is obtained by searching the FAQ library and the first reply statement has obtained negative feedback from the user. At this time, the question statement and the associated second reply statement in the FAQ library can be deleted, thereby improving the accuracy and reliability of the FAQ library.

[0146] The second type of feedback information may be negative feedback information. For example, the second type of information may be a user's thumbs-down information, or may be a user's negative evaluation information, etc., which is not limited in the present disclosure.

[0147] It should be noted that the second reply statement, like the first reply statement, is a reply statement stored in the FAQ library.

[0148] Combine the following Figure 7 , an example is given to illustrate the dialogue processing method based on the large model provided by the present disclosure, Figure 7 A flowchart of a large model-based dialogue processing method provided in an embodiment of the present disclosure.

[0149] like Figure 7 As shown, first obtain the multimodal information input by the user (such as Figure 7 The text query information query and / or image information in the question extraction module is first used to obtain the user's input sentence, and then based on the user's input sentence, the FAQ library is searched to perform question matching. After obtaining the answer that matches the input sentence (i.e., the first reply sentence), the answer can be directly returned to the user. In the case that the answer matching the input statement is not obtained, the input statement is processed through the question decomposition / expansion / rewriting module to obtain the query statement set corresponding to the input statement, and based on the query statement set, the text library and image library in the knowledge base are retrieved, and the retrieval data corresponding to the query statement set is obtained according to the retrieval strategy. Then, the model answer module generates the answer corresponding to the input statement based on the retrieval data, the input statement and the query statement set. Finally, the generated answer is evaluated through the evaluation module. If the evaluation passes, the answer is returned to the user. If the evaluation fails, if the evaluation conclusion (i.e., the evaluation result) indicates that the retrieval data fails, based on the evaluation conclusion, the retrieval enhancement generation (RAG) process is re-performed to generate a new query statement set and new retrieval data until a new answer is obtained. If the evaluation conclusion indicates that the answer fails, the answer is regenerated based on the evaluation conclusion. After returning the answer to the user, the user's feedback information is collected, and based on the feedback information, the QA pairs in the FAQ library are updated, cleaned and merged, that is, the QA pairs in the FAQ library are filtered and deduplicated, and the answers are updated.

[0150] Among them, the FAQ library provided by the present disclosure can be obtained based on the knowledge base. First, the questions in the knowledge base are extracted using the question extraction module, and then the answers corresponding to the questions are generated based on the QA generation module to obtain QA pairs, and finally the QA pairs are evaluated in multiple dimensions (such as quality, factuality, etc.), and the QA pairs that pass the evaluation are stored in the FAQ library, thereby improving the quality of the QA pairs in the FAQ library, and the present disclosure does not limit this.

[0151] Among them, after obtaining the knowledge base, multimodal information in the knowledge base can be extracted, such as extracting text, table, image and other information to obtain corresponding text library and image library, which is not limited in the present disclosure.

[0152] in, Figure 7The dark modules in the, such as question extraction module, QA generation module, evaluation module, question extraction module, question decomposition / expansion / rewriting module, model answering module, etc., can all be operated through the big model, and the present disclosure does not limit this.

[0153] In the disclosed embodiment, firstly, based on the acquired input sentence, the FAQ database of frequently asked questions is searched. When the FAQ database does not contain the answer corresponding to the input sentence, a query statement set is generated based on the input sentence, and based on the query statement set, a preset knowledge base is searched to obtain the search data. Then, based on the input sentence, the query statement set and the search data, a first reply statement is generated, and the first reply statement is output. Then, feedback information corresponding to the first reply statement is obtained. Finally, based on the feedback information, the input sentence and the first reply statement, the FAQ database is updated. Thus, after the answer corresponding to the input sentence is generated and output, feedback information corresponding to the answer is obtained, and the FAQ database is updated based on the feedback information, the input sentence and the answer, thereby improving the accuracy and reliability of the FAQ database and improving the reliability of the dialogue processing method based on the large model.

[0154] In order to implement the above embodiments, the present disclosure also proposes a dialogue processing device based on a large model.

[0155] Figure 8 A schematic diagram of the structure of a large model-based dialogue processing device provided in an embodiment of the present disclosure.

[0156] like Figure 8 As shown, the large model-based dialogue processing device 800 includes: a retrieval module 801, a first generation module 802, an acquisition module 803, a second generation module 804, and an output module 805.

[0157] A retrieval module 801 is used to retrieve a FAQ database based on the acquired input sentence;

[0158] A first generating module 802, configured to generate a query statement set based on the input statement when the FAQ library does not contain an answer corresponding to the input statement;

[0159] The acquisition module 803 is used to search the preset knowledge base based on the query statement set to obtain the search data;

[0160] The second generating module 804 is used to generate a first reply statement based on the input statement, the query statement set and the search data;

[0161] The output module 805 is used to output the first reply statement.

[0162] In a possible implementation of the present disclosure, the first generating module 802 is specifically configured to:

[0163] The input statement is processed by at least one of disassembling, expanding, and rewriting to generate a query statement set.

[0164] In a possible implementation of the present disclosure, the first generating module 802 is further configured to:

[0165] Perform intent analysis on the input sentence to determine the intent and keywords of the input sentence;

[0166] Get candidate words with similar semantics to the keywords;

[0167] Based on the intent of the input sentence, determine the current sentence generation template;

[0168] Generate a query statement set based on the intent of the input sentence, the sentence generation template, and the candidate words.

[0169] In a possible implementation of the present disclosure, the output module 805 is specifically used to:

[0170] Inputting the input sentence, the query sentence set, the search data and the first reply sentence into the evaluation model to obtain the evaluation result;

[0171] In a case where the evaluation result indicates that the first reply sentence passes the evaluation, the first reply sentence is output.

[0172] In a possible implementation of the present disclosure, the output module 805 is further configured to:

[0173] In the case where the evaluation result indicates that the first reply statement has not passed the evaluation, the operation of generating the first reply statement is returned based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement.

[0174] In a possible implementation of the present disclosure, the output module 805 is further configured to:

[0175] In the case where the evaluation result indicates that the retrieved data has not passed the evaluation, an operation of generating a query statement set is returned based on the evaluation result corresponding to the retrieved data until a new first reply statement is obtained.

[0176] In a possible implementation of the present disclosure, the output module 805 is further configured to:

[0177] Based on the evaluation results corresponding to the search data, the search strategy is updated;

[0178] Based on the updated search strategy and evaluation results, the operation of generating a query statement set is returned.

[0179] In a possible implementation of the present disclosure, the above-mentioned search strategy includes at least one of the following: the scale of the search data, the proportion of the search results corresponding to each query statement in the search data, and the sorting method of each search result in the search data.

[0180] In a possible implementation of the present disclosure, the output module 805 is further configured to:

[0181] Determine an evaluation result corresponding to the new first response statement;

[0182] If the new first reply statement still fails the evaluation, obtaining reference information from the new search data;

[0183] Reference information and indication information are returned, wherein the indication information is used to indicate the first reply statement corresponding to the reference information non-input statement.

[0184] In a possible implementation of the present disclosure, the output module 805 is further configured to:

[0185] Obtaining feedback information corresponding to the first reply statement;

[0186] The FAQ database is updated based on the feedback information, input sentence and first reply sentence.

[0187] In a possible implementation of the present disclosure, the output module 805 is further used for any of the following:

[0188] When the feedback information is the first type of information and the FAQ database does not contain a question statement matching the input statement, the input statement and the first answer statement are associated and stored in the FAQ database;

[0189] When the feedback information is the second type of information and the FAQ database contains a question statement matching the input statement, the question statement and the associated second answer statement in the FAQ database are deleted.

[0190] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, and will not be repeated here.

[0191] In the disclosed embodiment, firstly, based on the acquired input statement, the FAQ database of frequently asked questions is searched. If the FAQ database does not contain the answer corresponding to the input statement, a query statement set is generated based on the input statement. Then, based on the query statement set, a preset knowledge base is searched to obtain the search data. Then, based on the input statement, the query statement set and the search data, a first reply statement is generated, and finally, the first reply statement is output. Thus, by firstly searching the FAQ database based on the input statement, if the FAQ database does not contain the answer corresponding to the input statement, a query statement set corresponding to the input statement is obtained, and based on the query statement set, the knowledge base is searched to obtain the search data. Finally, based on the input statement, the query statement set and the search data, the answer is obtained and output, thereby improving the accuracy of the determined answer and improving the efficiency and reliability of the dialogue processing method based on the large model.

[0192] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0193] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0194] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0195] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0196] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a large model-based dialogue processing method. For example, in some embodiments, the large model-based dialogue processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the large model-based dialogue processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the large model-based dialog processing method in any other appropriate manner (eg, by means of firmware).

[0197] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0198] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable large-model-based dialog processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0199] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0200] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0201] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0202] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0203] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0204] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0205] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A conversation processing method based on a large model, comprising: Based on the obtained input sentence, retrieve the FAQ database; In the case that the FAQ library does not contain an answer corresponding to the input sentence, generating a query sentence set based on the input sentence; Based on the query statement set, a preset knowledge base is searched to obtain search data; Generate a first reply statement based on the input statement, the query statement set and the search data; Output the first reply sentence.

2. The method of claim 1, wherein: The step of generating a query statement set based on the input statement includes: The input statement is processed by at least one of disassembling, expanding, and rewriting to generate a query statement set.

3. The method of claim 2, wherein: The step of performing at least one of disassembling, expanding, and rewriting the input statement to generate a query statement set includes: Performing intent analysis on the input sentence to determine the intent and keywords of the input sentence; Obtain candidate words that are semantically similar to the keyword; Determining a current sentence generation template based on the intention of the input sentence; The query statement set is generated based on the intention of the input statement, the statement generation template, and the candidate words.

4. The method of claim 1, wherein: The outputting the first reply statement includes: Inputting the input sentence, the query sentence set, the search data and the first reply sentence into an evaluation model to obtain an evaluation result; In a case where the evaluation result indicates that the first reply sentence passes the evaluation, the first reply sentence is output.

5. The method of claim 4, wherein: After obtaining the evaluation result, the method further includes: In the case where the evaluation result indicates that the first reply statement has not passed the evaluation, the operation of generating the first reply statement is returned based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement.

6. The method of claim 4, wherein: After obtaining the evaluation result, the method further includes: In the case where the evaluation result indicates that the retrieval data has not passed the evaluation, the operation of generating the query statement set is returned based on the evaluation result corresponding to the retrieval data until a new first reply statement is obtained.

7. The method of claim 6, wherein: The operation of returning and executing the generation of the query statement set based on the evaluation result corresponding to the search data includes: Based on the evaluation results corresponding to the search data, the search strategy is updated; The operation of generating a query statement set is executed based on the updated search strategy and the evaluation result.

8. The method of claim 7, wherein: The search strategy includes at least one of the following: the scale of the search data, the proportion of the search results corresponding to each query statement in the search data, and the sorting method of each search result in the search data.

9. The method according to any one of claims 5 to 8, wherein: After obtaining the new first reply statement, the method further includes: Determine an evaluation result corresponding to the new first reply statement; If the new first reply statement still fails the evaluation, obtaining reference information from new search data; The reference information and indication information are returned, wherein the indication information is used to indicate that the reference information is not the first reply statement corresponding to the input statement.

10. The method according to any one of claims 1 to 9, wherein: After outputting the first reply statement, the method further includes: Obtaining feedback information corresponding to the first reply statement; The FAQ library is updated based on the feedback information, the input sentence and the first reply sentence.

11. The method of claim 10, wherein:The updating of the FAQ library based on the feedback information, the input sentence and the first reply sentence includes any one of the following: When the feedback information is information of the first type and the FAQ library does not contain a question sentence matching the input sentence, the input sentence and the first answer sentence are stored in the FAQ library in an associated manner; When the feedback information is the second type of information and the FAQ library contains a question statement matching the input statement, the question statement and the associated second answer statement in the FAQ library are deleted.

12. A dialogue processing device based on a large model, wherein: The device comprises: A retrieval module, used for retrieving a FAQ database based on the acquired input sentence; A first generating module, configured to generate a query statement set based on the input statement when the FAQ library does not contain an answer corresponding to the input statement; An acquisition module, used to search a preset knowledge base based on the query statement set to acquire search data; A second generating module, used for generating a first reply statement based on the input statement, the query statement set and the search data; An output module is used to output the first reply statement.

13. The device of claim 12, wherein: The first generating module is specifically used for: The input statement is processed by at least one of disassembling, expanding, and rewriting to generate a query statement set.

14. The device of claim 13, wherein: The first generating module is further used for: Performing intent analysis on the input sentence to determine the intent and keywords of the input sentence; Obtain candidate words that are semantically similar to the keyword; Determining a current sentence generation template based on the intention of the input sentence; The query statement set is generated based on the intention of the input statement, the statement generation template, and the candidate words.

15. The device of claim 12, wherein: The output module is specifically used for: Inputting the input sentence, the query sentence set, the search data and the first reply sentence into an evaluation model to obtain an evaluation result; In a case where the evaluation result indicates that the first reply sentence passes the evaluation, the first reply sentence is output.

16. The device of claim 15, wherein: The output module is further used for: In the case where the evaluation result indicates that the first reply statement has not passed the evaluation, the operation of generating the first reply statement is returned based on the evaluation result corresponding to the first reply statement to obtain a new first reply statement.

17. The device of claim 15, wherein: The output module is further used for: In the case where the evaluation result indicates that the retrieval data has not passed the evaluation, the operation of generating the query statement set is returned based on the evaluation result corresponding to the retrieval data until a new first reply statement is obtained.

18. The device of claim 17, wherein: The output module is further used for: Based on the evaluation results corresponding to the search data, the search strategy is updated; The operation of generating a query statement set is executed based on the updated search strategy and the evaluation result.

19. The device of claim 18, wherein: The search strategy includes at least one of the following: the scale of the search data, the proportion of the search results corresponding to each query statement in the search data, and the sorting method of each search result in the search data.

20. The device according to any one of claims 16 to 19, wherein: The output module is further used for: Determine an evaluation result corresponding to the new first reply statement; If the new first reply statement still fails the evaluation, obtaining reference information from new search data; The reference information and indication information are returned, wherein the indication information is used to indicate that the reference information is not the first reply statement corresponding to the input statement.

21. The device according to any one of claims 12 to 20, wherein: The output module is further used for: Obtaining feedback information corresponding to the first reply statement; The FAQ library is updated based on the feedback information, the input sentence and the first reply sentence.

22. The device of claim 21, wherein: The output module is also used for any of the following: When the feedback information is the first type of information and the FAQ library does not contain a question sentence matching the input sentence, the input sentence and the first answer sentence are stored in the FAQ library in an associated manner; When the feedback information is the second type of information and the FAQ library contains a question statement matching the input statement, the question statement and the associated second answer statement in the FAQ library are deleted.

23. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that may be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images