Replay generation system, method and equipment

Abstract

The invention provides a reply generation system, method and equipment, and the system comprises a reference search assembly which is used for obtaining reply information, and searching knowledge matched with the reply information in a knowledge base; wherein the reply information is obtained by a trained preset model according to input information of a first user; and the reference judgment component is used for judging the authenticity of the reply information according to the knowledge so as to determine information output to the first user according to a judgment result. The reply information can be subjected to reference search, whether the reply information can be supported by reference knowledge or not is judged, it is ensured that the reply information returned to the user is true and credible, the accuracy of the whole system is effectively improved, and user experience is improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and more particularly to a response generation system, method and device. Background Art

[0002] With the continuous development of artificial intelligence, various machine learning models for reasoning generation have been increasingly widely used. Based on machine learning models, systems suitable for different scenarios can be built to achieve intelligent interaction with users.

[0003] Taking the search generation scenario as an example, after receiving the question input by the user, the search component can search for knowledge related to the question from the knowledge base, and the model generates the corresponding answer based on the searched knowledge.

[0004] The shortcoming of this solution is that the model answers questions based on the searched knowledge, and the upper limit of the accuracy of the answer of the entire system will not be higher than the upper limit of the accuracy of the search component, resulting in limited accuracy of the answer and difficulty in meeting the actual needs of users. Summary of the invention

[0005] The present application provides a reply generation system, method and device for improving the accuracy of reply information output to users.

[0006] In a first aspect, an embodiment of the present application provides a reply generation system, comprising:

[0007] A reference search component is used to obtain reply information and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model based on input information of the first user;

[0008] The reference determination component is used to determine the authenticity of the reply information according to the knowledge, so as to determine the information output to the first user according to the determination result.

[0009] In a second aspect, an embodiment of the present application provides a reply generation system, including:

[0010] A query search component, used to obtain query information of the first user, and search the knowledge base for documents matching the query information; wherein the knowledge base includes a plurality of documents corresponding to products provided by the enterprise;

[0011] A rejection component, used to determine whether the document searched by the query search component can answer the query information;

[0012] a generation component, for obtaining the query information and the document when the judgment result of the rejection component is yes, inputting the query information and the document into a preset model, and obtaining corresponding reply information; and / or, for obtaining the query information when the judgment result of the rejection component is no, inputting the query information into a preset model, and obtaining corresponding reply information;

[0013] A reference search component is used to obtain the reply information and search the knowledge base for documents matching the reply information;

[0014] The reference determination component is used to determine the authenticity of the reply information based on the documents searched by the reference search component, so as to determine the answer output to the first user based on the judgment result.

[0015] In a third aspect, an embodiment of the present application provides a reply generation method, comprising:

[0016] Acquire reply information, and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model according to input information of the first user;

[0017] The authenticity of the reply information is judged according to the knowledge, so as to determine the information output to the first user according to the judgment result.

[0018] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0019] at least one processor; and

[0020] a memory communicatively coupled to the at least one processor;

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to execute the method described in any one of the above aspects.

[0022] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in any of the above aspects is implemented.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any of the above aspects.

[0024] The reply generation system, method and device provided in the embodiments of the present application can obtain reply information obtained by a trained preset model based on the input information of the first user, search for knowledge matching the reply information in the knowledge base through a reference search component, and judge the authenticity of the reply information based on the knowledge through a reference judgment component, so as to determine the information output to the first user based on the judgment result. In this way, even if the correct knowledge cannot be recalled from the knowledge base based on the input information, the preset model can still generate reply information based on the knowledge learned during the training process, and can perform a reference search on the reply information and judge whether the reply information can be supported by the referenced knowledge, thereby ensuring that the reply information returned to the user is authentic and reliable, increasing the probability of effective replies, and improving the accuracy of the entire system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] Figure 1 A schematic diagram of the principle of generating answers through a generative model;

[0027] Figure 2 A schematic diagram of the principle of generative search;

[0028] Figure 3 A schematic diagram of the principle of a posteriori reference provided in an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of a reply generation system provided in an embodiment of the present application;

[0030] Figure 5 A schematic diagram of the principle of a query search component provided in an embodiment of the present application;

[0031] Figure 6 A schematic diagram of the training principle of a rejection model provided in an embodiment of the present application;

[0032] Figure 7 A schematic diagram of an application of a preset model provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of the principle of a reference search component provided in an embodiment of the present application;

[0034] Fig. 9 A schematic diagram of the principle of a reference determination component provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of an interactive interface provided in an embodiment of the present application;

[0036] Fig.11 A flowchart of a reply generation method provided in an embodiment of the present application;

[0037] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0038] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0039] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0040] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] First, the terms involved in this application are explained:

[0042] Generative models: Large-scale training-based models for inference generation.

[0043] Generative search: A search technique that uses generative models to generate answers.

[0044] A posteriori reference: An embodiment of the present application provides a method for determining the authenticity of a generated answer, which supports the generated answer by verifying relevant information in a reference knowledge base.

[0045] Knowledge base: contains a large amount of knowledge, which can be used as a basis for verifying whether the answer is true in the embodiment of the present application. For example, for an enterprise providing product services, the constructed knowledge base can include multiple documents of the product, such as interface documents, update logs, FAQs (frequently-asked questions), etc.

[0046] First user: In the embodiment of the present application, the first user refers to the user who initiates the input of information, for example, the user may be a customer or employee of the enterprise. Unless otherwise specified, the users appearing in the embodiment of the present application refer to the first user.

[0047] Second user: In the embodiment of the present application, the second user refers to the user who configures and maintains the system and knowledge base, and can generally be the management personnel of the enterprise. The second user can configure the system's processing strategy, update the knowledge base, etc.

[0048] In the field of artificial intelligence, generative models can usually be used to generate answers that match questions input by users. For example, generative models can be large language models. Among them, large language models refer to deep learning language models with large-scale language model parameters, which usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion language model parameters. Large language models can also be called foundation language models / foundation models (Foundation Model l). Large language models are pre-trained through large-scale unlabeled corpora to produce pre-trained language models with more than 100 million parameters. This language model can adapt to a wide range of downstream tasks, and the language model has good generalization capabilities, such as large-scale language models (LargeLanguage Model, LLM), multi-modal pre-training language models (multi-modal pre-training model), etc.

[0049] In actual application, only a small number of samples are needed to fine-tune the pre-trained language model and it can be applied to different tasks. The large language model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large language model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0050] Figure 1 This is a schematic diagram of the principle of generating answers through a generative model. Figure 1As shown in the figure, when the first user inputs a question, the generative model can automatically generate an answer based on the knowledge learned in the pre-training process. However, this solution relies solely on the model's own ability to generate answers, which may produce false answers. This is because the generative model has a problem of illusory knowledge, and the generated answers may be fabricated and lack authenticity. In addition, it is difficult for users to determine whether the generated answers have supporting reference information, and thus cannot verify the authenticity and credibility of the answers.

[0051] In order to reduce the hallucination problem of the model, a knowledge base can be introduced as a constraint for the model to generate answers. Figure 2 This is a schematic diagram of the principle of generative search. Figure 2 As shown, after the first user inputs a question, the search component can search for knowledge related to the question in the knowledge base. After searching for relevant knowledge, the generation model can generate a corresponding answer based on the question and related knowledge and display it to the user.

[0052] Generative search can effectively improve the authenticity of answers, but there is a problem of limited accuracy when providing answers, because the accuracy of the answer depends on the recall ability of the search component. If the search component cannot correctly recall the relevant knowledge from the knowledge base, the generative model cannot generate accurate answers. Therefore, the upper limit of the answer accuracy of the entire system will not exceed the search accuracy upper limit of the search component. For some questions that have correct answers in the knowledge base, if the relevant knowledge is not accurately recalled by the search component, the system cannot return the correct answer to the user.

[0053] In view of this, an embodiment of the present application provides a response generation system. By introducing a posteriori references, even if the search component cannot accurately recall relevant knowledge, the system can still generate credible answers, thereby improving the upper limit of the accuracy of the entire system.

[0054] Figure 3 A schematic diagram of the principle of a posteriori reference provided in the embodiment of the present application. Figure 3 As shown, after the first user inputs a question, the query search component searches for relevant knowledge in the knowledge base. After searching for relevant knowledge, the rejection component determines whether the knowledge can answer the question. If it is determined that it can answer the question, the generation component generates a corresponding answer based on the question and knowledge through a preset model.

[0055] Optionally, the preset model may be any model that can obtain response information based on input information. Exemplarily, the preset model may be a generation model.

[0056] If it is determined that the answer cannot be given, the generation component will generate a corresponding answer based on the question. After the answer is generated, the reference search component searches for knowledge related to the answer in the knowledge base, and the reference determination component determines whether the knowledge supports the answer. If it does, it means that the answer is true, and the answer can be output to the user.

[0057] The system provided by the embodiment of the present application combines the query search component and the generation component, and can search for relevant knowledge from the knowledge base when the user asks a question, and use the preset model to generate accurate answers. The introduction of the rejection component to check whether the searched knowledge can support answering the user's question can effectively reduce wrong answers. When the searched knowledge does not support answering the user's question, or when the query search component cannot search for relevant knowledge, the generation component can directly generate an answer based on the question, and verify the accuracy of the answer through the method of a posteriori reference.

[0058] In the knowledge base, there may be some knowledge that can be used to answer the user's questions. However, when the query search component searches from the knowledge base according to the question, it may not be able to search out the relevant knowledge, or the relevant knowledge searched out may not be accurate. In this case, the preset model can generate the corresponding answer based on its own knowledge, and search from the knowledge base based on the answer, and then verify whether the searched knowledge supports the answer. In this way, even if the query search component cannot recall accurate knowledge, the generation component may still generate credible answers, thereby increasing the number of valid answers, because part of the valid answers returned to the user are generated based on the knowledge recalled by the query search component, and part are directly generated based on the question and verified by the knowledge base, thereby increasing the probability of valid answers, improving the accuracy of the entire system, and thus improving the user experience.

[0059] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the case where there is no conflict between the embodiments, the following embodiments and the features in the embodiments can be combined with each other. In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0060] Figure 4 A schematic diagram of the structure of a reply generation system provided in an embodiment of the present application. The system in this embodiment can be deployed in any location, for example, on the cloud, locally, on a client, or on an IOT (Internet of Things) device, etc. The system can be in the form of software, hardware, or a combination of software and hardware. Figure 4 As shown, the system may include:

[0061] A reference search component is used to obtain reply information and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model based on input information of the first user;

[0062] The reference determination component is used to determine the authenticity of the reply information according to the knowledge, so as to determine the information output to the first user according to the determination result.

[0063] The authenticity, which may also be referred to as correctness or validity in the embodiment of the present application, is used to indicate whether the knowledge supports the reply information. Specifically, given a knowledge base, if the reply information can find relevant evidence in the knowledge base, that is, there is some knowledge in the knowledge base that can support the reply information, then it means that the reply information is correct in the context of the given knowledge base, and the reply information can be considered to be authentic and credible.

[0064] If the reply information is judged to be true, the reply information can be output to the user. If the reply information is judged to be false, the reply information can be not output to the user, or the reply information can be modified before output to the user. If the reply information contains more content, the authenticity of each part can be judged separately and then output in a targeted manner.

[0065] Optionally, the system provided in the embodiment of the present application can be provided for enterprises, and the knowledge base can be configured by the enterprise to assist in replying to the input information of the first user, and the searched knowledge can be part of the content in the knowledge base. The first user can be a customer or employee of the enterprise. Of course, the embodiment of the present application is not limited to this, and any user who can input questions and expect to get answers can be the first user.

[0066] The input information of the first user may be recorded as query, which may be information in any form input by the first user, for example, the input information may be a question, a demand, query information, or a statement in a chat scenario, etc. The reply information may be information in any form to be displayed to the user, for example, an answer to a question or demand, or a reply or question to a user's statement in a chat scenario, etc.

[0067] The reply information may be generated by a preset model, and the preset model may be a trained machine learning model, such as a large language model.

[0068] Optionally, the system may further include: a generation component, configured to obtain input information of the first user, and generate reply information corresponding to the input information based on a trained preset model.

[0069] Furthermore, the generation component may combine the knowledge in the knowledge base to generate corresponding reply information. Optionally, the system may also include:

[0070] A query search component, used to obtain input information of a first user, and search the knowledge base for knowledge matching the input information;

[0071] The rejection component is used to determine whether the knowledge searched by the query search component can answer the input information.

[0072] Optionally, the generation component is specifically used to obtain the input information and knowledge matching the input information when the judgment result of the rejection component is yes, input the input information and knowledge into a preset model, and obtain corresponding reply information; and / or, when the judgment result of the rejection component is no, obtain the input information, input the input information into a preset model, and obtain corresponding reply information.

[0073] Combine the following Figure 4 To illustrate the inputs, outputs, and functions of each component in the system.

[0074] Query search component: takes the first user's input information (query) as input, searches for relevant knowledge in the knowledge base, and returns search results (results) as output.

[0075] Rejection component: receives query and response as input, determines whether the response can answer the query based on the query and response, and obtains the rejection result (is_reject) as output.

[0076] Generation component: contains a preset model, which is pre-trained based on the knowledge base so that the preset model can learn knowledge in related fields. There are two inputs to the generation component: receiving query and result as input, and generating reply information (reply) as output based on query and result; receiving query as input, and directly generating reply based on query.

[0077] Reference search component: receives reply as input, searches for related knowledge in the knowledge base, and returns the knowledge corresponding to the reply.

[0078] Reference determination component: It can also be called reference validity determination component, which is used to determine the validity, that is, authenticity, of the reply based on the knowledge referenced by the reply. Specifically, the reference determination component receives the reply and the corresponding knowledge as input, determines whether the knowledge can correctly support the reply, and outputs the determination result (i s_support).

[0079] When the system obtains the query of the first user, the processing flow of the entire system is as follows:

[0080] Step a: The query search component searches the knowledge base according to the query of the first user and returns results.

[0081] Step b: The rejection component gives a rejection result is_reject based on the query and results.

[0082] Step c: if is_reject is false, it means that the searched knowledge can answer the user's query. At this time, the query and results are given to the generation component to generate reply information; if is_reject is true, it means that the searched knowledge cannot answer the user's query. At this time, the query is directly given to the generation component to generate reply information.

[0083] Step d: For the generated reply, use the reference search component to search for relevant knowledge from the knowledge base.

[0084] Step e: for the reply and the corresponding knowledge, use the reference determination component to determine whether the knowledge can support the reply, and obtain the determination result is_support.

[0085] Step f: return the reply information indicating that the judgment result is_support is true to the user.

[0086] By using the above solution, for queries that can search for correct knowledge, the preset model can give correct replies as usual; for queries that do not search for correct knowledge, the preset model can answer based on the knowledge learned during the training process and ensure the authenticity of the reply returned to the user through a posteriori reference, making the upper limit of the accuracy of the entire system exceed the upper limit of the accuracy of the query search component.

[0087] In this embodiment, the query search component is combined with the generation component so that the generated reply information can find the corresponding reference information in the knowledge base, and the user can verify the authenticity of the reply information. A rejection component is introduced to determine whether the knowledge searched by the query search component can answer the user's input information, thereby reducing the situation where the reply information is incorrect due to inaccurate knowledge searched by the query, and improving the accuracy of the reply information.

[0088] In actual applications, the system may include the above five components, or some of them may be omitted. For example, the rejection component may be omitted, and the corresponding reply information may be directly generated based on the knowledge searched by the query search component and the input information. Alternatively, the query search component may be omitted, and the corresponding reply information may be directly generated based on the input information. Alternatively, the generation component may be omitted, and the reference search component and the reference judgment component in the system may be connected to an external preset model and verify the authenticity of the reply information generated by the preset model.

[0089] Through the system provided in this embodiment, reply information generated by a trained preset model according to the input information of the first user can be obtained, knowledge matching the reply information can be searched in the knowledge base through a reference search component, and the authenticity of the reply information can be judged based on the knowledge through a reference judgment component, so as to determine the information output to the first user based on the judgment result. In this way, even if the correct knowledge cannot be recalled from the knowledge base based on the input information, the preset model can still generate reply information based on the knowledge learned during the training process, and can ensure that the reply information returned to the user is authentic and reliable by performing a reference search on the reply information and judging whether the reply information can be supported by the referenced knowledge, thereby effectively improving the accuracy of the entire system and enhancing the user experience.

[0090] In an embodiment of the present application, the knowledge base may include one or more files, and the file may include content in at least one of the following modalities: text, image, audio, video; the knowledge searched by the reference search component or the query search component is at least one file in the knowledge base, or is part of the content in a file; the input information and the reply information both contain content in at least one modality; the preset model can be a model that can handle one modality, or a model that can handle multiple modalities.

[0091] Optionally, the knowledge base may be configured by a second user, who may be a user responsible for maintaining the system, for example, a manager of an enterprise. The processing capability of the model involved may also be configured by the second user.

[0092] Exemplarily, the enterprise is a technical service enterprise, and the knowledge base includes product documents, which may specifically include documents corresponding to various functions of the product, update logs, FAQs, and any other documents related to the product, and the specific form is not limited. When the information input by the first user may be a question or query information about the product, for example, "how to use a certain function", or "does a certain interface support returning a certain field". The query search component can search for relevant documents from the knowledge base, or search for part of the content of a document, and output it as knowledge matching the input information to other components for processing.

[0093] The specific implementation schemes of each component provided in the embodiments of the present application are described below.

[0094] Figure 5 A schematic diagram of the principle of a query search component provided in an embodiment of the present application. Figure 5 As shown, the query search component is specifically used for:

[0095] Obtaining input information of the first user;

[0096] Searching for knowledge corresponding to the input information in a knowledge base by means of semantic recall and / or vector recall;

[0097] If multiple pieces of knowledge are found, the matching degree between each piece of knowledge and the input information is determined based on at least one indicator, and the multiple pieces of knowledge are sorted based on the matching degree, and the knowledge matching the input information is determined based on the sorting result.

[0098] Optionally, the semantic recall includes: searching for knowledge whose similarity with the input information satisfies a first preset condition; the vector recall includes: searching for knowledge whose distance with the embedding vector of the input information satisfies a second preset condition according to the embedding vector corresponding to each piece of knowledge;

[0099] The at least one indicator includes at least one of the following: a semantic matching score, used to indicate the similarity between the input information and the knowledge; an edit distance score, used to indicate the distance between the embedding vector of the input information and the embedding vector of the knowledge; an entity relevance score, used to indicate the relevance between the entities contained in the input information and the entities contained in the knowledge;

[0100] For any piece of knowledge, the matching degree between the knowledge and the input information is: the result obtained by weighted summation of the at least one indicator of the knowledge.

[0101] Specifically, the query search component can mainly realize two functions: recall and sorting. Recall includes: after obtaining input information, corresponding knowledge can be searched from the knowledge base through semantic recall, vector recall, or semantic recall plus vector recall.

[0102] Among them, semantic recall refers to searching for knowledge whose similarity with the input information meets the first preset condition. The similarity here can refer to semantic similarity. Taking text as an example, the similarity can be expressed as the degree of text matching. The first preset condition can be greater than a certain threshold, or taking the top N (topN) with the largest similarity. In this way, at least one knowledge similar to the input information can be selected from the knowledge base, for example, at least one file.

[0103] Vector recall refers to searching for knowledge that meets a second preset condition with the embedding vector of the input information, wherein the embedding vector can be used to represent the position of the input information or knowledge in the vector space, and when generating reply information, the preset model can specifically generate reply information based on the embedding vector of the input information and the embedding vector of the knowledge. The distance between the embedding vector of the input information and the embedding vector of the knowledge can reflect the relevance of the input information and the knowledge from a feature dimension. The second preset condition can be that the distance between the embedding vector of the input information and the embedding vector of the knowledge is less than a certain threshold, or the first N with the smallest distance are taken, so that the searched knowledge and the embedding vector of the input information are as close as possible.

[0104] Optionally, the search may be performed by multi-way recall, which means using semantic recall and vector recall at the same time. For example, semantic recall takes the first 5 results, and vector recall takes the first 5 results, thereby obtaining 10 pieces of knowledge corresponding to the input information.

[0105] After obtaining the corresponding knowledge, the knowledge can be sorted to determine the knowledge to be output to the generation component or the rejection component. For example, three pieces of knowledge A, B, and C that meet the preset conditions are obtained through recall, and the first two most relevant pieces of knowledge are output through sorting: knowledge B and knowledge A.

[0106] Among them, multi-dimensional features can be referred to when sorting, and the multi-dimensional features can be specifically represented by different indicators. When the indicators include semantic matching score, edit distance score and entity relevance score, the scores corresponding to the three indicators can be calculated for each piece of knowledge.

[0107] For example, for a certain knowledge, the similarity between the knowledge and the input information can be calculated to obtain the semantic matching score. The semantic matching score is positively correlated with the similarity. The higher the similarity, the higher the semantic matching score. The edit distance score is calculated based on the distance between the embedding vector of the input information and the embedding vector of the knowledge. The edit distance score is negatively correlated with the distance. The closer the distance, the higher the edit distance score. Entities are extracted from the input information and knowledge respectively, and the entity relevance score is calculated based on the relevance between the entities contained in the input information and the entities contained in the knowledge. The entity relevance score is positively correlated with the relevance. The relevance can be represented by similarity. That is, the higher the proportion of entities in the input information and entities in the knowledge that are identical or similar, the higher the entity relevance score. After obtaining each indicator, the weighted sum of each indicator of the knowledge is taken to obtain the matching degree between the knowledge and the input information.

[0108] After determining the matching degree between each piece of knowledge and the input information, the knowledge can be sorted from high to low according to the matching degree, and one or more pieces of knowledge with the highest ranking can be selected and output to the generation component or the rejection component as the knowledge matching the input information.

[0109] Through the above scheme, knowledge that meets the requirements can be obtained through a variety of recall strategies and multi-dimensional sorting strategies to improve the accuracy of the search. In addition, in actual applications, the second user can configure a specific query search strategy, for example, which recall method to use, the amount of recalled knowledge, which indicators to use for sorting, the weight of each indicator, the amount of knowledge output after sorting, etc., to facilitate knowledge search in different scenarios and meet the personalized needs of users.

[0110] The rejection component is described below.

[0111] Optionally, the rejection component is specifically used to: input the input information and the matching knowledge into a rejection model, and determine whether the knowledge can answer the input information through the rejection model.

[0112] Optionally, the rejection model may be a trained machine learning model. The training samples of the rejection model may be constructed manually or with the assistance of a preset model.

[0113] Figure 6 A schematic diagram of the training principle of a rejection model provided in an embodiment of the present application. Figure 6 As shown, the rejection model can be trained in the following manner: according to the preset model, a training sample is obtained, and the training sample includes: an input information sample, a knowledge sample and a label, and the label is used to indicate whether the knowledge sample can answer the input information sample; according to the input information sample, the knowledge sample and the label, the rejection model is trained.

[0114] In one example, the input information sample and the knowledge sample can be obtained first, and the corresponding label can be determined according to the preset model. Specifically, the input information sample and the knowledge sample can be collected from the knowledge base or other platforms in the network, or can be manually input. The input information sample and the knowledge sample are input into the preset model. The preset model has a certain information processing capability and can be used to determine whether the knowledge sample supports replying to the input information sample, so that the corresponding label can be obtained through the preset model. The input information sample, the knowledge sample and the label constitute a training sample for training the rejection component.

[0115] In another example, a knowledge sample may be first obtained, and an input information sample that the knowledge sample can support answering may be generated through a preset model, and the corresponding label may be set to be able to answer; and / or, an input information sample that the knowledge sample cannot answer may be generated through a preset model, and the corresponding label may be set to be unable to answer, thereby obtaining a training sample for training the rejection component.

[0116] Optionally, after pre-training, the preset model itself has the ability to process natural language. When using the preset model, you can guide the work of the preset model by adding prompts. For example, you can prompt the preset model: "Please generate a question that this document (knowledge) can answer", or "Please determine whether this document can be used to answer this question", so that the preset model can obtain the desired output.

[0117] Optionally, after using the preset model to generate training samples, the training samples can be modified, screened, etc. manually to further improve the accuracy of the training samples.

[0118] Through the above method, the ability of the preset model can be used to construct training samples to train the rejection model, thereby reducing the manual burden and improving the training efficiency. In addition, the preset model can be a large model, and the rejection model can be a small model. The small model is a concept relative to the large model, which means that the rejection component has a simpler structure and fewer parameters than the preset model. When using the system, the rejection model can be directly used for judgment, which can effectively improve the processing efficiency during use.

[0119] In an optional implementation, the rejection component may be specifically used for:

[0120] Inputting the input information, knowledge and a first prompt instruction into a rejection model, wherein the first prompt instruction is used to prompt the rejection model to extract a plurality of knowledge points from the knowledge;

[0121] For each extracted knowledge point, input the knowledge point, input information and a second prompt instruction into a rejection model, wherein the second prompt instruction is used to prompt the rejection model to determine whether the knowledge point can answer the input information;

[0122] If there is at least one knowledge point that can answer the input information, it is determined that the knowledge can answer the input information.

[0123] Optionally, the rejection model may be a large language model, for example, the aforementioned preset model may be directly reused, or it may be another trained model. When making a rejection judgment, the knowledge and the first prompt instruction may be first input into the rejection model, prompting the rejection model to analyze the knowledge and extract multiple knowledge points from the knowledge.

[0124] The knowledge point, which may also be called a fact point, is the content extracted from the knowledge by the rejection model. For example, the first prompt instruction may be: "What content does the above knowledge describe? Please answer point by point."

[0125] After obtaining multiple knowledge points output by the rejection model, the rejection model can be asked for each knowledge point whether the knowledge point can answer the input information. As long as there is a knowledge point that can answer the input information, the rejection component can determine that the searched knowledge can answer the input information.

[0126] Through the above method, the model can be used to first analyze the knowledge points, and then determine whether each knowledge point can answer the input information. The final result can be gradually obtained in a step-by-step manner to improve the accuracy of the rejection component.

[0127] The generation components are described below.

[0128] Optionally, the preset model used by the generation component may be a preset model trained with domain data, wherein the domain data may include the knowledge base, so that the preset model learns relevant domain knowledge and improves the accuracy and credibility of the generated reply information.

[0129] Optionally, the knowledge base may be used for pre-training, and the pre-trained preset model may be obtained for subsequent reply information generation operations. Alternatively, a preset model may be obtained after pre-training based on general knowledge, and then fine-tuned according to the knowledge base. Alternatively, the preset model may be pre-trained and fine-tuned according to the knowledge base. In addition to using the knowledge base during training, other domain data may be further added, and the specific scope of the training samples may be configured by the second user.

[0130] When constructing training samples, you can collect existing data as training samples, construct training samples through manual annotation, or use the capabilities of the preset model to automatically construct training samples.

[0131] Specifically, in the knowledge base, FAQ can be used to construct training samples. FAQ itself has questions and corresponding answers, which can greatly reduce the workload of constructing training samples. In addition, training samples can also be constructed in the following ways:

[0132] Extracting knowledge from a knowledge base, inputting the knowledge and prompt text into a preset model to obtain input information, wherein the prompt text is used to prompt the preset model to generate input information that the knowledge can answer;

[0133] Input the generated input information (and corresponding knowledge) into the preset model to obtain corresponding reply information;

[0134] Determine the correlation between the input information and the reply information, where the correlation is used to indicate whether the reply information can correctly answer the input information. If so, add the input information (and corresponding knowledge) and the reply information to the training sample. If not, discard the input information and the reply information. Alternatively, modify the reply information or regenerate the reply information based on the input information, perform the correlation judgment again, and add it to the training sample after the correlation meets the requirements.

[0135] Among them, relevance judgment can be performed manually or through some text evaluation indicators.

[0136] After the training is completed, during use, depending on the judgment results of the rejection component, there are two cases for the input of the generation component: one is input information, and the other is input information and knowledge. Optionally, in order to enable the preset model to distinguish between two different inputs, it can be processed in combination with the prompt information.

[0137] In the first case, when the input information and knowledge are input into the preset model to obtain corresponding reply information, the generation component is specifically used to: input the input information, the knowledge and the prompt information into the preset model to obtain corresponding reply information, and the prompt information is used to prompt the preset model to generate reply information corresponding to the input information based on the knowledge.

[0138] The prompt information may be obtained through a template. For example, the prompt information may be: "Human: Please remember the following knowledge, which may be helpful in answering certain questions. If you use a certain knowledge, please add the number before the knowledge in your reply:\n{doc}\nAssistant: OK, I have memorized the above information. Is there any question I can help you answer?\nHuman:{text_q}\nAssistant:".

[0139] Among them, {doc} is used to replace with knowledge, and {text_q} is used to replace with the input information (query) of the first user, so that the preset model can respond to the input information under the guidance of knowledge.

[0140] In the second case, when the input information is input into the preset model to obtain the corresponding reply information, the generation component is specifically used to: input the input information and prompt information into the preset model to obtain the corresponding reply information, and the prompt information is used to prompt the preset model to generate the reply information corresponding to the input information according to the knowledge learned during training; or, input the input information alone into the preset model to obtain the corresponding reply information.

[0141] Specifically, during the pre-training and fine-tuning training process, the preset model has learned some knowledge in related fields. When input information is input into the preset model, the model can be prompted to respond to the input information based on its own knowledge. Alternatively, since the preset model itself has the ability to answer questions, the input information can be directly input into the preset model without prompting information, and the preset model can obtain corresponding response information based on the input information.

[0142] Through the above scheme, the preset model can be equipped with the ability to distinguish different inputs, and can process different inputs in a targeted manner in combination with prompt information, thereby meeting the usage requirements of the preset model in different situations.

[0143] When deploying the system, the trained preset model can be directly obtained and used. In order to balance efficiency and accuracy, the embodiment of the present application also provides a solution for performing different operations according to the base capabilities of the preset model.

[0144] Optionally, before generating reply information based on input information, the generation component is also used to: obtain a trained preset model; if the preset model meets the preset requirements, save the preset model, and / or, if the preset model does not meet the preset requirements, fine-tune the preset model according to the knowledge base, and save the preset model after fine-tuning; wherein the preset requirement is that the preset model has the ability to generate reply information based on the prompt information.

[0145] Accordingly, when the input information and knowledge are input into the preset model to obtain corresponding reply information, the preset model is specifically used to:

[0146] If the saved preset model is a preset model that meets the preset requirements and has not been fine-tuned, then according to the input information, knowledge and prompt information, reply information is obtained through at least one round of interaction with the preset model; wherein, if there are multiple rounds of interaction, the prompt information corresponding to each round of interaction is different;

[0147] If the saved preset model is a preset model that has undergone the fine-tuning training, the input information, knowledge and prompt information are input into the preset model to directly obtain the corresponding reply information.

[0148] Figure 7 This is a schematic diagram of an application of a preset model provided in an embodiment of the present application. Figure 7 As shown, a trained preset model can be obtained, where the trained preset model may refer to a pre-trained preset model.

[0149] If the base capability of the preset model is weak, that is, it does not have the ability to generate reply information according to the prompt information, the preset model can be fine-tuned and trained according to the knowledge base. The fine-tuning training can be based on the needs of the current scene, so that the preset model has the ability to generate reply information according to the knowledge, input information and prompt information, and then the preset model after fine-tuning and training is saved. When in use, the input information, knowledge and prompt information are directly input into the preset model to obtain the corresponding reply information.

[0150] If the preset model has a strong base capability, that is, it has the ability to generate reply information according to the prompt information, fine-tuning training can be omitted and the preset model can be directly saved. When in use, although the preset model has a certain base capability, it has not undergone fine-tuning training of the knowledge base in this field, so it may not be able to adapt to the current scene well when generating reply information. Therefore, the preset model can be guided to generate reply information through multiple rounds of interaction.

[0151] Optionally, if the knowledge matching the input information includes at least one file, when the reply information is obtained after at least one round of interaction with the preset model, the generating component is specifically used to:

[0152] Inputting at least one file matching the input information, the input information and first prompt information into a preset model, wherein the first prompt information is used to prompt the preset model to select a target file related to the input information from the at least one file;

[0153] Determine a target file selected by the preset model, input the target file, input information, and second prompt information into the preset model, wherein the second prompt information is used to prompt the preset model to select a target paragraph related to the input information from the target file;

[0154] Determine the target paragraph selected by the preset model, input the target paragraph, input information and third prompt information into the preset model, and the third prompt information is used to prompt the preset model to generate corresponding reply information according to the target paragraph and the input information.

[0155] Specifically, when the reply information is obtained after at least one round of interaction with the preset model, the corresponding prompt information may include first prompt information, second prompt information and third prompt information, which are respectively used to prompt the preset model to find relevant files, relevant paragraphs and generate reply information. The relevant files / paragraphs may refer to files / paragraphs that the preset model recognizes from at least one file and can answer the input information.

[0156] Exemplarily, the query search component searches for a list of three documents, and the rejection component will uniformly determine whether the three documents can answer the input information. If they can, the list of three documents and the input information will be given to the preset model, prompting the preset model to extract relevant documents from the three documents, and then the documents and input information will be given to the preset model, prompting the preset model to extract relevant paragraphs from the documents, and finally the relevant paragraphs and input information will be given to the preset model, prompting the preset model to generate reply information based on the relevant paragraphs and input information.

[0157] In this way, through multiple rounds of interaction, the preset model can be guided step by step to generate reply information, thereby utilizing the preset model with certain base capabilities to gradually generate accurate reply information, effectively improving the quality of generation.

[0158] In addition, for preset models with strong base capabilities, a solution without fine-tuning and multiple rounds of interaction can be used. For preset models with weak base capabilities, a solution with fine-tuning and single round of interaction can be used. According to the capabilities of the preset models obtained, fine-tuning strategies and usage strategies can be executed in a targeted manner, taking into account the efficiency and accuracy of the fine-tuning and usage processes, thereby improving the overall efficiency and accuracy of the system.

[0159] Based on the technical solutions provided in the embodiments of the present application, optionally, the generating component is further used for:

[0160] If the judgment result of the rejection component is yes, after obtaining corresponding reply information according to the input information and the matching knowledge, the reply information is output to the first user;

[0161] If the determination result of the rejection component is negative, after obtaining corresponding reply information according to the input information, the reply information is output to the reference search component, so as to determine the information output to the first user through the reference search component and the reference determination component.

[0162] Specifically, if the judgment result of the rejection component is yes, it means that the knowledge can answer the input information. After the reply information is obtained according to the knowledge and the input information, the reply information can be directly output to the first user without using the reference search component and the reference determination component for processing.

[0163] If the judgment result of the rejection component is no, it means that the knowledge cannot answer the input information. After obtaining the reply information based on the input information, the reference search component and the reference judgment component can be used for a posteriori processing, that is, the reference search component searches for knowledge matching the reply information in the knowledge base, and the reference judgment component judges the authenticity of the reply information based on the knowledge, so as to determine the information output to the first user based on the judgment result.

[0164] By using the judgment result of the rejection component to decide whether to directly output the reply information after it is generated or to trigger the reference search component and the reference determination component for processing, the overall processing efficiency of the system can be effectively improved while ensuring a certain degree of accuracy.

[0165] The reference search component and the reference determination component are described below.

[0166] Optionally, the reference search component is specifically used to: obtain reply information, split the reply information to obtain multiple sub-information; and search the knowledge base for knowledge matching each sub-information. Correspondingly, the reference determination component is specifically used to: determine the authenticity of each sub-information based on each sub-information and the corresponding knowledge, and determine the information output to the first user based on the authenticity of each sub-information.

[0167] Exemplarily, the reply information can be a paragraph of text, then the reply information is divided into sentences to obtain multiple sentences as sub-information. For each sentence, relevant knowledge can be searched in the knowledge base, and the specific implementation principle of the search can refer to the query search component. Specifically, the query search component searches for knowledge that matches the input information, and the reference search component searches for knowledge that matches the reply information. It is only necessary to replace the input information in the processing scheme of the query search component with the reply information to obtain the processing scheme of the reference search component. Of course, the processing scheme of the reference search component may also be different from that of the query search component, as long as it can realize the search for relevant knowledge based on the reply information. After the reference search component searches for the knowledge corresponding to each sentence, the authenticity of each sentence can be judged separately by the reference judgment component.

[0168] Optionally, the reference judgment component is specifically used to: for any sub-information, input the sub-information and corresponding knowledge into the judgment model to obtain a judgment result of the sub-information, the judgment result includes: true, false, and irrelevant, respectively indicating that the knowledge supports the sub-information, the sub-information contradicts the knowledge, and the knowledge and the sub-information are irrelevant; determine the information to be output to the first user based on the judgment result of each sub-information and the output strategy configured by the second user.

[0169] Optionally, the judgment model may be a trained machine learning model, the input of which is the sub-information and knowledge, and the output is the authenticity judgment result. Among them, true means that the sub-information is contained in the corresponding knowledge, that is, the corresponding sub-information can be judged to be true and correct based on the knowledge; false means that the meaning of the sub-information and the knowledge are contradictory, so the sub-information is false and wrong; irrelevant means that the sub-information and the knowledge are not related, and it is impossible to confirm whether the sub-information is correct or wrong.

[0170] Both the determination model and the rejection model can be regarded as specific classification models. Therefore, the training process of the determination model can also refer to the training process of the rejection model. For example, the corresponding label (true, false, irrelevant) can be determined by a preset model or a sub-information sample can be generated to construct a training sample to train the determination model.

[0171] After the judgment model obtains the judgment result of each sub-information, it can decide whether to output each sub-information to the first user according to the judgment result of each sub-information. Optionally, only the sub-information with a true judgment result can be output, or the sub-information with a true judgment result or irrelevant can also be output.

[0172] Considering that different fields, different enterprises and different scenarios have different output requirements, the output policy configured by the second user can be combined to determine the sub-information to be output, wherein the output policy is used to indicate whether to output sub-information with a false or irrelevant judgment result.

[0173] Figure 8 A schematic diagram of the principle of a reference search component provided in an embodiment of the present application, Fig. 9 The following is a schematic diagram of the principle of a reference determination component provided in an embodiment of the present application. Figure 8 and Fig. 9 As shown, for a text-type reply message, it can be split by referencing the search component to obtain sentence 1, sentence 2, and sentence 3. For each sentence, the corresponding knowledge is searched, for example, knowledge A, knowledge B, and knowledge C are obtained respectively.

[0174] The reference judgment component can judge the authenticity of each sentence based on the corresponding knowledge, that is, whether the corresponding knowledge can support the sentence, and obtain the judgment result of each sentence. Then, based on the judgment result of each sentence and the pre-configured output strategy, the information output to the first user is determined.

[0175] The above explanation uses sentences as sub-information as an example. When the reply information contains other modalities besides text, the reply information can also be split accordingly to obtain sub-information of other modalities. For example, the reply information can be a segment of audio. The audio can be split to obtain multiple audio segments for subsequent processing as sub-information.

[0176] Through the above method, the reply information can be split into multiple sub-information, which can be searched and judged in a more fine-grained manner, generate more refined results, and facilitate users to verify each sub-information according to the knowledge corresponding to each sub-information, thereby improving the user experience. In addition, the final output information can be determined based on the authenticity of the sub-information and the pre-configured output strategy. For scenarios with stricter requirements such as law and medical care, it can be configured to prohibit the output of sub-information with false or irrelevant judgment results, so as to avoid giving the first user some unfounded suggestions, such as suggestions on drugs, which affect property and life safety. For other relatively relaxed scenarios, such as cultural and entertainment scenarios, it can be allowed to output irrelevant sub-information, and even output false sub-information when certain conditions are met, so as to meet the usage needs of different scenarios.

[0177] On the basis of the technical solutions provided in the above embodiments, the system may optionally further include: an output component for outputting the reply information and a link to the knowledge corresponding to the reply information to the first user through an interactive interface when the reply information is judged to be true based on the knowledge, so that the first user can verify the reply information based on the link.

[0178] Fig.10 A schematic diagram of an interactive interface provided in an embodiment of the present application. Fig.10 As shown, the interactive interface displayed to the first user may display the input information of the first user, such as "S library creates triggers and shadow tables". After the corresponding reply information is generated, it may also be displayed on the interactive interface, such as "According to the search results provided, S library does not support the creation of custom objects, including triggers and shadow tables[1], by submitting change applications". [] is used to represent the serial number of the referenced knowledge (such as a document). A link to the knowledge with the corresponding serial number may be displayed below. After the user clicks the corresponding link, the relevant document may be opened to verify the authenticity of the reply information.

[0179] By displaying the reply information and the link to the corresponding knowledge on the interactive interface, it is convenient for users to verify the reply information. When the reply information is split into multiple sub-information, the link to the knowledge corresponding to each sub-information can be displayed separately to ensure that each sub-information has a basis, and it is convenient for the first user to further view and learn related content after opening the corresponding knowledge, thereby improving the user experience.

[0180] Optionally, the output component can be used to:

[0181] When it is determined that the reply information is not true according to the knowledge, outputting target information to the first user through an interactive interface, the target information being used to indicate that no reply information matching the input information has been searched, and prompting the first user to try to search again;

[0182] After obtaining the re-search instruction input by the first user, outputting a regeneration instruction, wherein the regeneration instruction is used to instruct to regenerate reply information according to the input information using the preset model;

[0183] When the reference search component searches for knowledge matching the regenerated reply information and the reference determination component determines that the regenerated reply information is true based on the knowledge, the regenerated reply information is output to the first user through the interactive interface.

[0184] Specifically, when the reply information does not meet the requirements, the user can click to search again on the interactive interface. At this time, new reply information can be obtained again based on the previous input information, and a posteriori reference operation can be performed on the new reply information. If the new reply information meets the requirements, that is, the new reply information is true, or partially true, then the true reply information can be output to the user.

[0185] Optionally, when the reply message is split into multiple sub-information and the authenticity is judged separately, if multiple sub-information are not true, then the reply message is considered to be untrue, and the first user is prompted to try to search again; if at least one sub-information among the multiple sub-information is true, then it is considered that it does not belong to the case of "the reply information is not true", and the true sub-information can be output at this time.

[0186] In the case where the first user confirms to re-search, if the regenerated reply information is split into multiple sub-information and the authenticity is judged separately, only the true sub-information can be output, or the final output information can be determined more flexibly according to the output strategy.

[0187] Through the above method, when the reply information does not meet the requirements, it can be regenerated under the instruction of the first user. Since the regenerated reply information is not exactly the same as the original reply information, although the original reply information does not meet the requirements, the regenerated reply information may meet the requirements. In this way, the generation ability of the model and the guarantee ability of the a posteriori reference can be utilized to try to generate reply information that meets the requirements for the user, meet the user's usage needs, and improve the user experience.

[0188] On the basis of the technical solutions provided in the above embodiments, optionally, the system further includes: a configuration component, the configuration component being used to provide the second user with at least one of the following:

[0189] A query search configuration interface, used for the second user to configure a recall method of the query search component and / or at least one indicator used in sorting;

[0190] A training strategy configuration interface, used for the second user to determine the training strategy of any one or more of the preset model, the rejection model and the determination model used by the response generation system;

[0191] A knowledge base update interface, used to display input information corresponding to the reply information to the second user when the reference determination component determines that the reply information is true, so that the second user updates the knowledge base according to the input information;

[0192] The output policy configuration interface is used for the second user to configure the output policy.

[0193] The second user may configure any one or more components involved in the system, may configure any one or more models, and may configure and update the knowledge base.

[0194] When configuring a component, the second user may be allowed to configure the input, output, and processing strategy of the component. For example, the recall method, sorting method, etc. of the query search component and the reference search component may be configured, or the output strategy of the reference determination component may be configured.

[0195] When configuring the model, the second user may be allowed to adjust the training strategy of the model through the relevant interface, including but not limited to: model input, model output, model structure, model training process, etc.

[0196] For example, in terms of model input, the second user can adjust the input format and input mode, and add, modify, and filter the input information in the training sample. In terms of model output, the user can adjust the output format and mode, and can also modify and filter the labels. In terms of model structure, the user can adjust the specific structure and parameter quantity of the model, and add, reduce, or modify one or more modules in the model. In terms of the model training process, the user can plan the training stage of the model, select the training data set and input and output strategy, loss function, and training stop condition used in each stage.

[0197] Optionally, relevant information to assist the user in making a choice may be displayed on the interface, such as various information available for selection, or detailed content of various information available for selection. In addition, various dynamic information during the model training process may be displayed to the user, such as the intermediate results of the model training, so that the user can update the strategy in a timely manner.

[0198] When configuring the knowledge base, the second user can choose to add, modify, delete files, etc. in the knowledge base. At the same time, the knowledge base can also be updated according to the input information of the first user during actual use.

[0199] Specifically, in actual applications, there may be such a situation: according to the input information of the first user (such as a question), no relevant knowledge can be found in the knowledge base. This is because there may be no text similar to the question in the knowledge base, resulting in empty search results or irrelevant knowledge searched. However, after the preset model obtains the corresponding reply information (such as an answer) according to the question, there may be text similar to the answer in the knowledge base, which can be used as the knowledge corresponding to the answer. When the answer is judged to be true according to the corresponding knowledge, it means that the knowledge base can answer the corresponding question. Therefore, the question can be put into the knowledge base. The next time the first user inputs the same or similar question, the relevant knowledge can be directly searched according to the question, which effectively improves the overall processing efficiency of the system.

[0200] Optionally, when no relevant knowledge can be found based on the question but an answer that can be supported by the knowledge base is generated, the question, answer and related knowledge can be displayed to a second user to allow the second user to decide whether to add the question to the knowledge base and how to add it to the knowledge base, thereby ensuring the accuracy of the knowledge in the knowledge base.

[0201] In an embodiment of the present application, various configuration interfaces are provided to the second user, allowing the second user to configure components, models, knowledge bases, etc. separately, which can meet the various actual needs of the second user, is conducive to improving the overall efficiency and accuracy of the system in specific scenarios, and has strong applicability.

[0202] Optionally, when the reply information is confirmed to be false, the generation component may input the false reply information, input information and prompt information into the preset model, and generate updated reply information through the preset model, wherein the prompt information is used to inform the preset model that the input reply information is false and prompt it to output true reply information. For example, the prompt information may be: "Here is a question and a corresponding answer, but the answer is wrong. Please generate a correct answer based on the question and the wrong answer."

[0203] Optionally, the corresponding knowledge may also be input into the preset model at the same time, so that the preset model can modify the false reply information with the assistance of the corresponding knowledge to obtain updated reply information.

[0204] After obtaining the true reply information, the authenticity of the reply information can be determined by re-performing the reference search component and the reference determination component based on the true reply information. If it is true, it can be output.

[0205] Since false reply information refers to information that contradicts knowledge, when false reply information is obtained, the true reply information can be obtained by reversing the false reply information and verifying it through the knowledge base, thereby increasing the effective answer rate and improving the user experience.

[0206] Optionally, when the reply message is split into multiple sub-messages, if there are false sub-messages, the false sub-messages can be reversed and back-tested respectively. In addition, the reversal strategy can also be set, for example, only when all sub-messages are false or irrelevant, the false sub-messages are reversed; if there are true sub-messages, the true sub-messages are output and the false sub-messages are ignored, so as to reduce the system operations and improve the system efficiency while ensuring the accuracy of the system.

[0207] Based on the solutions provided in the embodiments of the present application, optionally, a rewriting component may be added before the query search component, and the rewriting component is used to:

[0208] Acquire input information of the first user, and rewrite the input information of the first user;

[0209] The rewritten input information of the first user is sent to the query search component.

[0210] Among them, rewriting includes: supplementation and / or splitting. Supplementation means completing the input information according to the context. For example, the user's input information is "how much is this?" According to the above, it can be determined that the user is asking about model B of product A, so the input information can be supplemented as "how much is model B of product A?" Splitting means that the user's input information may contain multiple questions, so multiple questions can be split from the input information. For example, the user's input information is "how much is product A, how long will it take to ship after placing an order, and how long is the shelf life?", it can be split into three input information: "how much is product A", "how long will it take to ship after placing an order for product A", and "how long is the shelf life of product A".

[0211] Optionally, the rewrite operation may also be implemented through a machine learning model, which may be implemented through a large model such as a preset model, or through a small model.

[0212] In the embodiments of the present application, the preset model used by the generation component must have relatively complex processing capabilities, so a large model can be used to implement it. Other models, such as rejection models, judgment models, rewrite models, etc., can be implemented using a large model. For example, the preset model can be reused, and by giving the preset model certain prompts, the preset model can be allowed to implement rejection judgment, authenticity judgment, rewrite and other operations. Alternatively, these operations can also use small models, for example, a single model can be trained to implement rewrite operations to improve processing efficiency.

[0213] The above uses text as an example to illustrate the processing of the system. On this basis, the knowledge base and the system can be expanded to multimodal implementation. When searching, you can search for pictures based on pictures, search for text based on text, etc. When using models to process multimodal information, you can first convert other modal information into text, and then process it based on the text. Alternatively, the preset models and other models used can be multimodal models that have multimodal processing capabilities. In order to enable the model to learn multimodal processing capabilities, multimodal samples can be constructed during the training phase to train the model.

[0214] The present application also provides a reply generation system, including:

[0215] A query search component, used to obtain input information of a first user, and search the knowledge base for knowledge matching the input information;

[0216] A rejection component, used to determine whether the knowledge searched by the query search component can answer the input information;

[0217] A generation component is used to obtain the input information and knowledge matching the input information when the judgment result of the rejection component is yes, input the input information and knowledge into a preset model, and obtain corresponding reply information; and / or, when the judgment result of the rejection component is no, obtain the input information, input the input information into a preset model, and obtain corresponding reply information.

[0218] Among them, the specific implementation principles and processing procedures of each component in the reply generation system can be found in the aforementioned embodiments and will not be repeated here.

[0219] The reply generation system provided in this embodiment combines the query search component with the generation component, so that the generated reply information can find the corresponding reference information in the knowledge base, and the user can verify the authenticity of the reply information. A rejection component is introduced to determine whether the knowledge searched by the query search component can answer the user's input information, thereby reducing the situation where the reply information is incorrect due to inaccurate knowledge searched by the query, and improving the accuracy of the reply information.

[0220] The embodiment of the present application also provides a reply generation system, which is applied to the scenario of a technical service enterprise, and the system includes:

[0221] A query search component, used to obtain query information of the first user, and search the knowledge base for documents matching the query information; wherein the knowledge base includes a plurality of documents corresponding to products provided by the enterprise;

[0222] A rejection component, used to determine whether the document searched by the query search component can answer the query information;

[0223] a generation component, for obtaining the query information and the document when the judgment result of the rejection component is yes, inputting the query information and the document into a preset model, and obtaining corresponding reply information; and / or, for obtaining the query information when the judgment result of the rejection component is no, inputting the query information into a preset model, and obtaining corresponding reply information;

[0224] A reference search component, used to obtain the reply information and search the knowledge base for documents matching the reply information;

[0225] The reference determination component is used to determine the authenticity of the reply information based on the documents searched by the reference search component, so as to determine the answer output to the first user based on the judgment result.

[0226] Among them, the specific implementation principles and processing procedures of each component in the reply generation system can be found in the aforementioned embodiments and will not be repeated here.

[0227] The reply generation system provided in this embodiment can be connected to the enterprise's knowledge base, and through the cooperation of the above-mentioned components, it can realize effective replies to the input information of enterprise users, ensure that the reply information returned to the user is authentic and reliable, effectively improve the accuracy of the entire system, and enable users to use the technical services provided by the enterprise efficiently and accurately to meet the actual needs of the enterprise.

[0228] Corresponding to the above system, an embodiment of the present application also provides a reply generation method. Fig.11 A flowchart of a reply generation method provided in an embodiment of the present application. Fig.11 As shown, the method includes:

[0229] Step 1101: Acquire reply information, and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model based on the input information of the first user. Step 1102: Determine the authenticity of the reply information based on the knowledge, and determine the information output to the first user based on the determination result.

[0230] Among them, step 1101 and step 1102 can be implemented by a reference search component and a reference determination component respectively.

[0231] Optionally, the method further includes:

[0232] Acquire input information of the first user through a query search component, and search the knowledge base for knowledge matching the input information;

[0233] Determine, by a rejection component, whether the knowledge searched by the query search component can answer the input information;

[0234] When the judgment result of the rejection component is yes, the generation component obtains the input information and the knowledge matching the input information, inputs the input information and the knowledge into a preset model, and obtains corresponding reply information; and / or, when the judgment result of the rejection component is no, obtains the input information, inputs the input information into a preset model, and obtains corresponding reply information.

[0235] Optionally, the knowledge base includes one or more files, and the files include content in at least one of the following modes: text, image, audio, video;

[0236] The knowledge searched by the reference search component or the query search component is at least one file in the knowledge base, or is part of the content in a file;

[0237] The input information and the reply information both contain at least one modal content;

[0238] Acquiring input information of a first user, and searching the knowledge base for knowledge matching the input information, comprises:

[0239] Obtain input information of the first user; search the knowledge base for knowledge corresponding to the input information by semantic recall and / or vector recall; if multiple knowledge are found, determine the degree of match between each knowledge and the input information according to at least one indicator, sort the multiple knowledge according to the degree of match, and determine the knowledge matching the input information according to the sorting result;

[0240] Wherein, semantic recall includes: searching for knowledge whose similarity with the input information satisfies a first preset condition; vector recall includes: searching for knowledge whose distance with the embedding vector of the input information satisfies a second preset condition according to the embedding vector corresponding to each piece of knowledge;

[0241] The at least one indicator includes at least one of the following: a semantic matching score, used to indicate the similarity between the input information and the knowledge; an edit distance score, used to indicate the distance between the embedding vector of the input information and the embedding vector of the knowledge; an entity relevance score, used to indicate the relevance between the entities contained in the input information and the entities contained in the knowledge;

[0242] For any piece of knowledge, the matching degree between the knowledge and the input information is: the result obtained by weighted summation of the at least one indicator of the knowledge.

[0243] Optionally, judging whether the knowledge searched by the query search component can answer the input information includes: inputting the input information and the matching knowledge into a rejection model, and judging whether the knowledge can answer the input information through the rejection model;

[0244] The rejection model is trained in the following way:

[0245] According to the preset model, a training sample is obtained, wherein the training sample includes: an input information sample, a knowledge sample and a label, wherein the label is used to indicate whether the knowledge sample can answer the input information sample;

[0246] The rejection model is trained according to the input information samples, knowledge samples and labels.

[0247] Optionally, inputting the input information and the knowledge into a preset model to obtain corresponding reply information includes: inputting the input information, the knowledge and prompt information into a preset model to obtain corresponding reply information, the prompt information being used to prompt the preset model to generate reply information corresponding to the input information according to the knowledge;

[0248] and / or,

[0249] The input information is input into a preset model to obtain corresponding reply information, including: inputting the input information and prompt information into a preset model to obtain corresponding reply information, wherein the prompt information is used to prompt the preset model to generate reply information corresponding to the input information according to the knowledge learned during training; or, inputting the input information alone into the preset model to obtain corresponding reply information.

[0250] Optionally, before generating reply information according to the input information, the method further includes:

[0251] Get a trained preset model;

[0252] If the preset model meets the preset requirements, the preset model is saved, and / or, if the preset model does not meet the preset requirements, the preset model is fine-tuned according to the knowledge base, and the fine-tuned preset model is saved;

[0253] The preset requirement is that the preset model has the ability to generate reply information according to the prompt information;

[0254] Accordingly, the input information and knowledge are input into the preset model to obtain corresponding reply information, including:

[0255] If the saved preset model is a preset model that meets the preset requirements and has not been fine-tuned, then according to the input information, knowledge and prompt information, reply information is obtained through at least one round of interaction with the preset model; wherein, if there are multiple rounds of interaction, the prompt information corresponding to each round of interaction is different;

[0256] and / or,

[0257] If the saved preset model is a preset model that has undergone the fine-tuning training, the input information, knowledge and prompt information are input into the preset model to directly obtain the corresponding reply information.

[0258] Optionally, if the knowledge matching the input information includes at least one file, then after at least one round of interaction with the preset model, the reply information is obtained, including:

[0259] Inputting at least one file matching the input information, the input information and first prompt information into a preset model, wherein the first prompt information is used to prompt the preset model to select a target file related to the input information from the at least one file;

[0260] Determine a target file selected by the preset model, input the target file, input information, and second prompt information into the preset model, wherein the second prompt information is used to prompt the preset model to select a target paragraph related to the input information from the target file;

[0261] Determine the target paragraph selected by the preset model, input the target paragraph, input information and third prompt information into the preset model, and the third prompt information is used to prompt the preset model to generate corresponding reply information according to the target paragraph and the input information.

[0262] Optionally, the method further includes:

[0263] If the judgment result of the rejection component is yes, after obtaining corresponding reply information according to the input information and the matching knowledge, the reply information is output to the first user;

[0264] If the determination result of the rejection component is negative, after obtaining corresponding reply information according to the input information, the reply information is output to the reference search component, so as to determine the information output to the first user through the reference search component and the reference determination component.

[0265] Optionally, obtaining the reply information and searching the knowledge base for knowledge matching the reply information includes: obtaining the reply information, splitting the reply information to obtain multiple sub-information; searching the knowledge base for knowledge matching each sub-information;

[0266] The authenticity of the reply information is judged according to the knowledge to determine the information output to the first user according to the judgment result, including: judging the authenticity of each sub-information according to each sub-information and the corresponding knowledge, and determining the information output to the first user according to the authenticity of each sub-information.

[0267] Optionally, judging the authenticity of each sub-information according to each sub-information and corresponding knowledge, and determining the information output to the first user according to the authenticity of each sub-information, includes:

[0268] For any sub-information, the sub-information and the corresponding knowledge are input into the judgment model to obtain the judgment result of the sub-information, and the judgment result includes: true, false, and irrelevant, which respectively indicate that the knowledge supports the sub-information, the sub-information contradicts the knowledge, and the knowledge is irrelevant to the sub-information;

[0269] Determine the information to be output to the first user according to the judgment result of each sub-information and the output strategy configured by the second user;

[0270] The output strategy is used to indicate whether to output sub-information that the judgment result is false or irrelevant.

[0271] Optionally, the method further includes:

[0272] When the output component determines that the reply information is true according to the knowledge, the reply information and a link to the knowledge corresponding to the reply information are output to the first user through an interactive interface, so that the first user can verify the reply information according to the link;

[0273] and / or,

[0274] When it is judged that the reply information is not true based on the knowledge, target information is output to the first user through an interactive interface, and the target information is used to indicate that reply information matching the input information has not been searched, and the first user is prompted to try to search again; after obtaining the re-search instruction input by the first user, a regeneration instruction is output, and the regeneration instruction is used to instruct to use the preset model to regenerate the reply information according to the input information; when knowledge matching the regenerated reply information is searched through the reference search component, and the regenerated reply information is judged to be true based on the knowledge through the reference judgment component, the regenerated reply information is output to the first user through the interactive interface.

[0275] Optionally, the method further includes: providing at least one of the following to the second user through the configuration component:

[0276] A query search configuration interface, used for the second user to configure a recall method of the query search component and / or at least one indicator used in sorting;

[0277] A training strategy configuration interface, used for the second user to determine the training strategy of any one or more of the preset model, the rejection model and the determination model used by the response generation system;

[0278] A knowledge base update interface, used to display input information corresponding to the reply information to the second user when the reference determination component determines that the reply information is true, so that the second user updates the knowledge base according to the input information;

[0279] The output policy configuration interface is used for the second user to configure the output policy.

[0280] The method provided in the embodiment of the present application can be implemented by the system provided in the aforementioned embodiment. The specific implementation principles and effects can be found in the aforementioned embodiment and will not be repeated here.

[0281] Fig.12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig.12 As shown, the electronic device of this embodiment may include:

[0282] at least one processor 1201; and

[0283] A memory 1202 communicatively connected to the at least one processor;

[0284] The memory 1202 stores instructions that can be executed by the at least one processor 1201, and the instructions are executed by the at least one processor 1201 to enable the electronic device to execute the method described in any of the above embodiments.

[0285] Optionally, the memory 1202 may be independent or integrated with the processor 1201 .

[0286] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0287] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.

[0288] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.

[0289] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0290] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0291] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, referred to as CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0292] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0293] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0294] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0295] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0296] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0297] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A reply generation system, characterized in that: include: A reference search component is used to obtain reply information and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model based on input information of the first user; The reference determination component is used to determine the authenticity of the reply information according to the knowledge, so as to determine the information output to the first user according to the determination result.

2. The system according to claim 1, characterized in that Also includes: A query search component, used to obtain input information of a first user, and search the knowledge base for knowledge matching the input information; A rejection component, used to determine whether the knowledge searched by the query search component can answer the input information; A generating component, for obtaining the input information and knowledge matching the input information when the judgment result of the rejecting component is yes, inputting the input information and knowledge into a preset model, and obtaining corresponding reply information; And / or, when the determination result of the rejection component is negative, the input information is obtained, and the input information is input into a preset model to obtain corresponding reply information.

3. The system according to claim 2, characterized in that The knowledge base includes one or more files, and the files include content in at least one of the following modalities: text, image, audio, video; The knowledge searched by the reference search component or the query search component is at least one file in the knowledge base, or is part of the content in a file; The input information and the reply information both contain at least one modal content; The query search component is specifically used to: obtain input information of the first user; search the knowledge base for knowledge corresponding to the input information by means of semantic recall and / or vector recall; If multiple pieces of knowledge are found, the matching degree between each piece of knowledge and the input information is determined based on at least one indicator, and the multiple pieces of knowledge are sorted based on the matching degree, and the knowledge matching the input information is determined based on the sorting result.

4. The system according to claim 2, characterized in that When the input information and knowledge are input into the preset model to obtain corresponding reply information, the generating component is specifically used to: input the input information, the knowledge and the prompt information into the preset model to obtain corresponding reply information, and the prompt information is used to prompt the preset model to generate reply information corresponding to the input information according to the knowledge; and / or, When the input information is input into the preset model to obtain corresponding reply information, the generating component is specifically used to: input the input information and prompt information into the preset model to obtain corresponding reply information, and the prompt information is used to prompt the preset model to generate reply information corresponding to the input information according to the knowledge learned during training; Alternatively, the input information is input into a preset model separately to obtain corresponding reply information.

5. The system according to claim 2, characterized in that Before generating reply information according to the input information, the generating component is further used to: Get a trained preset model; If the preset model meets the preset requirements, the preset model is saved, and / or, if the preset model does not meet the preset requirements, the preset model is fine-tuned according to the knowledge base, and the fine-tuned preset model is saved; The preset requirement is that the preset model has the ability to generate reply information according to the prompt information; Accordingly, when the input information and knowledge are input into the preset model to obtain corresponding reply information, the preset model is specifically used to: If the saved preset model is a preset model that meets the preset requirements and has not been fine-tuned, then according to the input information, knowledge and prompt information, reply information is obtained through at least one round of interaction with the preset model; wherein, if there are multiple rounds of interaction, the prompt information corresponding to each round of interaction is different; and / or, If the saved preset model is a preset model that has undergone the fine-tuning training, the input information, knowledge and prompt information are input into the preset model to directly obtain the corresponding reply information.

6. The system according to claim 5, characterized in that If the knowledge matching the input information includes at least one file, when the reply information is obtained after at least one round of interaction with the preset model, the generating component is specifically used to: Inputting at least one file matching the input information, the input information and first prompt information into a preset model, wherein the first prompt information is used to prompt the preset model to select a target file related to the input information from the at least one file; Determine a target file selected by the preset model, input the target file, input information, and second prompt information into the preset model, wherein the second prompt information is used to prompt the preset model to select a target paragraph related to the input information from the target file; Determine the target paragraph selected by the preset model, input the target paragraph, input information and third prompt information into the preset model, and the third prompt information is used to prompt the preset model to generate corresponding reply information according to the target paragraph and the input information.

7. The system according to claim 2, characterized in that The generation component is also used to: If the judgment result of the rejection component is yes, after obtaining corresponding reply information according to the input information and the matching knowledge, the reply information is output to the first user; If the determination result of the rejection component is negative, after obtaining corresponding reply information according to the input information, the reply information is output to the reference search component, so as to determine the information output to the first user through the reference search component and the reference determination component.

8. The system according to claim 1, characterized in that The reference search component is specifically used to: obtain reply information, split the reply information to obtain multiple sub-information; search the knowledge base for knowledge matching each sub-information; The reference determination component is specifically used to determine the authenticity of each sub-information according to each sub-information and corresponding knowledge, and determine the information output to the first user according to the authenticity of each sub-information.

9. The system according to claim 8, characterized in that The reference determination component is specifically used for: For any sub-information, the sub-information and the corresponding knowledge are input into the judgment model to obtain the judgment result of the sub-information, and the judgment result includes: true, false, and irrelevant, which respectively indicate that the knowledge supports the sub-information, the sub-information contradicts the knowledge, and the knowledge is irrelevant to the sub-information; Determine the information to be output to the first user according to the judgment result of each sub-information and the output strategy configured by the second user; The output strategy is used to indicate whether to output sub-information that the judgment result is false or irrelevant.

10. The system according to any one of claims 1 to 9, characterized in that: Also includes: An output component, the output component is used to: When the reply information is judged to be true according to the knowledge, the reply information and a link to the knowledge corresponding to the reply information are output to the first user through an interactive interface, so that the first user can verify the reply information according to the link; and / or, When it is judged that the reply information is not true based on the knowledge, target information is output to the first user through an interactive interface, and the target information is used to indicate that reply information matching the input information has not been searched, and the first user is prompted to try to search again; after obtaining the re-search instruction input by the first user, a regeneration instruction is output, and the regeneration instruction is used to instruct to use the preset model to regenerate the reply information according to the input information; when knowledge matching the regenerated reply information is searched through the reference search component, and the regenerated reply information is judged to be true based on the knowledge through the reference judgment component, the regenerated reply information is output to the first user through the interactive interface.

11. The system according to any one of claims 2 to 9, characterized in that: Also includes: A configuration component, the configuration component being used to provide at least one of the following to the second user: A query search configuration interface, used for the second user to configure a recall method of the query search component and / or at least one indicator used in sorting; A training strategy configuration interface, used for the second user to determine the training strategy of any one or more of the preset model, the rejection model and the determination model used by the response generation system; A knowledge base update interface, used to display input information corresponding to the reply information to the second user when the reference determination component determines that the reply information is true, so that the second user updates the knowledge base according to the input information; The output policy configuration interface is used for the second user to configure the output policy.

12. A reply generation system, characterized in that: The system comprises: A query search component, used to obtain query information of the first user, and search the knowledge base for documents matching the query information; wherein the knowledge base includes a plurality of documents corresponding to products provided by the enterprise; A rejection component, used to determine whether the document searched by the query search component can answer the query information; a generation component, for obtaining the query information and the document when the judgment result of the rejection component is yes, inputting the query information and the document into a preset model, and obtaining corresponding reply information; and / or, for obtaining the query information when the judgment result of the rejection component is no, inputting the query information into a preset model, and obtaining corresponding reply information; A reference search component, used to obtain the reply information and search the knowledge base for documents matching the reply information; The reference determination component is used to determine the authenticity of the reply information based on the documents searched by the reference search component, so as to determine the answer output to the first user based on the judgment result.

13. A reply generation method, characterized in that: include: Acquire reply information, and search the knowledge base for knowledge matching the reply information; wherein the reply information is obtained by a trained preset model according to input information of the first user; The authenticity of the reply information is judged according to the knowledge, so as to determine the information output to the first user according to the judgment result.

14. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method of claim 13.

Images