Problem processing method and device based on large language model

Abstract

The embodiment of the invention provides a problem processing method and device based on a large language model, and the method comprises the steps: carrying out the text screening in a preset text set according to a user problem, obtaining an intermediate text, and carrying out the first filtering processing of the intermediate text based on the relevancy between the user problem and the intermediate text, and obtaining a first remaining text; inputting the first residual text and the user question into a large language model to generate an answer to obtain a question answer of the user question; performing second filtering processing on the intermediate text based on the relevancy to obtain second residual texts, and selecting at least one of the second residual texts and a preset document abstract according to the number of the second residual texts; and inputting at least one, the user question and the question answer into a large language model for question recommendation processing to obtain a recommended question. By adopting the embodiment of the invention, the accuracy of answers to the questions of the user can be improved, and the initiative of question and answer interaction of the user is improved.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a problem processing method and device based on a large language model. Background Art

[0002] With the continuous development of Internet technology and the continuous advancement of social information, more and more service scenarios have begun to use large language models to generate answers to user questions, such as customer service scenarios; the large language model is a deep learning model trained using a large amount of text data and belongs to the field of natural language processing.

[0003] At present, in the process of generating answers to user questions through a large language model, the knowledge obtained after knowledge retrieval of the user question is often input into the large language model to obtain answers that meet the user question within the preset knowledge range output by the large language model. Due to the single knowledge retrieval method, the accuracy and comprehensiveness of the answers to user questions are poor. At the same time, the output answers to user questions are not extensible and cannot guide users to conduct multiple rounds of conversations. Summary of the invention

[0004] The purpose of one embodiment of the present specification is to provide a question processing method and device based on a large language model to improve the accuracy of answers to user questions and enhance the user's initiative in question-and-answer interaction.

[0005] To solve the above technical problems, an embodiment of this specification is implemented as follows: In a first aspect, an embodiment of the present specification provides a problem processing method based on a large language model, the method comprising: Screening a preset text set according to the user question to obtain an intermediate text, and performing a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text; Inputting the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question; Performing a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The at least one, the user question and the answer to the question are input into a large language model for question recommendation processing to obtain a recommended question.

[0006] The problem processing method based on the large language model provided in this embodiment performs text screening in a preset text set according to the user's question to obtain an intermediate text, performs a first filtering process on the intermediate text according to the relevance between the user's question and the intermediate text to obtain a first remaining text, inputs the first remaining text and the user's question into the large language model to generate an answer, and obtains the answer to the user's question, thereby further screening the intermediate text recalled based on the user's question to improve the effectiveness of the screened first remaining text, and then improve the accuracy of the answer to the user's question, on this basis, performs a second filtering process based on the relevance between the user's question and the intermediate text to obtain a second remaining text, and through the data of the second remaining text The purpose is to select at least one of the second remaining text and the preset document summary, input the at least one, the user question and the answer to the question into the large language model for question recommendation processing, and obtain recommended questions, so as to further generate recommended questions on the basis of generating the answer to the user question, so as to guide the user to conduct multiple rounds of dialogue, improve the user's initiative in question-and-answer interaction, and improve the user's retention rate. At the same time, at least one of the second remaining text and the preset document summary is selected according to the number of the second remaining texts, so as to improve the flexibility of the question recommendation processing, and the comprehensiveness and richness of the recommended questions are improved by combining the user question, the answer to the question and at least one of the second remaining text and the preset document summary for question recommendation processing.

[0007] In a second aspect, another embodiment of the present specification provides a problem processing device based on a large language model, the device comprising: A text screening module, configured to screen texts in a preset text set according to a user question to obtain an intermediate text, and to perform a first filtering process on the intermediate text based on a relevance between the user question and the intermediate text to obtain a first residual text; An answer generation module, used for inputting the first remaining text and the user question into a large language model to generate an answer, so as to obtain an answer to the user question; a filtering processing module, configured to perform a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and select at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The question recommendation module is used to input the at least one, the user question and the answer to the question into a large language model for question recommendation processing to obtain a recommended question.

[0008] In a third aspect, another embodiment of the present specification provides a problem processing device based on a large language model, the device comprising: a memory, a processor, and computer executable instructions stored on the memory and executable on the processor, wherein the computer executable instructions, when executed by the processor, implement the steps of the problem processing method based on the large language model as described in the first aspect above.

[0009] In a fourth aspect, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the problem processing method based on a large language model as described in the first aspect above are implemented.

[0010] In a fifth aspect, another embodiment of the present specification provides a computer program product, wherein the computer program product includes a call exception handling program, and the call exception handling program is executed by a processor to implement the steps of the problem handling method based on a large language model as described in the first aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 A processing flow chart of a problem processing method based on a large language model provided in one embodiment of this specification; Figure 2 A schematic diagram of an answer to a user question and a recommended question provided by an embodiment of the present specification; Figure 3 A flowchart of a problem processing method based on a large language model applied to a dialogue scenario provided by one embodiment of this specification; Figure 4 A schematic diagram of a problem processing device based on a large language model provided by one embodiment of this specification; Figure 5 A schematic diagram of the structure of a question processing device based on a large language model provided in one embodiment of this specification. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0013] An embodiment of a problem processing method based on a large language model provided in this specification is as follows: The problem processing method based on the large language model provided in this embodiment can be executed by a computer device, which can be a terminal or a server. The terminal can include a mobile phone, a laptop computer, an intelligent interactive device, etc., and the server can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0014] Reference Figure 1 The problem processing method based on the large language model provided in this embodiment specifically includes the following steps S102 to S108.

[0015] Step S102, filtering texts in a preset text set according to the user question to obtain intermediate texts, and performing a first filtering process on the intermediate texts based on the relevance between the user question and the intermediate texts to obtain first remaining texts.

[0016] The user questions described in this embodiment include questions input by the user for dialogue interaction. The user questions may be user questions input through a touch control, or user questions obtained by semantically recognizing a user voice input through a voice control. The touch control here may be a touch control for text input, or a recommended question control, that is, the user may choose to trigger one or more recommended question controls from multiple recommended question controls, and then submit the corresponding recommended questions to the dialogue system.

[0017] This embodiment can be applied to a dialogue system; the user question can be a user question in any field, such as a user question related to the field of SIM authentication (Subscriber Identity Module, using a SIM card for identity authentication), a question related to the field of call fee packages, etc. The intermediate text refers to a text obtained by text screening, and the intermediate text can be one or more texts. The relevance includes the relevance degree or relevance score between the intermediate text and the user question. The preset text set can be a text feature set or a word segmentation and text mapping table.

[0018] In practical applications, before generating answers to user questions, due to the large amount of data in knowledge documents, directly using knowledge documents to generate answers to user questions may result in a large amount of calculations and a low efficiency in answer generation. To this end, in order to improve the efficiency of answer generation, feature extraction can be performed on user questions to obtain question features, and the intermediate text corresponding to the text features that match the question features can be screened out from the text feature set; in the first optional implementation provided by this embodiment, in the process of obtaining the intermediate text by performing text screening in a preset text set according to the user question, the following operations are performed: Extract features of user questions to obtain question features, and calculate similarity between question features and each text feature in the text feature set; According to the calculated similarity, the intermediate text corresponding to the text feature is filtered out from the text feature set.

[0019] The question feature includes the question feature vector, and the similarity includes the similarity between the question feature and each text feature.

[0020] Specifically, in the process of screening out the intermediate texts corresponding to the text features in the text feature set according to the calculated similarity, the similarities between the question features and each text feature can be sorted, and the intermediate texts corresponding to the text features corresponding to the similarities whose sorting positions are before the preset positions are determined in the sorting results.

[0021] It should be noted that the text feature set may also be stored in a vector database. In this case, similarity calculation may be performed based on the question feature and each text feature in the text feature set in the vector database.

[0022] On this basis, in an optional implementation manner provided by this embodiment, the text feature set is obtained in the following manner: The knowledge document is split to obtain multiple texts, and features are extracted from the multiple texts to obtain multiple text features; A feature set is constructed according to multiple texts and multiple text features to obtain a text feature set.

[0023] The knowledge document refers to a pre-collected relevant knowledge document for question-answering scenarios; the knowledge document may be one or more. The multiple texts may be multiple document paragraphs in the knowledge document. The text feature set includes each text in the multiple texts and the corresponding text features.

[0024] In order to improve the comprehensiveness and effectiveness of the intermediate text, the above-mentioned recall of the intermediate text matched with the user question by means of text features can be performed by text segmentation in this embodiment to recall the intermediate text matched with the user question, and the intermediate text is recalled from two dimensions of features and segmentation, so as to improve the flexibility and comprehensiveness of the recalled intermediate text; in the process of obtaining the intermediate text by text screening in a preset text set according to the user question, the user question can also be segmented to obtain multiple segmentations, and the text where each segmentation is located is queried in the segmentation and text mapping table, and the matching degree between the user question and the text is calculated, and the intermediate text is screened out from the text based on the calculated matching degree; specifically, in the second optional implementation manner provided by this embodiment, in the process of obtaining the intermediate text by text screening in a preset text set according to the user question, the following operations are performed: Perform word segmentation on the user's question to obtain multiple word segments, query the text where each word segment is located in the word segmentation and text mapping table, and calculate the distribution information of each word segment for the text and the text length of the text; The matching degree between the user question and the text is calculated according to the distribution information and the text length, and the intermediate text is filtered out in the text based on the calculated matching degree.

[0025] Among them, each segmentation refers to each segmentation in multiple segmentations; the distribution information of each segmentation for the text includes the frequency of each segmentation in the text and / or the number of texts in the text. The frequency of each segmentation in the text includes the number of occurrences of each segmentation in the text. The text in which each segmentation is located can be one or more. The text length of the text can be the number of characters contained in the text.

[0026] Specifically, in the process of filtering out intermediate texts in a text based on the calculated matching degrees, the calculated matching degrees may be sorted to obtain sorting results, and texts corresponding to matching degrees whose sorting positions are before a preset position may be determined in the sorting results as intermediate texts.

[0027] It should be noted that the above second optional implementation can be replaced by inputting the user question and the word segmentation and text mapping table into the text matching algorithm for text matching processing to obtain the intermediate text matching the user question; the text matching algorithm here can be the BM25 (Best Matching 25) algorithm.

[0028] On this basis, the word segmentation and text mapping table can also be constructed in advance; in an optional implementation manner provided in this embodiment, the word segmentation and text mapping table is obtained in the following manner: The knowledge document is split to obtain multiple texts, and each text is segmented to obtain the word segments of each text; The word segmentation and text mapping table is constructed according to the texts and the word segments of the texts.

[0029] The word segmentation and text mapping table may be an inverted index table.

[0030] In addition, in the process of obtaining intermediate texts by filtering texts in a preset text set according to user questions, the intermediate texts can be recalled for user questions by combining features and word segmentation, thereby improving the effectiveness and comprehensiveness of the recalled intermediate texts. In a third optional implementation manner provided by this embodiment, in the process of obtaining intermediate texts by filtering texts in a preset text set according to user questions, feature extraction can be performed on user questions to obtain question features, a first intermediate text corresponding to a text feature matching the question feature can be queried in a text feature set, the user question can be word segmented to obtain a plurality of word segments, a second intermediate text matching the user question can be screened out from each text included in a word segmentation and text mapping table based on the plurality of word segments, and the first intermediate text and the second intermediate text can be merged to obtain an intermediate text. Specifically, the following operations can be performed: Extracting features of the user's question to obtain question features, and searching the text feature set for a first intermediate text corresponding to a text feature matching the question features; Perform word segmentation on the user question to obtain multiple word segments, determine the text where each word segment is located in the word segmentation and text mapping table, and calculate the matching degree between the user question and the text based on the distribution information of each word segmentation for the text and the text length; A second intermediate text matching the user question is selected from the text according to the matching degree, and the first intermediate text and the second intermediate text are merged to obtain an intermediate text.

[0031] The distribution information of each word segment for the text includes the frequency of each word segment in the text and / or the number of texts in the text. The frequency of each word segment in the text includes the number of occurrences of each word segment in the text. The text in which each word segment is located can be one or more. The text length of the text can be the number of characters contained in the text. The first intermediate text can be zero, one or more; the second intermediate text can be zero, one or more; the intermediate text can be one or more.

[0032] Specifically, in the process of searching for the first intermediate text corresponding to the text feature matching the question feature in the text feature set, similarity calculation can be performed based on the question feature and each text feature in the text feature set, and the first intermediate text corresponding to the text feature can be filtered out from the text feature set based on the calculated similarity; the specific implementation process of filtering out the first intermediate text corresponding to the text feature in the text feature set based on the calculated similarity is similar to the implementation process in the first optional implementation mode mentioned above, and this embodiment will not be repeated here.

[0033] In the process of selecting the second intermediate text matching the user question from the text according to the matching degree, the matching degree between the user question and the text may be sorted to obtain a sorting result, and the second intermediate text corresponding to the matching degree whose sorting position is before the preset position is determined in the sorting result. In the process of merging the first intermediate text and the second intermediate text to obtain the intermediate text, the first intermediate text and the second intermediate text may be directly merged to obtain the intermediate text, or the first intermediate text and the second intermediate text may be deduplicated and the deduplicated results may be merged to obtain the intermediate text.

[0034] For example, the first intermediate text is {Di}, the second intermediate text is {Fi}, and the intermediate text obtained by merging the first intermediate text {Di} and the second intermediate text {Fi} is {Gi}.

[0035] On this basis, in an optional implementation manner provided by this embodiment, the relevance between the intermediate text and the user question is calculated in the following manner: Reading the similarity between the text features corresponding to the user question and the first intermediate text, and reading the matching degree between the user question and the second intermediate text; The relevance of the intermediate text to the user question is calculated based on the read similarity and the read matching degree.

[0036] Specifically, in the process of calculating the relevance between the intermediate text and the user question based on the read similarity and the read matching degree, the similarity and matching degree between the intermediate text and the user question can be determined based on the read similarity and the read matching degree, and the relevance between the intermediate text and the user question can be calculated based on the similarity and matching degree between the intermediate text and the user question; specifically, the similarity between the read user question and the text features corresponding to the first intermediate text can be used as the similarity of the first intermediate text in the intermediate text, the similarity between the second intermediate text in the intermediate text and the user question can be determined to be 0, the matching degree between the read user question and the second intermediate text can be used as the matching degree of the second intermediate text in the intermediate text, the matching degree between the first intermediate text in the intermediate text and the user question can be determined to be 0, the square of the similarity between the intermediate text and the user question can be calculated, and the square of the matching degree between the intermediate text and the user question can be calculated, the sum of the square of the similarity and the square of the matching degree can be calculated, and then the square root calculation of the sum can be performed to obtain the relevance between the intermediate text and the user question.

[0037] In addition, the similarity between the text features corresponding to the user question and the first intermediate text, as well as the matching degree between the user question and the second intermediate text can be input into the relevance algorithm for correlation calculation to obtain the relevance between the first intermediate text and the second intermediate text in the intermediate text and the user question.

[0038] For example, relevance algorithms include:

[0039] in, Represents the relevance of the i-th intermediate text to the user's question; Represents the similarity between the text features of the user question and the intermediate text i; Represents the matching degree between the user question and the intermediate text i.

[0040] During the specific execution process, after obtaining the intermediate text by filtering the text in the preset text set according to the user question, in order to further improve the effectiveness of the text for subsequent answer generation, thereby improving the efficiency of answer generation and reducing the time cost of answer generation, the intermediate text can be first filtered based on the relevance of the user question and the intermediate text to obtain the first remaining text; specifically, based on the relevance of the user question and the intermediate text, the intermediate text with a relevance less than a first relevance threshold can be deleted from the intermediate text to obtain the first remaining text; the first remaining text here can be zero, one or more, for example, the first relevance threshold is R1, and the intermediate text with a relevance less than R1 can be deleted from the intermediate text {Gi} according to the relevance of the user question and the intermediate text to obtain the first remaining text H1.

[0041] Optionally, if the number of the first remaining texts is less than or equal to the number threshold, a default answer to the question is output, and specifically, the default answer to the question is output to the user; the number threshold here can be 0 or other values; for example, if the number of the first remaining texts is less than or equal to the number threshold, the default answer to the question "Sorry, no relevant answer to your question was found" is sent to the user; if the number of the first remaining texts is greater than the number threshold, the following step S104 can be executed.

[0042] Step S104: input the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question.

[0043] The above-mentioned method obtains the intermediate text by filtering the text in the preset text set according to the user question, and performs a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain the first remaining text. In this step, the first remaining text and the user question are input into the large language model to generate the answer to obtain the answer to the user question.

[0044] In a specific implementation, in the process of inputting the first remaining text and the user question into a large language model to generate an answer and obtaining the answer to the user question, a prompt can be constructed based on the first remaining text and the user question, and the first prompt information obtained by the construction is input into the large language model to generate an answer and obtain the answer to the user question; in the process of constructing a prompt based on the first remaining text and the user question, the first remaining text and the user question can be written into a first prompt template to obtain the first prompt information.

[0045] For example, the first prompt template is: Known information: "{context}"; Answer the user's question concisely and professionally based on the known information above. If you cannot get an answer from it, please say "I cannot answer this question based on the known information" or "Not enough relevant information is provided". Fabrication is not allowed in the answer. Please use Chinese for the answer. The user's question is: "{question}".

[0046] The first remaining text may be written into the context in the first prompt template, the user question may be written into the question in the first prompt template to obtain first prompt information, and the first prompt information may be input into the large language model to generate an answer to obtain an answer to the user question.

[0047] Step S106: performing a second filtering process on the intermediate text based on the relevance to obtain second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text.

[0048] The preset document summary described in this embodiment refers to a pre-generated document summary; optionally, the preset document summary is obtained after the knowledge document is input into a large language model for summary generation.

[0049] During the specific implementation process, in order to improve the efficiency of generating the preset document summary, a large language model can be introduced. Specifically, a prompt can be constructed based on the knowledge document to obtain the second prompt information, and the second prompt information can be input into the large language model for summary generation to obtain the document summary of the knowledge document as the preset document summary. Specifically, in the case where the knowledge document is a plurality of documents, the preset document summary can also be the document summary of each document in the plurality of documents; the large language model for summarizing and the large language model for answer generation can be a large language model of the same structure or a large language model of different structures.

[0050] Specifically, in the process of constructing a prompt based on a knowledge document to obtain the second prompt information, the knowledge document may be written into a second prompt template to obtain the second prompt information; for example, the second prompt template is: You are a text content summary generator. Please generate a summary of no more than xx words based on the known content.

[0051] Known content is: "{content}".

[0052] The knowledge document may be written into the content in the second prompt template to obtain the second prompt information.

[0053] After generating answers to user questions, in order to improve the interaction rate with users and improve user retention rate, recommended questions can continue to be generated for multiple rounds of conversations with users; in the process of performing a second filtering process on the intermediate text based on the relevance to obtain the second remaining text, based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than a second relevance threshold can be deleted from the intermediate text to obtain the second remaining text; optionally, the second relevance threshold is less than or equal to the first relevance threshold; for example, the second relevance threshold is R2, and based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than the second relevance threshold R2 is deleted from the intermediate text {Gi} to obtain the second remaining text H2; the second remaining text can be zero, one or more texts.

[0054] In order to improve the flexibility and richness of the subsequent question recommendation processing, while ensuring the validity of the information subsequently input into the large language model, the amount of information subsequently input into the large language model is reduced, that is, while ensuring the validity of the recommended questions generated by the large language model, the amount of calculation in the question recommendation processing of the large language model is also reduced; in an optional implementation manner provided by this embodiment, in the process of selecting at least one of the second remaining text and the preset document summary according to the number of the second remaining text, the following operations are performed: If the number is greater than a preset number threshold, selecting the second remaining text from the second remaining text and the preset document summary; If the number is less than or equal to a preset number threshold, the second remaining text and the preset document summary are taken as at least one.

[0055] The preset number threshold may be 3, 4, 5 or other number thresholds.

[0056] Step S108: input the at least one, the user question and the answer to the question into a large language model for question recommendation processing to obtain a recommended question.

[0057] The recommended questions described in this embodiment refer to questions to be recommended, that is, questions recommended to users.

[0058] In specific implementation, prompts can be constructed based on at least one of the user's questions and answers to obtain prompt information, and the prompt information can be input into the large language model for question recommendation processing to obtain recommended questions. After obtaining the recommended questions, the answers to the user's questions and the recommended questions can be output. Specifically, the answers to the user's questions and the recommended questions can be sent to the user, so that they can be displayed in the dialogue service running on the user's terminal device; the user can further select the target recommended question from the recommended questions to submit or re-enter the user question to submit, so as to conduct the next round of dialogue, until the keyword entered by the user is detected to terminate the dialogue, and the keyword can be a keyword of the dialogue terminal, such as goodbye.

[0059] For example, Figure 2 The answers to the user questions and recommended questions shown are used to generate answers for the user question "What is SIM authentication?", and the answers to the questions are "SIM authentication is a product developed based on the security authentication and proactive messaging capabilities of the SIM card quick confirmation card application. It can realize one-click PC login of the IT system using a mobile phone number as an account, or business approval or authorization functions", as well as recommended questions "What are the significant advantages of SIM authentication products?", "What is a SIM card application?", "Does SIM authentication login require activation or payment?".

[0060] In actual applications, in the process of performing question recommendation processing through user questions and question answers, since both the user questions and the question answers are from the current round, the richness of the recommended questions may be insufficient. In view of this, in order to improve the richness and comprehensiveness of the recommended questions, an optional implementation method provided in this embodiment is to input at least one of the user questions and the question answers into a large language model for question recommendation processing, and in the process of obtaining the recommended questions, perform the following operations: Constructing a prompt based on at least one of the user question, the answer to the question, and historical conversation information to obtain prompt information; The prompt information is input into the large language model for question recommendation processing to obtain recommended questions.

[0061] The historical conversation information may be historical conversation information between a user and a conversation service; the historical conversation information includes historical user questions and / or historical question answers; and the historical user questions and historical question answers may be one or more.

[0062] Specifically, when at least one is the second remaining text, the second remaining text, user questions, answers to questions, and historical conversation information can be written into a third prompt template to obtain prompt information; when at least one is the second remaining text and a preset document summary, the second remaining text, user questions, answers to questions, historical conversation information, and a preset document summary can be written into a fourth prompt template to obtain prompt information.

[0063] For example, the third prompt template is: The historical conversation information between you and the user is "{history}" Known information: "{content}" User question is "{question}" The answer to the question is "{answer}" Please provide no more than x questions that users may ask based on the known information and the answers you give to them. The questions you provide can be answered from the known information and cannot be similar to the user's questions. Additional explanation: Please reply in the format of "Users may ask further questions as follows: "n1.xxx\n2.xxx", with one line representing one question. If you cannot find the question from the known information, reply "No question".

[0064] The second remaining text H2 can be written into the content in the third prompt template, the user question can be written into the question in the third prompt template, the answer to the question can be written into the answer in the third prompt template, and the historical dialogue information can be written into the history in the third prompt template to obtain the prompt information.

[0065] For another example, the fourth prompt template is: The historical conversation information between you and the user is "{history}" Known information: "{content}" The default document summary is: "{summary}" User question is "{question}" The answer to the question is "{answer}" Please provide no more than x questions that users may ask based on the known information and the answers you give to them. The questions you provide can be answered from the known information or the preset document summary, and the questions you provide cannot be similar to the user's questions. Additional explanation: Please reply in the format of "Users may ask further questions as follows: "n1.xxx\n2.xxx", with one line representing one question. If you cannot find the question from the known information, reply "No question".

[0066] The second remaining text H2 can be written into the content in the fourth prompt template, the user question can be written into the question in the fourth prompt template, the answer to the question can be written into the answer in the fourth prompt template, the historical conversation information can be written into the history in the fourth prompt template, and the preset document summary can be written into the summary in the fourth prompt template to obtain the prompt information.

[0067] It should be supplemented that each optional implementation method or each feasible execution method in steps S102 to S108 provided in this embodiment can be executed independently as needed, or can be combined and referenced with each other. At the same time, each specific execution step in each optional implementation method or each feasible execution method can also be executed independently, and this embodiment does not make specific limitations.

[0068] In summary, the question processing method based on the large language model provided in this embodiment firstly extracts the features of the user question to obtain the question features, searches the text feature set for the first intermediate text corresponding to the text features matching the question features, performs word segmentation processing on the user question to obtain multiple word segments, and determines the text where each word segment is located in the word segmentation and text mapping table, calculates the matching degree between the user question and the text based on the distribution information of each word segmentation for the text and the text length, selects the second intermediate text matching the user question from the text according to the matching degree, and merges the first intermediate text and the second intermediate text to obtain the intermediate text; thereby, the text is recalled from the two dimensions of question features and word segmentation to obtain the intermediate text with higher effectiveness; Secondly, based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than a first relevance threshold is deleted from the intermediate text to obtain the first remaining text, the first remaining text and the user question are input into the large language model for answer generation, and the answer to the user question is obtained; the accuracy of the answer to the question is improved by the first remaining text with higher effectiveness; on this basis, based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than a second relevance threshold is deleted from the intermediate text to obtain the second remaining text, and at least one of the second remaining text and the preset document summary is selected according to the number of the second remaining text, and the at least one, the user question, the answer to the question and the historical dialogue information are input into the large language model for question recommendation processing to obtain the recommended question, so as to further generate the recommended question on the basis of generating the answer to the user question, so as to guide the user to conduct multiple rounds of dialogue and enhance the user's initiative in question-answering interaction. At the same time, at least one of the second remaining text and the preset document summary is selected according to the number of the second remaining text, so as to enhance the flexibility of the question recommendation processing, and the comprehensiveness and richness of the recommended question are enhanced by combining the user question, the answer to the question and at least one of the second remaining text and the preset document summary for question recommendation processing.

[0069] The following combination Figure 3 , taking the application of the problem processing method based on the large language model provided in this embodiment in a dialogue scenario as an example, the problem processing method based on the large language model provided in this embodiment is further explained. Figure 3 ,The problem processing method based on a large language model applied to dialogue scenarios specifically includes the following steps.

[0070] Step S302, obtaining a user question input by the user in the dialogue service, and performing text screening in a preset text set according to the user question to obtain an intermediate text.

[0071] A dialog service can be a dialog application or a dialog subprogram (dialog applet).

[0072] Step S304: based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than a first relevance threshold is deleted from the intermediate text to obtain a first remaining text.

[0073] Step S306: Input the first remaining text and the user question into the large language model to generate an answer, thereby obtaining an answer to the user question.

[0074] Step S308: based on the relevance between the user question and the intermediate text, the intermediate text with a relevance less than a second relevance threshold is deleted from the intermediate text to obtain a second remaining text.

[0075] Optionally, the second correlation threshold is less than or equal to the first correlation threshold.

[0076] Step S310: If the number of the second remaining texts is greater than a preset number threshold, the second remaining texts, user questions, answers to the questions and historical dialogue information are input into the large language model for question recommendation processing to obtain recommended questions.

[0077] Step S312: If the number of the second remaining texts is less than or equal to a preset number threshold, the second remaining texts, the preset document summary, the user question, the answer to the question and the historical dialogue information are input into the large language model for question recommendation processing to obtain a recommended question.

[0078] Step S314: output the answer to the user's question and the recommended question to the user.

[0079] It should be noted that any one of steps S302 to S314 or any combination of multiple steps can be selected from steps S102 to S108 to form a new implementation method according to the needs of implementation deployment; and any one or multiple technical features in the technical scheme composed of steps S302 to S314 can be selected from steps S102 to S108 to form a new implementation method according to the needs of actual deployment, or the technical features in one or more optional implementation methods provided by steps S102 to S108 can be selected to combine into a new implementation method, which will not be elaborated here one by one.

[0080] Figure 4 A schematic diagram of a problem processing device based on a large language model provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device comprises: A text screening module 402 is used to screen texts in a preset text set according to the user question to obtain an intermediate text, and to perform a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first residual text; An answer generation module 404 is used to input the first remaining text and the user question into a large language model to generate an answer, so as to obtain an answer to the user question; A filtering processing module 406 is used to perform a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and select at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The question recommendation module 408 is used to input the at least one, the user question and the answer to the question into a large language model for question recommendation processing to obtain a recommended question.

[0081] The question processing device based on the large language model provided in this embodiment first performs text screening in a preset text set according to a user question to obtain an intermediate text by running a text screening module 402, and performs a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text, and then runs an answer generation module 404 to input the first remaining text and the user question into the large language model for answer generation to obtain an answer to the user question, and then runs a filtering processing module 406 to perform a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selects at least one of the second remaining text and a preset document summary according to the number of the second remaining texts, and finally runs a question recommendation module 408 to input at least one, the user question and the answer to the question into the large language model for question recommendation processing to obtain a recommended question, so as to improve the accuracy of the answer to the user question and improve the user's initiative in question-answering interaction.

[0082] The problem processing device based on a large language model provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0083] Furthermore, an embodiment of the present specification also provides a problem processing device based on a large language model, Figure 5 A schematic diagram of a problem processing device based on a large language model provided in an embodiment of this specification is shown in FIG. Figure 5As shown, the device includes: a memory 501, a processor 502, a bus 503 and a communication interface 504. The memory 501, the processor 502 and the communication interface 504 communicate through the bus 503, and the communication interface 504 may include an input and output interface, which includes but is not limited to a keyboard, a mouse, a display, a microphone, a loudspeaker, etc.

[0084] Figure 5 In the embodiment, the memory 501 stores computer executable instructions that can be run on the processor 502. When the computer executable instructions are executed by the processor 502, the following process is implemented: Screening a preset text set according to the user question to obtain an intermediate text, and performing a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text; Inputting the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question; Performing a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The at least one, the user question and the answer to the question are input into a large language model for question recommendation processing to obtain a recommended question.

[0085] The large language model-based question processing device provided in this embodiment, through the cooperation of the memory 501, the processor 502, the bus 503 and the communication interface 504, performs text screening in a preset text set according to the user question to obtain an intermediate text, performs a first filtering process on the intermediate text according to the relevance between the user question and the intermediate text to obtain a first remaining text, inputs the first remaining text and the user question into the large language model to generate an answer to obtain the answer to the user question, performs a second filtering process based on the relevance between the user question and the intermediate text to obtain a second remaining text, selects at least one of the second remaining text and the preset document summary according to the number of the second remaining text, inputs the at least one, the user question and the answer to the question into the large language model for question recommendation processing to obtain a recommended question, so as to improve the accuracy of the answer to the user question and improve the user's initiative in question-answering interaction.

[0086] The problem processing device based on a large language model provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0087] Furthermore, another embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented: Screening a preset text set according to the user question to obtain an intermediate text, and performing a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text; Inputting the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question; Performing a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The at least one, the user question and the answer to the question are input into a large language model for question recommendation processing to obtain a recommended question.

[0088] The computer-readable storage medium provided in this embodiment performs text screening in a preset text set according to a user question to obtain an intermediate text, performs a first filtering process on the intermediate text according to the relevance between the user question and the intermediate text to obtain a first remaining text, inputs the first remaining text and the user question into a large language model to generate an answer to obtain an answer to the user question, performs a second filtering process based on the relevance between the user question and the intermediate text to obtain a second remaining text, selects at least one of the second remaining text and a preset document summary according to the number of the second remaining texts, inputs the at least one, the user question, and the answer to the question into a large language model for question recommendation processing to obtain a recommended question, so as to improve the accuracy of the answer to the user question and enhance the user's initiative in question-answering interaction.

[0089] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0090] A computer-readable storage medium provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0091] Furthermore, another embodiment of the present specification also provides a computer program product, which implements the following process when executed by a processor: Screening a preset text set according to the user question to obtain an intermediate text, and performing a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text; Inputting the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question; Performing a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The at least one, the user question and the answer to the question are input into a large language model for question recommendation processing to obtain a recommended question.

[0092] The computer program product provided in this embodiment performs text screening in a preset text set according to a user question to obtain an intermediate text, performs a first filtering process on the intermediate text according to the relevance of the user question to the intermediate text to obtain a first remaining text, inputs the first remaining text and the user question into a large language model to generate an answer to obtain an answer to the user question, performs a second filtering process based on the relevance of the user question to the intermediate text to obtain a second remaining text, selects at least one of the second remaining text and a preset document summary according to the number of the second remaining texts, inputs the at least one, the user question and the answer to the question into a large language model for question recommendation processing to obtain a recommended question, so as to improve the accuracy of the answer to the user question and enhance the user's initiative in question-answering interaction.

[0093] A computer program product provided in an embodiment of this specification can implement each process in the aforementioned method embodiment and achieve the same functions and effects, which will not be repeated here.

[0094] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0095] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] The memory may include non-permanent storage in a computer-readable storage medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable storage medium.

[0100] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0103] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A problem processing method based on a large language model, characterized in that: The method comprises: Screening a preset text set according to the user question to obtain an intermediate text, and performing a first filtering process on the intermediate text based on the relevance between the user question and the intermediate text to obtain a first remaining text; Inputting the first remaining text and the user question into a large language model to generate an answer, thereby obtaining an answer to the user question; Performing a second filtering process on the intermediate text based on the relevance to obtain a second remaining text, and selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text; The at least one, the user question and the answer to the question are input into a large language model for question recommendation processing to obtain a recommended question.

2. The problem processing method based on a large language model according to claim 1, characterized in that: The step of filtering texts in a preset text set according to the user's question to obtain intermediate texts includes: Extracting features of the user question to obtain question features, and calculating similarity between the question features and each text feature in the text feature set; The intermediate text corresponding to the text feature is selected from the text feature set according to the calculated similarity.

3. The problem processing method based on a large language model according to claim 2 is characterized in that: The text feature set is obtained in the following way: Splitting the knowledge document to obtain multiple texts, and extracting features from the multiple texts to obtain multiple text features; A feature set is constructed according to the multiple texts and the multiple text features to obtain the text feature set.

4. The problem processing method based on a large language model according to claim 1, characterized in that: The step of filtering texts in a preset text set according to the user's question to obtain intermediate texts includes: Perform word segmentation processing on the user question to obtain multiple word segments, query the text where each word segment is located in the word segmentation and text mapping table, and calculate the distribution information of each word segment for the text and the text length of the text; The matching degree between the user question and the text is calculated according to the distribution information and the text length, and the intermediate text is screened out from the text based on the calculated matching degree.

5. The problem processing method based on a large language model according to claim 4 is characterized in that: The word segmentation and text mapping table is obtained in the following way: The knowledge document is split to obtain multiple texts, and each text is segmented to obtain the word segments of each text; The word segmentation and text mapping table is constructed according to the texts and the word segments of the texts.

6. The problem processing method based on a large language model according to claim 1, characterized in that: The step of filtering texts in a preset text set according to the user's question to obtain intermediate texts includes: Extracting features of the user question to obtain question features, and searching a text feature set for a first intermediate text corresponding to a text feature matching the question features; Performing word segmentation processing on the user question to obtain multiple word segments, determining the text where each word segment is located in a word segmentation and text mapping table, and calculating the matching degree between the user question and the text based on the distribution information of each word segmentation for the text and the text length; A second intermediate text matching the user question is selected from the text according to the matching degree, and the first intermediate text and the second intermediate text are merged to obtain the intermediate text.

7. The problem processing method based on a large language model according to claim 6, characterized in that: The correlation is calculated in the following way: Reading the similarity between the text features corresponding to the user question and the first intermediate text, and reading the matching degree between the user question and the second intermediate text; The relevance between the intermediate text and the user question is calculated according to the read similarity and the read matching degree.

8. The problem processing method based on a large language model according to claim 1, characterized in that: If the number of the first remaining texts is less than or equal to the number threshold, outputting a default answer to the question; The preset document summary is obtained after the knowledge document is input into the large language model for summary generation.

9. The problem processing method based on a large language model according to claim 1, characterized in that: The selecting at least one of the second remaining text and a preset document summary according to the amount of the second remaining text comprises: If the number is greater than a preset number threshold, selecting the second remaining text from the second remaining text and the preset document summary; If the number is less than or equal to the preset number threshold, the second remaining text and the preset document summary are used as the at least one.

10. The problem processing method based on a large language model according to claim 1, characterized in that: The step of inputting the at least one, the user question and the answer to the question into a large language model for question recommendation processing to obtain a recommended question includes: Constructing a prompt according to the at least one, the user question, the answer to the question and the historical conversation information to obtain prompt information; The prompt information is input into a large language model for question recommendation processing to obtain the recommended question.

Images