Question and answer method and device based on large model, electronic equipment and medium

Abstract

The invention provides a question and answer method and device based on a large model, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, intelligent search, dialogue systems and the like. According to the specific implementation scheme, a to-be-processed problem is obtained; disassembling the problem to be processed to obtain a plurality of sub-problems and a logic sequence among the plurality of sub-problems; for each sub-question, retrieving a candidate result matched with the sub-question from the candidate results to serve as a retrieval result corresponding to the sub-question; inputting the to-be-processed question and a first preset cue word into a first large language model, so that the first large language model generates an answer of the to-be-processed question; wherein the first preset cue word comprises the plurality of sub-questions, the retrieval results corresponding to the plurality of sub-questions respectively and the logic sequence among the plurality of sub-questions. Therefore, through the scheme, the answering efficiency and the answering effect can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning, intelligent search, and dialogue systems. Background Art

[0002] As data security needs continue to increase, industrial enterprises need to regularly conduct data security risk assessments, risk detection, and compliance audits.

[0003] Currently, when faced with regulatory responses to tasks such as data security risk assessments, a manual response method is usually adopted in which compliance experts and business experts collaborate. Summary of the invention

[0004] The present disclosure provides a question-answering method, device, electronic device and medium based on a large model.

[0005] A first aspect of the embodiments of the present disclosure provides a question-answering method based on a large model, comprising:

[0006] Get pending issues;

[0007] Decomposing the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems;

[0008] For each sub-question, a candidate result matching the sub-question is retrieved from each candidate result as the retrieval result corresponding to the sub-question;

[0009] The question to be processed and a first preset prompt word are input into a first large language model so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes the multiple sub-questions, the retrieval results corresponding to the multiple sub-questions respectively, and the logical ordering between the multiple sub-questions.

[0010] A second aspect of the embodiments of the present disclosure provides a question-answering device based on a large model, including:

[0011] The acquisition module is used to obtain pending issues;

[0012] A disassembly module, used for disassembling the problem to be processed to obtain multiple sub-problems and a logical order between the multiple sub-problems;

[0013] A retrieval module is used to retrieve, for each sub-question, a candidate result matching the sub-question from each candidate result as a retrieval result corresponding to the sub-question;

[0014] A generation module is used to input the question to be processed and a first preset prompt word into a first large language model so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes the multiple sub-questions, the search results corresponding to the multiple sub-questions respectively, and the logical ordering between the multiple sub-questions.

[0015] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model-based question-answering method described in the first aspect.

[0019] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the large model-based question-answering method according to the first aspect.

[0020] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the large model-based question-answering method according to the first aspect.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flow chart of a question-answering method based on a large model provided in an embodiment of the present disclosure;

[0024] Figure 2 It is a schematic diagram of answer logic of an intelligent response provided by an embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram of a specific example of an intelligent response provided by an embodiment of the present disclosure;

[0026] Figure 4 is a flow chart of another large model-based question-answering method provided by an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of a data flywheel provided by an embodiment of the present disclosure;

[0028] Figure 6 is a structural diagram of a question-answering device based on a large model provided in an embodiment of the present disclosure;

[0029] Figure 7 The invention is a block diagram of an electronic device used to implement the large model-based question-answering method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] With the promulgation and implementation of laws and regulations related to data security, industrial enterprises need to regularly carry out data security risk assessment, risk detection and compliance audits. Currently, enterprises face many difficulties and challenges in implementing data security compliance management:

[0032] (1) There is a shortage of professional talents and the evaluation is difficult: the evaluation work requires a group of professional talents who understand both policies and business, which makes it difficult for enterprises to carry out the evaluation work.

[0033] (2) A large number of people are required and the evaluation cost is high: Business changes rapidly and systems are updated frequently, which requires a large amount of manpower to complete the evaluation task, resulting in high evaluation costs.

[0034] (3) Knowledge is difficult to reuse and assessment efficiency is low: There are many types of risk assessments, and experiential knowledge such as internal and external regulations, evidence archiving, and risk management is difficult to reuse, resulting in low assessment efficiency.

[0035] At present, when faced with regulatory responses to tasks such as data security risk assessments, a manual response method is usually adopted in which compliance experts and business experts collaborate. This requires a lot of manpower costs and relies on manual experience, resulting in low response efficiency.

[0036] In order to solve the above problems, the embodiment of the present disclosure provides a question-answering method based on a large model, which is applied to a client with an intelligent answering function in an electronic device. For example, the electronic device can be a server, a desktop computer, or a laptop computer, etc., which has data processing capabilities. The present disclosure does not specifically limit the device form of the electronic device. Figure 1As shown, the question-answering method based on a large model provided in the embodiment of the present disclosure includes steps S101-S104:

[0037] S101, obtaining pending issues;

[0038] It is understandable that in a compliance response scenario, the pending issues may be compliance check items regarding data security; in other business scenarios, the pending issues may be issues related to the business scenario. For example, the other business scenario may be an information query scenario in a predetermined field, and the pending issues may be issues for indicating information query. The embodiments of the present disclosure do not limit specific scenarios and pending issues. In addition, the method of obtaining the pending issues may include: receiving the pending issues input by the user through the interactive interface to obtain the pending issues; or, calling the audio acquisition module to collect voice information sent by the user to indicate the pending issues, and so on.

[0039] S102, breaking down the problem to be processed into multiple sub-problems and obtaining a logical order between the multiple sub-problems;

[0040] In this embodiment, in order to accurately answer the problem to be processed, the problem to be processed can be decomposed to obtain multiple sub-problems and the logical order between the multiple sub-problems. Among them, the logical order between the sub-problems represents the order relationship between the sub-problems. It can be understood that in order to generate an accurate answer, when decomposing, it is necessary to have a deep understanding of the problem to be processed to determine the thinking steps for processing the problem to be processed, and then decompose the problem to be processed into multiple sub-problems that can be processed independently according to the thinking steps.

[0041] For example, if the problem to be solved is "formulate a data strategy that can reflect the business development needs of the entire organization", the thinking steps for solving this problem can be determined as follows: first clarify what data strategy-related documents the company has, then determine what the company's data strategy is, then determine what specific organizational business plans the company has, and finally understand how the company's data strategy and organizational business are combined. After breaking down the problem to be solved according to this thinking step, we can get "1. What documents related to data strategy has the company issued? 2. What is the company's data strategy? 3. What specific organizational business plans does the company have? 4. How are the company's data strategy and organizational business combined?". The serial number corresponding to each sub-problem indicates the logical order between the sub-problems, that is, "What documents related to data strategy has the company issued?" is the sub-problem with the first logical order, and "How are the company's data strategy and organizational business combined?" is the sub-problem with the last logical order.

[0042] Optionally, in an implementation, decomposing the problem to be processed to obtain multiple sub-problems and the logical ordering between the multiple sub-problems may include:

[0043] The problem to be processed and the second preset prompt word are input into the second largest language model, so that the second largest language model decomposes the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems; wherein the second preset prompt word includes a problem decomposition rule and at least one problem decomposition example that conforms to the problem decomposition rule.

[0044] It is understandable that the second largest language model and the first largest language model mentioned below are large language models that are distinguished only by naming. The second largest language model and the first largest language model can be the same model or different models. The embodiments of the present disclosure do not limit the specific types of the first largest language model and the second largest language model. Exemplarily, the second largest language model can be Wenxinyiyan, ChatGPT (Chat Generative Pre-trained Transformer), and the like.

[0045] In this embodiment, the problem decomposition rule is a rule for decomposing the problem to be processed into multiple logically continuous sub-problems, so that the answers to the multiple sub-problems obtained by decomposition can be summarized as the answer to the original question. Exemplarily, in one way, the specific content of the problem decomposition rule may include "given a problem, decompose it according to the thought chain and output the result in 'json' (a lightweight data exchange format), wherein the thought chain decomposition includes: given a problem, split the problem into multiple logically continuous sub-problems, so that the answers to these sub-problems can be summarized as the answer to the original question, and the number of split sub-problems shall not exceed 5, and the description of the split sub-problems needs to be concise and direct, and shall not exceed 30 words." Exemplarily, in another way, the specific content of the problem decomposition rule may include "given a problem, split the problem into multiple logically continuous sub-problems, so that the answers to these sub-problems can be summarized as the answer to the original question, and the split sub-problems need to use concise language and must not be vague."

[0046] It can be understood that since the large language model has a strong contextual learning capability, that is, it can learn from the examples contained in the prompt, therefore, by adding at least one problem decomposition example to the second preset prompt, the second largest language model can learn the logic of decomposing the problem from the problem decomposition example, so that it can decompose the problem to be processed according to the problem decomposition example, and the second largest language model can output the result of decomposing the problem to be processed according to the output format of the problem decomposition example.

[0047] For example, the specific content of a problem decomposition example may include "Question: Provide resource assurance according to the data strategy formulated by the organization; Output: ["What documents related to data strategy resource assurance has the company issued?", "What resource assurance has the company provided for the implementation of the data strategy?", "What funds, personnel, equipment, and technology does the resource assurance include?"]". The positional relationship of each sub-question in the output forms a logical ordering between the sub-questions, that is, in the output, the sub-question with the first logical ordering is "What documents related to data strategy resource assurance has the company issued?", and the sub-question with the last logical ordering is "What funds, personnel, equipment, and technology does the resource assurance include?".

[0048] Since the second preset prompt word includes a problem decomposition rule and at least one problem decomposition example that conforms to the problem decomposition rule, after the problem to be processed and the second preset prompt word are input into the second largest language model, the second largest language model can decompose the problem to be processed into multiple sub-problems with a logical chain (i.e., having a logical relationship) and MECE (Mutually Exclusive Collectively Exhaustive, mutually independent and completely exhaustive) by learning the at least one problem decomposition example, and obtain the multiple sub-problems and the logical ordering between the multiple sub-problems.

[0049] It should be noted that the embodiments of the present disclosure do not limit the method of decomposing the problem to be processed. For example, in another implementation method, keywords in the problem to be processed can be extracted, and then different sub-problems can be generated by reasoning based on the keywords using a large language model. This is all reasonable.

[0050] S103, for each sub-question, searching for a candidate result matching the sub-question from each candidate result as a search result corresponding to the sub-question;

[0051] In this embodiment, each candidate result can be a reference content for the question-and-answer result under the business field to which the problem to be processed belongs. For example, if the business field to which the problem to be processed belongs is the data security field, each candidate result can be a text block from each document in a database such as a legal database or an enterprise system database related to data security, or a semantic label from each image in the above database, etc.

[0052] Exemplarily, each document may be a PDF (Portable Document Format), a Word document, etc., and each picture may be a JPG (Joint Photographic Experts Group) picture, a PNG (Portable Network Graphics) picture, a BMP (Bitmap, standard image file) picture, etc. The embodiments of the present disclosure do not limit the types of documents and pictures.

[0053] Exemplarily, for each sub-question, sparse search, dense search, mixed search, image-text search and other search methods can be used to search for candidate results matching the sub-question from each candidate result as the search result corresponding to the sub-question. Among them, the candidate results matching the sub-question in each candidate result can be the first several search results that are closest to the semantics of the sub-question, or the first several search results that hit the most word segments included in the sub-question, which are all reasonable.

[0054] Optionally, in one implementation, each candidate result includes: each text block in each candidate document in the business field to which the problem to be processed belongs, and / or, a semantic label of each candidate image in the business field to which the problem to be processed belongs;

[0055] In this implementation, the semantic label of each candidate image can be obtained by manually marking according to the image content in advance, or generated by semantically understanding the image content of the candidate image using a deep learning model, which is reasonable. Each text block can be a block obtained by dividing the content in the document into paragraphs, chapters, etc.

[0056] Exemplarily, for each sub-question, a candidate result matching the sub-question is retrieved from each candidate result, and the retrieval result corresponding to the sub-question may include:

[0057] For each sub-question, according to the first search method and / or the second search method, a candidate result matching the sub-question is retrieved from each candidate result as a search result corresponding to the sub-question;

[0058] The first search method includes: vectorizing the sub-question to obtain a search vector; using the search vector to search in a pre-built vector library to obtain a first preset number of candidate vectors with the highest similarity to the search vector; wherein the vector library includes candidate vectors obtained by vectorizing multiple candidate results respectively;

[0059] The second search method includes: dividing the sub-question to obtain multiple participles; using the multiple participles to search from each candidate result to obtain a correlation score between each participle and each candidate result; for each candidate result, calculating the sum of the correlation scores between the candidate result and each participle as the sum value corresponding to the candidate result; selecting the first second preset number of candidate results with the highest corresponding sum value among the candidate results.

[0060] In the first retrieval method, the encoder of a pre-trained deep learning model, such as the encoder of a model such as BERT (Bidirectional Encoder Representations from Transformers) or T5 (Text-to-Text Transfer Transformer), can be used to encode the sub-questions to obtain retrieval vectors, and the above encoder can be used to encode each candidate result in the vector library to obtain a candidate vector. It should be noted that the embodiment of the present disclosure does not limit the method for vectorizing each sub-question and candidate result, and any encoding method that can encode text into a dense vector can be used in this solution.

[0061] It is understandable that after obtaining the search vector, when using the search vector to search in the pre-built vector library, the cosine similarity between each candidate vector in the vector library and the search vector can be calculated to obtain the similarity between each candidate vector and the search vector. In one implementation, if only the first search method is used to search from each candidate result, the candidate results corresponding to the first preset number of candidate vectors with the highest similarity can be used as the search results corresponding to the sub-question. Exemplarily, the first preset number can be 100, 200, and so on.

[0062] It is understandable that by vectorizing the sub-problem to obtain the retrieval vector, the sub-problem can be converted into a continuous vector representation in a high-dimensional space, and the retrieval vector can be used to search in the pre-built vector library, and the similarity between the vectors can be used to retrieve the candidate results related to the sub-problem. This method can capture the deep semantic information of the sub-problem, so as to find the candidate results that are close to the semantics of the sub-problem from each candidate result.

[0063] In the second retrieval method, for each sub-question, word segmentation algorithms such as jieba word segmentation can be used to segment the sub-question to obtain multiple segmented words. It can be understood that in practical applications, the obtained multiple segmented words can be the segmented words after removing the segmented words without semantics such as "de" (的) and "le" (了). Use the multiple segmented words to retrieve from each candidate result, that is, count the word frequencies of the segmented words that appear in each candidate result, and determine the correlation score between the segmented word and each candidate result according to the word frequency of each segmented word in each candidate result. Among them, the correlation score between any segmented word and a candidate result is positively correlated with the word frequency of the segmented word in the candidate result.

[0064] After obtaining the correlation score between each segmented word and each candidate result, the sum of the correlation scores between each candidate result and each segmented word can be counted as the sum value corresponding to the candidate result. Then, select the first second preset number of candidate results with the highest corresponding sum values among each candidate result. In one implementation, if only the second retrieval method is used to retrieve from each candidate result, the first second preset number of candidate results with the highest sum values corresponding to each candidate result can be used as the retrieval result corresponding to the sub-question. Exemplarily, the second preset number can be 100, 200, etc. It can be understood that in practical applications, the first preset number and the second preset number can be set by relevant technical personnel according to requirements, and the embodiments of the present disclosure do not limit this.

[0065] It can be understood that by segmenting the sub-question to obtain multiple segmented words and using the multiple segmented words to retrieve from each candidate result, it is possible to retrieve candidate results related to the sub-question through the word frequencies of the segmented words, that is, it is possible to perform an exact query using the keywords included in the sub-question, so that candidate results that can match the most keywords in the sub-question can be quickly retrieved from each candidate result, and the retrieval efficiency is high.

[0066] Exemplarily, in practical applications, algorithms such as TF-IDF (term frequency–inverse document frequency) algorithm and BM25 (Best Match 25) algorithm can be used to implement the above second retrieval method.

[0067] In another implementation, a mixed search can be performed from each candidate result in combination with the first search method and the second search method. At this time, the candidate results retrieved by the two search methods can be deduplicated to obtain the final search result. It can be understood that when searching according to the first search method, each candidate result corresponds to a similarity, and when searching according to the second search method, each candidate result corresponds to a sum value. In this method, the weights corresponding to the first search method and the second search method can be set respectively, and then the similarity and sum value corresponding to each candidate result can be weighted and summed according to the set weights to obtain the weighted sum value corresponding to each candidate result, and the first third preset number of candidate results can be determined as the search result according to the sorting of the weighted sum values ​​corresponding to each candidate result. Among them, the third preset number can be 100, 200, and so on. Hybrid search can combine vector similarity search with keyword matching to obtain more accurate and relevant search results.

[0068] S104, inputting the question to be processed and the first preset prompt word into the first large language model, so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes multiple sub-questions, search results corresponding to the multiple sub-questions respectively, and a logical ordering between the multiple sub-questions.

[0069] Exemplarily, the first language model may be Wenxinyiyan, ChatGPT, and so on.

[0070] It is understandable that, since the large language model has a strong context learning capability, and the first preset prompt word includes the search results corresponding to the multiple sub-questions, the first large language model can use the search results of each sub-question as a reference for answering each sub-question to infer the answer to each sub-question. Moreover, according to the logical ordering between the multiple sub-questions included in the first preset prompt word, the first large language model can summarize the answers to the sub-questions according to the logical ordering between the multiple sub-questions, thereby generating an answer to the question to be processed.

[0071] Exemplarily, the content of the first preset prompt word may include "Thinking step 1: text 1; thinking step 2: text 2; ...", where thinking step 1 is the sub-question with the first logical order, thinking step 2 is the sub-question with the second logical order, and text 1 and text 2 are the search results corresponding to each sub-question.

[0072] It is understandable that by breaking down the problem to be processed into multiple sub-problems with a logical chain, when the first large language model answers the problem to be processed, it can use the search results of multiple sub-problems as a reference to infer the answers to each sub-problem, and then summarize the answers to each sub-problem according to the logical order between the sub-problems to obtain the answer to the problem to be processed. Compared with directly using a large language model to answer the problem to be processed, since the search results of each sub-problem are referenced when answering, the content of the answer contains richer information and has a large information gain; and, the answers to each sub-problem are summarized according to the logical order between the multiple sub-problems, so that the final generated answer is logical and the answer effect is better.

[0073] In addition, when the first language model summarizes the answers to the sub-questions according to the logical order between the multiple sub-questions, it can also generate a summarized answer according to the structure of total score or total score total.

[0074] Optionally, in one implementation, the first preset prompt word also includes a preset checking rule; the checking rule includes a rule constraining the vocabulary and / or format of the generated answer.

[0075] It can be understood that in order to enable the first language model to generate more standardized answers, the first preset prompt word also includes preset verification rules, so that after the first language model generates the answer to the question to be processed, it can also verify the generated answer according to the verification rules, so as to correct the generated answer when the generated answer does not meet the verification rules, so that the first language model can reflect on and correct the generated answer according to the preset verification rules.

[0076] For example, the specific content of the checking rules can be "1. The answer should be professional and rigorous, narrated from a third-party perspective, and avoid the use of first-person pronouns, such as "we" and other words; 2. Do not state in the answer that this is a summary output, and the output content should not contain similar words such as "answer", "summary", etc.; 3. Output in a whole paragraph, do not divide it into paragraphs, and the output should not wrap, and it should be a complete paragraph."

[0077] It should be noted that the embodiment of the present disclosure does not limit the content included in the first preset prompt word. For example, in order to achieve a better answering effect, the first preset prompt word may also include content requirements and format requirements when summarizing, etc.

[0078] In the disclosed embodiment, the problem to be processed is disassembled to obtain multiple sub-problems and the logical order between the multiple sub-problems. When the first large language model is used to answer the problem to be processed, the search results of each sub-problem and the logical order between the sub-problems are input into the first large language model as prompt words, so that the first large language model can use the search results of multiple sub-problems as a reference to infer the answers to each sub-problem, and then summarize the answers to each sub-problem according to the logical order between the sub-problems to generate the answer to the problem to be processed. This solution can realize intelligent answering, and compared with the manual answering method, it can reduce labor costs and improve answering efficiency. Compared with directly using a large language model to answer the problem to be processed, since the search results of each sub-problem are referred to when answering, the content of the answer contains richer information and has a large information gain; and the answers to each sub-problem are summarized according to the logical order between the multiple sub-problems, so that the answer finally generated is logical. Compared with the large language model answering according to random logic, the answer generated by this solution is more accurate and the answer effect is better. It can be seen that this solution can improve the answering efficiency and answering effect.

[0079] Optionally, in another embodiment of the present disclosure, each sub-question corresponds to multiple search results;

[0080] Optionally, in one implementation, before inputting the question to be processed and the first preset prompt word into the first language model in the above step S104, the process further includes:

[0081] For each sub-question, among the search results corresponding to the sub-question, the search results whose similarity with the sub-question is lower than a first preset threshold are deleted.

[0082] It is understandable that, since multiple search results can be retrieved for each sub-question, if there are too many search results, the processing cost of the first language model will increase, and the model's answering efficiency will be reduced. Therefore, in order to further improve the answering efficiency, this solution can also denoise the search results, that is, delete the search results corresponding to each sub-question whose similarity with the sub-question is lower than the first preset threshold. Exemplarily, the first preset threshold can be 70%, 80%, and so on.

[0083] Exemplarily, if the first preset threshold is 70%, for a sub-question, 100 search results are retrieved from each candidate result, then the similarity between the sub-question and each search result can be calculated, and then the search results with a corresponding similarity lower than 70% are deleted. Exemplarily, the similarity can be calculated by vectorizing the sub-question and each search result, and then calculating the cosine similarity between the vectors.

[0084] Optionally, in another implementation, before inputting the question to be processed and the first preset prompt word into the first language model in the above step S104, steps A1-A2 are further included:

[0085] A1, for each sub-question, inputting the search results corresponding to the sub-question and the sub-question into the third language model, so that the third language model can identify whether the sub-question is related to the corresponding search results, and obtain the search results related to the sub-question;

[0086] A2, among the search results corresponding to the sub-question, the other search results except the search results related to the sub-question are deleted.

[0087] In this implementation, the third language model is used to judge the relevance between each sub-question and the corresponding search results, and the search results are denoised according to the relevance judgment result.

[0088] In this implementation, the search results corresponding to each sub-question and the sub-question are input into the third language model, so that the third language model can understand the semantics of the sub-question and the corresponding search results, and output the search results related to the sub-question. Then, the search results that are not related to the sub-question are deleted from the search results corresponding to the sub-question.

[0089] It should be noted that the third largest language model and the first largest language model and the second largest language model are distinguished only in terms of naming. The third largest language model may be the same model as the first largest language model or the second largest language model, or may be a different model. The embodiments of the present disclosure do not limit the specific type of the third largest language model.

[0090] It is understandable that by denoising the search results in the above two ways, the number of search results subsequently input into the first language model can be reduced, the processing cost of the first language model can be reduced, and the answer efficiency can be improved. In addition, since search results with low similarity or irrelevant to each sub-question will lead to poor answering effect, the answering effect can be improved by deleting search results with low similarity or irrelevant to each sub-question.

[0091] Optionally, in another embodiment of the present disclosure, each sub-question corresponds to multiple search results; before the question to be processed and the first preset prompt word are input into the first language model in the above step S104, steps B1-B2 are also included:

[0092] B1, for each sub-question, sort the search results corresponding to the sub-question according to the importance, and obtain the re-ranking results corresponding to the sub-question;

[0093] B2, the search results corresponding to each sub-question input into the first language model are input in a specified order; the specified order is the sorting order between the search results represented by the re-ranking results corresponding to the sub-questions.

[0094] It can be understood that, since the first language model refers to the search results corresponding to each sub-question to infer the answer to the sub-question, the search results with a higher sorting order input into the first language model have a greater impact on the answer output by the first language model. Therefore, in order to further improve the answer effect, the search results of each sub-question can also be sorted according to importance, so that the first language model can focus on referring to the search results with high importance when inferring the answers to each sub-question.

[0095] In addition, it is understandable that if a retrieval algorithm based on the HNSW (Hierarchical Navigable Small World graphs) index structure is used when searching among candidate results for each sub-question, the TopK (i.e., the top K retrieval results with the highest similarity) of similarity retrieval in a multi-copy environment may be different, and idempotence cannot be guaranteed, that is, it cannot be guaranteed that the TopK obtained by executing the same retrieval operation multiple times are the same, resulting in a different input order each time when the retrieval results are subsequently input into the first language model in order of similarity from high to low. Therefore, sorting the retrieval results according to their importance so that each sorted retrieval result has a certain sorting position can solve the problem of not being able to guarantee idempotence when searching based on the HNSW index structure.

[0096] For example, if the search results corresponding to a sub-question include result A, result B, and result C, and after sorting them according to importance, the re-sorted results are result C, result A, and result B. Then when the search results corresponding to the sub-question are input into the first largest language model, the search results are input in the sorted order of result C, result A, and result B.

[0097] Optionally, in one implementation, for each sub-question in step B1, sorting the search results corresponding to the sub-question according to the importance, and obtaining the re-sorted results corresponding to the sub-question may include:

[0098] For each sub-question, the search results are sorted according to the preset index corresponding to the document or image from which the search results corresponding to the sub-question come, and a re-sorted result is obtained;

[0099] In this implementation, each document or picture from which the search result originates corresponds to a preset index, and the preset index corresponding to each document or picture is used to indicate the importance of the document or picture. For example, in actual applications, the preset index can be obtained by relevant technicians marking the importance of the document or picture in advance according to business rules.

[0100] For example, in the business field of data security, business rules may include the timeliness, effectiveness level, compliance of data security-related regulations, and the enforcement strength of data security-related systems, etc. For example, if the document A from which the search result A comes is a national-level regulation, and the document B from which the search result B comes is a provincial-level regulation, then the importance indicated by the preset index corresponding to the document A is higher than the importance indicated by the preset index corresponding to the document B. Then, after sorting the search results according to the preset index, in the re-sorted results, the sorting order of the search result A is before the search result B.

[0101] Optionally, in another implementation, for each sub-question in step B1, sorting the search results corresponding to the sub-question according to the importance, and obtaining the re-sorted results corresponding to the sub-question may include:

[0102] For each sub-question, the retrieval result corresponding to the sub-question is input into a pre-trained re-ranking model to obtain a re-ranking result; wherein the re-ranking model is a neural network model trained based on the sample retrieval results.

[0103] In this implementation, the pre-trained re-ranking model can be used to re-rank the retrieval results corresponding to each sub-question. Exemplarily, the training process of the re-ranking model may include: inputting multiple sample retrieval results into the initial re-ranking model, each sample retrieval result has a label representing the true value of the importance, according to the predicted importance of each sample retrieval result output by the model, calculating the difference between the predicted importance of each sample retrieval result and the label of each sample retrieval result, obtaining the model loss value, adjusting the model parameters by minimizing the model loss value, until the model converges and the training is completed, and obtaining the re-ranking model. Exemplarily, the initial re-ranking model can be CNN (Convolutional Neural Networks), DNN (Deep Neural Networks), etc. The embodiments of the present disclosure do not limit the specific type of the re-ranking model.

[0104] Optionally, in one implementation, the first large language model has a preset input character limit value; before inputting the question to be processed and the first preset prompt word into the first large language model in step S104, the method further includes:

[0105] If the total number of characters related to the question to be processed and the first preset prompt word is greater than the input character limit value, the search results corresponding to each sub-question are filtered according to the reordering result corresponding to the sub-question, so that the total number of characters related to the question to be processed and the first preset prompt word after filtering is not greater than the input character limit value.

[0106] In this implementation, the first large language model has a preset input character limit value, which is the maximum number of tokens that can be processed at one time when the first large language model processes input data.

[0107] It is understandable that if the total number of characters of the question to be processed and the first preset prompt word is greater than the input character limit value, the search results corresponding to each sub-question can be filtered according to the re-ranking result corresponding to the sub-question, that is, the search results with a lower ranking among the search results corresponding to the sub-question are deleted to retain the search results with high importance. Exemplarily, the search results can be deleted one by one from the back to the front according to the re-ranking result until the total number of characters of the question to be processed and the first preset prompt word is not greater than the input character limit value. Thus, when the input length of the large language model is limited, the search results with a higher degree of importance can be input first to improve the answer effect of the model.

[0108] Optionally, in another embodiment of the present disclosure, in the above step S104, the question to be processed and the first preset prompt word are input into the first large language model so that the first large language model generates an answer to the question to be processed, which may include steps C1-C3:

[0109] C1, inputting the multiple sub-questions and the search results corresponding to the multiple sub-questions into the first large language model, so that the first large language model performs reasoning based on the search results corresponding to the sub-questions to obtain answers corresponding to the sub-questions;

[0110] C2, correct the answers corresponding to each sub-question to obtain the corrected answers corresponding to each sub-question;

[0111] C3, input the problem to be processed, the corrected answers corresponding to each sub-problem, and the logical order between multiple sub-problems into the first largest language model, so that the first largest language model summarizes the answers corresponding to each sub-problem according to the logical order to obtain the answer to the problem to be processed.

[0112] In this embodiment, multiple sub-questions and the search results corresponding to the multiple sub-questions can be input into the first large language model, so that the first large language model uses the search results corresponding to the sub-questions as a reference to infer the answers corresponding to the sub-questions.

[0113] After obtaining the answers corresponding to each sub-question, the answers corresponding to each sub-question can be modified, for example, the exquisite words in the answers corresponding to each sub-question can be filtered to obtain the modified answers corresponding to each sub-question. For example, in practical applications, the exquisite words in the answers corresponding to each sub-question can be filtered using a preset exquisite word library, that is, by keyword matching, it is searched whether there are exquisite words contained in the preset exquisite word library in each answer, and when the exquisite words are retrieved, the exquisite words in the answer are deleted to obtain the modified answer.

[0114] It should be noted that the preset exquisite word library can be constructed by relevant technical personnel based on experience, and the embodiments of the present disclosure do not limit the exquisite words contained in the exquisite word library. For example, in the compliance question-and-answer scenario in the field of data security, it is necessary to generate rigorous answers, so any semantically imprecise word in this scenario can be used as an exquisite word.

[0115] It is understandable that after the answers corresponding to each sub-question are corrected, the corrected answers obtained are more in line with the needs of the current question-answering scenario. Then, the question to be processed, the corrected answers corresponding to each sub-question, and the logical order between multiple sub-questions are input into the first language model, so that after the first language model summarizes the answers corresponding to each sub-question according to the logical order, the answer to the question to be processed generated is more in line with the needs of the current question-answering scenario, thereby improving the answer effect.

[0116] For example, in one approach, in a regulatory response scenario for data security risk assessment, the issue to be addressed is a compliance check item on data security. The answer logic for intelligently responding to compliance check items using a large language model is as follows: Figure 2 As shown in the figure, the first step is to understand the inspection items, and then query the regulatory database, the enterprise system database, and the compliance evidence database based on the understanding of the inspection items, and finally summarize and organize the answers, that is, summarize the query results and organize the answers. However, the above answer logic has the following problems when dealing with intelligent responses to complex compliance inspection items: serious retrieval homogeneity recall and low information gain; random answer organization logic and poor controllability; and it is impossible to automatically generate screenshots for evidence.

[0117] In order to provide intelligent responses to complex compliance issues, thereby automating and intelligentizing risk assessment, reducing the risk assessment costs of industrial enterprises, and improving the effectiveness of responses and the efficiency of risk assessment, in another implementation method, the response logic of "planning-retrieval-reasoning-summarization-reflection" can be used to enable the large language model to provide in-depth responses and improve the response effect. A specific example of intelligent response based on the above-mentioned "planning-retrieval-reasoning-summarization-reflection" response logic is as follows: Figure 3As shown, it includes five steps: ① planning, ② retrieval, ③ reasoning, ④ summary, and ⑤ reflection.

[0118] Among them, ① Planning: Use the thinking chain technology to break down complex compliance issues into sub-issues with logical chains and MECE; Figure 3 As shown, the complex compliance issue is: "Develop a data strategy that can reflect the business development needs of the entire organization." After processing step ①, we get "1. What documents related to data strategy has the company issued? 2. What is the company's data strategy? 3. What specific organizational business plans does the company have? 4. How are the company's data strategy and organizational business combined?"

[0119] ② Search: For each sub-question, hybrid search (dense search and sparse search in a specific ratio) and image-text search are performed. That is, each sub-question is used to perform hybrid search and image-text search on PDF documents, Word documents, JPG images, and PNG images in databases such as data security-related regulations and corporate system libraries.

[0120] ③ Reasoning: including denoising, sorting, model reasoning and special word filtering. Denoising: Denoising the results according to the similarity threshold at the time of retrieval or the relevance judgment of the large language model, so as to reduce the impact of inaccurate or irrelevant recall results (i.e., retrieval results) placed in the context of the large language model on the answer effect. Sorting: The context length of the large language model is limited. When the context length is exceeded, the recall results with higher importance are placed first. In addition, if the retrieval algorithm based on the HNSW index structure is used, the TopK of similarity retrieval in a multi-copy environment may be different, and idempotence cannot be guaranteed. Therefore, a re-ranking model is often used to re-rank the recall results, or the recall results are re-ranked based on business rules, such as regulations (timeliness, effectiveness level, compliance, etc.), systems (enforcement strength, etc.). Model reasoning: Reasoning answers based on the recall results to obtain the answers to each sub-question. Special word filtering: Because the large language model often generates some exquisite words when answering, and compliant question and answer is a rigorous scenario, it is necessary to filter out the exquisite words.

[0121] ④ Summary: Generate answers according to the planned path, that is, generate summarized answers according to the logical chain between the sub-questions and specific logic (total score or total score total).

[0122] ⑤ Reflection: The large language model self-corrects to obtain the final result. According to the verification criteria (corresponding to the verification rules above), the large language model is used to reflect on and correct the answer results. In addition, after the final answer is generated, the answer result output by the large language model and each recall result can be compared for similarity. When the similarity is greater than a certain threshold, an automatic screenshot is taken according to the page number of the source document from which the recall result comes. It should be noted that the method of automatic screenshot is introduced in the following embodiments and will not be repeated here.

[0123] It can be seen that intelligent answering is carried out according to the answer logic of "planning-retrieval-reasoning-summarization-reflection". Since complex questions are broken down into multiple sub-questions for retrieval, the recall results are rich and the information gain is large; and the answers to each sub-question are summarized according to the planned path, making the answer organization logical and highly controllable; the citations in the answers can be automatically screenshoted for evidence, which can ensure the consistency of the answer content and the screenshots for evidence.

[0124] Optionally, in another embodiment of the present disclosure, Figure 1 Based on the embodiment shown, Figure 4 As shown, the above-mentioned question-answering method based on the big model may further include steps S401-S403:

[0125] S401, determining the similarity between each search result included in the first preset prompt word and the answer to the question to be processed;

[0126] S402, if the corresponding search result with a similarity greater than the second preset threshold value originates from a document, determining the source page number of the corresponding search result with a similarity greater than the second preset threshold value in the document from which it originates;

[0127] S403, taking a screenshot of the page where the determined source page number is located to obtain a screenshot result.

[0128] In this embodiment, after obtaining the answer to the question to be processed, the similarity between each search result included in the first preset prompt word and the answer to the question to be processed can be determined. Exemplarily, the similarity between each search result and the answer to the question to be processed can be calculated by vectorizing each search result and the answer to the question to be processed, and then calculating the cosine similarity between the vector corresponding to each search result and the vector corresponding to the answer to the question to be processed, so as to obtain the similarity between each search result and the answer to the question to be processed.

[0129] It is understandable that since the answer to the question to be processed is generated by referring to each search result, the search result with higher similarity has a greater contribution to the generated answer. Then, the search result with high similarity can be used as the basis for generating the answer and a screenshot can be kept as evidence.

[0130] In this embodiment, the second preset threshold may be 60%, 70%, etc. It should be noted that in practical applications, the second preset threshold may be set by relevant technicians based on experience, and the embodiment of the present disclosure is not limited thereto.

[0131] Exemplarily, in actual applications, each candidate result can be pre-marked with the source page number from the document, so that after determining the search result whose corresponding similarity is greater than the second preset threshold, the source page number of the search result in the document from which it originated can be determined according to the mark corresponding to the search result. After obtaining the source page number, you can call screenshot tools such as snipaste (a screenshot tool that supports regional screenshots, pasting, etc.) and snagit (a screenshot tool that provides rich screenshot and editing functions) to take a screenshot of the page where the source page number is located to obtain a screenshot result. It should be noted that the embodiment of the present disclosure does not limit the method of taking the screenshot.

[0132] Optionally, in one implementation, the issue to be processed is a compliance check item regarding data security, and the method further includes:

[0133] The screenshot result is saved in a preset database as evidence for the compliance check item; accordingly, in this implementation, the method further includes:

[0134] If the search result with the corresponding similarity greater than the second preset threshold comes from a picture, the picture from which the search result with the corresponding similarity greater than the second preset threshold comes is saved in the preset database as evidence for the compliance check item.

[0135] It is understandable that if the corresponding search result with a similarity greater than the second preset threshold is derived from an image, the image can be directly saved in the preset database as evidence for the compliance check item. By saving the above screenshot results and the image from which the search result with a similarity greater than the second preset threshold is derived in the preset database, intelligent evidence can be collected from the preset database when responding to compliance checks such as data security risk assessment, so as to improve the efficiency of risk assessment.

[0136] Optionally, in one implementation, the method further includes steps D1-D2:

[0137] D1, if an evidence supplement instruction for any compliance check item is detected, the evidence supplement instruction is sent to the designated client, so that the designated client will feedback the supplementary file for the compliance check item;

[0138] D2, save the supplementary documents for the compliance check item as evidence in the preset database.

[0139] In this implementation, the designated client is a client used by the business side to supplement evidence. Exemplarily, when conducting a compliance check, if the staff on the compliance management side uses the intelligent answer to any compliance check item output by the intelligent answer client to perform intelligent evidence collection in the preset database and finds that evidence is missing, then the evidence supplement instruction can be entered in the current intelligent answer client. If an evidence supplement instruction for any compliance check item is detected, an evidence supplement instruction is sent to the designated client so that the staff on the business side can supplement the business evidence for the compliance check item where the evidence is missing, that is, the supplementary file of the compliance check item is input to the designated client, so that the designated client will feedback the supplementary file for the compliance check item. After receiving the supplementary file fed back by the designated client, the intelligent answer client saves the supplementary file as evidence about the compliance check item in the preset database for use in the next evaluation, thereby realizing evidence reuse. It can be understood that in actual applications, the designated client can also be an intelligent answer client.

[0140] For example, in practical applications, a data flywheel cycle update mechanism can be used to implement evidence reuse. Figure 5 As shown, it is a schematic diagram of the data flywheel construction of the end-to-end business scenario. It includes the intelligent response client on the compliance management side and the designated client on the business side. Among them, the staff on the compliance management side is responsible for creating projects, adding inspection items, and intelligent evidence collection based on the compliance evidence library. When the inspection items are non-compliant, partially compliant, or evidence is missing, the staff on the compliance management side can issue a bill to the designated client on the business side through the intelligent response client (corresponding to the evidence supplement instruction mentioned above), and the staff on the compliance business side will supplement the business evidence and feedback to the compliance management side, and the compliance management side will conduct evidence review. If the review fails, return to the step of issuing the bill. If the review passes, the evidence supplemented by the business side (corresponding to the supplementary file mentioned above) will be archived, that is, stored in the compliance evidence library (corresponding to the preset database mentioned above). Through multiple rounds of interaction between the business side and the compliance management side, until all the inspection items of the currently created project are reviewed and passed. After all the inspection items are reviewed and passed, the currently created project will be archived, that is, the current inspection items and supplemented evidence will be archived to the compliance evidence library for the next intelligent evaluation.

[0141] It can be seen that through this solution, a data flywheel cycle update mechanism of "compliance evidence query-evidence supplement-evidence archiving" is constructed, which can realize evidence reuse and accumulate compliance experience knowledge. Thus, the efficiency of risk assessment can be further improved.

[0142] Corresponding to the above method embodiment, the present disclosure also provides a question-answering device based on a large model, such as Figure 6 As shown, including:

[0143] An acquisition module 610 is used to acquire a problem to be processed;

[0144] A decomposition module 620 is used to decompose the problem to be processed to obtain multiple sub-problems and a logical order between the multiple sub-problems;

[0145] A retrieval module 630 is used to retrieve, for each sub-question, a candidate result matching the sub-question from each candidate result as a retrieval result corresponding to the sub-question;

[0146] A generation module 640 is used to input the question to be processed and a first preset prompt word into a first large language model so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes the multiple sub-questions, the search results corresponding to the multiple sub-questions respectively, and the logical ordering between the multiple sub-questions.

[0147] Optionally, the disassembly module is specifically used to:

[0148] The problem to be processed and the second preset prompt word are input into the second largest language model so that the second largest language model decomposes the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems; wherein the second preset prompt word includes a problem decomposition rule and at least one problem decomposition example that conforms to the problem decomposition rule.

[0149] Optionally, each candidate result includes: each text block in each candidate document in the business field to which the problem to be processed belongs, and / or, a semantic label of each candidate image in the business field to which the problem to be processed belongs;

[0150] The retrieval module is specifically used for:

[0151] For each sub-question, according to the first search method and / or the second search method, a candidate result matching the sub-question is retrieved from each candidate result as a search result corresponding to the sub-question;

[0152] The first search method includes: vectorizing the sub-question to obtain a search vector; using the search vector to search in a pre-built vector library to obtain a first preset number of candidate vectors with the highest similarity to the search vector; wherein the vector library includes candidate vectors obtained by vectorizing multiple candidate results respectively;

[0153] The second search method includes: dividing the sub-question to obtain multiple participles; using the multiple participles to search from each candidate result to obtain a correlation score between each participle and each candidate result; for each candidate result, calculating the sum of the correlation scores between the candidate result and each participle as the sum value corresponding to the candidate result; and selecting the first second preset number of candidate results with the highest corresponding sum values ​​from each candidate result.

[0154] Optionally, each sub-question corresponds to multiple search results; the device further includes:

[0155] A deleting module is used for, before the generating module executes the inputting of the to-be-processed question and the first preset prompt word into the first large language model, for each sub-question, deleting the retrieval results corresponding to the sub-question whose similarity with the sub-question is lower than the first preset threshold value; or, for each sub-question, inputting the retrieval results corresponding to the sub-question and the sub-question into the third large language model so that the third large language model identifies whether the sub-question is related to the corresponding retrieval results, and obtaining the retrieval results related to the sub-question; and deleting the retrieval results corresponding to the sub-question except the retrieval results related to the sub-question from the retrieval results corresponding to the sub-question.

[0156] Optionally, each sub-question corresponds to multiple search results; the device further includes:

[0157] a reordering module, used for, before the generating module executes the inputting of the to-be-processed question and the first preset prompt word into the first large language model, for each sub-question, sorting the search results corresponding to the sub-question according to the importance, so as to obtain a reordering result corresponding to the sub-question;

[0158] The retrieval results corresponding to each sub-question input into the first language model are input in a specified order; the specified order is the sorting order among the retrieval results represented by the re-ranking results corresponding to the sub-questions.

[0159] Optionally, the first language model has a preset input character limit value; and the device further comprises:

[0160] A filtering module is used to filter the search results corresponding to each sub-question according to the reordering result corresponding to the sub-question before inputting the question to be processed and the first preset prompt word into the first large language model, if the total number of characters related to the question to be processed and the first preset prompt word is greater than the input character limit value, so that the total number of characters related to the question to be processed and the first preset prompt word after filtering is not greater than the input character limit value.

[0161] Optionally, the reordering module is specifically used to:

[0162] For each sub-question, the search results are sorted according to the preset index corresponding to the document or picture from which the search results corresponding to the sub-question come, to obtain a re-sorted result;

[0163] or,

[0164] For each sub-question, the retrieval result corresponding to the sub-question is input into a pre-trained re-ranking model to obtain a re-ranking result; wherein the re-ranking model is a neural network model trained based on the sample retrieval result.

[0165] Optionally, the generating module includes:

[0166] a reasoning submodule, configured to input the plurality of sub-questions and the search results respectively corresponding to the plurality of sub-questions into the first large language model, so that the first large language model performs reasoning based on the search results respectively corresponding to the sub-questions to obtain answers respectively corresponding to the sub-questions;

[0167] A correction submodule is used to correct the answers corresponding to each sub-question to obtain a corrected answer corresponding to each sub-question;

[0168] The summarizing submodule is used to input the pending question, the revised answers corresponding to each sub-question, and the logical ordering between the multiple sub-questions into the first large language model, so that the first large language model summarizes the answers corresponding to each sub-question according to the logical ordering to obtain the answer to the pending question.

[0169] Optionally, the device further comprises:

[0170] A first determination module, used to determine the similarity between each search result included in the first preset prompt word and the answer to the question to be processed;

[0171] A second determination module is used to determine the source page number of the corresponding search result with a similarity greater than the second preset threshold in the document from which it comes, if the search result with a similarity greater than the second preset threshold comes from the document;

[0172] The screenshot module is used to take a screenshot of the page where the determined source page number is located to obtain a screenshot result.

[0173] Optionally, the problem to be processed is a compliance check item regarding data security, and the device further includes:

[0174] A first evidence retention module, used to save the screenshot result as evidence for the compliance check item in a preset database;

[0175] The device also includes:

[0176] The second evidence retention module is used to save the picture from which the search result with the corresponding similarity greater than the second preset threshold is derived as evidence for the compliance check item in the preset database if the search result with the corresponding similarity greater than the second preset threshold is derived from the picture.

[0177] Optionally, the device further comprises:

[0178] A supplementing module, configured to, if an evidence supplementing instruction for any compliance check item is detected, send the evidence supplementing instruction to a designated client, so that the designated client will feedback the supplementary file for the compliance check item;

[0179] The third evidence retention module is used to save the supplementary documents for the compliance check item as evidence in the preset database.

[0180] Optionally, the first preset prompt word also includes preset checking rules; the checking rules include rules for constraining the vocabulary and / or format of the generated answer.

[0181] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of information included in the legal database, corporate system database and compliance evidence database involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0182] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0183] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0184] at least one processor; and

[0185] a memory communicatively connected to the at least one processor; wherein,

[0186] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned large model-based question-answering methods.

[0187] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any of the above-mentioned large model-based question-answering methods.

[0188] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any of the above-mentioned large model-based question-answering methods when executed by a processor.

[0189] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0190] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0191] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0192] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a large model-based question-answering method. For example, in some embodiments, the large model-based question-answering method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the large model-based question-answering method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the large model-based question-answering method in any other appropriate manner (eg, by means of firmware).

[0193] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0195] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0197] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0198] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0199] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0200] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A question answering method based on a large model, comprising: Get pending issues; Decomposing the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems; For each sub-question, a candidate result matching the sub-question is retrieved from each candidate result as the retrieval result corresponding to the sub-question; The question to be processed and a first preset prompt word are input into a first large language model so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes the multiple sub-questions, the retrieval results corresponding to the multiple sub-questions respectively, and the logical ordering between the multiple sub-questions.

2. The method according to claim 1, wherein: The problem to be processed is decomposed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems, including: The problem to be processed and the second preset prompt word are input into the second largest language model so that the second largest language model decomposes the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems; wherein the second preset prompt word includes a problem decomposition rule and at least one problem decomposition example that conforms to the problem decomposition rule.

3. The method according to claim 1 or 2, wherein: Each candidate result includes: each text block in each candidate document in the business field to which the problem to be processed belongs, and / or, the semantic label of each candidate image in the business field to which the problem to be processed belongs; For each sub-question, retrieving a candidate result matching the sub-question from each candidate result as the retrieval result corresponding to the sub-question includes: For each sub-question, according to the first search method and / or the second search method, a candidate result matching the sub-question is retrieved from each candidate result as a search result corresponding to the sub-question; Wherein, the first search method includes: Vectorize the sub-problem to obtain a search vector; use the search vector to search in a pre-built vector library to obtain a first preset number of candidate vectors with the highest similarity to the search vector; wherein the vector library includes candidate vectors obtained by vectorizing multiple candidate results respectively; The second search method includes: The sub-problem is divided into multiple participles; the multiple participles are used to search from each candidate result to obtain a correlation score between each participle and each candidate result; for each candidate result, the sum of the correlation scores between the candidate result and each participle is calculated as the sum value corresponding to the candidate result; and the first second preset number of candidate results with the highest corresponding sum values ​​are selected from each candidate result.

4. The method according to claim 1 or 2, wherein: Each sub-question corresponds to multiple search results; before the question to be processed and the first preset prompt word are input into the first language model, the method further includes: For each sub-question, deleting the search results corresponding to the sub-question whose similarity with the sub-question is lower than a first preset threshold value; or, For each sub-question, inputting the search results corresponding to the sub-question and the sub-question into the third language model, so that the third language model identifies whether the sub-question is related to the corresponding search results, and obtains the search results related to the sub-question; Among the search results corresponding to the sub-question, the other search results except the search results related to the sub-question are deleted.

5. The method according to claim 1 or 2, wherein: Each sub-question corresponds to multiple search results; before the question to be processed and the first preset prompt word are input into the first language model, the method further includes: For each sub-question, the search results corresponding to the sub-question are sorted according to the importance, and the re-ranking results corresponding to the sub-question are obtained; The retrieval results corresponding to each sub-question input into the first language model are input in a specified order; the specified order is the sorting order among the retrieval results represented by the re-ranking results corresponding to the sub-questions.

6. The method according to claim 5, wherein: The first language model has a preset input character limit value; Before inputting the question to be processed and the first preset prompt word into the first language model, the method further includes: If the total number of characters related to the question to be processed and the first preset prompt word is greater than the input character limit value, the retrieval results corresponding to each sub-question are filtered according to the reordering result corresponding to the sub-question, so that the total number of characters related to the question to be processed and the first preset prompt word after filtering is not greater than the input character limit value.

7. The method according to claim 5, wherein: For each sub-question, the search results corresponding to the sub-question are sorted according to the importance, and the re-ranking results corresponding to the sub-question are obtained, including: For each sub-question, the search results are sorted according to the preset index corresponding to the document or picture from which the search results corresponding to the sub-question come, to obtain a re-sorted result; or, For each sub-question, the retrieval result corresponding to the sub-question is input into a pre-trained re-ranking model to obtain a re-ranking result; wherein the re-ranking model is a neural network model trained based on the sample retrieval result.

8. The method according to claim 1 or 2, wherein: Inputting the question to be processed and the first preset prompt word into a first large language model so that the first large language model generates an answer to the question to be processed, including: Inputting the plurality of sub-questions and the search results corresponding to the plurality of sub-questions into the first large language model, so that the first large language model performs reasoning based on the search results corresponding to the sub-questions to obtain answers corresponding to the sub-questions; Correct the answers corresponding to each sub-question to obtain the corrected answers corresponding to each sub-question; The question to be processed, the revised answers corresponding to each sub-question, and the logical ordering between the multiple sub-questions are input into the first large language model, so that the first large language model summarizes the answers corresponding to each sub-question according to the logical ordering to obtain the answer to the question to be processed.

9. The method according to claim 1 or 2, wherein: The method further comprises: Determining the similarity between each search result included in the first preset prompt word and the answer to the question to be processed; If the corresponding search result with a similarity greater than the second preset threshold value originates from a document, determining the source page number of the corresponding search result with a similarity greater than the second preset threshold value in the document from which it originates; A screenshot is taken of the page where the determined source page number is located to obtain a screenshot result.

10. The method according to claim 9, wherein: The problem to be processed is a compliance check item on data security, and the method further includes: The screenshot result is saved in a preset database as evidence for the compliance check item; If the search result with the corresponding similarity greater than the second preset threshold comes from a picture, the picture from which the search result with the corresponding similarity greater than the second preset threshold comes is saved in the preset database as evidence for the compliance check item.

11. The method according to claim 10, wherein: The method further comprises: If an evidence supplement instruction for any compliance check item is detected, the evidence supplement instruction is sent to a designated client, so that the designated client provides feedback with a supplementary document for the compliance check item; The supplementary document for the compliance check item is saved as evidence in the preset database.

12. The method according to claim 1 or 2, wherein: The first preset prompt word also includes preset checking rules; the checking rules include rules for constraining the vocabulary and / or format of the generated answer.

13. A question-answering device based on a large model, comprising: The acquisition module is used to obtain pending issues; A disassembly module, used for disassembling the problem to be processed to obtain multiple sub-problems and a logical order between the multiple sub-problems; A retrieval module is used to retrieve, for each sub-question, a candidate result matching the sub-question from each candidate result as a retrieval result corresponding to the sub-question; A generation module is used to input the question to be processed and a first preset prompt word into a first large language model so that the first large language model generates an answer to the question to be processed; wherein the first preset prompt word includes the multiple sub-questions, the search results corresponding to the multiple sub-questions respectively, and the logical ordering between the multiple sub-questions.

14. The device according to claim 13, wherein: The disassembly module is specifically used for: The problem to be processed and the second preset prompt word are input into the second largest language model so that the second largest language model decomposes the problem to be processed to obtain multiple sub-problems and a logical ordering between the multiple sub-problems; wherein the second preset prompt word includes a problem decomposition rule and at least one problem decomposition example that conforms to the problem decomposition rule.

15. The device according to claim 13 or 14, wherein: Each candidate result includes: each text block in each candidate document in the business field to which the problem to be processed belongs, and / or, the semantic label of each candidate image in the business field to which the problem to be processed belongs; The retrieval module is specifically used for: For each sub-problem, the sub-problem is vectorized to obtain a search vector; the search vector is used to search in a pre-built vector library to obtain a first preset number of candidate vectors with the highest similarity to the search vector, and the candidate results corresponding to the obtained candidate vectors are used as the search results corresponding to the sub-problem; wherein the vector library includes candidate vectors obtained by vectorizing multiple candidate results respectively; or, For each sub-question, the sub-question is segmented to obtain multiple participles; the multiple participles are used to search from each candidate result to obtain a correlation score between each participle and each candidate result; for each candidate result, the sum of the correlation scores between the candidate result and each participle is calculated as the sum value corresponding to the candidate result; and the first second preset number of candidate results with the highest corresponding sum value among the candidate results are selected.

16. The device according to claim 13 or 14, wherein: Each sub-question corresponds to multiple search results; the device also includes: A deleting module is used for, before the generating module executes the inputting of the to-be-processed question and the first preset prompt word into the first large language model, for each sub-question, deleting the retrieval results corresponding to the sub-question whose similarity with the sub-question is lower than the first preset threshold value; or, for each sub-question, inputting the retrieval results corresponding to the sub-question and the sub-question into the third large language model so that the third large language model identifies whether the sub-question is related to the corresponding retrieval results, and obtaining the retrieval results related to the sub-question; and deleting the retrieval results corresponding to the sub-question except the retrieval results related to the sub-question from the retrieval results corresponding to the sub-question.

17. The device according to claim 13 or 14, wherein: Each sub-question corresponds to multiple search results; the device also includes: a reordering module, used for, before the generating module executes the inputting of the to-be-processed question and the first preset prompt word into the first large language model, for each sub-question, sorting the search results corresponding to the sub-question according to the importance, so as to obtain a reordering result corresponding to the sub-question; The retrieval results corresponding to each sub-question input into the first language model are input in a specified order; the specified order is the sorting order among the retrieval results represented by the re-ranking results corresponding to the sub-questions.

18. The device according to claim 17, wherein: The first language model has a preset input character limit value; the device also includes: A filtering module is used to filter the search results corresponding to each sub-question according to the reordering result corresponding to the sub-question before inputting the question to be processed and the first preset prompt word into the first large language model, if the total number of characters related to the question to be processed and the first preset prompt word is greater than the input character limit value, so that the total number of characters related to the question to be processed and the first preset prompt word after filtering is not greater than the input character limit value.

19. The device according to claim 17, wherein: The reordering module is specifically used for: For each sub-question, the search results are sorted according to the preset index corresponding to the document or picture from which the search results corresponding to the sub-question come, to obtain a re-sorted result; or, For each sub-question, the retrieval result corresponding to the sub-question is input into a pre-trained re-ranking model to obtain a re-ranking result; wherein the re-ranking model is a neural network model trained based on the sample retrieval result.

20. The device according to claim 13 or 14, wherein: The generating module comprises: a reasoning submodule, configured to input the plurality of sub-questions and the search results respectively corresponding to the plurality of sub-questions into the first large language model, so that the first large language model performs reasoning based on the search results respectively corresponding to the sub-questions to obtain answers respectively corresponding to the sub-questions; A correction submodule is used to correct the answers corresponding to each sub-question to obtain a corrected answer corresponding to each sub-question; The summarizing submodule is used to input the pending question, the revised answers corresponding to each sub-question, and the logical ordering between the multiple sub-questions into the first large language model, so that the first large language model summarizes the answers corresponding to each sub-question according to the logical ordering to obtain the answer to the pending question.

21. The device according to claim 13 or 14, wherein: The device also includes: A first determination module, used to determine the similarity between each search result included in the first preset prompt word and the answer to the question to be processed; A second determination module is used to determine the source page number of the corresponding search result with a similarity greater than the second preset threshold in the document from which it comes, if the search result with a similarity greater than the second preset threshold comes from the document; The screenshot module is used to take a screenshot of the page where the determined source page number is located to obtain a screenshot result.

22. The device according to claim 21, wherein The problem to be processed is a compliance check item on data security, and the device further includes: A first evidence retention module, used to save the screenshot result as evidence for the compliance check item in a preset database; The second evidence retention module is used to save the picture from which the corresponding search result with a similarity greater than a second preset threshold is derived as evidence for the compliance check item in the preset database if the search result with a similarity greater than a second preset threshold is derived from the picture.

23. The device according to claim 22, wherein: The device also includes: A supplementing module, configured to, if an evidence supplementing instruction for any compliance check item is detected, send the evidence supplementing instruction to a designated client, so that the designated client will feedback the supplementary file for the compliance check item; The third evidence retention module is used to save the supplementary documents for the compliance check item as evidence in the preset database.

24. The device according to claim 13 or 14, wherein: The first preset prompt word also includes preset checking rules; the checking rules include rules for constraining the vocabulary and / or format of the generated answer.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images