Question answering method and system based on question answering system

Abstract

The invention relates to an answering method and system based on a question answering system, and the method comprises the steps: collecting a question, determining one or more vectors corresponding to the collected question in a vector database, and taking a text document corresponding to the one or more vectors as an answering corpus; determining cue words input into the large model according to the collected questions and answer corpora; generating an abstract on the basis of the cue word and the large model, taking the abstract as a final question answer of the question, taking a text document corresponding to one or more vectors as an answer corpus at the moment, further realizing automatic answer of the question, and at the same time, improving the answer efficiency of the question. Determining cue words input into the large model according to the collected questions and answer corpora; the abstract is generated based on the cue word and the large model, and the abstract is used as the final answer of the question, so that the answer is further refined and presented through the abstract, and the automation level of the question answering system is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of answering based on a question-answering system, and in particular to an answering method and system based on a question-answering system. Background Art

[0002] In some practical scenarios, such as automobiles, medical care, and finance, there may often be a need for expert consultation. For example, in the use of cars, users may have many questions about car maintenance, so there is a need for professional answers.

[0003] In the prior art, the conventional practice is to consult the corresponding after-sales staff or search and inquire through the Internet. However, the conventional practices cannot achieve automatic answers and the answers are not further refined. Summary of the invention

[0004] Based on this, it is necessary to provide a method and system for automatically answering the above technical problems based on a question-answering system.

[0005] A method for answering a question based on a question answering system, comprising:

[0006] Get the preprocessed text document;

[0007] Determine a corresponding vector based on the preprocessed text document, and construct a vector database based on the text document and the vector;

[0008] Collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0009] Determine the prompt words for input into the large model based on the collected questions and answer corpus;

[0010] A summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question.

[0011] A question-answering system-based answering system, wherein the question-answering system-based answering system is applied to the question-answering system-based answering method, and the question-answering system-based answering system comprises:

[0012] An acquisition module is used to acquire the preprocessed text document;

[0013] A vector database module, used to determine corresponding vectors based on the preprocessed text document, and to construct a vector database based on the text document and the vectors;

[0014] A vector module, used for collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0015] The prompt word module is used to determine the prompt words input into the large model based on the collected questions and answer corpus;

[0016] A summary module is used to generate a summary based on the prompt word and the large model, and use the summary as the final answer to the question.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Get the preprocessed text document;

[0019] Determine a corresponding vector based on the preprocessed text document, and construct a vector database based on the text document and the vector;

[0020] Collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0021] Determine the prompt words for input into the large model based on the collected questions and answer corpus;

[0022] A summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Get the preprocessed text document;

[0025] Determine a corresponding vector based on the preprocessed text document, and construct a vector database based on the text document and the vector;

[0026] Collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0027] Determine the prompt words for input into the large model based on the collected questions and answer corpus;

[0028] A summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question.

[0029] The above-mentioned answering method and system based on the question-answering system determine one or more vectors corresponding to the collected questions in the vector database, and use the text documents corresponding to the one or more vectors as the answer corpus, thereby realizing automatic answering of the questions. At the same time, the prompt words input into the large model are determined according to the collected questions and the answer corpus; a summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question, so as to further refine the answer and present it through the summary, thereby ensuring the accuracy of the question and answer, and thus improving the automation level of the question and answer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flowchart of an answering method based on a question-answering system in an embodiment of the present invention;

[0031] Figure 2 is a flow chart of S11 in the answering method based on the question-answering system in an embodiment of the present invention;

[0032] Figure 3 is a flow chart of S12 in the answering method based on the question-answering system in an embodiment of the present invention;

[0033] Figure 4 is a flowchart of S13 in the answering method based on the question-answering system in an embodiment of the present invention;

[0034] Figure 5 is a flowchart of S14 in the answering method based on the question-answering system in an embodiment of the present invention;

[0035] Figure 6 is a flow chart of S15 in the answering method based on the question-answering system in an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of the structural composition of an answering system based on a question-answering system in an embodiment of the present invention;

[0037] Figure 8 The figure is a hardware diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] In one embodiment, see Figures 1 to 8 , a question-answering system-based answering method, applied to a question-answering system-based answering scenario; the question-answering system-based answering method comprises:

[0040] Step S11: Obtain the preprocessed text document;

[0041] Step S12: determining a corresponding vector based on the preprocessed text document, and constructing a vector database based on the text document and the vector;

[0042] Step S13: Collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0043] Step S14: Determine the prompt words for input into the large model according to the collected questions and answer corpus;

[0044] Step S15: Generate a summary based on the prompt word and the large model, and use the summary as the final answer to the question.

[0045] In an embodiment of the present invention, through the method in the embodiment of the present invention, one or more vectors corresponding to the collected questions are determined in a vector database, and the text documents corresponding to the one or more vectors are used as answer corpus, thereby realizing automatic answering of the questions. At the same time, the prompt words for input into the large model are determined according to the collected questions and the answer corpus; a summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question, so as to further refine the answer and present it through the summary, thereby ensuring the accuracy of the question and answer, and thereby improving the automation level of the question and answer system.

[0046] In step S11, the preprocessed text document is obtained;

[0047] In the specific implementation process of the present invention, the specific steps may be:

[0048] S111: Obtaining a text document;

[0049] S112: Perform segmentation processing based on the content corresponding to the text document to determine a pre-processed text document.

[0050] In an embodiment of the present application, a text document is obtained, and subsequent processing is triggered for the text document. At this time, the content corresponding to the text document is presented so as to perform segmentation processing or division processing according to the content corresponding to the text document. At this time, in the segmentation processing, a sliding window is determined, and a text block size is determined based on the sliding window. The text document is segmented according to the sliding window and the text block size to obtain multiple text blocks corresponding to the text document, wherein two adjacent text blocks have associated repeated characters;

[0051] Alternatively, the text document is segmented based on the semantic topics corresponding to the text document, and one or more text blocks are divided based on the semantic topics.

[0052] Furthermore, segmentation or division processing is performed based on the content corresponding to the text document to determine the key area, so as to fix the key area, and then the text document is preprocessed according to the keywords corresponding to the key area to determine the preprocessed text document. The preprocessed text document includes at least one text block.

[0053] The specific steps include:

[0054] First, preprocess the text documents (including doc, docx, ppt, pptx, txt and other files). The specific steps include document format alignment, text preprocessing, cleaning whitespace, splitting long sentences into short sentences, splitting long paragraphs into shorter paragraphs, deleting duplicate content, and other operations. The main purpose of these methods in the field of NLP (natural language processing) is to obtain higher quality text corpus.

[0055] The segmentation method for long sentences is mainly based on some rules or algorithms. It includes:

[0056] 1.1 Rule method: Set a sliding window of chunk_size and set a parameter chunk_overlap for repeated characters in adjacent chunks. For example, the sentence:

[0057] “Cadillac is an American luxury automobile brand owned by General Motors.”

[0058] If chunk_size is set to 10 and chunk_overlap is set to 3, the above sentence can be split into:

[0059] [“Cadillac is an American luxury car brand,” “American luxury car brand affiliated with,” “Brand affiliated with General Motors.”]

[0060] The purpose of doing this is to ensure that some key information is not split into two chunks.

[0061] 1.2 Algorithm method: The main idea is to segment the text according to the differences at the semantic level based on the text similarity of semantic topics. The advantage of this method is that different topic texts can be divided, and different blocks do not necessarily need the same text length, but it requires additional calculations and some misjudgments of topics. The embodiment mentioned in the patent of this invention will adopt the rule segmentation method of rule chunk sliding window.

[0062] In step S12, a corresponding vector is determined based on the preprocessed text document, and a vector database is constructed based on the text document and the vector;

[0063] In the specific implementation process of the present invention, the specific steps may be:

[0064] S121: performing vectorization encoding on the text blocks corresponding to the preprocessed text document based on a vectorization model to determine a vector corresponding to each of the text blocks;

[0065] S122: constructing a vector database based on one or more of the text blocks and the vectors corresponding to each of the text blocks, wherein each of the vectors and the text blocks has a corresponding index relationship in the vector database.

[0066] In an embodiment of the present application, preprocessed text documents are collected, and corresponding vectorization models are matched based on the preprocessed text documents, thereby ensuring subsequent processing of the preprocessed text documents by the vectorization model and ensuring the processing accuracy of the preprocessed text documents.

[0067] At this time, the preprocessed text document is vectorized and encoded based on the vectorization model to determine the vector corresponding to each text block; a vector database is constructed based on one or more text blocks and the vectors corresponding to each text block, and each vector and the text block has a corresponding index relationship in the vector database.

[0068] The specific steps include:

[0069] 2.1 Select a suitable text embedding model, such as text2vec, m3e, bge and other pre-trained embedding models based on the Transformer architecture. The selection methods here mainly include research methods (such as comparing the evaluation results of public models), independent domain test set comparison methods (mainly based on the data set designed in the field, scoring comparison), etc.

[0070] 2.2 Use the text vectorization Embedding model determined in 2.1 to vectorize the text block output by 1 and store it in the vector database. Among them, the vector database can select milvus, faiss, etc. The specific selected solution can be combined with the data volume scale, search tool considerations, etc. The embodiment of the patent of the present invention is determined to be milvus based on comprehensive considerations such as scenario and data volume. It should be noted that it includes but is not limited to this selection.

[0071] 2.3 The original text of each block and the corresponding Embedding are associated through a certain index relationship, and the mutual mapping between Embedding and text can be realized later. Usually, vector databases come with some indexing methods.

[0072] In step S13, questions are collected, one or more vectors corresponding to the collected questions are determined in the vector database, and text documents corresponding to the one or more vectors are used as answer corpus;

[0073] In the specific implementation process of the present invention, the specific steps may be:

[0074] S131: Determine the problem vector corresponding to the collected problem;

[0075] S132: Searching the vector database for one or more vectors that match the question vector, and using the text blocks corresponding to the one or more vectors found as answer corpus.

[0076] In an embodiment of the present application, a question is collected. The question can be manually input or can be text-based based on sound. At this time, a question statement is defined based on the question, so as to further control the question statement, thereby determining the corresponding vector based on the question statement and the vector database. At this time, one or more vectors matching the question vector are searched in the vector database, and the text blocks corresponding to the one or more vectors found are used as answer corpus. Therefore, the corresponding text document is matched based on the vector, and the text document is used as the answer corpus. The question statement and the corresponding vector have a corresponding mapping relationship in the database. Optionally, one or more vectors matching the question vector are searched in the vector database based on a similarity algorithm or a clustering algorithm.

[0077] The specific steps include:

[0078] 3.1 When a query is input, the same text embedding model will be used to vectorize the input query. Then, the vector retrieval method provided by the vector database (generally based on similarity algorithm, clustering algorithm, etc.) can be used to quickly find the top k text block vectors most relevant to the query. The index in 2.3 can be used to find the text block corresponding to the recalled vector, which can be used as the answer corpus for subsequent steps.

[0079] 3.2 Among them, the top k and other parameters mentioned in step 3.1 are adjustable parameters to control the number of recalled text blocks.

[0080] S14: Determine the prompt words to be input into the large model according to the collected questions and answer corpus;

[0081] In the specific implementation process of the present invention, the specific steps may be:

[0082] S141: defining a first prompt part based on the collected questions;

[0083] S142: defining a second prompt part based on the answer corpus;

[0084] S143: Combine the first prompt part and the second prompt part to determine the prompt word for inputting the large model.

[0085] In an embodiment of the present application, the question and the answer corpus are associated to facilitate control of the question and the answer corpus, thereby defining a first prompt part based on the question; defining a second prompt part based on the answer corpus; and combining the first prompt part and the second prompt part to determine the prompt word for inputting the large model.

[0086] S15: Generate a summary based on the prompt word and the large model, and use the summary as the final answer to the question;

[0087] In the specific implementation process of the present invention, the specific steps may be:

[0088] S151: Collect prompt words;

[0089] S152: inputting the prompt word into the large model, wherein the large model is trained based on historical prompt words and summaries;

[0090] S153: Recognize the prompt word based on the large model, generate a summary, and use the summary as the final answer to the question;

[0091] S154: Collecting a summary, and performing customized output based on the summary and the mode selected by the user to output an optimized summary.

[0092] In the embodiment of the present application, the prompt word and the large model are associated so that the prompt word is input into the large model, and the large model is trained based on the previous prompt words and summaries. Further, the prompt word is recognized based on the large model, and a summary is generated, which is used as the final answer to the question; the summary is collected, and customized output is performed based on the summary and the mode selected by the user to output an optimized summary.

[0093] The specific steps include:

[0094] Based on the question (query) asked in step 2 and the top k most relevant text blocks recalled in step 3, prompt words are designed, and a summary is generated using a large model, which is generated as the answer to the question (query) and returned to the user.

[0095] The specific steps include:

[0096] The prompt word design mainly utilizes the prompt project in NLP. Through manually designed prompt words (a language description organized in human language, equivalent to language interaction with the big model), the user's input query, the top k related corpus recalled in step 3, and the final summary generation goal (usually let the big model (LLM) answer the user's query) are combined together. The main purpose is to use the big model (LLM)'s (LLM) understanding and generation capabilities to complete the tasks of refining, compressing, and answering the user's query as the final answer to the question.

[0097] You can design prompt words to let the large model (LLM) complete customized output goals. For example, output in different formats such as list, Excel, json, etc.

[0098] At this time, the corresponding vector is matched to the question, so that the text document can be traced back based on the vector to determine the answer corpus, thereby realizing automatic answering of the question. At the same time, the prompt words input into the large model are determined according to the collected questions and answer corpus; a summary is generated based on the prompt words and the large model to further refine the answer, and it is presented through the summary, which ensures the accuracy of the question and answer, thereby improving the automation level of the question and answer system.

[0099] The present invention can obtain more accurate information from the knowledge base, and use the large model to polish and integrate the search results to generate more coherent and accurate answers. This comprehensive application improves the quality and accuracy of the response, allowing users to obtain more satisfactory answers.

[0100] Example

[0101] See also Figure 7 , Figure 7 It is a schematic diagram of the structural composition of the answering system based on the question-answering system in an embodiment of the present invention.

[0102] like Figure 7 As shown, an answering system based on a question-answering system comprises:

[0103] An acquisition module 21 is used to acquire the preprocessed text document;

[0104] A vector database module 22, configured to determine corresponding vectors based on the preprocessed text document, and to construct a vector database based on the text document and the vectors;

[0105] A vector module 23, used for collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus;

[0106] A prompt word module 24, used to determine prompt words for input into the large model according to the collected questions and answer corpus;

[0107] The summary module 25 is used to generate a summary based on the prompt word and the large model, and use the summary as the final answer to the question.

[0108] Example

[0109] See also Figure 8 , refer to the following Figure 8 The electronic device 40 according to this embodiment of the present invention will be described. Figure 8 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0110] like Figure 8 As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: at least one processing unit 41, at least one storage unit 42, and a bus 43 connecting different system components (including the storage unit 42 and the processing unit 41).

[0111] The storage unit stores program codes, which can be executed by the processing unit 41, so that the processing unit 41 executes the steps according to various exemplary embodiments of the present invention described in the above “Embodiment Method” section of this specification.

[0112] The storage unit 42 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit 421 and / or a cache storage unit 422 , and may further include a read-only storage unit (ROM) 423 .

[0113] The storage unit 42 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0114] Bus 43 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0115] The electronic device 40 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 40, and / or any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 44. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., local area networks, wide area networks, and / or public networks, such as the Internet) via a network adapter 45. Figure 8 As shown, the network adapter 45 communicates with other modules of the electronic device 40 via the bus 43. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, tape drives, and data backup planning systems.

[0116] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0117] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: a read-only memory, a random access memory, a disk or an optical disk, etc. In addition, it stores computer program instructions, and when the computer program instructions are executed by a computer, the computer executes the above method.

[0118] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0119] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0120] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for answering a question based on a question-answering system, characterized in that: include: Get the preprocessed text document; Determine a corresponding vector based on the preprocessed text document, and construct a vector database based on the text document and the vector; Collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus; Determine the prompt words for input into the large model based on the collected questions and answer corpus; A summary is generated based on the prompt words and the large model, and the summary is used as the final answer to the question.

2. The answering method based on the question-answering system according to claim 1, characterized in that: The step of obtaining the preprocessed text document includes: Get the text document; Segmentation processing is performed based on the content corresponding to the text document to determine a pre-processed text document.

3. The answering method based on the question-answering system according to claim 2, characterized in that: The segmentation process based on the content corresponding to the text document includes: Determine the sliding window; Determine the text block size based on the sliding window, segment the text document according to the sliding window and the text block size, and obtain a plurality of text blocks corresponding to the text document, wherein two adjacent text blocks have associated repeated characters; Alternatively, the text document is segmented based on the semantic topics corresponding to the text document, and one or more text blocks are divided based on the semantic topics.

4. The answering method based on the question-answering system according to claim 3, characterized in that: The preprocessed text document includes at least one text block; The determining of the corresponding vector based on the preprocessed text document and constructing a vector database based on the text document and the vector include: Performing vectorization encoding on the text blocks corresponding to the preprocessed text document based on a vectorization model to determine a vector corresponding to each of the text blocks; A vector database is constructed based on one or more text blocks and the vectors corresponding to each text block, and each vector and the text block have a corresponding index relationship in the vector database.

5. The answering method based on the question-answering system according to claim 1, characterized in that: The collecting of questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus, includes: Determine a problem vector corresponding to the collected problem; One or more vectors matching the question vector are searched in the vector database, and the text blocks corresponding to the one or more vectors found are used as answer corpus.

6. The answering method based on the question-answering system according to claim 5, characterized in that: Searching the vector database for one or more vectors matching the question vector comprises: Based on a similarity algorithm or a clustering algorithm, one or more vectors matching the question vector are searched in the vector database.

7. The answering method based on the question-answering system according to claim 1, characterized in that: Determining the prompt words for inputting into the large model according to the collected questions and answer corpus includes: Defining a first prompt part based on the collected questions; defining a second prompt part based on the answer corpus; The first prompt part and the second prompt part are combined to determine the prompt words for inputting the large model.

8. The answering method based on the question-answering system according to claim 7, characterized in that: The step of generating a summary based on the prompt word and the large model, and using the summary as the final answer to the question, includes: Collecting the prompt words; Inputting the prompt word into the large model, wherein the large model is trained based on historical prompt words and summaries; The prompt word is recognized based on the large model, and a summary is generated, and the summary is used as the final answer to the question.

9. The answering method based on the question-answering system according to claim 1, characterized in that: The step of generating a summary based on the prompt word and the large model, and using the summary as the final answer to the question, includes: A summary is collected, and customized output is performed based on the summary and a mode selected by the user to output an optimized summary.

10. An answering system based on a question-answering system, characterized in that: The answering system based on the question-answering system is applied to the answering method based on the question-answering system as claimed in any one of claims 1 to 9, and the answering system based on the question-answering system includes: An acquisition module is used to acquire the preprocessed text document; A vector database module, used to determine corresponding vectors based on the preprocessed text document, and to construct a vector database based on the text document and the vectors; A vector module, used for collecting questions, determining one or more vectors corresponding to the collected questions in the vector database, and using text documents corresponding to the one or more vectors as answer corpus; The prompt word module is used to determine the prompt words input into the large model based on the collected questions and answer corpus; A summary module is used to generate a summary based on the prompt word and the large model, and use the summary as the final answer to the question.

Images