Question and answer method, device, related equipment and computer program product

Abstract

The present application discloses a question-answering method, device, related equipment and computer program product. The present application relates to the field of artificial intelligence technology. Through each first large model in the configured large model set, the answer content corresponding to the question information is generated respectively. The present application also pre-configures the second large model, makes full use of the reasoning ability of the second large model, performs consistency detection on the answer content generated by the aforementioned first large models, and obtains the judgment result of whether the answer content in the answer content set is consistent. If the judgment result indicates that there is at least one pair of consistent answer content, the consistent answer content can be output as the final answer. Considering that a single large model has errors, but if there is at least one pair of consistent answer content, the accuracy of the consistent answer content will be greatly improved, so the consistent answer content can be output as the final answer, which can greatly improve the accuracy of the answer result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a question-answering method, apparatus, related equipment, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, artificial intelligence models are increasingly being used in various industries, such as intelligent question-answering scenarios. In particular, with the rise of big models, the processing efficiency of various types of tasks has been greatly improved with the powerful natural semantic understanding, logical reasoning, and text generation capabilities of big models.

[0003] Currently, there are many types of big models on the market, and the network structures, training strategies, and capabilities of different big models are different. The performance of a single big model is limited, and it is easy to generate wrong answers in some question-answering scenarios. Therefore, how to improve the performance of big models in intelligent question-answering scenarios has become an urgent problem to be solved. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a question-answering method, apparatus, related equipment and computer program product to achieve the purpose of improving the effect of large models in intelligent question-answering scenarios. The specific scheme is as follows:

[0005] In a first aspect of the present application, a question-answering method is provided, comprising:

[0006] Get problem information;

[0007] Calling each first large model in the configured large model set to instruct each of the first large models to generate answer content corresponding to the question information, thereby obtaining an answer content set, wherein the large model set includes more than two different first large models;

[0008] Calling the configured second largest model to instruct the second largest model to determine whether the answer contents in the answer content set are consistent with each other based on the question information, and obtaining a determination result of whether the answer contents in the answer content set are consistent with each other;

[0009] If the judgment result indicates that there is at least one pair of consistent answer contents, the consistent answer contents are output as the final answer.

[0010] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0011] If the judgment result indicates that the answer contents in the answer content set are inconsistent, refer to the advantage processing question types of each first large model in the configured large model set, and select the first large model whose advantage processing question type includes the question information as the target first large model;

[0012] The answer content corresponding to the question information generated by the target first model is output as the final answer.

[0013] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the process of calling the configured second largest model to obtain a judgment result of whether the answer contents of two pairs in the answer content set are consistent includes:

[0014] Continuously detecting whether each of the first large models has completed generating answer content, and when obtaining answer content generated by two of the first large models, starting to combine the generated answer content in pairs, and calling the second large model to instruct the second large model to determine whether two answer contents in each answer content combination are consistent, until obtaining a determination result of whether two answer contents in the answer content set are consistent;

[0015] If the judgment result indicates that there is at least one pair of consistent answer contents, the process of outputting the consistent answer contents as the final answer includes:

[0016] The judgment result output by the second largest model is continuously detected, and when the judgment result is detected to be consistent for the first time, the consistent answer content is output as the final answer.

[0017] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0018] According to the judgment result, the risk level of the final answer is determined and output, and the risk level is used to characterize the risk size of the final answer being wrong.

[0019] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0020] Obtaining the accuracy rate of the first target model for the question type of the question information obtained in a pre-test;

[0021] The risk level of the final answer is determined and output according to the accuracy rate, and the risk level is used to characterize the risk of errors in the final answer.

[0022] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of determining the risk level of the final answer according to the judgment result includes:

[0023] The risk level of the final answer is determined according to the number of answer contents consistent with the final answer in the judgment result, and the greater the number, the smaller the risk of error in the final answer represented by the corresponding risk level.

[0024] In one possible design, in another implementation of the first aspect of the embodiments of the present application, the second large model is a model obtained by fine-tuning the general large model using question and answer training data, wherein the question and answer training data includes sample question information, answer content pairs corresponding to the sample question information, and a judgment result of whether the answer content pairs are consistent.

[0025] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the method further includes:

[0026] The consistency judgment results of each pair of answer contents in the answer content set are cross-validated. If it is determined that there are answer contents that fail the verification, the answer contents that fail the verification are marked as training data for updating the training of the second largest model.

[0027] A second aspect of the present application provides a question-answering device, comprising:

[0028] A question acquisition unit, used for acquiring question information;

[0029] A first large model calling unit, used for calling each first large model in the configured large model set to instruct each of the first large models to generate answer content corresponding to the question information, and obtain an answer content set, wherein the large model set includes more than two different first large models;

[0030] A second large model calling unit, used to call the configured second large model to instruct the second large model to determine whether the answer contents in the answer content set are consistent with each other based on the question information, and obtain a determination result of whether the answer contents in the answer content set are consistent with each other;

[0031] The first answer output unit is used to output the consistent answer content as the final answer if the judgment result indicates that there is at least one pair of consistent answer contents.

[0032] According to a third aspect of the present application, an electronic device is provided, comprising: a memory and a processor;

[0033] The memory is used to store programs;

[0034] The processor is used to execute the program to implement the question-answering method described in any one of the first aspects of the present application.

[0035] According to a fourth aspect of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the question-and-answer method described in any one of the first aspects of the present application is implemented.

[0036] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the question-and-answer method described in any one of the first aspects of the present application.

[0037] By means of the above technical solution, the present application integrates multiple large models for question and answer processing. Specifically, each first large model in the configured large model set generates the answer content corresponding to the question information. Furthermore, the present application also pre-configures the second large model, which is used to perform consistency detection on the answer content generated by the aforementioned first large models, and obtains the judgment result of whether the answer content in the answer content set is consistent. If the judgment result indicates that there is at least one pair of consistent answer content, the consistent answer content can be output as the final answer. Considering that the answer content generated by a single large model may be wrong, but by cross-validating the consistency of the answer content generated by any two large models, if there is at least one pair of consistent answer content, the accuracy of the consistent answer content will be greatly improved, so the consistent answer content can be output as the final answer, which can greatly improve the accuracy of the answer result.

[0038] In addition, when verifying whether the answer contents generated by different first-largest models are consistent, the present application makes use of the capabilities of the second-largest model. By combining the second-largest model with the contextual information of the question information, the consistency of the answer contents between each pair is detected. This can fully utilize the reasoning ability of the second-largest model and improve the accuracy of consistency detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of an implementation system architecture of the question-answering method provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a terminal structure provided in an embodiment of the present application;

[0042] Figure 3 A schematic diagram of a server structure provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of a process flow of a question-and-answer method provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of a question-and-answer device provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] Reference numerals:

[0047] 100-terminal; 200-server; 110-RF unit; 120-memory; 130-input unit; 131-touch screen, 132-other input devices; 140-display unit; 150-camera; 160-audio circuit; 161-speaker; 162-microphone; 163-headphone jack; 170-processor; 180-external interface; 190-power supply; 201-bus; 202-processing circuit; 203-communication interface; 204-storage circuit; 601-processing device; 602-ROM; 603-RAM; 604-public transmission line; 605-I / O interface; 606-input device; 607-output device; 608-storage device; 609-communication device; 11-question acquisition unit; 12-first large model calling unit; 13-second large model calling unit; 14-first answer output unit. DETAILED DESCRIPTION

[0048] Before introducing this application solution, the relevant concepts involved in this article are first explained:

[0049] prompt: Instructions. When interacting with AI (such as an artificial intelligence model), you need to send instructions to AI. It can be a text description, such as "Please recommend me a pop song" when you interact with AI, or it can be a parameter description in a certain format. For example, if you want AI to draw in a certain format, you need to describe the relevant drawing parameters.

[0050] Large models: In the field of artificial intelligence, large models usually refer to large-scale pre-trained models. Such models are called "large" because they can be pre-trained on a large amount of data and can be transferred to a variety of downstream tasks. Their full English name is Large Pre-Trained Models or Large-Scale Pre-Training Models. Large models are characterized by their large scale and contain billions or even more parameters, which help them learn complex patterns in the data. The emerging capabilities of large models include but are not limited to: contextual learning, instruction following, code generation, step-by-step reasoning capabilities, etc. Large models can include large language models (LLMs) and multimodal large models. Large language models are mainly used to process text modal data. Multimodal large models further integrate multimodal capabilities on the basis of large language models and can process information in multiple modalities, such as images, text, audio, etc.

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] This application provides a question-and-answer method that can be applied to a variety of question-and-answer scenarios, especially question-and-answer scenarios for closed questions. Users can ask questions and answers to closed questions, and this application uses the power of a large model to give answers to questions. Among them, closed questions refer to those questions whose answers are clearly defined or preset, and can only be selected from given options, or whose answers are strictly controlled within a specific range. Such questions usually have clear, unique or limited answer options.

[0053] The question-answering method provided in this application can be applied to Figure 1 The system architecture shown in FIG. 1 may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 A server is included as an example for explanation).

[0054] The terminal 100 or the server 200 can be used alone to execute the question-answering method provided in the embodiment of the present application. In addition, the terminal 100 and the server 200 can also be used in collaboration to execute the question-answering method provided in the embodiment of the present application.

[0055] Next describe Figure 1 The product form of the mid-terminal 100;

[0056] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a learning machine, a teaching screen, a wearable device, a vehicle-mounted device, a conference terminal, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0057] Figure 2 An optional hardware structure diagram of the terminal 100 is shown.

[0058] refer to Figure 2As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160, a speaker 161, a microphone 162, an earphone jack 163 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2 These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0059] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 and / or other input devices 132. The touch screen 131 can collect the user's touch operation on or near it (such as the user's operation on or near the touch screen using any suitable object such as a finger, joint, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices 132. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0060] Among them, other input devices 132 can receive input data and so on.

[0061] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display various interactive interfaces, processing results, etc. in the question-and-answer method.

[0062] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0063] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0064] Among them, the memory 120 can be used to store software codes related to the question-answering method, the processor 170 can execute the steps of the question-answering method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement corresponding functions.

[0065] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0066] In this embodiment of the present application, the RF unit 110 can send data to the server 200 and receive the processing result sent by the server 200. Exemplarily, the RF unit 110 sends the received user's question information to the server 200, and the server 200 obtains the answer corresponding to the question information and returns the answer to the terminal 100 for output.

[0067] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0068] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0069] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0070] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2 In the terminal 100 shown.

[0071] Next describe Figure 1 The product form of the server 200;

[0072] Figure 3 A structural diagram of a server 200 is provided, such as Figure 3 As shown, the server 200 includes a bus 201, a processing circuit 202, a communication interface 203, and a storage circuit 204. The processing circuit 202, the storage circuit 204, and the communication interface 203 communicate with each other via the bus 201.

[0073] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0074] The processing circuit 202 may also be referred to as a processor, and may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0075] The storage circuit 204 may also be referred to as a memory, and may include a volatile memory, such as a random access memory (RAM). The storage circuit 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive drive (HDD), or a solid state drive (SSD).

[0076] The storage circuit 204 may be used to store software codes related to the question-answering method, and the processing circuit 202 may execute the steps of the question-answering method and may also schedule other units to implement corresponding functions.

[0077] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processor 170 in the above-mentioned terminal 100 and the processing circuit 202 in the server 200 can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (digital signal processing, DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor 170 and the processing circuit 202 can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0078] The present application embodiment provides a question-answering method, which is applied to Figure 1 Taking the computer device in the example as an example, the computer device can be Figure 1 The terminal 100 or the system consisting of the terminal 100 and the server 200. Figure 4 , the question-answering method specifically includes the following steps:

[0079] Step S100: Obtain problem information.

[0080] Specifically, in a human-computer question-answering dialogue scenario, a computer device may obtain input question information. The question information may be input in the form of text, voice, image, video, or other modalities.

[0081] Taking the teaching question-and-answer scenario as an example, users can use the title as question information.

[0082] Step S110: Call each first large model in the configured large model set to instruct each of the first large models to generate answer content corresponding to the question information, so as to obtain an answer content set.

[0083] The large model set includes more than two different first large models. Here, "first" is only used to distinguish it from the "second large model" below. The large model set may include multiple different large models, such as large models developed by multiple different companies, or large models of different types and models developed by the same company. The large model set may include currently disclosed large models, and may also include large models that may be developed in the future.

[0084] Optionally, the present application may select multiple large models that each have advantages in different topic types and combine them into a large model set.

[0085] In this step, each first large model in the large model set can be called to instruct each first large model to generate answer content corresponding to the question information, so as to obtain a set of answer contents generated by each first large model in the large model set.

[0086] When calling the first large model, a corresponding prompt instruction can be generated according to the question information, and the prompt instruction prompt can be sent to the first large model to obtain the answer content generated by the first large model.

[0087] Step S120, calling the configured second largest model to instruct the second largest model to determine whether the answer contents in the answer content set are consistent with each other based on the question information, and obtain a determination result of whether the answer contents in the answer content set are consistent with each other.

[0088] In order to judge the consistency of any two answers, the applicant initially thought of converting the answer content into a vector representation, and then calculating the similarity between the two vector representations as a criterion for measuring whether the two answers are consistent. For example, through technologies such as word embeddings and sentence embeddings, the answer content text can be converted into a vector representation, and the similarity between them can be calculated. If the similarity exceeds the set similarity threshold, it means that the two are consistent, otherwise, it means that the two are inconsistent. Specifically, an embedding model can be used to convert the answer content into a vector representation. The embedding model includes but is not limited to BERT, GPT, etc.

[0089] Vector representation maps text into numerical vectors in a high-dimensional space. These vectors capture the semantic information of the text, not just the superficial word order and spelling. When comparing the similarity of two embedding vectors, we can better understand their similarity at the semantic level.

[0090] However, this approach has the following two difficulties:

[0091] 1. In practical applications, it is difficult to determine an appropriate similarity threshold to determine whether two answers express the same meaning. Choosing an appropriate pixel low threshold often requires experience and experimentation, and may vary depending on the specific application scenario.

[0092] 2. Semantic similarity and consistent answer content are two completely different concepts in some scenarios. For example, if the question information is a multiple-choice question, the answer given by model A is the specific content of option A, and the answer given by model B is the identifier of option A. Obviously, the two answers are essentially the same, but they are very different in semantics. When using the above method to judge whether the answers are consistent, it is easy to draw wrong conclusions.

[0093] To this end, this application provides a solution that uses a large model to determine whether the content of two answers is consistent.

[0094] The large model LLM has powerful natural language understanding and generation capabilities, enabling it to perform well in many complex natural language processing tasks. By carefully designing prompts, the large model can be guided to understand the specific context of the question and generate accurate answers.

[0095] LLM is trained on large-scale and diverse text datasets and has the ability to process diverse language expressions. This enables it to understand and associate different forms of answers and make accurate judgments.

[0096] LLM is pre-trained on large-scale data, so that it has rich language and common sense knowledge, which helps the model have a strong foundation when dealing with a wide range of problems. For example, when dealing with multiple-choice questions, LLM can use the knowledge accumulated during its pre-training process to understand common question-answering patterns and language structures, so as to make reasonable judgments.

[0097] LLM has a certain reasoning ability and can make inferences and judgments in a given context. Even if the options and answers are different, the model can infer their relationship.

[0098] Based on the capabilities of the above-mentioned large model, the present application can pre-configure a second large model, which is a large model different from the first large model. The present application can call the second large model, instructing the second large model to refer to the context information of the question information, determine whether the two answers in the answer content set are consistent, and obtain the consistency judgment result of the combination of the two answers.

[0099] This embodiment provides a specific example of a prompt instruction prompt as follows:

[0100] "Please refer to the question information given to determine whether the two answers given are consistent.

[0101] Question information: {question information slot}

[0102] Answer content 1: {answer content slot}

[0103] Answer content 2: {answer content slot}".

[0104] The question information obtained in the above steps can be filled into the question information slot in the prompt, and the two answer contents that need to be compared can be filled into the two answer content slots in the prompt respectively to obtain the edited prompt, which is sent to the second largest model to obtain the consistency judgment result of the two answer contents output by the second largest model.

[0105] Optionally, the second largest model in this embodiment can use a general large model. In addition, in order to further improve the performance of the second largest model in the task of judging the consistency of the answer content, the general large model can be fine-tuned using the question-answer training data, and the large model after fine-tuning training is used as the second largest model used in this step. The question training data includes sample question information, answer content pairs corresponding to the sample question information, and the judgment result of whether the answer content pairs are consistent.

[0106] By using the above question-answering training data to fine-tune the general large model, the "hallucination" phenomenon of the large model can be effectively suppressed, and the performance of the large model in the task of judging the consistency of the answer content can be improved.

[0107] Step S130: If the judgment result indicates that there is at least one pair of consistent answer contents, the consistent answer contents are output as the final answer.

[0108] Specifically, if there is at least one pair of consistent answer contents in the answer content set, the consistent answer contents may be output as the final answer.

[0109] The question-and-answer method provided in the embodiment of the present application integrates multiple large models for question-and-answer processing. Specifically, each first large model in the configured large model set generates answer content corresponding to the question information. Furthermore, the present application also pre-configures a second large model, which is used to perform consistency detection on the answer content generated by the aforementioned first large models, and obtains a judgment result on whether the answer content in the answer content set is consistent. If the judgment result indicates that there is at least one pair of consistent answer content, the consistent answer content can be output as the final answer. Considering that the answer content generated by a single large model may be wrong, but by cross-validating the consistency of the answer content generated by any two large models, if there is at least one pair of consistent answer content, the accuracy of the consistent answer content will be greatly improved, so the consistent answer content can be output as the final answer, which can greatly improve the accuracy of the answer result.

[0110] In addition, when verifying whether the answer contents generated by different first-largest models are consistent, the present application makes use of the capabilities of the second-largest model. By combining the second-largest model with the contextual information of the question information, the consistency of the answer contents between each pair is detected. This can fully utilize the reasoning ability of the second-largest model and improve the accuracy of consistency detection.

[0111] In order to further verify the effectiveness of this application scheme, refer to the experimental data provided below:

[0112] This application tested large models with public APIs such as GPT and Gemini, various open source large models, and mathematical models fine-tuned based on open source large models on the prepared comprehensive mathematics test set. After testing, it was found that even if the accuracy of two large models was only 60%, that is, they would most likely give wrong answers, but the reasons for the errors were often inconsistent, and the wrong answers would also be different.

[0113] Referring to Table 1 below, on the prepared comprehensive mathematics test set, the open source model glm-4-9b-chat has an accuracy rate of 70.01% and gpt3.5 has an accuracy rate of 66.28%. The second largest model provided by this application is used to compare the answers of these two models. Among them, there are 1026 questions with the same answers. The accuracy rate of the answers to these questions has risen to 94.05%, which is 10% higher than gpt4, which has the highest accuracy rate of single model answers.

[0114] In addition, the greater the difference in the reasoning process of the large model, the less likely it is that the wrong answers they give will be consistent when errors occur. For example, for the same large model, the first answer requires the model to answer directly. The second answer requires the model to generate Python code, run it on the computer, and then get the answer. When the answers of these two completely different paths are consistent, the probability of error will be lower.

[0115] Table 1

[0116]

[0117] As the number of the first largest models used in the large model set increases, the correct answer content can be screened out more accurately and a higher accuracy rate can be achieved.

[0118] In addition, considering that the processing speeds of multiple different first large models in the large model set are different, that is, the time consumed to generate answer content is different, in this embodiment, you can choose to wait until all the first large models have generated answer content, and then combine the answer contents in the answer content set in pairs, and then call the second large model in parallel to judge the consistency of each answer content combination, and obtain the judgment result of whether the answer contents in the answer content set are consistent.

[0119] In addition, in order to improve the response speed of computer devices and respond to user input more quickly, a streaming processing method can also be adopted in this embodiment, that is, continuously detecting whether each first model has completed generating the answer content, and when the answer content generated by the two first models is obtained for the first time, the generated answer content is combined in pairs, and the second largest model is called to instruct the second largest model to determine whether the two answer contents in each answer content combination are consistent, until the judgment result of whether the two answer contents in the answer content set are consistent is obtained.

[0120] That is, in the process of calling multiple different first-large models to generate answer content corresponding to the question information, once two first-large models have generated answer content, the second-largest model is called to judge the consistency of the two answer contents. Subsequently, as other first-largest models generate answer content, the answer contents that have not been combined will continue to be combined in pairs, and the second-largest model will be called to continue to judge the consistency of the newly combined answer contents, until the judgment result of whether the two answer contents in the answer content set are consistent is obtained.

[0121] In this process, the judgment results output by the second largest model are continuously tested, and when the judgment results are detected to be consistent for the first time, the consistent answer content is output as the final answer.

[0122] The streaming processing method introduced in this embodiment can speed up the response rate of the device and reduce the time the user spends waiting for the final answer.

[0123] In some embodiments of the present application, although the method described in the above embodiments can maximize the accuracy of the output answer content, the final output answer still inevitably has the risk of error, and the risk of the final answer output in different situations may not be exactly the same. For this reason, the present embodiment can further add a step of determining the risk level of the final answer output and outputting the risk level.

[0124] Specifically, after obtaining the judgment result of whether the answer contents are consistent with each other in the answer content set based on step S120, if the judgment result indicates that there is at least one pair of consistent answer contents, this embodiment can determine and output the risk level of the final answer based on the judgment result.

[0125] The risk level is used to characterize the risk of errors in the final answer. The risk level setting strategy can be determined according to business needs. For example, the larger the risk level, the greater the corresponding error risk.

[0126] In an optional implementation, the present application may determine the risk level of the final answer based on the number of answer contents in the judgment result that are consistent with the final answer.

[0127] Among them, the more the number of answers that are consistent with the final answer is, the smaller the risk that the final answer represented by the corresponding risk level is wrong.

[0128] An optional example is that if the judgment result indicates that all answers are consistent, the consistent answer content is used as the final answer, and the corresponding risk level can be set to 0 (wherein, the smaller the risk level value, the smaller the risk of error). If the judgment result indicates that at least one pair of answers is consistent, the consistent answer content can be used as the final answer, and the corresponding risk level can be set to 1.

[0129] In the solution introduced in this embodiment, the risk level of the final answer is set by considering the number of answers generated by each first model that are consistent with the final answer, which can more accurately measure the risk of errors in the final answer. On the basis of outputting the final answer, the risk level of the final answer can also be output to inform the user that the final answer has a risk of errors, so that the user can adopt the final answer with caution.

[0130] In addition, considering that conventional problem-solving models need to collect a large amount of data and spend a lot of manpower to mark these data in order to optimize the accuracy of problem-solving, the solution of this application introduces the concept of risk level, and the final answer will be automatically marked with the risk level based on the final answer corresponding to the output question.

[0131] For the vast majority (about 80%) of low-risk questions, these questions cannot be automatically distinguished from the data set using traditional methods, and a lot of manpower is required just to mark the data that the model itself can already do well. These manpower costs are worthless. However, by adopting the scheme of this application, these low-risk questions can be automatically distinguished, which greatly saves the manpower cost. In addition, this application can also mark high-risk questions, and further construct a training data set for these high-risk questions, and conduct targeted optimization training on the first large model to improve the accuracy of the first large model in solving such questions.

[0132] In some embodiments of the present application, considering the answer content set generated by each first large model in the large model set, it is possible that all the answer contents in the answer content set are inconsistent, that is, when the consistency of each two answer contents in the answer content set detected by calling the second large model is different, it means that each first large model in the large model set has not reached a common understanding of the question information. For this situation, this embodiment provides an optional solution:

[0133] The present application can obtain the advantage processing type of each first largest model in the large model set through experimental testing in advance, and the advantage processing type is the type of question that the first largest model is good at processing, that is, the type of question with a higher correct answer rate. The following Table 2 illustrates the correct answer rates of two different first largest models under different question types.

[0134] Table 2

[0135]

[0136] From the table above, we can see that the accuracy of the two models is still very different in all question types. However, it does not mean that the model with a low overall accuracy is meaningless. For the composite functions question type, the model with the lowest overall accuracy has the highest accuracy.

[0137] As can be seen from the above, each of the first models has its own strengths in handling questions, so the strengths of each of the first models in handling questions can be tested and summarized. Further, the accuracy data of the first model in handling strengths can also be recorded.

[0138] On this basis, when the answer contents in the answer content set are inconsistent, the first large model whose advantageous processing question type includes the question information is selected as the target first large model by referring to the advantageous processing question type of each first large model in the configured large model set. Then, the answer content corresponding to the question information generated by the target first large model is output as the final answer.

[0139] Obviously, when the answers generated by each first-largest model to the current question information are inconsistent, in order to give the answer content with a higher accuracy rate corresponding to the question information as much as possible, the target first-largest model that has an advantage in processing the current question information can be called in this embodiment, and the answer content corresponding to the question information generated by the target first-largest model can be output as the final answer, which can maximize the accuracy of the answer content.

[0140] Furthermore, in the case where the answer content generated by the above-mentioned first target model is used as the final answer, the corresponding risk level can also be further output.

[0141] Specifically, the above embodiment has explained that when obtaining the advantage processing question type of each first large model, the accuracy data of the first large model on the advantage processing question type can also be recorded, as shown in Table 2. On this basis, the accuracy of the question type of the target first large model for the question information can be queried, and then the risk level of the final answer can be determined and output according to the accuracy.

[0142] Specifically, a correctness threshold can be set in advance, such as 90%. If the correctness of the target first model for the question type corresponding to the current question information exceeds the correctness threshold, the first risk level can be set; if the correctness of the target first model for the question type of the current question information does not exceed the correctness threshold, the second risk level can be set, wherein the risk of error represented by the first risk level is lower than that of the second risk level.

[0143] In some possible implementations, if the answer contents in the answer content set are inconsistent, the risk level of the final answer determined at this time may represent a higher risk of error than the risk level of the final answer determined when there is at least one pair of answer contents in the answer content set.

[0144] Exemplarily, the risk levels are defined as 0, 1, 2, and 3, where a larger value indicates a higher risk of error.

[0145] 1) If the judgment result indicates that all answers are consistent, the consistent answers will be used as the final answer, and the corresponding risk level can be set to 0;

[0146] 2) If the judgment result indicates that at least one pair of answers are consistent, the consistent answers can be used as the final answer, and the corresponding risk level can be set to 1;

[0147] 3) If the judgment result indicates that all answers are inconsistent, the target first model is called to generate the final answer. If the accuracy of the target first model for the current question type exceeds the set accuracy threshold, the risk level corresponding to the final answer can be set to 2;

[0148] 4) If the judgment result indicates that all answers are inconsistent, the target first model is called to generate the final answer. If the accuracy of the target first model for the current question type does not exceed the set accuracy threshold, the risk level corresponding to the final answer can be set to 3.

[0149] The above is only an example of an optional setting method of the risk level.

[0150] In some embodiments of the present application, considering that the second largest model itself may also have certain errors, the judgment results it outputs may also have some logical errors, for example: the second largest model judges that answer content 1 is consistent with answer content 2, judges that answer content 2 is consistent with answer content 3, and judges that answer content 1 is inconsistent with answer content 3. Obviously, there are logical errors in the above three judgment results. Such data can be automatically marked and, after manual review, used as training data for subsequent update training of the second largest model.

[0151] Specifically, based on the above embodiments, the method of the present application may include the following steps:

[0152] The consistency judgment results of each pair of answer contents in the answer content set are cross-validated. If it is determined that there are answer contents that fail the verification, the answer contents that fail the verification are marked as training data for updating the training of the second largest model.

[0153] The above process of cross-validating the consistency judgment result may adopt pre-configured validation rules, such as the validation rules may include:

[0154] When answer content x is consistent with answer content y, and answer content y is consistent with answer content z, answer content x needs to be consistent with answer content z.

[0155] If the verification rule is not passed, the relevant answer contents may be marked as failed verification. For example, in the verification rule of the above example, the answer contents x, y, and z may all be marked as failed verification.

[0156] By adopting the method provided in this embodiment, manual labeling only needs to focus on a small portion of data that fails cross-validation, which can stably improve the accuracy of the second largest model and greatly save the cost of manual labeling.

[0157] In summary, the question-answering method provided by this application not only makes up for the accuracy defects of a single large model in the short term, but can also steadily benefit from the growth of each first large model. For example, large models with excellent performance on the market can be added to the large model collection of this application, thereby steadily improving the accuracy of the answers finally output by this application.

[0158] The question-and-answer device provided in an embodiment of the present application is described below. The question-and-answer device described below and the question-and-answer method described above can be referenced to each other.

[0159] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a question-and-answer device disclosed in an embodiment of the present application.

[0160] like Figure 5 As shown, the device may include:

[0161] A question acquisition unit 11, used for acquiring question information;

[0162] A first large model calling unit 12 is used to call each first large model in the configured large model set to instruct each of the first large models to generate an answer content corresponding to the question information to obtain an answer content set, wherein the large model set includes more than two different first large models;

[0163] The second large model calling unit 13 is used to call the configured second large model to instruct the second large model to determine whether the answer contents in the answer content set are consistent with each other based on the question information, and obtain a determination result of whether the answer contents in the answer content set are consistent with each other;

[0164] The first answer output unit 14 is configured to output the consistent answer contents as the final answer if the judgment result indicates that there is at least one pair of consistent answer contents.

[0165] In a possible implementation, the process of the second largest model calling unit calling the configured second largest model to obtain a judgment result of whether the answer contents in the answer content set are consistent with each other includes:

[0166] Continuously detecting whether each of the first large models has completed generating answer content, and when obtaining answer content generated by two of the first large models, starting to combine the generated answer content in pairs, and calling the second large model to instruct the second large model to determine whether two answer contents in each answer content combination are consistent, until obtaining a determination result of whether two answer contents in the answer content set are consistent;

[0167] If the judgment result indicates that there is at least one pair of consistent answer contents, the process of outputting the consistent answer contents as the final answer includes:

[0168] The judgment result output by the second largest model is continuously detected, and when the judgment result is detected to be consistent for the first time, the consistent answer content is output as the final answer.

[0169] In a possible implementation, the device of the present application may further include:

[0170] The first risk level determination unit is used to determine and output the risk level of the final answer according to the judgment result, and the risk level is used to characterize the risk size of the final answer being wrong.

[0171] In a possible implementation, the device of the present application may further include:

[0172] The second answer output unit is used for selecting a target first large model whose advantageous processing question type includes the question information if the judgment result indicates that the answer contents in the answer content set are inconsistent, referring to the advantageous processing question types of each first large model in the configured large model set, and outputting the answer content corresponding to the question information generated by the target first large model as the final answer.

[0173] In a possible implementation, the device of the present application may further include:

[0174] The second risk level determination unit is used to obtain the accuracy rate of the target first large model for the question type of the question information obtained in advance; determine and output the risk level of the final answer based on the accuracy rate, and the risk level is used to characterize the risk size of the final answer being wrong.

[0175] In a possible implementation, the process of the first risk level determination unit determining the risk level of the final answer according to the judgment result includes:

[0176] The risk level of the final answer is determined according to the number of answer contents consistent with the final answer in the judgment result, and the greater the number, the smaller the risk of error in the final answer represented by the corresponding risk level.

[0177] In one possible implementation, the second large model is a model obtained by fine-tuning the general large model using question and answer training data, wherein the question and answer training data includes sample question information, answer content pairs corresponding to the sample question information, and a judgment result of whether the answer content pairs are consistent.

[0178] In a possible implementation, the device of the present application may further include:

[0179] A cross-validation unit is used to cross-validate the consistency judgment results of each pair of answer contents in the answer content set. If it is determined that there are answer contents that fail the verification, the answer contents that fail the verification are marked as training data for updating the training of the second large model.

[0180] The present application also provides an electronic device in an embodiment. Figure 6 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, tablet computers, learning machines, teaching large screens, wearable devices, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0181] like Figure 6 As shown, the electronic device may include a processing device 601 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the ROM 602 (read-only memory) or the program loaded from the storage device 608 to the RAM 603 (random access memory) to implement the question-and-answer method of the aforementioned embodiment of the present application. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602 and RAM 603 are connected to each other via a common transmission line 604 (also referred to as a bus). The I / O interface 605 (input / output interface) is also connected to the common transmission line 604.

[0182] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0183] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the question-and-answer methods provided in the embodiments of the present application.

[0184] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any question-and-answer method provided in the embodiment of the present application.

[0185] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0186] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0187] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0188] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0189] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A question-answering method, It is characterized in that include: Get problem information; Calling each first large model in the configured large model set to instruct each of the first large models to generate answer content corresponding to the question information, thereby obtaining an answer content set, wherein the large model set includes more than two different first large models; Calling the configured second largest model to instruct the second largest model to determine whether the answer contents in the answer content set are consistent with each other based on the question information and the answer contents, and obtaining a determination result of whether the answer contents in the answer content set are consistent with each other; If the judgment result indicates that there is at least one pair of consistent answer contents, the consistent answer contents are output as the final answer; The method further comprises: The consistency judgment results of each pair of answer contents in the answer content set are cross-validated, and if it is determined that there are answer contents that fail the verification, the answer contents that fail the verification are marked to be used as training data for updating the training of the second large model; wherein, the consistency judgment results of each pair of answer contents in the answer content set are cross-validated using pre-configured verification rules, and the verification rules include that when answer content x is consistent with answer content y, and answer content y is consistent with answer content z, answer content x needs to be consistent with answer content z.

2. The method according to claim 1, It is characterized in that Also includes: If the judgment result indicates that the answer contents in the answer content set are inconsistent, refer to the advantage processing question types of each first large model in the configured large model set, and select the first large model whose advantage processing question type includes the question information as the target first large model; The answer content corresponding to the question information generated by the target first model is output as the final answer.

3. The method according to claim 1, It is characterized in that The process of calling the configured second largest model to obtain a judgment result of whether the answer contents of two pairs in the answer content set are consistent includes: Continuously detecting whether each of the first large models has completed generating answer content, and when obtaining answer content generated by two of the first large models, starting to combine the generated answer content in pairs, and calling the second large model to instruct the second large model to determine whether two answer contents in each answer content combination are consistent, until obtaining a determination result of whether two answer contents in the answer content set are consistent; If the judgment result indicates that there is at least one pair of consistent answer contents, the process of outputting the consistent answer contents as the final answer includes: The judgment result output by the second largest model is continuously detected, and when the judgment result is detected to be consistent for the first time, the consistent answer content is output as the final answer.

4. The method according to claim 1, It is characterized in that Also includes: According to the judgment result, the risk level of the final answer is determined and output, and the risk level is used to characterize the risk size of the final answer being wrong.

5. The method according to claim 2, It is characterized in that Also includes: Obtaining the accuracy rate of the first target model for the question type of the question information obtained in a pre-test; The risk level of the final answer is determined and output according to the accuracy rate, and the risk level is used to characterize the risk of errors in the final answer.

6. The method according to claim 4, It is characterized in that The process of determining the risk level of the final answer according to the judgment result includes: The risk level of the final answer is determined according to the number of answer contents consistent with the final answer in the judgment result, and the greater the number, the smaller the risk of error in the final answer represented by the corresponding risk level.

7. The method according to any one of claims 1 to 6, It is characterized in that The second large model is a model obtained by fine-tuning the general large model using question-answering training data, wherein the question-answering training data includes sample question information, answer content pairs corresponding to the sample question information, and a judgment result of whether the answer content pairs are consistent.

8. A question-answering device, It is characterized in that include: A question acquisition unit, used for acquiring question information; A first large model calling unit, used for calling each first large model in the configured large model set to instruct each of the first large models to generate answer content corresponding to the question information, and obtain an answer content set, wherein the large model set includes more than two different first large models; a second large model calling unit, configured to call the configured second large model to instruct the second large model to determine whether the answer contents in the answer content set are consistent with each other based on the question information and the answer contents, and obtain a determination result of whether the answer contents in the answer content set are consistent with each other; A first answer output unit, configured to output the consistent answer contents as the final answer if the judgment result indicates that there is at least one pair of consistent answer contents; The device also includes: A cross-validation unit is used to cross-validate the consistency judgment results of each pair of answer contents in the answer content set. If it is determined that there are answer contents that fail the verification, the answer contents that fail the verification are marked as training data for updating the training of the second large model; wherein, a pre-configured verification rule is used to cross-validate the consistency judgment results of each pair of answer contents in the answer content set, and the verification rule includes that when answer content x is consistent with answer content y, and answer content y is consistent with answer content z, answer content x needs to be consistent with answer content z.

9. An electronic device, It is characterized in that include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the question-answering method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the question-answering method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, each step of the question-answering method as described in any one of claims 1 to 7 is implemented.

Images