Large model answer fact verification method and device, electronic equipment and storage medium

Abstract

The invention discloses a large model answer fact verification method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a state with the highest priority from a priority queue as a first state, and determining an action reward value which can be obtained when a target verification action is executed in the first state to reach a second state; determining a priority corresponding to the second state according to the action reward value, and adding the second state into the priority queue under the condition that the second state is not added into the priority queue; and obtaining a state with the highest priority from the priority queue again as a first state, repeating the process until the first state is the target state or the priority queue is empty, and judging that the fact verification of the answer given by the large model is passed under the condition that the first state is the target state. According to the method and the device, the technical problem that the checking effect of a large model answer fact checking method for multi-hop reasoning answers in the related technology is poor is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a large-model answer fact verification method, device, electronic device and storage medium. Background Art

[0002] Due to the nature of statistical induction, large models are inherently prone to hallucinations, including factual hallucinations and faithfulness hallucinations. Factual hallucinations involve professional knowledge in various fields and usually require corresponding evidence retrieved from external knowledge bases to assist in verification. However, the fact-checking methods for large model answers in related technologies are not effective for multi-hop reasoning answers.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a large-model answer fact verification method, device, electronic device and storage medium to at least solve the technical problem that the large-model answer fact verification method in the related art has poor verification effect on multi-hop reasoning answers.

[0005] According to one aspect of an embodiment of the present application, a method for verifying facts answered by a large model is provided, comprising: decomposing the answer given by the large model into multiple fact points, and determining an initial state, and adding the initial state to a priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state each fact point is in a state to be verified; obtaining a state with the highest priority from the priority queue as the first state, and when the first state is not a preset target state, using a reinforcement learning algorithm network to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching the second state, wherein the target verification action is the reward value of the action reward that can be obtained by executing the ... The action is used to verify a fact point and obtain the verification result corresponding to the fact point. In the target state, the verification result corresponding to each fact point is verified. According to the action reward value, the priority corresponding to the second state is determined, and if the second state has not been added to the priority queue, the second state is added to the priority queue. A state with the highest priority is obtained from the priority queue again as the first state, and the above process of determining the action reward value and priority and updating the priority queue is repeated until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model is passed.

[0006] Optionally, after determining the reward value that can be obtained by executing the target verification action in the first state and reaching the second state, the method also includes: obtaining a first cumulative reward value corresponding to the first state in the reward hash table, wherein the first cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the first state; determining a second cumulative reward value corresponding to the second state based on the first cumulative reward value and the action reward value, wherein the second cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the second state; if the second cumulative reward value corresponding to the second state does not exist in the reward hash table, or the historical second cumulative reward value corresponding to the second state recorded in the reward hash table is less than the newly determined second cumulative reward value, updating the newly determined second cumulative reward value corresponding to the second state into the reward hash table.

[0007] Optionally, determining the priority corresponding to the second state includes: using a reinforcement learning algorithm network to determine a state value corresponding to the second state, wherein the state value is a reward value that can be obtained from the second state to the target state predicted by the reinforcement learning algorithm network; determining the priority corresponding to the second state based on the state value corresponding to the second state and the second cumulative reward value.

[0008] Optionally, after obtaining a state with the highest priority from the priority queue as the first state, the method also includes: determining whether the first state is the target state; when the first state is the target state, obtaining a sequence of states experienced in the process from the initial state to the target state; based on the sequence, determining a verification order for each fact point, and sending the verification order to the front-end interactive interface for display.

[0009] Optionally, the reinforcement learning algorithm network includes: a Q-learning algorithm network, wherein the Q-learning algorithm network includes: a feature extraction layer, a duel network and a state network, the feature extraction layer is used to extract the features of the state, the duel network is used to learn the relative advantages and disadvantages of performing various verification actions in different states, and the state network is used to learn the expected rewards that can be obtained in different states.

[0010] Optionally, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching the second state, including: using a state network to determine the value expectation corresponding to the first state; determining all verification actions that can be executed in the first state in the action space; using a duel network to determine the average value of the advantage value corresponding to executing each verification action in the first state; determining the action reward value based on the advantage value, value expectation, and average value of the advantage value corresponding to the verification action as the target verification action.

[0011] Optionally, the training steps of the reinforcement learning algorithm network include: obtaining a training data set, wherein the training data includes four-tuple data in the historical answer verification process, and the four-tuple data includes: a first state, a target verification action taken in the first state, a second state reached after taking the target verification action, and a real action reward value obtained by taking the target verification action to reach the second state; using an initial model to predict the reward value for performing the target verification action in the first state in the training data set to obtain a predicted action reward value; determining a loss function value based on the real action reward value and the predicted action reward value, and updating the model parameters of the initial model based on the loss function value to obtain a reinforcement learning algorithm network.

[0012] According to another aspect of the embodiment of the present application, a large model answer fact verification device is also provided, including: a state determination module, used to decompose the answer given by the large model into multiple fact points, and determine the initial state, and add the initial state to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state each fact point is a state to be verified; a reward determination module, used to obtain a state with the highest priority from the priority queue as the first state, and when the first state is not a preset target state, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing the target verification action in the first state and reaching the second state, wherein the target core The verification action is used to verify a fact point and obtain the verification result corresponding to the fact point. In the target state, the verification results corresponding to each fact point are all verified. The queue update module is used to determine the priority corresponding to the second state according to the action reward value, and add the second state to the priority queue if the second state has not been added to the priority queue. The cyclic verification module is used to obtain a state with the highest priority from the priority queue again as the first state, repeat the above process of determining the action reward value and priority and updating the priority queue until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model is passed.

[0013] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the large model answer fact verification method is executed when the program is run.

[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the large model answer fact verification method by running the computer program.

[0015] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, which implements the steps of the large model answer fact verification method when the computer program is executed by a processor.

[0016] In an embodiment of the present application, the answer given by the large model is decomposed into multiple fact points, and an initial state is determined, and the initial state is added to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state each fact point is in a state to be verified; a state with the highest priority is obtained from the priority queue as the first state, and when the first state is not the preset target state, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching the second state, wherein the target verification action is used to verify a fact point and obtain the verification result corresponding to the fact point, and the verification result corresponding to each fact point in the target state is passed; based on the action reward value, the priority corresponding to the second state is determined, and If the second state has not been added, the second state is added to the priority queue; a state with the highest priority is obtained from the priority queue again as the first state, and the above process of determining the action reward value and priority and updating the priority queue is repeated until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model has passed. By formalizing the multi-hop reasoning problem as the optimal reasoning path search problem in the solution space, the combination of reinforcement learning and the shortest path search algorithm is used to realize dynamic evaluation of the reasoning path, which is fully applicable to a variety of fact verification tasks and provides a certain degree of explainability for the final verification results, thereby solving the technical problem that the fact verification method of the large model answer in the related technology has poor verification effect on the multi-hop reasoning answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for verifying facts of large model answers provided in an embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of a method flow of answering fact verification using a large model provided in an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a reinforcement learning algorithm network provided according to an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a large-model fact-checking device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0025] A* algorithm: A heuristic shortest path search algorithm on a graph structure that searches for paths by balancing existing costs and estimated costs.

[0026] Q-Learning: A reinforcement learning algorithm that approximates the optimal action-value function by learning a deep network from historical data, thereby achieving action evaluation.

[0027] In the related technology, fact-checking or hallucination detection schemes can be divided into retrieval-based schemes and model-state-based schemes. Among them, the retrieval-based scheme usually splits the answer into multiple independent atomic units and verifies them one by one. The verification schemes range from simple traditional measurement schemes such as entity overlap and relationship overlap to deep learning models such as Natural Language Inference, and then to the use of large models for judgment. However, the assumption that atomic units are independent of each other cannot be generalized to the actual multi-hop reasoning answers. There is a causal relationship between the units, which leads to misjudgment. The model-state-based scheme judges whether there is a hallucination in the answer of the large model based on the uncertainty of the data distribution of the model output Logits, or the consistency of the model's multiple answers to the same question. However, this method is difficult to give which specific parts of the answer have which fact-type conflicts, and lacks explainability.

[0028] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0029] According to an embodiment of the present application, an embodiment of a method for answering fact verification with a large model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing a large model answer fact verification method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the large model answer fact verification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the large model answer fact verification method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0035] Under the above operating environment, the embodiment of the present application provides a method for verifying facts answered by a large model. Figure 2 is a schematic diagram of a method flow for answering fact verification using a large model according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0036] Step S202, decomposing the answer given by the large model into multiple fact points, and determining the initial state, and adding the initial state to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state, each fact point is in a state to be verified;

[0037] Step S204, obtaining a state with the highest priority from the priority queue as the first state, and using a reinforcement learning algorithm network to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching the second state when the first state is not the preset target state, wherein the target verification action is used to verify a fact point and obtain a verification result corresponding to the fact point, and the verification result corresponding to each fact point in the target state is verification passed;

[0038] Step S206, determining the priority corresponding to the second state according to the action reward value, and adding the second state to the priority queue if the second state has not been added to the priority queue;

[0039] Step S208, again obtain a state with the highest priority from the priority queue as the first state, repeat the above process of determining the action reward value and priority and updating the priority queue, until the first state is the target state or the priority queue is empty, and when the first state is the target state, determine that the fact verification of the answer given by the large model has passed.

[0040] Through the above steps, by formalizing the multi-hop reasoning problem as the optimal reasoning path search problem in the solution space, the combination of reinforcement learning and the shortest path search algorithm is used to realize the dynamic evaluation of the reasoning path, which achieves the purpose of being fully applicable to a variety of fact-checking tasks and providing a certain degree of explainability for the final verification results, thereby solving the technical problem of poor verification effect of large-model answer fact-checking methods for multi-hop reasoning answers in related technologies.

[0041] The following is a further introduction to the method for verifying facts answered by the large model in steps S202 to S208 of the embodiment of the present application.

[0042] In an embodiment of the present application, the A* algorithm can be used to find the shortest path. At the same time, in order to evaluate the global reasoning state, the optimal action value network and the optimal state value network obtained by reinforcement learning are used to guide the weights in the search path and accelerate the convergence speed of the path search. Among them, the A* algorithm searches for the shortest path by estimating historical values ​​and future hypothetical values, but it is difficult to design a suitable function to estimate historical values ​​and future hypothetical values. In an embodiment of the present application, for fact-checking tasks, the historical values ​​and future hypothetical values ​​are approximated by integrating Q-Learning, and the A* algorithm is transformed into a practically usable algorithm, as follows.

[0043] First, we can use the big model to decompose the answer given by the big model into a series of fact points (states) (f1, f2, ..., f n ), and initialize the state s0 = (0, ..., 0), that is, in the initial state, each fact point is in a state to be verified, and the initial state is added to the priority queue.

[0044] Then, for the current state, the A* algorithm is used to search for a path, as follows.

[0045] If the priority queue is not empty, a circular exploration is performed, that is, a node with the highest priority is obtained from the priority queue as the current state cur (that is, the first state mentioned above), and it is determined whether the current first state is the preset target state, that is, whether all fact points have been verified in the current first state. If so, the given task is completed in the current state, and all paths (operations) from the initial state to the current state are returned, as follows.

[0046] In some embodiments of the present application, after obtaining a state with the highest priority from the priority queue as the first state, the method also includes the following steps: determining whether the first state is the target state; when the first state is the target state, obtaining the sequence of each state experienced in the process from the initial state to the target state; based on the sequence, determining the verification order of each fact point, and sending the verification order to the front-end interactive interface for display.

[0047] If the current first state is not the preset target state, that is, when the first state is not the preset target state, iterate the verification action a that can be executed in the current state cur, and the state reached after executing the verification action a (that is, the second state) The verification action is used to sample a fact point f i , check whether it is true or not, and mark its corresponding status as the result of verification, for example, 1 means that the fact is true, 2 means that the fact is false, and 3 means that the fact cannot be judged true or false.

[0048] At the same time, the reward for reaching S1 (i.e., the second accumulated reward value corresponding to the second state) is calculated. In this embodiment, the calculation formula for the reward for reaching S1 is as follows:

[0049] rc=g[cur]+Q * (s,a)

[0050] Where g[cur] represents the accumulated reward from the initial state to the current first state (i.e., the first accumulated reward value corresponding to the first state), Q *(s,a) is the reward estimated by the reinforcement learning algorithm network (Q network) that can be obtained by taking action a from the current first state to reach the second state S1, that is, the action reward value.

[0051] The above g[cur] can be obtained from the reward hash table. g[i] in the reward hash table is used to record the maximum reward obtained from the initial state to the current state i. In the embodiment of the present application, the target verification action is performed in the first state, and the action reward value Q that can be obtained when reaching the second state is * (s,a) can be determined using a reinforcement learning algorithm network. The specific steps are as follows.

[0052] In some embodiments of the present application, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by performing a target verification action in a first state and reaching a second state, including the following steps: using a state network to determine the value expectation corresponding to the first state; determining all verification actions that can be performed in the first state in the action space; using a duel network to determine the average value of the advantage value corresponding to performing each verification action in the first state; determining the action reward value based on the advantage value, value expectation, and average value of the advantage value corresponding to the verification action as the target verification action.

[0053] Specifically, as shown in the following formula:

[0054]

[0055] Where D is the duel network, which accepts the current first state s and the verification action a as input, and its output is a vector with a dimension equal to the number of actions, representing the advantage function. V represents the state network, which accepts the current first state as input, and its output is used to represent the value expectation of the current first state (the experience value is not the actual expectation). N represents the total number of all verification actions that can be performed in the first state in the action space. It is the average value of the advantage value corresponding to each verification action, which is used to eliminate network uncertainty.

[0056] After calculating the reward for reaching the second state S1, the reward hash table and the priority queue may also be updated. The specific steps are as follows.

[0057] In some embodiments of the present application, after determining that in the first state, executing the target verification action and reaching the second state can obtain the reward value, the method further includes the following steps: obtaining a first cumulative reward value g[cur] corresponding to the first state in the reward hash table, wherein the first cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the first state; based on the first cumulative reward value g[cur] and the action reward value Q *(s, a), determine the second cumulative reward value rc corresponding to the second state, where the second cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the second state; if the second cumulative reward value corresponding to the second state does not exist in the reward hash table, or the historical second cumulative reward value corresponding to the second state recorded in the reward hash table is less than the newly determined second cumulative reward value, the newly determined second cumulative reward value corresponding to the second state is updated to the reward hash table.

[0058] Specifically, if the reward hash table does not record the cumulative reward value for reaching the second state S1, or the newly determined second cumulative reward value rc> the historical second cumulative reward value g[S1] corresponding to the second state recorded in the reward hash table, that is, if there is a higher reward, update g[S1] = rc. And determine the priority corresponding to the second state S1, the specific steps are as follows.

[0059] In some embodiments of the present application, determining the priority corresponding to the second state includes the following steps: using a reinforcement learning algorithm network to determine the state value corresponding to the second state, wherein the state value is the reward value that can be obtained from the second state to the target state predicted by the reinforcement learning algorithm network; determining the priority corresponding to the second state based on the state value corresponding to the second state and the second cumulative reward value, and inserting the second state and the corresponding priority (S1, P) into the priority queue to update the priority queue.

[0060] The calculation process of the priority P corresponding to the second state is specifically shown in the following formula:

[0061] P=rc+V * (S1)

[0062] Among them, V * (S1) is the possible reward of state S1 estimated by the state network in the reinforcement learning algorithm network, that is, the state value corresponding to the second state.

[0063] Afterwards, a state with the highest priority is obtained from the priority queue again as the first state. The large model is used to determine whether the current first state can determine that all knowledge points of the current answer are supported by evidence (that is, determine whether it is the target state and end the verification process). If it is not the target state, the corresponding processes of steps S204 and S206 are repeated until the first state newly taken out from the priority queue is the target state or the priority queue is empty. When the first state is the target state, it is determined that the fact verification of the answer given by the large model has passed.

[0064] The results and training process of the reinforcement learning network used in the embodiments of the present application are introduced below, taking the Q-learning algorithm network as an example, and the details are as follows.

[0065] In some embodiments of the present application, the reinforcement learning algorithm network includes: a Q-learning algorithm network, wherein the Q-learning algorithm network includes: a feature extraction layer, a duel network and a state network, the feature extraction layer is used to extract the features of the state, the duel network is used to learn the relative advantages and disadvantages of performing various verification actions in different states, and the state network is used to learn the expected rewards that can be obtained in different states.

[0066] Specific as Figure 3 As shown in the figure, the network consists of two parts, among which D is the duel network, which accepts the current state s and the verification action a as input, and its output is a vector with a dimension equal to the number of actions, representing the advantage function, and V represents the state network, which accepts the current state as input, and its output is used to represent the value expectation of the current first state. The two share the feature extraction layer. After the two are operated, the value obtained is the empirical approximation of the action value function, that is, the action reward value Q * (s, a) is used to measure the possible benefits of taking an action in a given state, so that when the network training is completed, it can be used to judge the most appropriate action in any state.

[0067] In this embodiment, the training process of the reinforcement learning network is specifically as follows.

[0068] In some embodiments of the present application, the training steps of the reinforcement learning algorithm network include: obtaining a training data set, wherein the training data includes four-tuple data in the historical answer verification process, and the four-tuple data includes: a first state, a target verification action taken in the first state, a second state reached after taking the target verification action, and a real action reward value obtained by taking the target verification action to reach the second state; using an initial model to predict the reward value for performing the target verification action in the first state in the training data set to obtain a predicted action reward value; determining a loss function value based on the real action reward value and the predicted action reward value, and updating the model parameters of the initial model based on the loss function value to obtain a reinforcement learning algorithm network.

[0069] Specifically, for fact-checking tasks, empirical quadruple data is collected based on real scenarios: (s t ,a t ,r t ,s t+1 ), where s t represents the verification state of the current answer (i.e., the first state), a represents the action to be taken for the next verification (i.e., the target verification action taken in the first state), and r t Verify the previous step to get s t Then, take action a from it t Arriving at a new state s t+1(i.e. the second state) (i.e. the real action reward value obtained by taking the target verification action to reach the second state).

[0070] For a verification operation of length N, when the entire answer is perfectly verified, the entire verification is completed. When the entire answer is judged to be correct, all rewards on the path are set to 1. If the judgment is wrong, all rewards are set to -1. If it cannot be judged, it is set to 0. The specific process is as follows.

[0071] Perform forward propagation on the initial model network to obtain the duel network output Select the best verification action based on the network output: a * = argmax a∈A Q(s t+1 ,a;W), take the maximum value as the optimal action; use the momentum to update the target network parameters for forward propagation, and get (indicates the estimated value at the next moment after selecting the optimal action); calculate the TD target: r t is the observed actual reward, i.e., the real action reward value, Estimate for the next moment; use MSE (Mean Square Error) as the loss function Use the back propagation algorithm to update the overall network parameter W; use momentum update to set the target network Alleviate the instability of network training caused by too fast gradient descent. Where W is the network parameter before the gradient update, are the network parameters after gradient update.

[0072] This application scheme uses the Q-Learning method to approximate the historical value and future hypothesis value functions required in the A* algorithm. In actual reasoning, the output value of the Q network can be directly used for the A* algorithm. On the one hand, the generalization of the algorithm in this scheme is enhanced, and it is fully applicable to a variety of fact-checking tasks; on the other hand, through the deterministic search algorithm and path, a certain degree of explainability is provided for the final result.

[0073] According to an embodiment of the present application, an embodiment of a large model answer fact verification device is also provided. Figure 4 Schematic diagram of a large model answer fact verification device provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0074] The state determination module 40 is used to decompose the answer given by the large model into multiple fact points, determine the initial state, and add the initial state to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state, each fact point is in a state to be verified;

[0075] The reward determination module 42 is used to obtain a state with the highest priority from the priority queue as the first state, and when the first state is not the preset target state, use the reinforcement learning algorithm network to determine the action reward value that can be obtained by executing the target verification action in the first state and reaching the second state, wherein the target verification action is used to verify a fact point and obtain the verification result corresponding to the fact point, and the verification result corresponding to each fact point in the target state is verification passed;

[0076] A queue updating module 44 is used to determine the priority corresponding to the second state according to the action reward value, and add the second state to the priority queue if the second state has not been added to the priority queue;

[0077] The loop verification module 46 is used to obtain a state with the highest priority from the priority queue again as the first state, and repeat the above process of determining the action reward value and priority and updating the priority queue until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model has passed.

[0078] Optionally, after determining that in the first state, executing the target verification action and reaching the reward value that can be obtained in the second state, it also includes: obtaining a first cumulative reward value corresponding to the first state in the reward hash table, wherein the first cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the first state; determining a second cumulative reward value corresponding to the second state based on the first cumulative reward value and the action reward value, wherein the second cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the second state; if the second cumulative reward value corresponding to the second state does not exist in the reward hash table, or the historical second cumulative reward value corresponding to the second state recorded in the reward hash table is less than the newly determined second cumulative reward value, the newly determined second cumulative reward value corresponding to the second state is updated to the reward hash table.

[0079] Optionally, determining the priority corresponding to the second state includes: using a reinforcement learning algorithm network to determine a state value corresponding to the second state, wherein the state value is a reward value that can be obtained from the second state to the target state predicted by the reinforcement learning algorithm network; determining the priority corresponding to the second state based on the state value corresponding to the second state and the second cumulative reward value.

[0080] Optionally, after obtaining a state with the highest priority from the priority queue as the first state, it also includes: determining whether the first state is the target state; when the first state is the target state, obtaining a sequence of states experienced in the process from the initial state to the target state; based on the sequence, determining the verification order of each fact point, and sending the verification order to the front-end interactive interface for display.

[0081] Optionally, the reinforcement learning algorithm network includes: a Q-learning algorithm network, wherein the Q-learning algorithm network includes: a feature extraction layer, a duel network and a state network, the feature extraction layer is used to extract the features of the state, the duel network is used to learn the relative advantages and disadvantages of performing various verification actions in different states, and the state network is used to learn the expected rewards that can be obtained in different states.

[0082] Optionally, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching the second state, including: using a state network to determine the value expectation corresponding to the first state; determining all verification actions that can be executed in the first state in the action space; using a duel network to determine the average value of the advantage value corresponding to executing each verification action in the first state; determining the action reward value based on the advantage value, value expectation, and average value of the advantage value corresponding to the verification action as the target verification action.

[0083] Optionally, the training steps of the reinforcement learning algorithm network include: obtaining a training data set, wherein the training data includes four-tuple data in the historical answer verification process, and the four-tuple data includes: a first state, a target verification action taken in the first state, a second state reached after taking the target verification action, and a real action reward value obtained by taking the second state; using an initial model to predict the reward value for performing the target verification action in the first state in the training data set to obtain a predicted action reward value; determining a loss function value based on the real action reward value and the predicted action reward value, and updating the model parameters of the initial model based on the loss function value to obtain a reinforcement learning algorithm network.

[0084] It should be noted that the various modules in the above-mentioned large model answer fact verification device can be a program module (for example, a set of program instructions that implement a certain specific function) or a hardware module. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0085] It should be noted that the large model answer fact verification device provided in this embodiment can be used to perform Figure 2The large model shown answers the fact verification method. Therefore, the relevant explanations and instructions on the above-mentioned large model answering fact verification method are also applicable to the embodiments of the present application and will not be repeated here.

[0086] The embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following large model answer fact verification method by running the computer program: decomposing the answer given by the large model into multiple fact points, and determining an initial state, adding the initial state to a priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state each fact point is a state to be verified; obtaining a state with the highest priority from the priority queue as the first state, and when the first state is not a preset target state, using a reinforcement learning algorithm network to determine that in the first state, a target verification action is executed to reach The action reward value that can be obtained in the second state, wherein the target verification action is used to verify a fact point and obtain the verification result corresponding to the fact point, and the verification result corresponding to each fact point in the target state is verified; based on the action reward value, determine the priority corresponding to the second state, and if the second state has not been added to the priority queue, add the second state to the priority queue; obtain a state with the highest priority from the priority queue again as the first state, repeat the above process of determining the action reward value and priority and updating the priority queue, until the first state is the target state or the priority queue is empty, and when the first state is the target state, determine that the fact verification of the answer given by the large model is passed.

[0087] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the large model answer fact verification method described in each embodiment of the present application: decomposing the answer given by the large model into multiple fact points, and determining an initial state, adding the initial state to a priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state each fact point is a state to be verified; obtaining a state with the highest priority from the priority queue as the first state, and when the first state is not the preset target state, using a reinforcement learning algorithm network to determine that in the first state, executing the target verification action to reach the second state is possible The target verification action is used to verify a fact point and obtain the verification result corresponding to the fact point. In the target state, the verification result corresponding to each fact point is verified. According to the action reward value, the priority corresponding to the second state is determined, and if the second state has not been added to the priority queue, the second state is added to the priority queue. A state with the highest priority is obtained from the priority queue again as the first state, and the above process of determining the action reward value and priority and updating the priority queue is repeated until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model is passed.

[0088] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0089] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0094] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A large model answer fact verification method, characterized in that: include: Decompose the answer given by the large model into multiple fact points, determine the initial state, and add the initial state to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state, each fact point is in a state to be verified; A state with the highest priority is obtained from the priority queue as the first state, and when the first state is not the preset target state, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing the target verification action in the first state and reaching the second state, wherein the target verification action is used to verify the fact of one of the fact points to obtain the verification result corresponding to the fact point, and the verification result corresponding to each of the fact points in the target state is all verified; Determine the priority corresponding to the second state according to the action reward value, and add the second state to the priority queue if the second state has not been added to the priority queue; A state with the highest priority is obtained from the priority queue again as the first state, and the process of determining the action reward value and priority and updating the priority queue is repeated until the first state is the target state or the priority queue is empty. When the first state is the target state, it is determined that the fact verification of the answer given by the large model has passed.

2. The method for verifying facts based on a large model answer according to claim 1, characterized in that: After determining the reward value that can be obtained by performing the target verification action in the first state and reaching the second state, the method further includes: Obtaining a first cumulative reward value corresponding to the first state in a reward hash table, wherein the first cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the first state; Determining a second cumulative reward value corresponding to the second state according to the first cumulative reward value and the action reward value, wherein the second cumulative reward value is used to represent the sum of the action reward values ​​accumulated from the initial state to the second state; If the second cumulative reward value corresponding to the second state does not exist in the reward hash table, or the historical second cumulative reward value corresponding to the second state recorded in the reward hash table is smaller than the newly determined second cumulative reward value, the newly determined second cumulative reward value corresponding to the second state is updated to the reward hash table.

3. The method for verifying facts based on a large model answer according to claim 2, characterized in that: Determining the priority corresponding to the second state includes: Using the reinforcement learning algorithm network, determining a state value corresponding to the second state, wherein the state value is a reward value that can be obtained from the second state to the target state as predicted by the reinforcement learning algorithm network; The priority corresponding to the second state is determined according to the state value corresponding to the second state and the second accumulated reward value.

4. The method for verifying facts based on a large model answer according to claim 1, characterized in that: After obtaining a state with the highest priority from the priority queue as the first state, the method further includes: Determining whether the first state is the target state; When the first state is the target state, obtaining a sequence of states experienced during a process from the initial state to the target state; Based on the sequence, the verification order of each of the fact points is determined, and the verification order is sent to the front-end interactive interface for display.

5. The method for verifying facts based on a large model answer according to claim 1 or 3, characterized in that: The reinforcement learning algorithm network includes: a Q-learning algorithm network, wherein the Q-learning algorithm network includes: a feature extraction layer, a duel network and a state network, the feature extraction layer is used to extract the characteristics of the state, the duel network is used to learn the relative advantages and disadvantages of performing various verification actions in different states, and the state network is used to learn the expected rewards that can be obtained in different states.

6. The method for verifying facts based on a large model answer according to claim 5, characterized in that: Using a reinforcement learning algorithm network, it is determined that in the first state, the target verification action is performed, and the action reward value that can be obtained when reaching the second state includes: Using the state network, determining a value expectation corresponding to the first state; Determining all verification actions that can be performed in the action space in the first state; Using the duel network, determining an average value of the advantage value corresponding to executing each of the verification actions in the first state; The action reward value is determined according to the advantage value corresponding to the verification action as the target verification action, the expected value, and the average value of the advantage value.

7. The method for verifying facts based on a large model answer according to claim 5, characterized in that: The training steps of the reinforcement learning algorithm network include: Acquire a training data set, wherein the training data includes four-tuple data in the process of historical answer verification, and the four-tuple data includes: the first state, the target verification action taken in the first state, the second state reached after taking the target verification action, and the real action reward value obtained by taking the target verification action to reach the second state; Using the initial model, predicting a reward value for executing the target verification action in the first state in the training data set to obtain a predicted action reward value; A loss function value is determined based on the real action reward value and the predicted action reward value, and a model parameter of the initial model is updated based on the loss function value to obtain the reinforcement learning algorithm network.

8. A large model answer fact verification device, characterized in that: include: A state determination module is used to decompose the answer given by the large model into multiple fact points, and determine an initial state, and add the initial state to the priority queue, wherein the state is used to characterize the verification state of each fact point, and in the initial state, each fact point is in a state to be verified; A reward determination module is used to obtain a state with the highest priority from the priority queue as the first state, and when the first state is not a preset target state, a reinforcement learning algorithm network is used to determine the action reward value that can be obtained by executing a target verification action in the first state and reaching a second state, wherein the target verification action is used to perform fact verification on one of the fact points to obtain a verification result corresponding to the fact point, and the verification result corresponding to each of the fact points in the target state is a verification pass; a queue updating module, configured to determine the priority corresponding to the second state according to the action reward value, and add the second state to the priority queue if the second state has not been added to the priority queue; A cyclic verification module is used to obtain a state with the highest priority from the priority queue again as the first state, and repeat the above process of determining the action reward value and priority and updating the priority queue until the first state is the target state or the priority queue is empty, and when the first state is the target state, it is determined that the fact verification of the answer given by the large model has passed.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the large model answer fact verification method described in any one of claims 1 to 7 when running.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the large model answer fact verification method described in any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model answer fact verification method described in any one of claims 1 to 7 are implemented.

Images