Large model question and answer method and device based on thinking chain

Abstract

The invention relates to the technical field of large models, and provides a thinking chain-based large model question and answer method and device. The method comprises the following steps: obtaining a subjective question text; inputting the subjective question text into a question and answer model to obtain a target answer output by the question and answer model; the target answer is an answer conforming to a social moral specification; the question and answer model is obtained by using a historical subjective question text and a historical target answer tag corresponding to the historical subjective question text to train the large model on the basis of expanding the reasoning process of the large model into a structured reasoning chain and applying budget constraint to the structured reasoning chain. According to the question and answer model, when subjective questions are processed, answers conforming to social moral specifications can be stably output, and the reasoning efficiency and accuracy are effectively improved in the reasoning process.

Description

Technical Field

[0001] This application relates to the field of large model technology, specifically to a large model question-answering method and apparatus based on thought chain. Background Technology

[0002] In current large-scale model research, thought chain distillation has been widely applied to objective judgment and text generation tasks. These methods improve the performance of models on complex tasks by guiding them to generate intermediate reasoning steps. However, existing research mainly focuses on objective problems, with relatively little discussion on maintaining performance for subjective problems and aligning with social and ethical content.

[0003] Existing large models may output answers that do not conform to social moral norms when dealing with subjective questions, and their reasoning efficiency and accuracy are also unsatisfactory. Summary of the Invention

[0004] This application provides a large-scale question-answering method and apparatus based on thought chain to solve the technical problem that existing large-scale models may output answers that do not conform to social moral norms when dealing with subjective questions, and that their reasoning efficiency and accuracy are also unsatisfactory during the reasoning process.

[0005] Firstly, embodiments of this application provide a large-scale question-answering method based on a thought chain model, including:

[0006] Obtain the text of the subjective question;

[0007] The subjective question text is input into the question-answering model to obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms.

[0008] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0009] In one embodiment, extending the reasoning process of the large model into a structured reasoning chain includes:

[0010] Add structured tags to the vocabulary of the large model to obtain the tagged vocabulary;

[0011] Multiple words in the generated sequence of the large model that match the labeled words are identified as multiple target words;

[0012] Nodes are generated for all words in the generated sequence, and node edges are generated based on the dependencies between word inferences. A tree structure is constructed based on the nodes and the node edges. Each node includes a corresponding target triple, which includes the inference stage type of the node, the inference stage content of the node, and the inference stage confidence score of the node.

[0013] The node whose vocabulary corresponds to the target vocabulary in the reasoning stage is determined as the starting node of the new reasoning stage, thus obtaining the target tree;

[0014] The target tree is identified as the structured reasoning chain.

[0015] In one embodiment, imposing a budget constraint on the structured inference chain includes:

[0016] A corresponding budget constraint is set for each reasoning branch in the structured reasoning chain; the budget constraint includes the number of reasoning steps, the number of words generated per reasoning step, and the number of reasoning stages.

[0017] Based on the aforementioned budget constraints, the reasoning reward for each reasoning branch is set.

[0018] The large model is controlled to perform inference with the goal of maximizing the expected reward of all inference branches until any inference branch reaches the corresponding budget constraint, at which point the large model is controlled to stop inference.

[0019] In one embodiment, the question-answering model is trained based on the following steps:

[0020] Input the historical subjective question text and its corresponding historical target answer label into the large model to obtain multiple historical reasoning answers output by multiple reasoning branches of the structured reasoning chain within the large model;

[0021] Calculate the total stage loss of the structured reasoning chain and the total relative advantage of the multiple historical reasoning answers;

[0022] When a balance is not achieved between minimizing the total stage loss and maximizing the total relative advantage, a target historical reasoning answer with a deviation from social moral norms is selected from the multiple historical reasoning answers.

[0023] After updating the parameters of the large model and adding subjective question text similar to the historical subjective question text corresponding to the target historical reasoning answer, return to the step of inputting the historical subjective question text and its corresponding historical target answer label into the large model until a balance is reached between minimizing the total stage loss and maximizing the total relative advantage, and then determine the large model at this time as the question answering model.

[0024] In one embodiment, calculating the total stage loss of the structured inference chain includes:

[0025] For each reasoning stage in the structured reasoning chain, the loss of the historical reasoning result of the reasoning stage relative to its historical target result is calculated to obtain multiple losses;

[0026] The total stage loss of the structured inference chain is obtained by weighted summation of the multiple losses.

[0027] In one embodiment, calculating the total relative advantage of the plurality of historical reasoning answers includes:

[0028] Obtain human ratings for the multiple reasoning responses based on social moral norms;

[0029] Calculate the mean and variance of the human ratings for the multiple reasoning responses;

[0030] For each of the multiple historical reasoning answers, based on the mean and variance, the relative advantage of the historical reasoning answer relative to the other historical reasoning answers is calculated to obtain multiple relative advantages;

[0031] Based on the multiple relative advantages and KL divergence penalty, the total relative advantage of the multiple historical reasoning answers is generated.

[0032] Secondly, embodiments of this application provide a large-scale question-answering device based on a thought chain, comprising:

[0033] The question acquisition module is used to: acquire subjective question text;

[0034] The answer generation module is used to: input the subjective question text into the question-answering model and obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms;

[0035] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0036] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the large model question-answering method based on thought chain described in the first aspect.

[0037] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the large model question-answering method based on thought chain described in the first aspect.

[0038] Fifthly, embodiments of this application provide a non-transitory computer-readable storage medium, including a computer program, which, when executed by a processor, implements the steps of the large model question-answering method based on thought chain described in the first aspect.

[0039] This application provides a large-scale question-answering method and apparatus based on a thought chain. It acquires subjective question text, inputs the subjective question text into a question-answering model, and obtains the target answer output by the model, which is an answer that conforms to social moral norms. The question-answering model in this application, on the one hand, extends the reasoning process of the large model from a single output to a structured reasoning chain. This guides the model to generate supervised, structured intermediate reasoning steps, facilitating phased supervision of the formation and changes of social moral norms during the model's reasoning process, thus enabling timely intervention and adjustment. On the other hand, imposing budget constraints on the structured reasoning chain guides the model to obtain the target answer that conforms to social moral norms under limited computational resources, improving the model's reasoning efficiency and accuracy. Furthermore, by training the large model using historical subjective question texts and their corresponding historical target answer labels, the model can correct model answers that do not conform to social moral norms during training, allowing the large model to fully learn social moral norms. The final question-answering model can accurately output the target answer that conforms to social moral norms. In summary, the question-answering model of this application can stably output answers that conform to social moral norms when processing subjective questions, and effectively improves reasoning efficiency and accuracy during the reasoning process. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts of the large model question-answering method based on mind chain provided in the embodiments of this application;

[0042] Figure 2 This is the second flowchart of the large-scale question-answering method based on the thought chain provided in the embodiments of this application;

[0043] Figure 3 This is the third flowchart of the large-scale question-answering method based on the thought chain provided in the embodiments of this application;

[0044] Figure 4 This is the fourth flowchart of the large-scale question-answering method based on the thought chain provided in the embodiments of this application;

[0045] Figure 5 This is a schematic diagram of the structure of the large-scale question-answering device based on the thought chain provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] It should be noted that in the description of the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, or integral connections; they can be mechanical connections or electrical connections; they can be direct connections or indirect connections through an intermediate medium; and they can be internal connections between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0049] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0050] Figure 1 This is one of the flowcharts illustrating the large-scale question-answering method based on thought chain provided in this application embodiment; see reference. Figure 1 This application provides a large-scale question-answering method based on thought chains, which may include:

[0051] 101. Obtain the text of the subjective questions;

[0052] 102. Input the subjective question text into the question-answering model to obtain the target answer output by the question-answering model.

[0053] The target response is one that conforms to social and ethical norms;

[0054] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0055] First, traditional thought chain reasoning is a linear, sequential reasoning process. Its basic idea is to generate a series of intermediate reasoning steps through a language model, making the model more capable of solving complex tasks, especially mathematical problems, logical reasoning, and common-sense reasoning. For example, when prompted "Please think step by step," the model might output a series of textual descriptions in sequence to complete the solution process, which unfolds linearly. In this embodiment, the traditional thought chain reasoning process is extended from linear reasoning to structured reasoning, thereby significantly improving the interpretability of the reasoning process and making it possible to understand and monitor the formation and changes of social moral norms throughout the reasoning process in stages.

[0056] Secondly, by imposing budget constraints on the structured inference chain, the model's inference process can be finely controlled to achieve efficient and accurate inference within a limited computational budget, making it particularly suitable for application scenarios with limited resources or high requirements for inference efficiency.

[0057] Furthermore, by using historical subjective question texts and their corresponding historical target answer labels to train the large model, we can make full use of the autoregressive characteristics of the large model, enabling the model to continuously learn the historical target answer labels, and gradually obtain a question-answering model with the ability to accurately output answers that conform to social moral norms in the process of continuously correcting the output answers.

[0058] The method in this embodiment improves the model's controllability over inference criteria and gives the model strong generalization and adaptability, making it transferable to most large language models.

[0059] This embodiment provides a large-scale question-answering method based on thought chains. It acquires subjective question text, inputs it into a question-answering model, and obtains the target answer output by the model. This target answer is one that conforms to social moral norms. The question-answering model in this application, on the one hand, extends the reasoning process of the large model from a single output to a structured reasoning chain. This guides the model to generate supervised, structured intermediate reasoning steps, facilitating phased supervision of the formation and changes of social moral norms during the model's reasoning process, thus enabling timely intervention and adjustment. On the other hand, imposing budget constraints on the structured reasoning chain guides the model to obtain the target answer that conforms to social moral norms under limited computing resources, improving the model's reasoning efficiency and accuracy. Furthermore, by training the large model using historical subjective question texts and their corresponding historical target answer labels, it is possible to correct model answers that do not conform to social moral norms during training, allowing the large model to fully learn social moral norms. The final question-answering model can accurately output the target answer that conforms to social moral norms. In summary, the question-answering model of this application can stably output answers that conform to social moral norms when processing subjective questions, and effectively improves reasoning efficiency and accuracy during the reasoning process.

[0060] Figure 2 This is the second flowchart illustrating the large-scale question-answering method based on thought chain provided in this application embodiment; see also... Figure 2 In one embodiment, extending the reasoning process of a large model into a structured reasoning chain may include:

[0061] 201. Add structured tags to the vocabulary of the large model to obtain the tagged vocabulary;

[0062] 202. Identify multiple words in the generated sequence of the large model that match the labeled words as multiple target words;

[0063] 203. Generate nodes for all words in the generated sequence, and generate node edges based on the dependencies between word inferences, and construct a tree structure based on the nodes and node edges;

[0064] Each node includes a corresponding target triplet, which includes the node's inference stage type, the node's inference stage content, and the node's inference stage confidence score.

[0065] 204. Determine the node whose vocabulary in the reasoning stage corresponds to the target vocabulary as the starting node of the new reasoning stage to obtain the target tree;

[0066] 205. Define the target tree as a structured reasoning chain.

[0067] In step 201, to achieve structured reasoning, structured tokens are added to the model's vocabulary to indicate the stages and levels of reasoning. Let VL be the model's vocabulary. A set of structured and tagged vocabulary.

[0068] In step 202, the generation sequence of the large model It can be represented as:

[0069]

[0070] Where i = 1, ..., m, Any word in Chinese i Each is a single token.

[0071] like Then s i Once identified as a target word, this target word can indicate the start of a new stage of reasoning.

[0072] In step 203, the values ​​are s1, s2, ..., s m Generate nodes, and based on s1, s2, ..., s m The dependency relationships in the reasoning process generate node edges between any two words. Based on all nodes and node edges, a tree structure is constructed, which can be represented as:

[0073]

[0074] in, It is a set of nodes, each node Represents a reasoning unit. It is a set of nodes and edges. Each node v includes a target triple, i.e., v = (τ, s, μ), where τ represents the inference stage type of the node, such as key semantic extraction, causal analysis, decision judgment, etc., and s represents the content of the inference stage of the node, i.e., s i The corresponding natural language description is that μ represents the confidence score of the inference stage of this node.

[0075] In steps 204 to 205, that is, the s corresponding to a certain node v. Then, node v is identified as a new inference stage node. After determining all new inference stage nodes in the tree structure based on the above method, the target tree is obtained, which is the expanded structured inference chain.

[0076] In this embodiment, structured tokens are embedded during the model generation and reasoning process. These tokens are automatically inserted at key stages to mark the model's thinking stage. These tokens are similar to "semantic anchors," helping subsequent modules of the model understand the current reasoning stage. By transforming the reasoning process from a linear structure of the traditional thought chain to a tree-like structure, key semantic extraction, causal analysis, decision-making, and other stages are marked, which is closer to the "orderly division of labor" process when humans solve problems. This makes each structured stage easier to understand and interpret, and thus more effectively monitored, so that timely intervention and adjustment can be made when deviations from social moral norms occur.

[0077] Furthermore, the structured thought chain in this embodiment can be used to automatically evaluate whether each reasoning stage is reasonable, and to separately evaluate whether the classification is correct and whether the cause analysis is sufficient. In supervised fine-tuning or reinforcement learning, more granular feedback signals can be designed, and feedback functions or scores can be calculated independently at each stage to improve the quality of supervised training. At the same time, it can also facilitate multi-model collaboration, such as agent intelligence. When multiple models cooperate, the structured output can serve as a standard communication protocol or status flag.

[0078] Figure 3 This is the third flowchart of the large-scale question-answering method based on thought chain provided in the embodiments of this application; see also... Figure 3 In one embodiment, imposing a budget constraint on the structured inference chain may include:

[0079] 301. Set corresponding budget constraints for each inference branch in the structured inference chain;

[0080] Budget constraints include constraints on the number of reasoning steps, the number of words generated per reasoning step, and the number of reasoning stages;

[0081] 302. Based on budget constraints, set the reasoning reward for each reasoning branch;

[0082] 303. With the goal of maximizing the expected reward of all reasoning branches, control the large model to reason until any reasoning branch reaches the corresponding budget constraint, then control the large model to stop reasoning.

[0083] In step 301, it is assumed that any reasoning branch in the structured reasoning chain is also any tree branch in the target tree. It is expressed as follows:

[0084] b j~π(x; B);

[0085] Where π(x; B) represents the model inputs x and b. j Given a budget constraint of B, and assuming B is the constraint on the maximum number of words generated per inference step, then |b| j |≤B indicates the length of the reasoning branch, meaning the maximum number of tokens on that reasoning branch is no greater than B.

[0086] To accommodate varying input sample complexities, a dynamic budget adjustment mechanism can be introduced. This mechanism estimates the budget constraint required for each input sample using a budget prediction function g, i.e.:

[0087] B = g(x);

[0088] The function g can be a trained model or a heuristic rule used to dynamically allocate budgets based on the features of the input samples, such as length, complexity, and other evaluation dimensions.

[0089] In step 302, for b j For example, its reasoning reward R(b) j It can be set as follows:

[0090] R(b j )=r(b j )-λ·max(0,|b j |-B);

[0091] Where r(b) j ) for b j The base reward is λ, which is the budget penalty coefficient. By introducing an inference reward that includes a penalty term, the model can be guided to generate high-quality inference trajectories under budget constraints.

[0092] In step 303, within the reinforcement learning framework, the objective is to maximize the expected reward of all inference branches. This objective function can be expressed as:

[0093]

[0094] in, Let x be the data distribution.

[0095] In the process of large model inference, if |b j If | = B, then a special token indicating the end of thinking is forcibly inserted to terminate the reasoning process.

[0096] This embodiment guides the model to complete inference tasks within a preset budget range by controlling the budget of each inference branch, thereby achieving optimal allocation of computing resources. Furthermore, a dynamic budget adjustment mechanism is introduced to guide the model to evaluate the complexity of samples before inference and generate appropriate budget constraints in advance. This allows the model to adaptively increase the inference depth and improve the accuracy of inference when encountering complex problems, and avoid over-analysis when encountering simple problems, thus avoiding token waste. Therefore, this embodiment can be applied to inference tasks of different complexities. For example, it can be directly applied to large language models with a parameter scale of 7B or more, or to more complex large language models. It can significantly improve inference efficiency and accuracy for inference tasks of each complexity.

[0097] Furthermore, since a structured token is introduced when generating the structured thought chain, each step of the model reasoning is more easily embedded into the budget constraint process of this embodiment.

[0098] Figure 4 This is the fourth flowchart of the large-scale question-answering method based on thought chain provided in the embodiments of this application; see also... Figure 4 In one embodiment, the question-answering model can be trained based on the following steps:

[0099] 401. Input the historical subjective question text and its corresponding historical target answer label into the large model to obtain multiple historical reasoning answers output by multiple reasoning branches of the structured reasoning chain within the large model;

[0100] 402. Calculate the total stage loss of the structured reasoning chain and the total relative advantage of multiple historical reasoning answers;

[0101] 403. When a balance is not achieved between minimizing the total stage loss and maximizing the total relative advantage, select the target historical reasoning answer with social moral norm deviation from multiple historical reasoning answers.

[0102] 404. Update the parameters of the large model, and add subjective question text similar to the historical subjective question text corresponding to the target historical reasoning answer, then return to step 401;

[0103] 405. When a balance is achieved between minimizing the total stage loss and maximizing the total relative advantage, the large model at this point is determined to be a question-and-answer model.

[0104] Before step 401, the structured thought chain is output in JSON format so that the automated script can extract the model output results.

[0105] In step 401, an automatic information collection method can be used to obtain a certain number of historical subjective question text examples. Then, a training set and a test set are established by using a high-performance large model and a semi-automatic annotation method that manually corrects the annotations of social morality norms. It should be noted that the proportion of each question category and label category in the training set and the test set should be basically even, so as to ensure that a relatively fixed social morality norm evaluation standard is maintained between the training set and the test set, and to minimize the uncertainty of manually annotated social morality norm data.

[0106] Then, the large model is trained and tested using historical subjective question texts and their corresponding historical target answer labels from the training and testing sets.

[0107] Step 402, calculating the total stage loss of the structured inference chain, may include:

[0108] 402a. For each reasoning stage in the structured reasoning chain, calculate the loss of the historical reasoning result of the reasoning stage relative to its historical target result to obtain multiple losses;

[0109] 402b. Weighted summation of multiple losses yields the total stage loss of the structured inference chain.

[0110] In step 402a, each inference stage can be independently supervised. For each inference stage, the historical inference result corresponding to the new inference stage node v is set as follows: The historical target result is y v Then the loss of that node It can be represented as:

[0111]

[0112] Where l represents the appropriate loss function, This can represent the loss of the inference stage corresponding to v.

[0113] In step 402b, after calculating the loss of each new inference stage node v based on the above method, the total stage loss of the structured inference chain can be obtained according to the following formula.

[0114]

[0115] Where, α v The weights of the corresponding nodes reflect their importance in the entire reasoning process.

[0116] Step 402, calculating the total relative advantage of multiple historical reasoning answers, may include:

[0117] 402A. Obtain human ratings for multiple reasoning responses based on social moral norms;

[0118] 402B. Calculate the mean and variance of human ratings for multiple reasoning responses;

[0119] 402C. For each historical reasoning answer among multiple historical reasoning answers, calculate the relative advantage of that historical reasoning answer relative to other historical reasoning answers based on the mean and variance, and obtain multiple relative advantages;

[0120] 402D: Based on multiple relative advantages and KL divergence penalty, the total relative advantage of multiple historical reasoning answers is generated.

[0121] Because the social moral norm orientation expressed for the same subjective question is unstable during the reasoning process of a large model's thought chain, the generation process will have multiple different reasoning branches. The large model itself assumes that these reasoning branches and their corresponding inference results are equivalent. However, in practice, even if the large model considers these reasoning branches to be very similar in its vector space constructed based on conditional probabilities, these reasoning branches and their corresponding inference results may not be equivalent in social moral norm judgments. Therefore, a reinforcement learning method based on comparing rewards for multiple reasoning responses can be used to identify and reinforce outputs that are more in line with social moral norm judgments through the diverse reasoning paths generated by the model itself, thereby correcting the social moral norms in the model's reasoning process, as detailed below:

[0122] In steps 402A to 402C, any inference branch b can be calculated using the following formula. j The relative advantage of the t-th token in the corresponding historical reasoning answer

[0123]

[0124] in, It is b j The human score for the t-th token in the corresponding historical reasoning response is based on the degree of conformity with social moral norms. yes The average human rating of all tokens in the historical reasoning responses corresponding to all reasoning branches. yes The variance of human ratings for all tokens in the historical reasoning responses corresponding to all reasoning branches.

[0125] In step 402D, the total relative advantage J(θ) can be expressed as follows:

[0126]

[0127] in, This represents the old update strategy for generating the historical reasoning answer set O when the input is x. Let O represent the k-th historical reasoning answer among the G historical reasoning answers in set O, that is, any historical reasoning answer among the G historical reasoning answers. These G historical reasoning answers are... The historical reasoning answers corresponding to all reasoning branches in the text, |o k | represents the number of tokens in the k-th historical reasoning answer, clip(·) represents the truncation function used to control the magnitude of policy updates, and β is the KL divergence D. KL The penalty coefficient, D KL It is the divergence between the old and new update strategies, used to measure the difference between the old and new strategies, and θ is the large model parameter.

[0128] In traditional reinforcement learning frameworks, policy updates typically rely on external value function estimators. In this embodiment, when dealing with subjective questions, external estimators may fail to accurately capture judgments about social moral norms. By comparing multiple reasoning answers generated for the same subjective question within a group, the model's own generated solutions can be directly used to calculate relative advantages, thus avoiding reliance on external estimators. Furthermore, by introducing a KL divergence penalty term, the magnitude of policy updates can be effectively controlled, preventing drastic model changes and ensuring the stability of the training process.

[0129] Furthermore, by correcting the model's social morality through intra-group comparative rewards, a large model is no longer needed as a reward feedback model, greatly reducing training resource consumption.

[0130] Furthermore, by examining historical reasoning responses that exhibit deviations from social moral norms in each training iteration, and by adding similar subjective question texts to the training, the stability and consistency of the large model in handling subjective questions are further enhanced.

[0131] Furthermore, since a structured token is introduced when generating the structured thought chain, each step of the model reasoning is more easily embedded into the model training process of this embodiment.

[0132] The following describes the large model question-answering device based on the thinking chain provided in the embodiments of this application. The large model question-answering device based on the thinking chain described below can be referred to in correspondence with the large model question-answering method based on the thinking chain described above.

[0133] Figure 5 This is a schematic diagram of the structure of the large-scale question-answering device based on the thought chain provided in the embodiments of this application. (Refer to...) Figure 5 This application provides a large-scale question-answering device based on a thought chain, which may include:

[0134] Problem Acquisition Module 501 is used to: acquire subjective problem text;

[0135] The answer generation module 502 is used to: input the subjective question text into the question-answering model to obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms;

[0136] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0137] This embodiment provides a large-scale question-answering device based on a thought chain. It acquires subjective question text, inputs it into a question-answering model, and obtains the target answer output by the model. This target answer conforms to social moral norms. The question-answering model in this application, on the one hand, extends the reasoning process of the large model from a single output to a structured reasoning chain. This guides the model to generate supervised, structured intermediate reasoning steps, facilitating phased supervision of the formation and changes of social moral norms during the model's reasoning process, thus enabling timely intervention and adjustment. On the other hand, imposing budget constraints on the structured reasoning chain guides the model to obtain a target answer conforming to social moral norms under limited computing resources, improving the model's reasoning efficiency and accuracy. Furthermore, by training the large model using historical subjective question texts and their corresponding historical target answer labels, it can correct model answers that do not conform to social moral norms during training, allowing the large model to fully learn social moral norms. The final question-answering model can accurately output a target answer that conforms to social moral norms. In summary, the question-answering model of this application can stably output answers that conform to social moral norms when processing subjective questions, and effectively improves reasoning efficiency and accuracy during the reasoning process.

[0138] In one embodiment, a reasoning extension module (not shown) is further included for:

[0139] Add structured tags to the vocabulary of the large model to obtain the tagged vocabulary;

[0140] Multiple words in the generated sequence of the large model that match the labeled words are identified as multiple target words;

[0141] Nodes are generated for all words in the generated sequence, and node edges are generated based on the dependencies between word inferences. A tree structure is constructed based on the nodes and the node edges. Each node includes a corresponding target triple, which includes the inference stage type of the node, the inference stage content of the node, and the inference stage confidence score of the node.

[0142] The node whose vocabulary corresponds to the target vocabulary in the reasoning stage is determined as the starting node of the new reasoning stage, thus obtaining the target tree;

[0143] The target tree is identified as the structured reasoning chain.

[0144] In one embodiment, a budget constraint module (not shown) is further included for:

[0145] A corresponding budget constraint is set for each reasoning branch in the structured reasoning chain; the budget constraint includes the number of reasoning steps, the number of words generated per reasoning step, and the number of reasoning stages.

[0146] Based on the aforementioned budget constraints, the reasoning reward for each reasoning branch is set.

[0147] The large model is controlled to perform inference with the goal of maximizing the expected reward of all inference branches until any inference branch reaches the corresponding budget constraint, at which point the large model is controlled to stop inference.

[0148] In one embodiment, a model training module (not shown in the figure) is further included for:

[0149] Input the historical subjective question text and its corresponding historical target answer label into the large model to obtain multiple historical reasoning answers output by multiple reasoning branches of the structured reasoning chain within the large model;

[0150] Calculate the total stage loss of the structured reasoning chain and the total relative advantage of the multiple historical reasoning answers;

[0151] When a balance is not achieved between minimizing the total stage loss and maximizing the total relative advantage, a target historical reasoning answer with a deviation from social moral norms is selected from the multiple historical reasoning answers.

[0152] After updating the parameters of the large model and adding subjective question text similar to the historical subjective question text corresponding to the target historical reasoning answer, return to the step of inputting the historical subjective question text and its corresponding historical target answer label into the large model until a balance is reached between minimizing the total stage loss and maximizing the total relative advantage, and then determine the large model at this time as the question answering model.

[0153] In one embodiment, the model training module is specifically used for:

[0154] For each reasoning stage in the structured reasoning chain, the loss of the historical reasoning result of the reasoning stage relative to its historical target result is calculated to obtain multiple losses;

[0155] The total stage loss of the structured inference chain is obtained by weighted summation of the multiple losses.

[0156] In one embodiment, the model training module is specifically used for:

[0157] Obtain human ratings for the multiple reasoning responses based on social moral norms;

[0158] Calculate the mean and variance of the human ratings for the multiple reasoning responses;

[0159] For each of the multiple historical reasoning answers, based on the mean and variance, the relative advantage of the historical reasoning answer relative to the other historical reasoning answers is calculated to obtain multiple relative advantages;

[0160] Based on the multiple relative advantages and KL divergence penalty, the total relative advantage of the multiple historical reasoning answers is generated.

[0161] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call the computer program in the memory 630 to execute the steps of the large-scale question-answering method based on thought chains, such as:

[0162] Obtain the text of the subjective question;

[0163] The subjective question text is input into the question-answering model to obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms.

[0164] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0165] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the large-scale question-answering method based on thought chains provided in the above embodiments, such as including:

[0167] Obtain the text of the subjective question;

[0168] The subjective question text is input into the question-answering model to obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms.

[0169] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0170] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program thereon, the computer program being used to cause a processor to execute the steps of the large-scale question-answering method based on thought chains provided in the above embodiments, for example including:

[0171] Obtain the text of the subjective question;

[0172] The subjective question text is input into the question-answering model to obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms.

[0173] The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

[0174] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A large-scale question-answering method based on thought chains, characterized in that, include: Obtain the text of the subjective question; Input the subjective question text into the question-answering model to obtain the target answer output by the question-answering model; The target answer is one that conforms to social and ethical norms; The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

2. The large-scale question-answering method based on thought chain as described in claim 1, characterized in that, The extension of the reasoning process of the large model into a structured reasoning chain includes: Add structured tags to the vocabulary of the large model to obtain the tagged vocabulary; Multiple words in the generated sequence of the large model that match the marked words are identified as multiple target words; Nodes are generated for all words in the generated sequence, and node edges are generated based on the dependencies between word inferences. A tree structure is constructed based on the nodes and the node edges. Each node includes a corresponding target triple, which includes the inference stage type of the node, the inference stage content of the node, and the inference stage confidence score of the node. The node whose vocabulary corresponds to the target vocabulary in the reasoning stage is determined as the starting node of the new reasoning stage, thus obtaining the target tree; The target tree is identified as the structured reasoning chain.

3. The large-scale question-answering method based on thought chain as described in claim 1, characterized in that, The imposition of budget constraints on the structured inference chain includes: A corresponding budget constraint is set for each reasoning branch in the structured reasoning chain; the budget constraint includes the number of reasoning steps, the number of words generated per reasoning step, and the number of reasoning stages. Based on the aforementioned budget constraints, the reasoning reward for each reasoning branch is set. The large model is controlled to perform inference with the goal of maximizing the expected reward of all inference branches until any inference branch reaches the corresponding budget constraint, at which point the large model is controlled to stop inference.

4. The large-scale question-answering method based on thought chain as described in claim 1, characterized in that, The question-answering model is trained based on the following steps: Input the historical subjective question text and its corresponding historical target answer label into the large model to obtain multiple historical reasoning answers output by multiple reasoning branches of the structured reasoning chain within the large model; Calculate the total stage loss of the structured reasoning chain and the total relative advantage of the multiple historical reasoning answers; When a balance is not achieved between minimizing the total stage loss and maximizing the total relative advantage, a target historical reasoning answer with a deviation from social moral norms is selected from the multiple historical reasoning answers. After updating the parameters of the large model and adding subjective question text similar to the historical subjective question text corresponding to the target historical reasoning answer, return to the step of inputting the historical subjective question text and its corresponding historical target answer label into the large model until a balance is reached between minimizing the total stage loss and maximizing the total relative advantage, and then determine the large model at this time as the question answering model.

5. The large-scale question-answering method based on thought chain as described in claim 4, characterized in that, Calculating the total stage loss of the structured inference chain includes: For each reasoning stage in the structured reasoning chain, the loss of the historical reasoning result of the reasoning stage relative to its historical target result is calculated to obtain multiple losses; The total stage loss of the structured inference chain is obtained by weighted summation of the multiple losses.

6. The large-scale question-answering method based on thought chain as described in claim 4, characterized in that, Calculating the total relative advantage of the multiple historical reasoning answers includes: Obtain human ratings for the multiple reasoning responses based on social moral norms; Calculate the mean and variance of the human ratings for the multiple reasoning responses; For each of the multiple historical reasoning answers, based on the mean and variance, the relative advantage of the historical reasoning answer relative to the other historical reasoning answers is calculated to obtain multiple relative advantages; Based on the multiple relative advantages and KL divergence penalty, the total relative advantage of the multiple historical reasoning answers is generated.

7. A large-scale question-and-answer device based on thought chain, characterized in that, include: The question acquisition module is used to: acquire subjective question text; The answer generation module is used to: input the subjective question text into the question-answering model and obtain the target answer output by the question-answering model; the target answer is an answer that conforms to social moral norms; The question-answering model is obtained by extending the reasoning process of the large model into a structured reasoning chain and imposing budget constraints on the structured reasoning chain, and by training the large model using historical subjective question texts and their corresponding historical target answer labels.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the large model question-answering method based on the thought chain as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the large model question-answering method based on the thought chain as described in any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the large model question-answering method based on the thought chain as described in any one of claims 1 to 6.

Images