Chain reasoning method, system and equipment based on enhanced feedback and medium

Abstract

The invention belongs to the technical field of chained reasoning, and particularly relates to a chained reasoning method, system and equipment based on teacher model enhanced feedback and a medium. In order to overcome the defects of poor accuracy and reliability of an existing chain reasoning model fusing multi-modal information, the invention adopts the following technical scheme: the chain reasoning method based on teacher model enhanced feedback comprises the following steps: acquiring a multi-modal data set for model training; modeling the chain reasoning process of each sample problem in the multi-modal data set; the reasoning ability of the chain reasoning model is optimized through a teacher model GPT-4; on the basis of the model after reinforcement learning optimization, a teacher model GPT-4 is used for correcting errors or incomplete parts in an inference chain, then supervision fine tuning is carried out, the performance of the model is further stabilized, and finally an enhanced multi-mode chain type inference model is formed. According to the method provided by the invention, high stability and accuracy can be kept when a complex problem is reasoned.

Description

Technical Field

[0001] The present invention belongs to the technical field of chain reasoning, and in particular relates to a chain reasoning method, system, device and medium based on reinforced feedback. Background Art

[0002] With the rapid development of artificial intelligence, multimodal data processing (such as text and images) has become an important research direction for intelligent systems. In many applications, such as scientific question answering and knowledge graph construction, models are required to be able to process information from multiple modalities simultaneously and generate logically coherent reasoning steps through chain reasoning. However, due to the complexity of multimodal data and the asymmetry of information, existing models often find it difficult to effectively integrate multimodal information during the reasoning process, and therefore fail to generate high-quality reasoning chains.

[0003] At present, the common methods for dealing with multimodal chain reasoning mainly rely on pre-trained deep learning models to derive the final answer by directly generating reasoning steps. Although these methods can solve the problem of multimodal information fusion to a certain extent, they still have the following obvious shortcomings: First, existing models often lack fine supervision of the reasoning process, and are prone to generating logically incoherent or erroneous steps in the reasoning chain; second, the model lacks an effective error correction mechanism, resulting in subsequent steps being difficult to correct once a reasoning error occurs, ultimately affecting the accuracy and reliability of the overall reasoning. In addition, the correlation between multimodal features is not fully utilized in existing models, which further limits the performance of the model in complex tasks. Summary of the invention

[0004] The present invention aims at the shortcomings of poor accuracy and reliability of the existing chain reasoning model integrating multimodal information, and provides a chain reasoning method based on enhanced feedback to improve the accuracy and reliability of the chain reasoning model when processing multimodal information. The present invention also provides a system, a computer device and a computer-readable storage medium based on enhanced feedback.

[0005] To achieve the above object, the present invention adopts the following technical solution: a chain reasoning method based on teacher model reinforcement feedback, the chain reasoning method based on teacher model reinforcement feedback includes:

[0006] Step S1, obtaining a multimodal dataset for model training, wherein the multimodal dataset includes a plurality of sample questions, each sample question is equipped with text modality information, image modality information and a plurality of answer options, and an algorithm objective is defined;

[0007] Step S2, modeling the chain reasoning process of each sample question in the multimodal data set, using the chain reasoning model to reason about the question, and obtaining an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features;

[0008] Step S3: Optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, so that it is more inclined to generate high-scoring reasoning steps;

[0009] Step S4: Based on the model optimized by reinforcement learning, the teacher model GPT-4 is used to correct errors or incomplete parts in the reasoning chain, followed by supervised fine-tuning to further stabilize the model performance and finally form an enhanced multimodal chain reasoning model.

[0010] The chain reasoning method based on teacher model reinforcement feedback of the present invention combines multimodal feature representation with teacher model scoring mechanism, which can effectively improve the generation quality of the reasoning chain; the combination of reinforcement learning and supervised fine-tuning enables the model to maintain high stability and accuracy when reasoning complex problems. The method of the present invention has good application prospects in multimodal tasks such as intelligent question answering and complex reasoning.

[0011] As an improvement, the algorithm goal is defined as: when a sample problem q is given and the corresponding multimodal information c = {c text ,c image} and answer option set o = {o 1 ,o 2 ,…,o K}, generating chained reasoning steps r = {r 1 ,r 2 ,…,r P}, and finally select the best answer in the answer option set as model output.

[0012] As an improvement, in step S2,

[0013] The text of the sample problem c text Input the pre-trained text encoder and extract the feature representation F of the text text ={f text1 ,f text2 ,…,f textM};

[0014] The image c of the sample problem image Input the pre-trained image encoder to get the image feature representation F image ={f image1 ,f image2 ,…,f imageM};

[0015] The extracted multimodal features F text and F image Input chain reasoning generation model, chain reasoning generation model generates initial reasoning chain R = {r 1 ,r2 ,…,r P}.

[0016] As an improvement, step S3 includes:

[0017] Step S31: according to the currently generated inference chain step r t = {r 1 ,r 2 ,…,r t}, use the teacher model GPT-4 to score each reasoning step, and the scoring result is represented as s t ={s 1 ,s 2 ,…,s t};

[0018] Step S32: According to the obtained scoring result s t , optimize the model’s strategy through reinforcement learning algorithms;

[0019] Step S33: During the optimization process, the model parameters are updated using the proximal strategy optimization algorithm so that the generated reasoning chain steps are more likely to obtain high scores.

[0020] As an improvement, in step S32, the reward function is defined as the cumulative score of each step:

[0021]

[0022] The optimization goal is to maximize this cumulative score R t , the corresponding loss function is:

[0023]

[0024] Among them, π θ is the policy function of the model, θ is the parameter of the model, In the strategy π θ The expected value below.

[0025] As an improvement, in step S33, the optimization process of the proximal strategy optimization algorithm is expressed as:

[0026]

[0027] in, is the advantage function, which is used to measure the improvement of the current strategy relative to the benchmark strategy. To make The largest parameter θ;

[0028] Through iterative optimization, the chain reasoning model gradually generates a reasoning chain that better meets the scoring criteria, and finally selects the reasoning step with the highest score to derive the answer.

[0029] As an improvement, step S4 includes:

[0030] Step S41: Based on the model optimized by reinforcement learning, the teacher model GPT-4 checks each step in the generated reasoning chain, identifies the wrong or incomplete parts, and generates corresponding correction suggestions c = {c 1 ,c 2 ,…,c P};

[0031] Step S42: The obtained correction suggestion c p Combined with the initial reasoning chain, a revised reasoning chain r′=r′ is generated 1 ,r′ 2 ,…,r′ P ;

[0032] Step S43: Perform supervised fine-tuning on the corrected inference chain r′ and define the loss function L SFT To correct the difference between the previous and next reasoning chains:

[0033]

[0034] Among them, θ is the parameter of the model,

[0035] Step S44: Minimize the loss function L through the back propagation algorithm SFT , update the model parameters:

[0036]

[0037] Among them, η is the learning rate, θ represents the parameters of the current model, and θ new represents the updated parameters, Represents the loss function L under the current parameter θ SFT The gradient of θ;

[0038] Step S45: After several rounds of iterative training, an enhanced multimodal chain reasoning model is finally formed, so that it has stronger reasoning ability and stability when processing complex multimodal tasks.

[0039] The chain reasoning system based on teacher model reinforcement feedback comprises:

[0040] A data acquisition module, used to acquire a multimodal data set for model training, wherein the multimodal data set includes a plurality of sample questions, each of which is equipped with text modality information, image modality information, and a plurality of answer options;

[0041] Algorithm target module, used to define algorithm targets;

[0042] An initial reasoning module, used to model the chain reasoning process of each sample question in the multimodal dataset, use the chain reasoning model to reason about the question, and obtain an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features;

[0043] The optimization module is used to optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, making it more inclined to generate high-scoring reasoning steps;

[0044] The correction module is used to correct errors or incomplete parts in the reasoning chain using the teacher model GPT-4 based on the model optimized by reinforcement learning;

[0045] The supervised fine-tuning module is used to perform supervised fine-tuning to further stabilize the model performance and ultimately form an enhanced multimodal chain reasoning model.

[0046] The computer device includes a processor and a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by the processor, the aforementioned chain reasoning method based on teacher model reinforcement feedback is implemented.

[0047] A computer-readable storage medium having a computer program stored thereon, which, when executed, implements the aforementioned chain reasoning method based on teacher model reinforcement feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a principle diagram of the chain reasoning method based on teacher model reinforcement feedback in an embodiment of the present invention.

[0049] Figure 2 It is a sample problem in the multimodal dataset of an embodiment of the present invention.

[0050] Figure 3 The method of the embodiment of the present invention is used to Figure 2 Reasoning about the problem generates an initial chain of reasoning.

[0051] Figure 4 The method of the embodiment of the present invention is used to Figure 3 Scoring of the initial chain of reasoning.

[0052] Figure 5 The method of the embodiment of the present invention is used to Figure 3 The inference answer obtained after correction and supervised fine-tuning of the initial inference chain.

[0053] Figure 6 yes Figure 2 Reference reasoning answers to the questions in . DETAILED DESCRIPTION

[0054] The technical solutions of the embodiments of the present invention are explained and described below, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0055] See also Figure 1 and Figure 2 , the chain reasoning method based on teacher model reinforcement feedback of an embodiment of the present invention, the chain reasoning method based on teacher model reinforcement feedback includes:

[0056] Step S1, obtaining a multimodal dataset for model training, wherein the multimodal dataset includes a plurality of sample questions, each sample question is equipped with text modality information, image modality information and a plurality of answer options, and an algorithm objective is defined;

[0057] Step S2, modeling the chain reasoning process of each sample question in the multimodal data set, using the chain reasoning model to reason about the question, and obtaining an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features;

[0058] Step S3: Optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, so that it is more inclined to generate high-scoring reasoning steps;

[0059] Step S4: Based on the model optimized by reinforcement learning, the teacher model GPT-4 is used to correct errors or incomplete parts in the reasoning chain, followed by supervised fine-tuning to further stabilize the model performance and finally form an enhanced multimodal chain reasoning model.

[0060] In this embodiment, the algorithm goal is defined as: when a sample question q is given and the corresponding multimodal information c={c text ,c image} and answer option set o = {o 1 ,o 2 ,…,o K}, generating chained reasoning steps r = {r 1 ,r 2 ,…,r P}, and finally select the best answer in the answer option set As the model output. Where q represents the problem description, c text Represents language modality context information, c image Represents the visual modality context information; o K represents the Kth answer option, K represents the total number of options; r p represents the pth reasoning step in the chain of reasoning steps, and P represents the total number of reasoning steps generated; The answer option selected for the final model is one of the answer options.

[0061] In this embodiment, in step S2, the text c of the sample question is text Input the pre-trained text encoder and extract the feature representation F of the text text ={f text1 ,f text2 ,…,f textM};

[0062] The image c of the sample problem image Input the pre-trained image encoder to get the image feature representation F image ={f image1 ,f image2 ,…,f imageM};

[0063] The extracted multimodal features F text and F image Input chain reasoning generation model, chain reasoning generation model generates initial reasoning chain R = {r 1 ,r 2 ,…,r P}. Among them, f textM Refers to the feature representation of the Mth text context, f imageM Refers to the feature representation of the Mth image context, r p represents the p-th reasoning step generated.

[0064] In this embodiment, step S3 includes:

[0065] Step S31: according to the currently generated inference chain step r t = {r 1 ,r 2 ,…,r t}, use the teacher model GPT-4 to score each reasoning step, and the scoring result is represented as s t ={s 1 ,s 2 ,…,s t};

[0066] Step S32: According to the obtained scoring result s t , optimize the model’s strategy through reinforcement learning algorithms;

[0067] Step S33: During the optimization process, the proximal policy optimization algorithm (PPO for short, developed by the OpenAI research team, aims to solve the problem that the traditional policy gradient method may cause excessive policy updates during training, resulting in unstable training. PPO limits the amplitude of policy updates to ensure that the new policy is close to the old policy, thereby maintaining the stability of training) is used to update the model parameters so that the generated reasoning chain steps are more inclined to obtain high scores.

[0068] In this embodiment, in step S32, the reward function is defined as the cumulative score of each step:

[0069]

[0070] The optimization goal is to maximize this cumulative score R t , the corresponding loss function is:

[0071]

[0072] Among them, π θ is the policy function of the model, θ is the parameter of the model, In the strategy π θ The expected value below.

[0073] In this embodiment, in step S33, the optimization process of the proximal strategy optimization algorithm is expressed as:

[0074]

[0075] in, is the advantage function, which is used to measure the improvement of the current strategy relative to the benchmark strategy. To make The largest parameter θ;

[0076] Through iterative optimization, the chain reasoning model gradually generates a reasoning chain that better meets the scoring criteria, and finally selects the reasoning step with the highest score to derive the answer.

[0077] In this embodiment, step S4 includes:

[0078] Step S41: Based on the model optimized by reinforcement learning, the teacher model GPT-4 checks each step in the generated reasoning chain, identifies the wrong or incomplete parts, and generates corresponding correction suggestions c = {c 1 ,c 2 ,…,c P};

[0079] Step S42: The obtained correction suggestion c pCombined with the initial reasoning chain, a revised reasoning chain r′=r′ is generated 1 ,r′ 2 ,…,r′ P ;

[0080] Step S43: Perform supervised fine-tuning on the corrected inference chain r′ and define the loss function L SFT To correct the difference between the previous and next reasoning chains:

[0081]

[0082] Among them, θ is the parameter of the model,

[0083] Step S44: Minimize the loss function L through the back propagation algorithm SFT , update the model parameters:

[0084]

[0085] Among them, η is the learning rate, θ represents the parameters of the current model, and θ new represents the updated parameters, Represents the loss function L under the current parameter θ sFT The gradient of θ;

[0086] Step S45: After several rounds of iterative training, an enhanced multimodal chain reasoning model is finally formed, so that it has stronger reasoning ability and stability when processing complex multimodal tasks.

[0087] The method of the embodiment of the present invention is described below in conjunction with specific sample problems.

[0088] Prepare the visual feature extraction model ViT, the language feature extraction model T5, the inference generation model Flan-AlpaceBase, and the teacher model GPT-4 in advance. Vision Transformer is an innovative architecture that introduces the Transformer model in natural language processing into the field of computer vision. It was proposed by Alexey Dosovitskiy and others from Google in 2021. The VIT model divides the image into several patches of fixed size, each of which is regarded as a "word", and then converts these patches into positional encodings through a linear projection layer, and finally inputs them into a model composed of multiple layers of Transformer encoders for processing, and finally obtains the classification result through a multi-layer perceptron (MLP). The language feature extraction model T5 (Text-To-Text Transfer Transformer) is a pre-trained language model for natural language processing proposed by the Google research team. The core concept of the T5 model is to unify all NLP tasks into a text-to-text format, that is, whether it is text classification, translation, summary generation, or question-answering tasks, the input and output are represented as text sequences. The teacher model GPT-4 is a language model released by OpenAI for the chatbot ChatGPT.

[0089] See also Figure 2 Suppose that in one of the sample questions, the text information is "The baby wants to know what is in the cabinet, her hand exerts a force on the door, and then the door opens", the picture information can be simply described as "a baby and a slightly open cabinet", the question is "What kind of force does the baby's hand use to open the cabinet door?", the options are "(A) pull (B) push", and the answer is "(A) pull".

[0090] Initial reasoning chain generation: The VIT model is used to extract image features, the T5 model is used to extract text features, and the image features and text features are input into the Flan-Alpace Base model to obtain Figure 3 Output.

[0091] Teacher model GPT-4 feedback scoring: Use the reinforcement learning PPO strategy to optimize the reasoning chain generation model Flan-AlpaceBase to generate high-scoring reasoning chains. See the scoring results for more information. Figure 4 .

[0092] Teacher model GPT-4 error correction: The reasoning chain after error correction is as follows Figure 5 Specifically, the reasoning chain generation model Flan-Alpace Base is fine-tuned using the corrected data, and the corrected reasoning chain is used as the target. The difference between the model-generated reasoning chain and the reference answer is calculated. The reasoning chain generated by the model, "the baby's hand touches the cabinet door handle", is encoded as r1 , "hand moves inwards and the door opens slowly" is coded as r 2 , "The baby's hand then released and the door opened completely" was coded as r 3 , “The direction of the force indicated that the infant used pushing” was coded as r 4 , the reasoning chain after the model error correction "the baby's hand touched the cabinet door handle" is encoded as r' 1 , “The hand moves inwards and the door opens slowly” is coded as r′ 2 , "the direction of the force is consistent with the pulling action" is encoded as r′ 3 , calculate the difference as the loss function, the specific formula is as follows:

[0093] L sFT (θ)=||r′ 1 -r 1 || 2 +||r′ 2 -r 2 || 2 +||r′ 3 -r 3 || 2 +‖r 4 ‖ 2 (8)

[0094] Use the back-propagation algorithm to calculate the gradient and update the parameters of the Flan-AlpaceBase model:

[0095]

[0096] Among them, θ is the model parameter and η is the learning rate. The fine-tuned inference chain generation model Flan-Alpace Base is used to generate the inference chain, and we get Figure 6 Content shown.

[0097] The chain reasoning method based on teacher model reinforcement feedback of the embodiment of the present invention is experimentally verified on the ScienceQA dataset. ScienceQA is a large-scale, multimodal scientific dataset used to evaluate the multi-step reasoning ability of the model. ScienceQA contains multiple categories, namely natural sciences, social sciences, language sciences, text contexts, image contexts, no context, levels 1 to 6, and levels 7 to 12. ScienceQA includes a chain of reasoning to get the final answer. Each question includes a question, a correct answer, and an annotated explanation or solution chain. The method of the embodiment of the present invention is compared with manual answers and the Multimodal-CoT model, and the results are shown in the following table.

[0098] Model Human Answer MM-CoT Embodiments of the present invention Natural Sciences 90.23 84.59 88.04 Social Sciences 84.97 92.46 93.02 Language Sciences 87.48 83.45 83.48 Text Context 89.60 83.87 84.03 Image Context 87.50 83.29 83.67 No context 88.10 85.64 89.07 Level 1 to 6 91.59 86.34 89.00 Level 7 to 12 82.42 85.23 87.79 Mean 88.40 85.95 88.57

[0099] It can be seen from the above table that the method of the embodiment of the present invention shows superior performance in most tasks, especially surpassing human answers in social science and context-free tasks, demonstrating its innovative advantages and adaptability in multimodal chain reasoning tasks.

[0100] The beneficial effects of the chain reasoning method based on teacher model reinforcement feedback of the embodiment of the present invention are: combining multimodal feature representation with the teacher model scoring mechanism, it can effectively improve the generation quality of the reasoning chain; the combination of reinforcement learning and supervised fine-tuning enables the model to maintain high stability and accuracy when reasoning complex problems; the VIT model is used to extract image features, the T5 model is used to extract text features, the Flan-Alpace Base is used as the chain reasoning model, the teacher model GPT-4 is used for reinforcement feedback, and PPO is used for optimization; the method of the embodiment of the present invention has good application prospects in multimodal tasks such as intelligent question answering and complex reasoning.

[0101] The embodiment of the present invention also discloses a chain reasoning system based on teacher model reinforcement feedback, and the chain reasoning system based on teacher model reinforcement feedback includes:

[0102] A data acquisition module, used to acquire a multimodal data set for model training, wherein the multimodal data set includes a plurality of sample questions, each of which is equipped with text modality information, image modality information, and a plurality of answer options;

[0103] Algorithm target module, used to define algorithm targets;

[0104] An initial reasoning module, used to model the chain reasoning process of each sample question in the multimodal dataset, use the chain reasoning model to reason about the question, and obtain an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features;

[0105] The optimization module is used to optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, making it more inclined to generate high-scoring reasoning steps;

[0106] The correction module is used to correct errors or incomplete parts in the reasoning chain using the teacher model GPT-4 based on the model optimized by reinforcement learning;

[0107] The supervised fine-tuning module is used to perform supervised fine-tuning to further stabilize the model performance and ultimately form an enhanced multimodal chain reasoning model.

[0108] The embodiment of the present invention also discloses a computer device, including a processor and a storage medium, wherein a computer program is stored in the storage medium, and when the computer program is executed by the processor, the aforementioned chain reasoning method based on teacher model reinforcement feedback is implemented.

[0109] The embodiment of the present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed, the aforementioned chain reasoning method based on teacher model reinforcement feedback is implemented.

[0110] The above is only a specific implementation of the invention, but the protection scope of the invention is not limited thereto. Those skilled in the art should understand that the invention includes but is not limited to the contents described in the above specific implementation. Any modification that does not deviate from the functional and structural principles of the invention will be included in the scope of the claims.

Claims

1. A chain reasoning method based on teacher model reinforcement feedback, characterized by: The chain reasoning method based on teacher model reinforcement feedback includes: Step S1, obtaining a multimodal dataset for model training, wherein the multimodal dataset includes a plurality of sample questions, each sample question is equipped with text modality information, image modality information and a plurality of answer options, and an algorithm objective is defined; Step S2, modeling the chain reasoning process of each sample question in the multimodal data set, using the chain reasoning model to reason about the question, and obtaining an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features; Step S3: Optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, so that it is more inclined to generate high-scoring reasoning steps; Step S4: Based on the model optimized by reinforcement learning, the teacher model GPT-4 is used to correct errors or incomplete parts in the reasoning chain, followed by supervised fine-tuning to finally form an enhanced multimodal chain reasoning model.

2. The chain reasoning method based on teacher model reinforcement feedback according to claim 1 is characterized in that: The algorithm goal is defined as: given a sample problem q, the corresponding multimodal information c = {c text ,c image } and answer option set o = {o1,o2,…,o K }, generating chained reasoning steps r = {r1, r2, ..., r P }, and finally select the best answer in the answer option set As the model output, c text Indicates text modal information, c image Represents image modality information.

3. The chain reasoning method based on teacher model reinforcement feedback according to claim 2 is characterized in that: In step S2, the text c of the sample question is text Input the pre-trained text encoder and extract the feature representation F of the text text ={f text1 ,f text2 ,…,f textM }; The image c of the sample problem image Input the pre-trained image encoder to get the image feature representation F image ={f image1 ,f image2 ,…,f imageM }; The extracted multimodal features F text and F image Input chain reasoning generation model, chain reasoning generation model generates initial reasoning chain R = {r1,r2,…,r P }.

4. The chain reasoning method based on teacher model reinforcement feedback according to claim 3 is characterized by: Step S3 include: Step S31: according to the currently generated inference chain step r t ={r1,r2,…,r t }, use the teacher model GPT-4 to score each reasoning step, and the scoring result is represented as s t ={s1,s2,…,s t }; Step S32: According to the obtained scoring result s t , optimize the model’s strategy through reinforcement learning algorithms; Step S33: During the optimization process, the model parameters are updated using the proximal strategy optimization algorithm so that the generated reasoning chain steps are more likely to obtain high scores.

5. The chain reasoning method based on teacher model reinforcement feedback according to claim 4 is characterized in that: In step S32, the reward function is defined as the cumulative score of each step: The optimization goal is to maximize this cumulative score R t , the corresponding loss function is: Among them, π θ is the policy function of the model, θ is the parameter of the model, In the strategy π θ The expected value below.

6. The chain reasoning method based on teacher model reinforcement feedback according to claim 5 is characterized in that: In step S33, the optimization process of the proximal strategy optimization algorithm is expressed as: in, is the advantage function, which is used to measure the improvement of the current strategy relative to the benchmark strategy. To make The maximum parameter θ; Through iterative optimization, the chain reasoning model gradually generates a reasoning chain that better meets the scoring criteria, and finally selects the reasoning step with the highest score to derive the answer.

7. The chain reasoning method based on teacher model reinforcement feedback according to claim 6 is characterized in that: Step S4 includes: Step S41: Based on the model optimized by reinforcement learning, the teacher model GPT-4 checks each step in the generated reasoning chain, identifies the wrong or incomplete parts, and generates corresponding correction suggestions c = {c1, c2, ..., c P }; Step S42: The obtained correction suggestion c p Combined with the initial reasoning chain, a revised reasoning chain r is generated ′ =r ′ 1,r ′ 2,…,r ′ P ; Step S43: Correct the inference chain r ′ Perform supervised fine-tuning and define the loss function L SFT To correct the difference between the previous and next reasoning chains: Among them, θ is the parameter of the model; Step S44: Minimize the loss function L through the back propagation algorithm SFT , update the model parameters: Among them, η is the learning rate, θ represents the parameters of the current model, and θ new represents the updated parameters, Represents the loss function L under the current parameter θ SFT The gradient of θ; Step S45: After several rounds of iterative training, an enhanced multimodal chain reasoning model is finally formed, so that it has stronger reasoning ability and stability when processing complex multimodal tasks.

8. A chained reasoning system based on teacher model reinforcement feedback, characterized by: The chain reasoning system based on teacher model reinforcement feedback includes: A data acquisition module, used to acquire a multimodal data set for model training, wherein the multimodal data set includes a plurality of sample questions, each of which is equipped with text modality information, image modality information, and a plurality of answer options; Algorithm target module, used to define algorithm targets; An initial reasoning module, used to model the chain reasoning process of each sample question in the multimodal dataset, use the chain reasoning model to reason about the question, and obtain an initial reasoning chain of the sample question, wherein the chain reasoning process includes reasoning steps generated based on text and image features; The optimization module is used to optimize the reasoning ability of the chain reasoning model through the teacher model GPT-4, making it more inclined to generate high-scoring reasoning steps; The correction module is used to correct errors or incomplete parts in the reasoning chain using the teacher model GPT-4 based on the model optimized by reinforcement learning; The supervised fine-tuning module is used to perform supervised fine-tuning to further stabilize the model performance and ultimately form an enhanced multimodal chain reasoning model.

9. A computer device comprising a processor and a storage medium, wherein the storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, the chain reasoning method based on teacher model reinforcement feedback as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the chain reasoning method based on teacher model reinforced feedback as described in any one of claims 1 to 7 is implemented.

Images