Sample construction method and device, large model training method and device, electronic equipment and medium

Abstract

The invention provides a sample construction method, a large model training method and device, electronic equipment and a medium, and relates to the fields of computer vision, deep learning, large models and the like. According to the specific implementation scheme, the method comprises the steps of determining an initial reasoning chain for answering a question text according to an associated image and the question text; performing scrambling processing on the initial reasoning chain to obtain a scrambled reasoning chain meeting a predetermined abnormal condition; based on the large model, according to the initial reasoning chain and the scrambled reasoning chain, a target reasoning chain is determined, the target reasoning chain comprises a reference text and the scrambled reasoning chain, and the reference text is used for describing abnormal information in the scrambled reasoning chain; and constructing a training sample according to the image, the problem text and the target reasoning chain.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, deep learning, and large models. More specifically, the present disclosure provides a sample construction method, a multimodal large model training method, a sample construction device, a multimodal large model training device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the development of artificial intelligence, more and more application scenarios will use multimodal large models to perform tasks, such as using multimodal large models to perform tasks such as question answering and translation. It is understandable that improving the reasoning ability of multimodal large models can further enhance their application prospects. Summary of the invention

[0003] The present disclosure provides a sample construction method, a training method for a multimodal large model, a sample construction device, a training device for a multimodal large model, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a sample construction method is provided, including: determining an initial reasoning chain for answering a question text based on an associated image and a question text; scrambling the initial reasoning chain to obtain a scrambled reasoning chain that satisfies a predetermined abnormal condition; determining a target reasoning chain based on a large model and the initial reasoning chain and the scrambled reasoning chain, the target reasoning chain including a reference text and the scrambled reasoning chain, the reference text being used to describe abnormal information in the scrambled reasoning chain; and constructing a training sample based on the image, the question text and the target reasoning chain.

[0005] According to another aspect of the present disclosure, a method for training a multimodal large model is provided, comprising: obtaining training samples, the training samples comprising images, question texts and target reasoning chains; and training the multimodal large model using the training samples; wherein the training samples are obtained using the above-mentioned sample construction method.

[0006] According to another aspect of the present disclosure, a sample construction device is provided, including: an initial reasoning chain determination module, a scrambling module, a target reasoning chain determination module and a construction module. The initial reasoning chain determination module is used to determine the initial reasoning chain for answering the question text according to the associated image and question text. The scrambling module is used to scramble the initial reasoning chain to obtain a scrambled reasoning chain that meets a predetermined abnormal condition. The target reasoning chain determination module is used to determine the target reasoning chain based on the initial reasoning chain and the scrambled reasoning chain based on a large model, the target reasoning chain includes a reference text and the scrambled reasoning chain, and the reference text is used to describe the abnormal information in the scrambled reasoning chain. The construction module is used to construct a training sample according to the image, the question text and the target reasoning chain.

[0007] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, comprising: an acquisition module and a training module. The acquisition module is used to acquire training samples, and the training samples include images, question texts, and target reasoning chains. The training module is used to train the multimodal large model using the training samples. The training samples are constructed by the above-mentioned sample construction module.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 It is a schematic diagram of an application scenario of a sample construction method, a multimodal large model training method and a device according to an embodiment of the present disclosure;

[0014] Figure 2 is a schematic flow chart of a sample construction method according to an embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a sample construction method according to an embodiment of the present disclosure;

[0016] Figure 4 is a schematic flow chart of a training method for a multimodal large model according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic structural block diagram of a sample construction device according to an embodiment of the present disclosure;

[0018] Figure 6is a schematic structural block diagram of a training device for a multimodal large model according to an embodiment of the present disclosure; and

[0019] Figure 7 It is a structural block diagram of an electronic device used to implement the sample construction method and the multimodal large model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0021] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0022] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0023] Training a large multimodal model requires the use of multimodal training samples. One type of training sample includes images, question texts, and answer texts. The answer texts are relatively simple, for example, the answer texts only contain the answers, or they may also include some simple reasoning chains. Such simple reasoning chains generally only show a direct path from the question to the answer.

[0024] Since the training samples lack information such as error recognition, reflection, and error correction, using this type of training samples to train multimodal large models will result in the multimodal large models lacking the ability to perform complex reasoning. For example, multimodal large models lack the ability to recognize errors and are less sensitive to potential contradictions or errors in reasoning. For another example, multimodal large models lack reflection and correction, and when faced with complex tasks, multimodal large models lack a mechanism to reflect on and correct errors. For another example, multimodal large models have limited logical reasoning capabilities and perform poorly on tasks that require in-depth reasoning (such as solving science problems or multi-step complex question-answering).

[0025] In summary, the lack of training samples will result in poor reasoning capabilities of multimodal large models in error recognition, reflection correction, and multi-path reasoning. The above defects of multimodal large models are particularly obvious in dealing with tasks in specific fields (such as educational assessment and scientific problem solving), which limits the application potential of multimodal large models in high-level reasoning tasks.

[0026] The disclosed embodiment aims to provide a sample construction method, which can construct a training sample including a target reasoning chain, and the target reasoning chain includes a scrambled reasoning chain of predetermined abnormal conditions, and the scrambled reasoning chain may, for example, contain some errors, inconsistent with the image or text, or incomplete information. At the same time, the target reasoning chain also includes a reference text, and the reference text may, for example, explain the abnormal information in the scrambled reasoning chain, so that the target thinking chain in the training sample is a long thinking chain with thinking, error recognition, reflection, and correction characteristics. The deeper reasoning steps in the long thinking chain can improve the comprehensive reasoning ability of the multimodal large model. Therefore, using the training sample to train the multimodal large model can improve the reasoning ability of the multimodal large model in multimodal understanding, such as improving the error recognition, reflection, correction and other capabilities of the multimodal large model.

[0027] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 It is a schematic diagram of application scenarios of the sample construction method, multimodal large model training method and device according to the embodiments of the present disclosure.

[0029] It should be noted that Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptops, desktop computers, etc.

[0032] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results to the terminal device.

[0033] The system architecture 100 may include a database 106 , which may store pre-collected image and question text data pairs, and may also store constructed training samples.

[0034] For example, in the sample generation stage, the user can input the data pair of the image and the question text through the terminal devices 101, 102, 103, or obtain the data pair of the image and the question text stored in advance from the database 106. The server 105 can deploy a large model, which can include a large language model, a multimodal large model, etc. The image and the question text can be processed based on the large model to generate a target reasoning chain, and then a training sample is constructed, and the training sample can also be stored in the database 106.

[0035] For example, in the model training phase, the server 105 can obtain pre-built training samples from the database 106, and then train the multimodal large model to be trained until the multimodal large model converges. After obtaining the trained multimodal large model, the multimodal large model can be deployed on the server 105.

[0036] For example, in the model application stage, the server 105 can obtain the image and question text to be processed through the terminal devices 101, 102, and 103, and then use the pre-deployed trained multimodal large model to process the obtained image and question text to generate the answer text. The server 105 can also return the answer text to the terminal devices 101, 102, and 103 for display to the user.

[0037] It should be noted that the sample construction method and the multimodal large model training method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the sample construction device and the multimodal large model training device provided in the embodiments of the present disclosure can generally be set in the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0039] Figure 2 is a schematic flow chart of a sample construction method according to an embodiment of the present disclosure.

[0040] like Figure 2 As shown, the sample construction method 200 may include operations S210 to S240.

[0041] In operation S210 , an initial reasoning chain for answering the question text is determined based on the associated image and question text.

[0042] For example, the question text may ask a question about the image, such as "what does the picture tell you about". For example, the initial reasoning chain may include multiple reasoning steps, and the initial reasoning chain may be determined based on a large model. This embodiment does not limit the process of determining the initial reasoning chain.

[0043] In operation S220, the initial inference chain is scrambled to obtain a scrambled inference chain that meets a predetermined abnormal condition.

[0044] For example, the scrambling process is used to modify the content of the initial reasoning chain. The scrambled reasoning chain satisfies the predetermined abnormal condition, which can indicate that the scrambled initial reasoning chain contains some abnormal content, such as logical errors, common sense errors, inconsistency with the image content, inconsistency with the question text, and other abnormal content.

[0045] In operation S230, based on the large model, a target reasoning chain is determined according to the initial reasoning chain and the scrambled reasoning chain, the target reasoning chain includes a reference text and the scrambled reasoning chain, and the reference text is used to describe abnormal information in the scrambled reasoning chain.

[0046] For example, the initial reasoning chain and the scrambled reasoning chain can be compared to determine the location of the error in the scrambled reasoning chain, the cause of the error, the answer to correct the error, and other information, and the reference text can explain this information.

[0047] For example, the initial reasoning chain and the scrambled reasoning chain can be input into the large model, and the prompt information can be used to guide the large model to determine the abnormal information in the scrambled reasoning chain and output the target reasoning chain, or output the reference text and then combine the reference text with the scrambled reasoning chain into the target reasoning chain. The large model can be, for example, a large language model.

[0048] For another example, the image, question text, initial reasoning chain, and scrambled reasoning chain can all be input into the big model, and another prompt information can be used to guide the big model to determine abnormal information in the scrambled reasoning chain and output the target reasoning chain, or output the reference text and then combine the reference text with the scrambled reasoning chain into the target reasoning chain. The big model can be a multimodal big model, for example.

[0049] In operation S240, training samples are constructed according to the image, the question text, and the target reasoning chain.

[0050] For example, an image, a question text, and a target reasoning chain may be combined as a training sample. Alternatively, the target reasoning chain may be post-processed and then the image, the question text, and the post-processed target reasoning chain may be combined as a training sample.

[0051] According to the technical solution provided by the embodiment of the present disclosure, the training sample obtained by the sample construction method includes a target reasoning chain, and the target reasoning chain includes a scrambled reasoning chain of predetermined abnormal conditions, and the scrambled reasoning chain may, for example, contain some errors, inconsistent with the image or text, or incomplete information. At the same time, the target reasoning chain also includes a reference text, and the reference text may, for example, explain the abnormal information in the scrambled reasoning chain, so that the target thinking chain in the training sample is a long thinking chain with the characteristics of thinking, error recognition, reflection, and correction. The deeper reasoning steps in the long thinking chain can improve the comprehensive reasoning ability of the multimodal large model. Therefore, using the training sample to train the multimodal large model can improve the reasoning ability of the multimodal large model in multimodal understanding, such as improving the error recognition, reflection, correction and other capabilities of the multimodal large model.

[0052] Next, the process of determining the initial reasoning chain is described.

[0053] In one example, the question text may not be classified, but general prompt information may be used to guide the multimodal large model to output an initial reasoning chain. For example, the image, question text and general prompt information are used to generate an initial reasoning chain based on the multimodal large model.

[0054] In another example, the question text can be classified to obtain the category of the question text, and then the target prompt information matching the category is determined from multiple candidate prompt information, and then the initial reasoning chain is generated based on the multimodal large model according to the image, the question text and the target prompt information. For example, the question text and the image can be classified using a pre-trained classification model, and the categories include, for example, reasoning category, problem solving category, etc. The correspondence between the category and the prompt information can also be pre-configured, and different categories of question texts correspond to different prompt information. The image, the question text, and the target prompt information corresponding to the category of the question text are input into the multimodal large model, and the multimodal large model is used to generate the initial reasoning chain. The prompt information is used to guide the multimodal large model to consider from different angles such as perception, cognition, and reasoning, and output the reasoning steps that it considers reasonable, construct a linear reasoning path from the question to the answer, and then output the initial reasoning chain. This embodiment performs classification and selects the target prompt information matching the category, so that the multimodal large model can be targeted to output the initial reasoning chain. By selecting more matching target prompt information, the reasoning accuracy of the multimodal large model can be improved, thereby improving the quality of the initial reasoning chain.

[0055] In another example, multiple initial reasoning chains can be generated for the data pair consisting of the same image and question text, and then some high-quality initial reasoning chains can be screened from the multiple initial reasoning chains. In practical applications, multiple initial reasoning chains can be generated in a variety of ways. For example, a single multimodal large model can be used to require the multimodal large model to generate multiple initial reasoning chains with different reasoning logics in different ways. For another example, multiple multimodal large models can be used to process images, question texts, and target prompt information respectively, so as to obtain multiple initial reasoning chains output by multiple multimodal large models. After generating multiple initial reasoning chains, some high-quality initial reasoning chains can be screened from the multiple initial reasoning chains, and then only the screened initial reasoning chains are scrambled to obtain the scrambled reasoning chains. In this embodiment, multiple initial reasoning chains are first generated, and then screened, and then only the high-quality initial reasoning chains are scrambled, so that the quality of the scrambled reasoning chains can be improved by selecting high-quality initial reasoning chains.

[0056] The above describes the process of generating an initial inference chain.

[0057] Next, the process of selecting a high-quality initial reasoning chain from multiple initial reasoning chains is described.

[0058] In the disclosed embodiment, multiple initial reasoning chains can be generated in advance, and then for each of the multiple initial reasoning chains, the initial reasoning chain is evaluated from at least one predetermined dimension according to the image, the question text, and the content of the initial reasoning chain to obtain an evaluation result characterizing the quality of the initial reasoning chain. Then, according to the evaluation results of each of the multiple initial reasoning chains, some of the initial reasoning chains are screened from the multiple initial reasoning chains to obtain the screened initial reasoning chains.

[0059] For example, the evaluation dimensions may include content consistency, logical rationality, depth of the thinking chain, etc. Content consistency indicates a degree of consistency between the initial reasoning chain and the content of the image and question text. For example, the image, question text, initial reasoning chain and prompt information for evaluation may be input into the multimodal large model. The prompt information may be used to guide the multimodal large model to evaluate the initial reasoning chain from the above dimensions and output the evaluation results. The evaluation results may include, for example, evaluation values ​​of each dimension, weighted average and other operations are performed on the evaluation values ​​of each dimension, and the budget result is used as the evaluation result. After obtaining the evaluation results of each initial reasoning chain, several initial reasoning chains with the highest evaluation results may be retained, or the initial reasoning chains with calculated operation results higher than the threshold may be retained, and other initial reasoning chains may be filtered out and no longer processed.

[0060] This embodiment evaluates the initial reasoning chain from several predetermined dimensions based on the image, the question text and the content of the initial reasoning chain, and screens the initial reasoning chain based on the evaluation result, so that high-quality initial reasoning chains can be accurately screened out.

[0061] In one example, the evaluation result includes a first evaluation result. In the process of determining the first evaluation result, the initial reasoning chain can be evaluated from at least one predetermined dimension based on the image, the question text, and the initial reasoning chain based on multiple first evaluation models, to obtain an evaluation sub-result determined by each of the multiple first evaluation models. Then, the first evaluation result of the initial reasoning chain is determined based on the evaluation sub-results of each of the multiple first evaluation models.

[0062] For example, the first evaluation model can be a multimodal model, and the process of evaluating the initial reasoning chain by a single first evaluation model can refer to the above. For example, the image, question text, initial reasoning chain and prompt information for guiding the multimodal large model to perform evaluation can be input into the first evaluation model to obtain an evaluation sub-result. In this embodiment, for the same initial reasoning chain, multiple first evaluation models can be used for evaluation, each first evaluation model can output an evaluation sub-result, and then the first evaluation result is determined based on the multiple evaluation sub-results output by the multiple first evaluation models. In this way, multiple first evaluation models can more accurately determine the first evaluation result through voting, weighting, cross-validation or other processing.

[0063] In another example, the evaluation result includes a second evaluation result. In the process of determining the second evaluation result, the image can be identified to obtain the object in the image, and then the initial reasoning chain is evaluated based on the second evaluation model according to the object text, question text and initial reasoning chain associated with the object to obtain the second evaluation result; the evaluation result includes the second evaluation result.

[0064] For example, a high-precision visual operator can be used to identify the image, and the visual operator can include an optical character recognition model, a target detection model, etc., and the object in the image can be extracted by the visual operator. The object text associated with the object can be used to describe the category, characteristics, etc. of the object. For example, the second evaluation model can be a large language model. For example, the object text, the question text, the initial reasoning chain, and the prompt information for guiding the large language model to evaluate can be input into the second evaluation model, and the second evaluation model is used to evaluate the rationality of the initial reasoning path, the consistency of the initial reasoning chain with the visual information in the image, the rationality of the thinking logic, etc. If the second evaluation result indicates that the quality of the initial reasoning chain is poor, the initial reasoning chain can be filtered out. In this embodiment, the object in the image is first extracted, and then the object is converted into an object text, and then evaluated based on the object text, the question text and the initial reasoning chain. In this way, a large language model can be used as the second evaluation model. Compared with the multimodal large model, the language logic ability, the computing ability, etc. of the large language model are stronger, so the evaluation is more accurate.

[0065] It should be noted that, in practical applications, one of the first evaluation model and the second evaluation model can be used for evaluation, or the first evaluation model and the second evaluation model can be used together for evaluation. For example, the first evaluation model is used for evaluation first, and part of the initial reasoning chain is screened, and then the second evaluation model is used to evaluate the remaining initial reasoning chain again, thereby screening part of the reasoning chain again. Or for the same initial reasoning chain, the first evaluation result and the second evaluation result are determined respectively by the first evaluation model and the second evaluation model, and then the first evaluation result and the second evaluation result are weighted or other operations are performed on the first evaluation result and the second evaluation result to determine the evaluation result of the initial reasoning chain.

[0066] The above describes the process of selecting a high-quality initial reasoning chain from multiple initial reasoning chains.

[0067] After the initial inference chain is obtained, the initial inference chain may be scrambled. Next, the scrambling process is described.

[0068] In one example, the entity in the initial reasoning chain can be replaced with other entities. For example, a text in the initial reasoning chain is "There is a cat in the picture", and after replacement, the text is "There is a dog in the picture". Entity replacement can be achieved in many ways. For example, the reasoning chain can be processed by named entity recognition or keyword extraction to determine the entity to be replaced. The correspondence between entities can also be pre-determined, and the entity to be replaced can be replaced based on the correspondence. For another example, a multimodal model can be used for entity replacement. For example, an image, a question text, an initial reasoning chain, and prompt information for guiding the multimodal model to perform entity replacement are input into the multimodal model, and the multimodal model outputs the scrambled reasoning chain. In this embodiment, the text in the reasoning chain is feature misread or fuzzy described, so that hallucination simulation can be performed from the perception level, so that the scrambled reasoning chain meets the abnormal conditions, such as making the scrambled reasoning chain wrong, or the scrambled reasoning chain is inconsistent with the image and the question text.

[0069] In another example, the context text that satisfies the logical relationship in the initial reasoning chain can be modified so that the modified context text does not satisfy the logical relationship. For example, a text in the initial reasoning chain is "Since the two straight lines are parallel, the two straight lines do not intersect", and after modification, the text is "Since the two straight lines are perpendicular, the two straight lines do not intersect", or "Since the two straight lines are not parallel, the two straight lines intersect". The modification of the logical relationship can be achieved in many ways. For example, a specific keyword in the reasoning chain can be modified to other words, such as adding negative words, deleting negative words, or replacing keywords. For another example, a multimodal model can be used, and the image, question text, initial reasoning chain and prompt information for guiding the multimodal model to generate erroneous logic can be input into the multimodal model, and the multimodal model outputs the scrambled reasoning chain. In this embodiment, the assumption or misunderstanding based on the error logic is embedded in the reasoning chain, so that the hallucination simulation can be performed from the cognitive level, so that the scrambled reasoning chain meets the abnormal conditions, such as making the scrambled reasoning chain wrong, or the scrambled reasoning chain is inconsistent with the image and the question text.

[0070] In another example, some of the reasoning steps in the initial reasoning chain may be deleted. For example, the initial reasoning chain includes several reasoning steps, and some of the steps may be randomly deleted from the reasoning steps. For example, the initial reasoning chain includes steps 1 to 5, and steps 1 and 2 may be deleted, or step 3 may be deleted.

[0071] It should be noted that the above provides a variety of means for scrambling the initial reasoning chain. In practical applications, the various means can be used in combination or individually. This embodiment does not limit the combination of the various scrambling means.

[0072] The above describes the scrambling process of the initial reasoning chain. Through the scrambling process, the thinking bias in the initial reasoning chain can be disturbed, thereby simulating the potential bias in human thinking.

[0073] It should be noted that the purpose of the scrambling process is to change the initial reasoning chain so that the scrambled reasoning chain meets the abnormal condition.

[0074] In one example, the abnormal condition includes that the scrambled inference chain is inconsistent with the image, for example, the image contains a cat, while the inference chain describes that the image contains a dog.

[0075] In another example, the abnormal condition includes that the scrambled reasoning chain is inconsistent with the question text, for example, the question text asks for clothing recommendations, while the reasoning chain answers content related to food recommendations.

[0076] In another example, the abnormal condition includes: the scrambled reasoning chain contains erroneous content, and the erroneous content may include common sense errors, logical errors, etc. For example, the initial reasoning chain describes "because tomorrow's temperature is 35 degrees Celsius, it is recommended to wear light clothing", while the description in the scrambled reasoning chain is "because tomorrow's temperature is -5 degrees Celsius, it is recommended to wear light clothing".

[0077] In another example, the abnormal condition includes: the scrambled reasoning chain lacks some reasoning steps relative to the initial reasoning chain. For example, the initial reasoning chain includes steps 1 to 5, while the scrambled reasoning chain only includes steps 3 to 5.

[0078] It should be noted that this embodiment does not limit the specific means of scrambling processing, as long as the scrambled reasoning chain can meet the abnormal condition.

[0079] According to another embodiment of the present disclosure, after obtaining the scrambled reasoning chain, a target reasoning chain can be determined, which contains a reference text. In the actual processing process, the target reasoning chain can be generated based on multimodal guidance. For example, the image, the question text, the initial reasoning chain before scrambling, the reasoning chain after scrambling, and the relevant prompt information are input into the multimodal large model, and the multimodal large model outputs the target reasoning chain, wherein the prompt information can provide clear reflection and thinking methods, and can also provide some samples to guide the multimodal large model to identify errors through logical deduction and generate reference text. In this embodiment, the initial reasoning chain before scrambling is equivalent to the correct answer. By introducing the correct answer and the reflection error correction mechanism, the multimodal large model is guided to gradually return from the erroneous thinking path to the correct reasoning direction.

[0080] It should be noted that the multimodal large model does not correct the abnormal content in the scrambled reasoning chain, that is, it does not modify the erroneous content, but describes the erroneous information. In one example, the reference information can describe the location of the abnormal step in the scrambled reasoning chain, such as describing that the third reasoning step is wrong. In another example, the reference information can describe the abnormal reason for the abnormal step in the scrambled reasoning chain, for example, "Because the temperature is -5 degrees Celsius, the temperature is low, and light clothing has poor warmth retention, so the clothing recommendation is unreasonable." In another example, the reference information can describe the correction result for the abnormal step in the scrambled reasoning chain, such as "the calculation process 1+1=3, the correct calculation result should be 1+1=2".

[0081] In this embodiment, the initial reasoning chain before scrambling only provides forward reasoning steps and does not have the characteristics of thinking and identifying errors. In this embodiment, the errors after scrambling are described by reference text, which can simulate the human's ability to reconstruct thinking after interference, so that the target reasoning chain containing the reference text has the characteristics of reflection. At the same time, based on practical findings, the reasoning chain with reflective characteristics can make the large model think slowly, so that the large model has stronger reasoning ability.

[0082] According to another embodiment of the present disclosure, after obtaining the target reasoning chain, the image, the question text and the target reasoning chain can be directly combined into a training sample.

[0083] In another embodiment of the present disclosure, after obtaining the target reasoning chain, the target reasoning chain can be post-processed to obtain an adjusted target reasoning chain. Then, the image, the question text, and the adjusted target reasoning chain are combined into a training sample. During the post-processing of the target reasoning chain, the target reasoning chain can be adjusted based on the large language model according to the question text and the target reasoning chain to obtain an adjusted reasoning chain.

[0084] In one example, the adjustment process for the target reasoning chain is used to adjust the number of reasoning steps in the target reasoning chain. For example, a large language model can be used, and the input of the large language model includes the question text, the target reasoning chain, and the prompt information for post-processing, and the prompt information can contain examples, and the examples include questions, reasoning paths (the reasoning paths can provide reasoning results), and adjusted reasoning chains. The large language model is guided by the examples in the prompt information, so that the adjusted reasoning chain output by the large language model is similar to the adjusted reasoning chain in the example. In addition, the prompt information can also gradually optimize the reasoning chain from the aspects of perception (such as extraction of basic features), cognition (such as modeling of relationships between features), reasoning (such as derivation of causal chains), etc., and then refine the target reasoning chain, and control the depth of the reasoning path. In this way, the adjusted reasoning chain can not only cover the core reasoning process of complex problems, but also maintain a reasonable length to avoid being too lengthy or one-sided.

[0085] In another example, the adjustment process for the target reasoning chain is used to insert a description text that is semantically consistent with a predetermined text into the target reasoning chain to adjust the expression of the target reasoning chain.

[0086] For example, through anthropomorphic thinking patterns, multiple reasoning paths can be generated to make them closer to human thinking. For example, the question text, target reasoning chain and prompt information can be input into the large language model, and the large language model can output the adjusted reasoning chain. The prompt information can guide the large language model to simulate the common "hypothesis-verification-correction" mode of humans in the process of thinking chain. The prompt information can also provide examples. For example, an example in the prompt information is as follows:

[0087] Initial observation and hypothesis: "Wait, the flower in the picture doesn't seem to be a camellia. The shape of the flower above is very similar to a camellia."; Further reasoning based on knowledge: "Let me think about it, the characteristics of a camellia include feature 1 and feature 2, and the flower in the picture is very close in these features."; Discover contradictions and make corrections: "However, the texture of the leaves of the flower in the picture seems inconsistent, which may indicate that it is not a camellia."; Final verification and conclusion: "Combined with observation and feature comparison, the flower in the picture may be a rhododendron, because its leaf texture and characteristics are more consistent."

[0088] In this embodiment, some descriptive texts similar to "wait", "let me think about it", "seems inconsistent", etc. can be inserted into the reasoning chain. These descriptive texts are in line with the reflection and correction mechanism of human logic, thereby optimizing the naturalness and logic of multiple reasoning paths and enhancing the intuitiveness and persuasiveness of the reasoning chain.

[0089] Figure 3 is a schematic diagram of a sample construction method according to an embodiment of the present disclosure.

[0090] In this embodiment, multiple initial reasoning chains 331, 332, and 333 can be generated based on the image 310 and the question text 320 through multiple generation models M1.1, M1.2, and M1.3. Taking the generation model M1.1 as an example, the generation model M1.1 can be a multimodal large model. The input of the generation model M1.1 includes the image 310, the question text 320, and the first prompt information prompt1. The first prompt information prompt1 can be determined based on the category of the question text 320, or it can be a general prompt information unrelated to the category. The output of the multimodal large model is the initial reasoning chain.

[0091] Next, the multiple initial reasoning chains 331 , 332 , 333 may be evaluated by the evaluation model M2 , where the evaluation model M2 , for example, includes multiple first evaluation models M2 . 1 and second evaluation models M2 . 2 .

[0092] The first evaluation model M2.1 can be a multimodal large model. The input of each first evaluation model M2.1 includes an image 310, a question text 320, an initial reasoning chain and a second prompt information prompt2. The evaluation sub-results can be determined based on the output of the first evaluation model M2.1. Multiple evaluation substructures of multiple first evaluation models M2.1 can calculate the first evaluation results, and then preliminary screening can be performed based on the first evaluation results to filter out some initial reasoning chains, such as filtering out the initial reasoning chain 332.

[0093] For the remaining initial reasoning chains, visual operators can be used to detect objects in the image 310, and then the second evaluation model M2.2 can be used for evaluation again. The second evaluation model M2.2 can be a large language model. The input of the second evaluation model M2.2 includes the object text associated with the object, the question text 320, the initial reasoning chain and the third prompt information prompt3. The rationality of the initial reasoning chain is evaluated by the second evaluation model M2.2, and the high-quality initial reasoning chain is screened again. For example, the initial reasoning chain 331 is retained after screening.

[0094] For the initial reasoning chain 331 after screening, the initial reasoning chain 331 can be scrambled to obtain the scrambled reasoning chain 340. During the scrambling process, hallucination simulation can be performed at the perceptual level, such as feature misreading or vague description of the text in the reasoning chain. Hallucination simulation can also be performed at the cognitive level, for example, assumptions or misunderstandings based on biased logic can be embedded in the thinking chain. It is also possible to randomly delete some reasoning steps in the reasoning chain based on rules. The scrambling process can be implemented based on a scrambling model, and the scrambling model is, for example, a multimodal large model. The input of the scrambling model includes an image 310, a question text 320, an initial reasoning chain, and the fourth prompt information prompt4. The output of the scrambling model is the scrambled reasoning chain 340. The scrambling process makes the scrambled reasoning chain 340 meet the predetermined abnormal condition.

[0095] After obtaining the scrambled reasoning chain 340, the target reasoning chain 360 can be determined based on the large model according to the initial reasoning chain and the scrambled reasoning chain 340. The target reasoning chain 360 includes the reference text 350 and the scrambled reasoning chain 340. The reference text 350 is used to describe the abnormal information in the scrambled reasoning chain 340. The large model used in this process can be a multimodal large model. The input of the multimodal large model includes the image 310, the question text 320, the reasoning chain before scrambling, the reasoning chain 340 after scrambling, and the fifth prompt information prompt5. The target reasoning chain 360 is generated by the multimodal large model. The target reasoning chain 360 is equivalent to inserting reference information into the scrambled reasoning chain 340. The reference information is used to describe the abnormal information in the scrambled reasoning chain 340, such as describing the location of the abnormal step in the scrambled reasoning chain 340, the abnormal cause and the correction result.

[0096] After obtaining the target reasoning chain 360, the target reasoning chain 360 may be post-processed. The post-processing mainly adjusts the number of reasoning steps in the target reasoning chain 360 through adjustment processing, or inserts a description text that is consistent with the semantics of the predetermined text into the target reasoning chain 360 to adjust the expression of the target reasoning chain 360.

[0097] For example, the post-processing process can be implemented based on a large language model, and the input of the large language model includes the question text 320, the target reasoning chain 360, and the sixth prompt information prompt6, and the sixth prompt information prompt6 can include an example. It can be understood that the sixth prompt information prompt6 used to guide the large language model to adjust the reasoning depth and guide the large language model to adjust the expression method can be different.

[0098] By post-processing the target reasoning chain 360 , an adjusted reasoning chain 370 may be obtained. Next, the image 310 , the question text 320 , and the adjusted reasoning chain 370 may be combined into a training sample 380 .

[0099] It should be noted that the above-mentioned multiple processes use multiple models, such as multimodal guidance, large language models, etc., wherein the large models used in different processes can be the same or different. For example, the first evaluation model and the large model used for scrambling can use the same multimodal large model, or different multimodal large models. In addition, the prompt information used by each large model is flexibly configured according to actual needs. This embodiment does not limit the format, content, and expression of the prompt information, and it is sufficient to guide the large model to achieve the corresponding function.

[0100] By adopting the sample construction method provided in this embodiment, a training sample 380 for training a multimodal large model can be obtained, and the training sample 380 includes long thought chain data with characteristics such as thinking, error recognition, reflection, and correction.

[0101] Figure 4 It is a schematic flowchart of a training method for a multimodal large model according to an embodiment of the present disclosure.

[0102] like Figure 4 As shown, the training method 400 of the multimodal large model may include operations S410 to S420.

[0103] In operation S410, a training sample is obtained, the training sample including an image, a question text, and a target reasoning chain. For example, the training sample is obtained using the above-mentioned sample construction method.

[0104] In operation S420, a multimodal large model is trained using the training samples.

[0105] This embodiment uses training samples of long thinking chains with characteristics such as thinking, error recognition, reflection, and correction to train the multimodal large model, which can improve the reasoning ability of the multimodal large model, so as to understand and deal with complex reasoning scenarios. The slow thinking ability of the multimodal large model can also be enhanced. For example, the multimodal large model can simulate the "slow thinking" process in reasoning, gradually generate hypotheses, verify hypotheses, identify errors and correct them, thereby improving its multi-step reasoning performance in complex problems. In addition, compared to manually annotating reasoning steps, since the training samples are automatically generated, the cost of sample generation is reduced, thereby reducing the model cost.

[0106] Figure 5 is a schematic structural block diagram of a sample construction device according to an embodiment of the present disclosure.

[0107] like Figure 5 As shown, the sample construction device 500 may include an initial reasoning chain determination module 510 , a scrambling module 520 , a target reasoning chain determination module 530 , and a construction module 540 .

[0108] The initial reasoning chain determination module 510 is used to determine an initial reasoning chain for answering the question text according to the associated image and question text.

[0109] The scrambling module 520 is used to perform scrambling processing on the initial reasoning chain to obtain a scrambled reasoning chain that meets a predetermined abnormal condition.

[0110] The target reasoning chain determination module 530 is used to determine the target reasoning chain based on the large model, according to the initial reasoning chain and the scrambled reasoning chain. The target reasoning chain includes a reference text and the scrambled reasoning chain. The reference text is used to describe abnormal information in the scrambled reasoning chain.

[0111] The construction module 540 is used to construct training samples according to the image, question text and target reasoning chain.

[0112] According to another embodiment of the present disclosure, the scrambling module includes at least one of the following: a first scrambling submodule, a second scrambling submodule and a third scrambling submodule. The first scrambling submodule is used to replace entities in the initial reasoning chain with other entities. The second scrambling submodule is used to modify the context text that satisfies the logical relationship in the initial reasoning chain so that the modified context text does not satisfy the logical relationship. The third scrambling submodule is used to delete some reasoning steps in the initial reasoning chain.

[0113] According to another embodiment of the present disclosure, the abnormal condition includes at least one of the following: the scrambled reasoning chain is inconsistent with the image; the scrambled reasoning chain is inconsistent with the question text; the scrambled reasoning chain contains erroneous content; and the scrambled reasoning chain lacks some reasoning steps relative to the initial reasoning chain.

[0114] According to another embodiment of the present disclosure, the reference text is used to describe at least one of the following information: the location of the abnormal step in the scrambled reasoning chain; the abnormal cause of the abnormal step in the scrambled reasoning chain; and the correction result for the abnormal step in the scrambled reasoning chain.

[0115] According to another embodiment of the present disclosure, the construction module includes: an adjustment submodule and a construction submodule. The adjustment submodule is used to adjust the target reasoning chain based on the large language model according to the question text and the target reasoning chain to obtain an adjusted reasoning chain. The construction submodule is used to construct a training sample for training a multimodal large model according to the image, the question text and the adjusted reasoning chain. The adjustment process is used to adjust at least one of the following: the number of reasoning steps in the target reasoning chain, inserting a descriptive text that is consistent with the predetermined text semantics in the target reasoning chain to adjust the expression of the target reasoning chain.

[0116] According to another embodiment of the present disclosure, the number of initial reasoning chains is multiple; the scrambling module includes: an evaluation submodule, a screening submodule and a scrambling submodule. The evaluation submodule is used to evaluate each of the multiple initial reasoning chains from at least one predetermined dimension according to the image, the question text and the content of the initial reasoning chain, and obtain an evaluation result characterizing the quality of the initial reasoning chain. The screening submodule is used to screen some of the initial reasoning chains from the multiple initial reasoning chains according to the respective evaluation results of the multiple initial reasoning chains, and obtain the screened initial reasoning chains. The scrambling submodule is used to scramble the screened initial reasoning chains to obtain the scrambled reasoning chains.

[0117] According to another embodiment of the present disclosure, the evaluation submodule includes: an evaluation unit and a first evaluation result determination unit. The evaluation unit is used to evaluate the initial reasoning chain from at least one predetermined dimension based on the image, the question text and the initial reasoning chain based on multiple first evaluation models, and obtain an evaluation sub-result determined by each first evaluation model in the multiple first evaluation models. The first evaluation result determination unit is used to determine the first evaluation result of the initial reasoning chain according to the evaluation sub-results of each of the multiple first evaluation models; the evaluation result includes the first evaluation result.

[0118] According to another embodiment of the present disclosure, the evaluation submodule includes: a recognition unit and a second evaluation result determination unit. The recognition unit is used to recognize the image and obtain the object in the image. The second evaluation result determination unit is used to evaluate the initial reasoning chain based on the second evaluation model according to the object text, the question text and the initial reasoning chain associated with the object to obtain a second evaluation result; the evaluation result includes the second evaluation result.

[0119] According to another embodiment of the present disclosure, the initial reasoning chain determination module includes: a classification submodule, a prompt determination submodule and a generation submodule. The classification submodule is used to classify the question text to obtain the category of the question text. The prompt determination submodule is used to determine the target prompt information matching the category from multiple candidate prompt information. The generation submodule is used to generate an initial reasoning chain based on the multimodal large model according to the image, the question text and the target prompt information.

[0120] Figure 6 It is a schematic structural block diagram of a training device for a multimodal large model according to an embodiment of the present disclosure.

[0121] like Figure 6 As shown, the training device 600 of the multimodal large model may include an acquisition module 610 and a training module 620.

[0122] The acquisition module 610 is used to acquire training samples. The training samples include images, question texts and target reasoning chains. The training samples are obtained using the above-mentioned sample construction device.

[0123] The training module 620 is used to train the multimodal large model using training samples.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute at least one of the above-mentioned sample construction method and multimodal large model training method.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute at least one of the above-mentioned sample construction method and multimodal large model training method.

[0126] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, including a computer program, which implements at least one of the above-mentioned sample construction method and multimodal large model training method when executed by a processor.

[0127] Figure 7 It is a block diagram of an electronic device used to implement the sample construction method and the training method of the multimodal large model of the embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0129] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as at least one of the above-mentioned sample construction method and the training method of the multimodal large model. For example, in some embodiments, at least one of the above-mentioned sample construction method and the training method of the multimodal large model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of at least one of the above-mentioned sample construction method and the training method of the multimodal large model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured in any other appropriate manner (for example, by means of firmware) to execute at least one of the above-mentioned sample construction method and multimodal large model training method.

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A sample construction method, comprising: Determining an initial chain of reasoning for answering the question text based on the associated image and question text; Scrambling the initial reasoning chain to obtain a scrambled reasoning chain that meets a predetermined abnormal condition; Based on the large model, determining a target reasoning chain according to the initial reasoning chain and the scrambled reasoning chain, wherein the target reasoning chain includes a reference text and the scrambled reasoning chain, and the reference text is used to describe abnormal information in the scrambled reasoning chain; as well as A training sample is constructed according to the image, the question text and the target reasoning chain.

2. The method according to claim 1, wherein: The scrambling process includes at least one of the following: replacing entities in the initial chain of reasoning with other entities; Modifying the context text satisfying the logical relationship in the initial reasoning chain so that the modified context text does not satisfy the logical relationship; as well as Some of the reasoning steps in the initial reasoning chain are deleted.

3. The method according to claim 1, wherein: The abnormal condition includes at least one of the following: The scrambled chain of reasoning is inconsistent with the image; The scrambled reasoning chain is inconsistent with the question text; The scrambled inference chain contains erroneous content; as well as The scrambled reasoning chain lacks some reasoning steps relative to the initial reasoning chain.

4. The method according to claim 1, wherein: The reference text is used to describe at least one of the following information: the location of the abnormal step in the scrambled chain of reasoning; The abnormal reason of the abnormal step in the scrambled reasoning chain; and The correction result for the abnormal step in the scrambled reasoning chain.

5. The method according to claim 1, wherein: The constructing of training samples according to the image, the question text and the target reasoning chain comprises: According to the question text and the target reasoning chain, adjusting the target reasoning chain based on a large language model to obtain an adjusted reasoning chain; and Constructing a training sample for training a multimodal large model according to the image, the question text and the adjusted reasoning chain; The adjustment process is used to adjust at least one of the following: the number of reasoning steps in the target reasoning chain, inserting a description text consistent with the semantics of a predetermined text into the target reasoning chain to adjust the expression of the target reasoning chain.

6. The method according to claim 1, wherein: The number of the initial reasoning chains is multiple; the scrambling of the initial reasoning chains to obtain the scrambled reasoning chains that meet the predetermined abnormal conditions includes: For each of the multiple initial reasoning chains, evaluating the initial reasoning chain from at least one predetermined dimension according to the image, the question text, and the content of the initial reasoning chain, to obtain an evaluation result representing the quality of the initial reasoning chain; According to the evaluation results of each of the multiple initial reasoning chains, select some initial reasoning chains from the multiple initial reasoning chains to obtain selected initial reasoning chains; The scrambling process is performed on the screened initial inference chain to obtain the scrambled inference chain.

7. The method according to claim 6, wherein: The step of evaluating the initial reasoning chain from at least one predetermined dimension based on the image, the question text, and the content of the initial reasoning chain to obtain an evaluation result representing the quality of the initial reasoning chain includes: Based on a plurality of first evaluation models, respectively evaluating the initial reasoning chain from at least one predetermined dimension according to the image, the question text, and the initial reasoning chain, to obtain an evaluation sub-result determined by each of the plurality of first evaluation models; and A first evaluation result of the initial reasoning chain is determined according to the evaluation sub-results of each of the plurality of first evaluation models; the evaluation result includes the first evaluation result.

8. The method according to claim 6, wherein: The step of evaluating the initial reasoning chain from at least one predetermined dimension based on the image, the question text, and the content of the initial reasoning chain to obtain an evaluation result representing the quality of the initial reasoning chain includes: Recognizing the image to obtain an object in the image; and According to the object text associated with the object, the question text and the initial reasoning chain, the initial reasoning chain is evaluated based on a second evaluation model to obtain a second evaluation result; the evaluation result includes the second evaluation result.

9. The method according to claim 1, wherein: Determining an initial reasoning chain for answering the question text according to the associated image and question text includes: Classifying the question text to obtain the category of the question text; Determining target prompt information matching the category from a plurality of candidate prompt information; and The initial reasoning chain is generated based on the multimodal large model according to the image, the question text and the target prompt information.

10. A method for training a multimodal large model, comprising: Acquire a training sample, wherein the training sample includes an image, a question text, and a target reasoning chain; as well as Using the training samples, training a large multimodal model; Wherein, the training samples are constructed using the method described in any one of claims 1 to 9.

11. A sample construction device, comprising: An initial reasoning chain determination module, used to determine an initial reasoning chain for answering the question text according to the associated image and question text; A scrambling module, used for scrambling the initial reasoning chain to obtain a scrambled reasoning chain that meets a predetermined abnormal condition; a target reasoning chain determination module, configured to determine a target reasoning chain based on the large model and according to the initial reasoning chain and the scrambled reasoning chain, wherein the target reasoning chain includes a reference text and the scrambled reasoning chain, and the reference text is used to describe abnormal information in the scrambled reasoning chain; as well as A construction module is used to construct a training sample according to the image, the question text and the target reasoning chain.

12. The device according to claim 11, wherein The scrambling module includes at least one of the following: A first scrambling submodule, configured to replace entities in the initial reasoning chain with other entities; A second scrambling submodule, used for modifying the context text satisfying the logical relationship in the initial reasoning chain, so that the modified context text does not satisfy the logical relationship; as well as The third scrambling submodule is used to delete some reasoning steps in the initial reasoning chain.

13. The device according to claim 11, wherein: The abnormal condition includes at least one of the following: The scrambled chain of reasoning is inconsistent with the image; The scrambled reasoning chain is inconsistent with the question text; The scrambled inference chain contains erroneous content; as well as The scrambled reasoning chain lacks some reasoning steps relative to the initial reasoning chain.

14. The device according to claim 11, wherein: The reference text is used to describe at least one of the following information: the location of the abnormal step in the scrambled chain of reasoning; The abnormal reason of the abnormal step in the scrambled reasoning chain; and The correction result for the abnormal step in the scrambled reasoning chain.

15. The device according to claim 11, wherein The building blocks include: an adjustment submodule, configured to adjust the target reasoning chain based on the large language model according to the question text and the target reasoning chain to obtain an adjusted reasoning chain; and A construction submodule, configured to construct a training sample for training a multimodal large model according to the image, the question text and the adjusted reasoning chain; The adjustment process is used to adjust at least one of the following: the number of reasoning steps in the target reasoning chain, inserting a description text consistent with the semantics of a predetermined text into the target reasoning chain to adjust the expression of the target reasoning chain.

16. The device according to claim 11, wherein The number of the initial reasoning chains is multiple; the scrambling module includes: an evaluation submodule, configured to evaluate each of the plurality of initial reasoning chains from at least one predetermined dimension according to the image, the question text, and the content of the initial reasoning chain, and obtain an evaluation result representing the quality of the initial reasoning chain; A screening submodule, configured to screen some initial reasoning chains from the plurality of initial reasoning chains according to the respective evaluation results of the plurality of initial reasoning chains, to obtain screened initial reasoning chains; The scrambling submodule is used to perform the scrambling process on the screened initial reasoning chain to obtain the scrambled reasoning chain.

17. The device according to claim 16, wherein: The evaluation submodule includes: an evaluation unit, configured to evaluate the initial reasoning chain from at least one predetermined dimension based on the image, the question text, and the initial reasoning chain based on a plurality of first evaluation models, and obtain an evaluation sub-result determined by each of the plurality of first evaluation models; and A first evaluation result determining unit is used to determine a first evaluation result of the initial reasoning chain according to the evaluation sub-results of each of the plurality of first evaluation models; the evaluation result includes the first evaluation result.

18. The device according to claim 16, wherein: The evaluation submodule includes: a recognition unit, configured to recognize the image and obtain an object in the image; and A second evaluation result determination unit is used to evaluate the initial reasoning chain based on a second evaluation model according to the object text associated with the object, the question text and the initial reasoning chain to obtain a second evaluation result; the evaluation result includes the second evaluation result.

19. The device according to claim 11, wherein: The initial reasoning chain determination module comprises: A classification submodule, used for classifying the question text to obtain the category of the question text; a prompt determination submodule, configured to determine target prompt information matching the category from a plurality of candidate prompt information; and A generation submodule is used to generate the initial reasoning chain based on the multimodal large model according to the image, the question text and the target prompt information.

20. A multimodal large model training device, comprising: An acquisition module, used to acquire training samples, wherein the training samples include images, question texts, and target reasoning chains; as well as A training module, used to train a multimodal large model using the training samples; Wherein, the training samples are constructed using the device described in any one of claims 11 to 19.

21. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images