Model training method, related device, image processing method and related device

Abstract

The embodiment of the invention discloses a model training method, a related device, an image processing method and a related device. The model training method comprises the steps of generating a first training text based on context information, a question and question answer options corresponding to a training image, and inputting the first training text and the training image into a multi-modal model to obtain an intermediate reasoning logic output by the multi-modal model; generating a second training text based on the intermediate reasoning logic, the training question and the question answer option, and inputting the second training text into a language model to obtain a predicted answer output by the language model; determining a training loss value according to the intermediate reasoning logic, the predicted answer and a real answer corresponding to the question; and performing parameter optimization on the multi-modal model and the language model based on the training loss value. According to the embodiment of the invention, the reasoning accuracy of the large model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a model training method and related devices, an image processing method and related devices. Background Art

[0002] Artificial intelligence has developed rapidly in recent years, from the widespread application of diverse small-scale models in multiple fields (including computer vision, natural language processing, search advertising recommendations, etc.) to the current era dominated by large models. The technological innovations of these large models have brought new development opportunities to all walks of life. They can cope with a variety of scenarios including image classification, visual question answering, speech recognition and synthesis, video understanding and generation, and effectively solve text, image, audio and other related problems encountered in actual business. Especially in the field of AI generation and language / vision large models, generating correct answers based on the single-modal or multi-modal data input by users and improving accuracy have always been the goals of large model optimization. Summary of the invention

[0003] To solve the above technical problems, the embodiments of the present application respectively provide a model training method, a model training device, an image processing method, an image processing device, an electronic device, a computer-readable storage medium and a computer program product.

[0004] According to one aspect of an embodiment of the present application, a model training method is provided, including: generating a first training text based on context information, questions, and question answer options corresponding to a training image, and inputting the first training text and the training image into a multimodal model to obtain an intermediate reasoning logic output by the multimodal model; generating a second training text based on the intermediate reasoning logic, the training question, and the question answer options, and inputting the second training text into a language model to obtain a predicted answer output by the language model, wherein the predicted answer is an answer text in the question answer options; determining a training loss value based on the intermediate reasoning logic, the predicted answer, and the true answer corresponding to the question; and optimizing parameters of the multimodal model and the language model based on the training loss value.

[0005] In another exemplary embodiment, determining the training loss value based on the intermediate reasoning logic, the predicted answer, and the true answer corresponding to the question includes: calculating a first cross-entropy loss value based on the intermediate reasoning logic and the true answer, and calculating a second cross-entropy loss value based on the predicted answer and the true answer; determining the training loss value based on the first cross-entropy loss value and the second cross-entropy loss value.

[0006] In another exemplary embodiment, the question answer options include answer texts of at least two options, and one of the answer texts is a true answer corresponding to the training image.

[0007] In another exemplary embodiment, the method further includes: inputting the training image into an image context extraction model to obtain the context information output by the image context extraction model; and optimizing parameters of the multimodal model and the language model based on the training loss value, including: optimizing parameters of the image context extraction model, the multimodal model and the language model based on the training loss value.

[0008] According to one aspect of an embodiment of the present application, there is provided an image processing method, comprising: extracting context information of an image to be processed, and obtaining a question text and question answer options corresponding to the image to be processed; generating a first input text based on the context information, the question text and the question answer options, and inputting the first input text and the image to be processed into a trained multimodal model to obtain an intermediate reasoning logic output by the multimodal model; generating a second input text based on the intermediate reasoning logic, the question text and the question answer options, and inputting the second input text into a trained language model to obtain a prediction result output by the language model, wherein the prediction result is an answer text in the question answer options.

[0009] In another exemplary embodiment, the method further includes: inputting the image to be processed into a trained image context extraction model to obtain the context information output by the image context extraction model.

[0010] According to one aspect of an embodiment of the present application, a model training device is provided, comprising: a first processing module, configured to generate a first training text based on context information, questions and question answer options corresponding to a training image, and input the first training text and the training image into a multimodal model to obtain an intermediate reasoning logic output by the multimodal model; a second processing module, configured to generate a second training text based on the intermediate reasoning logic, the training question and the question answer options, and input the second training text into a language model to obtain a predicted answer output by the language model, wherein the predicted answer is an answer text in the question answer options; a loss determination module, configured to determine a training loss value based on the intermediate reasoning logic, the predicted answer and the true answer corresponding to the question; and a model optimization model, configured to perform parameter optimization on the multimodal model and the language model based on the training loss value.

[0011] According to one aspect of an embodiment of the present application, an image processing device is provided, comprising: an acquisition module, configured to extract context information of an image to be processed, and obtain a question text and question answer options corresponding to the image to be processed; an inference module, configured to generate a first input text based on the context information, the question text and the question answer options, and input the first input text and the image to be processed into a trained multimodal model to obtain an intermediate inference logic output by the multimodal model; a prediction module, configured to generate a second input text based on the intermediate inference logic, the question text and the question answer options, and input the second input text into a trained language model to obtain a prediction result output by the language model, wherein the prediction result is an answer text in the question answer options.

[0012] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements the model training method or image processing method as described above.

[0013] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the model training method or image processing method as described above.

[0014] According to one aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the model training method or image processing method as described above.

[0015] The technical solution provided in the embodiment of the present application introduces the technical means of thought chain reasoning in the process of processing data in both visual and language modalities. Specifically, the intermediate reasoning logic of the multimodal model on the image problem is obtained through the context information of the image, and then the answer to the question of the language model on the image problem is obtained through the intermediate reasoning logic. This not only improves the interpretability of the multimodal model, but also explicitly generates the intermediate reasoning logic, which can not only improve the reasoning accuracy, but also visualize the reasoning logic, solving the black box characteristics of the large model.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 is a schematic diagram of an exemplary implementation environment involved in the present application;

[0019] Figure 2 is a schematic diagram of a large model processing architecture shown in an exemplary embodiment of the present application;

[0020] Figure 3 is a flow chart of a model training method shown in an exemplary embodiment of the present application;

[0021] Figure 4 is based on Figure 3 A flow chart of a model training method further proposed in the illustrated embodiment;

[0022] Figure 5 is a flowchart of an image processing method shown in an exemplary embodiment of the present application;

[0023] Figure 6 is a block diagram of a model training device shown in an exemplary embodiment of the present application;

[0024] Figure 7 is a block diagram of an image processing device shown in an exemplary embodiment of the present application;

[0025] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0029] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0030] First of all, it should be noted that the big model is also called the pre-training model (PTM) or the cornerstone model. It refers to a deep neural network (DNN) with large parameters. It is trained on massive unlabeled data. The function approximation ability of the large-parameter DNN is used to enable the big model to extract common features from the data. After fine-tuning, parameter efficient fine-tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the big model can achieve ideal results in few-shot or zero-shot scenarios.

[0031] According to the data modality processed, the large model can be divided into language models (such as ELMO, BERT, GPT), visual models (such as swin-transformer, ViT, V-MOE), speech models (such as VALL-E), multimodal models (such as ViBERT, CLIP, Flamingo, Gato), etc., where the multimodal model refers to a model that establishes two or more data modality feature representations. The large model is an important tool for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface to connect multiple specific task models. The embodiments of the present application relate to the application of language models and multimodal models.

[0032] In the fields of AI generation, language / vision and other large models, generating correct answers and improving accuracy based on the single-modal or multi-modal data input by users have always been the goal of large model optimization. How to extract the information required by users from massive data, reduce the illusion effect of large models, and utilize data in multiple modalities are the key points in the field of large models.

[0033] The embodiments of the present application address the problems of low accuracy and poor interpretability of existing large model algorithms for complex problems, and propose a technical solution based on multimodal feature fusion. This technical solution establishes a multimodal large model, which extracts and fuses the feature information of visual and language modal data, thereby improving the large model's ability to understand the real world, which improves the accuracy of the large model to a certain extent. In addition, in view of the black box characteristics of the large model that lacks an intermediate logical reasoning process, the embodiments of the present application introduce the thought chain reasoning technology, which visualizes the problem reasoning logic by generating an intermediate reasoning process, thereby improving the accuracy of the large model.

[0034] The following is a detailed introduction to the embodiments of the present application:

[0035] First see Figure 1 , Figure 1 1 is a schematic diagram of an exemplary implementation environment involved in the present application. The implementation environment includes a terminal 110 and a server 120, and the terminal 110 and the server 120 communicate with each other in a wired or wireless manner.

[0036] The terminal 110 may be a smart phone, a tablet, a laptop, a computer, a smart home appliance, a smart terminal, etc., and is not limited here. The server 120 may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services, and is not limited here.

[0037] like Figure 1 As shown, the terminal 110 is used to upload the image and the question to the server 120, so as to obtain the answer to the question about the image through the multimodal model and language model deployed in the server 120. The multimodal model and language model deployed in the server 120 are obtained through pre-training, and the training process will be described in detail in subsequent embodiments, which will not be described in detail here.

[0038] It should be understood that in other implementation environments, the trained multimodal model and language model can also be deployed on the terminal 110, so that after the terminal 110 obtains the image and the answer, it directly obtains the relevant answer based on the locally deployed multimodal model and language model. The multimodal model and language model can also be deployed together on the terminal 110 and the server 120, which is not limited here.

[0039] Please continue reading Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a large model processing architecture shown in an exemplary embodiment of the present application. Figure 2As shown, the large model mentioned in the embodiment of the present application is composed of an image context extraction model, a multimodal model and a language model. However, it should be noted that in some exemplary embodiments, the large model may not include the image context extraction model. The image context extraction model is an additional model and is not limited to being a necessary component of the large model.

[0040] Among them, the image context extraction model is used to extract the context information of the image. The context information of the image can refer to the relationship between the various elements in the image (such as objects, backgrounds, colors, textures, etc.) and the environment and scenes in which they are located. This information is crucial for understanding the content of the image, performing image analysis and recognition. For example, the image context may include but is not limited to the relative position and spatial relationship between different objects in the image, the semantic connection between objects in the image, the scene presented by the image, the color and texture information in the image, dynamic elements (such as motion trajectory, speed changes, etc.), and one or more information in context reasoning. Exemplarily, the image context extraction model can adopt models such as ViT (Vision Transformer) and CLIP (Contrastive Language-Image Pre-training), and the specific structure of the image context extraction model is not limited here.

[0041] The multimodal model is used to output intermediate reasoning logic according to the text and image input into it. Figure 2 As shown, according to the thinking template, the text content input into the multimodal model is composed of questions, context information and answer options, and the image is also input into the multimodal model, so as to obtain the intermediate reasoning logic output by the multimodal model, and the intermediate reasoning logic refers to the derivation process for obtaining the answer to the question about the image. Exemplarily, the multimodal model can adopt models such as CLIP and ViLBERT (Vision-and-Language BERT), and the specific structure of the multimodal model is not limited here.

[0042] By learning and training a large amount of text data, the language model can capture the statistical laws and patterns of the language, and thus output text that conforms to the laws of the language. Figure 2 As shown, the text data input to the language model is composed of questions, intermediate reasoning logic, and question answer options according to the reasoning template, and the language model outputs the answer to the question in the image accordingly, and the answer is one of the options in the question answer options. Exemplarily, the language model can adopt LLAMA (LargeLanguage Model Meta AI), GPT (Generative Pre-trained Transformer), Qwen and other models, which are not limited here.

[0043] exist Figure 2 In the large model processing architecture shown, the image is first directly input into the image context extraction model to extract image features (context information), and then the extracted image features are input into the thinking template as context information, and the text corresponding to the image is also input into the thinking template to obtain the text content that is finally input into the multimodal model; wherein, the thinking template includes three components: question, context information, and question answer options; the image is also directly input into the multimodal model, so that the multimodal model uses the input text content and image to generate the image's intermediate reasoning logic about the question; the intermediate reasoning logic will continue to be input into the reasoning template to generate text content that is input into the language model; wherein, the reasoning template includes three components: question, intermediate reasoning logic, and question answer options; the language model outputs the final answer for the text content input therein, and the final answer is one of the answer options in the question answer options.

[0044] It can be concluded that the large model processing architecture disclosed in this application extracts and fuses feature information from both visual and language modal data, which can enhance the large model's perception of multimodal data, thereby improving the accuracy of reasoning. Specifically, in the multimodal model reasoning part, the text content of the input multimodal model is constructed through thinking templates, which improves the interpretability of the multimodal large model in processing actual business, and the answers are traceable; in the language model reasoning part, the text content of the input language model is constructed through reasoning templates, and the explicit generation of the large model logical reasoning intermediate process is realized, which can not only improve the accuracy of reasoning, but also visualize the reasoning logic and solve the black box characteristics of the large model. In general, based on the large model processing architecture disclosed in this application, it is possible to use a large model to automatically and accurately get answers to questions about image content.

[0045] It should also be noted that Figure 2 The large model processing architecture shown is suitable for large model training and application stages. Figure 2 The structure content of the example describes in detail the process of large model training and application.

[0046] First see Figure 3 , Figure 3 is a flowchart of a model training method shown in an exemplary embodiment of the present application. It should be noted that this method can be applied to Figure 1The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0047] like Figure 3 As shown, in an exemplary embodiment, the model training method includes S310-S340, which are described in detail as follows:

[0048] S310, based on the context information corresponding to the training image, the training questions and the answer options of the questions, generate a first training text, and input the first training text and the training image into the multimodal model to obtain the intermediate reasoning logic output by the multimodal model.

[0049] First of all, it should be noted that sufficient training data sets should be prepared in advance during the model training phase so that the large model can be trained using the training data sets. The training data sets can be multimodal question-answering data sets, such as ScienceQA (Science Question Answering) data sets, which are not limited here.

[0050] Furthermore, the training data in the training data set can also be changed into a data set form that is compatible with the large model processing architecture disclosed in this application, that is, compatible with Figure 2 The diagram shows the dataset form of a thinking template. Each piece of training data contains a training image, as well as text content consisting of the question corresponding to the training image, contextual information, and answer options.

[0051] Alternatively, in some embodiments, each piece of training data includes a training image, a question corresponding to the training image, and options for answering the question. The context information corresponding to the training image is obtained by extracting features from the training image.

[0052] In the model training stage, the first training text is generated based on the context information, questions, and answer options corresponding to the training image according to the preset thinking template. The thinking template is used to specify the specific format of the first training text composed of the context information, questions, and answer options corresponding to the training image. For example, the thinking template can specify that the first training text is composed of the context information, questions, and answer options corresponding to the training image in sequence, or other composition methods, which are not limited in this embodiment.

[0053] As mentioned above, the context information corresponding to the training image can be obtained from a training data set, or it can be the text content output by the image context extraction model after the training image is input into the image context extraction model, which is not limited here.

[0054] In this embodiment, the first training text and the training image are input into the multimodal model, so that the multimodal model outputs the intermediate reasoning logic accordingly. The intermediate reasoning logic can be understood as a derivation process for answering the corresponding question based on the image content of the training image, and therefore includes two parts: derivation and conclusion. Moreover, the conclusion part included in the intermediate reasoning process is the answer text corresponding to one of the answer options of the question.

[0055] Therefore, this embodiment introduces image context information to construct the text content of the input multimodal model, thereby improving the interpretability of the large multimodal model in processing actual business, and the answers are traceable, which is conducive to promoting more accurate answers to image questions.

[0056] S320, based on the intermediate reasoning logic, the training questions and the question answer options, generate a second training text, and input the second training text into the language model to obtain a predicted answer output by the language model, where the predicted answer is the answer text in the question answer options.

[0057] In the model training stage, the present embodiment also generates a second training text according to a preset reasoning template, based on the intermediate reasoning logic obtained in S310, and the question and question answer options corresponding to the training image. The thinking template is used for the specific format of the second training text composed of the intermediate reasoning logic, the question, and the question answer options. For example, the reasoning template may stipulate that the second training text is composed of the intermediate reasoning logic, the question, and the question answer options in sequence, or it may be other composition methods, and the present embodiment does not limit this.

[0058] In this embodiment, the second training text is input into the language model, so that the language model predicts which option in the question answer options the answer to the question is based on the intermediate reasoning logic, and uses the answer text corresponding to the predicted option as the predicted answer. Alternatively, the language model can also predict the probability of each option in the question answer options based on the intermediate reasoning logic, and output the final predicted answer based on the predicted probability, such as using the answer text corresponding to the option with the maximum probability as the predicted answer, which is not limited here. It can also be seen that the predicted answer output by the language model is the answer text in the question answer options.

[0059] For example, in some exemplary embodiments, the question answer option may only contain an answer text, which is usually the true answer to the question. In this case, the language model can predict the probability that the final answer is the question answer option, and if the probability is higher than a preset probability threshold, the question answer option is output as the predicted answer.

[0060] In another exemplary embodiment, the question answer options may only include at least two answer texts, and one of the answer texts is the real answer to the question. In this case, the language model can directly predict which option in the question answer options is the predicted answer, or can separately predict the probability that the answer corresponding to the question is each option in the question answer options, thereby determining the final predicted answer based on the obtained probabilities.

[0061] Therefore, this embodiment can explicitly generate the intermediate process of logical reasoning of the large model by inputting the intermediate reasoning logic output by the multimodal model into the language model, which can not only improve the accuracy of reasoning, but also visualize the reasoning logic and solve the black box characteristics of the large model.

[0062] S330, determining a training loss value according to the intermediate reasoning logic, the answer text, and the true answer corresponding to the training question.

[0063] In this embodiment, the large model training loss is composed of the losses of the multimodal model and the language model respectively. Therefore, the training loss value is determined based on the intermediate reasoning process of the multimodal output, the predicted answer output by the language model, and the actual answer to the question corresponding to the training image.

[0064] It can be understood that the loss of the multimodal language model can be determined based on the deviation between the conclusion content contained in the intermediate reasoning process and the true answer to the question. In this embodiment, the loss of the modal language model is referred to as the first cross-entropy loss value; the loss of the language model can be determined based on the deviation between the predicted answer output by the language model and the true answer to the question. In this embodiment, the loss of the language model is referred to as the second cross-entropy loss value.

[0065] Exemplarily, the process of calculating the first cross entropy loss value according to the intermediate reasoning logic and the true answer to the question can be expressed as the following formula:

[0066]

[0067] Among them, Loss R Represents the first cross entropy loss value, p R represents the category distribution of the true answer to the question, Q R represents the category distribution of the conclusion content about the problem contained in the intermediate reasoning logic, H R represents the cross entropy loss, and i represents the answer category.

[0068] Similarly, the process of calculating the second cross entropy loss value based on the predicted answer and the true answer to the question can be expressed as the following formula:

[0069]

[0070] Among them, Loss A Represents the second cross entropy loss value, p A represents the category distribution of the true answer to the question, Q A represents the category distribution of the predicted answer, H A represents the cross entropy loss, and i represents the answer category.

[0071] In some exemplary embodiments, the sum of the first cross entropy loss value and the second cross entropy loss value can be used as the training loss value, or the first cross entropy loss value and the second cross entropy loss value can be weighted and calculated to obtain the training loss value. The weights corresponding to the first cross entropy loss value and the second cross entropy loss value are usually preset values, and the sum of the weights corresponding to the first cross entropy loss value and the second cross entropy loss value is 1. For example, the weights corresponding to the first cross entropy loss value and the second cross entropy loss value can be 0.5 respectively, or other values, which are not limited here.

[0072] S340, optimizing parameters of the multimodal model and the language model based on the training loss value.

[0073] The process of optimizing the parameters of the multimodal model and the language model based on the training loss value in this embodiment can be implemented by gradient descent and other methods, and the specific method of parameter optimization is not limited here. Moreover, model training is an iterative process, and this embodiment does not limit the stopping conditions of model training. For example, training can be stopped when the training loss value is less than or equal to a preset loss threshold, or when the number of iterations reaches a preset number.

[0074] It can be concluded that the model training method disclosed in this embodiment extracts and fuses feature information from both visual and language modal data, which can enhance the perception ability of the large model (composed of a multimodal model and a language model) on multimodal data, thereby improving the accuracy of reasoning. Specifically, by inputting the image context into the multimodal model, the interpretability of the multimodal large model in handling actual business is improved, and the answers are traceable; by inputting the intermediate reasoning logic output by the multimodal model into the language model, the explicit generation of the large model logical reasoning intermediate process is achieved, which can not only improve the accuracy of reasoning, but also visualize the reasoning logic and solve the black box characteristics of the large model. Therefore, based on the model training method disclosed in this embodiment, the prediction accuracy of the large model can be greatly improved.

[0075] Figure 4is based on Figure 3 The flowchart of the model training method further proposed in the embodiment shown is as follows. Figure 4 As shown, in another exemplary embodiment, the model training method further includes S410, and the process of optimizing the parameters of the multimodal model and the language model based on the training loss value further includes S420, which is described in detail as follows:

[0076] S410: Input the training image into the image context extraction model to obtain context information output by the image context extraction model.

[0077] In this embodiment, the context information of the training image is automatically extracted by the image context extraction model. By inputting the training image into the image context extraction model, the context information corresponding to the output of the image context extraction model can be obtained.

[0078] S420, based on the training loss value, optimizing parameters of the image context extraction model, the multimodal model, and the language model.

[0079] In the technical solution involved in this embodiment, the large model includes not only the multimodal model and the language model, but also the image context extraction model. Since the image context extraction model will also cause a certain degree of training loss in the process of extracting context information of the training image, after obtaining the training loss value, it is necessary to optimize the parameters of the image context extraction model, the multimodal model and the language model based on the training loss value, so as to realize the training of the large model.

[0080] Therefore, in the accurate stage of training the data set, this embodiment does not need to prepare additional context information for each training image, but only requires questions and answer options for each training image, which reduces the complexity of large model training.

[0081] In other exemplary embodiments, considering that when it comes to specific fields, the training data obtained from public resources may not be sufficient, the scarcity of data may become a huge obstacle to training large models. To this end, generative models such as GAN (Generative Adversarial Networks) and Stable Diffusion can be used to automatically generate training data containing images, questions, and answer options. The generative model is also pre-trained with a small amount of training data. The data generated by the generative model can also be strictly manually reviewed to ensure its authenticity and diversity, and is suitable for the training needs of large models.

[0082] It is also possible to utilize existing training data (including images, questions, and answer options) and generate new training data through context reconstruction techniques, such as modifying, replacing, or redrawing the image content in the image, or modifying the questions and answer options, without limitation.

[0083] In other exemplary embodiments, the model training method may further include a process of fine-tuning the instructions of the trained large model, so that the large model after fine-tuning the instructions is more suitable for the actual application scenario. The instruction fine-tuning data set is usually derived from the actual application scenario, or is related to the actual application scenario.

[0084] In the instruction fine-tuning stage, the focus is on strengthening the model's instruction-following ability. To this end, at least one of the following methods can be used to expand and enrich the instruction fine-tuning dataset. For example, instructions from actual application scenarios can be refined into multiple specific scenarios, and the dataset for each instruction scenario contains detailed instruction descriptions and corresponding training data examples to ensure that the large model can accurately execute instructions in various instruction scenarios. For another example, an instruction dataset containing multiple rounds of dialogue can be designed so that the model can understand and execute complex instructions in continuous interactions. A mechanism for dynamically adjusting instructions can also be introduced to enable the model to flexibly respond to changes in instructions by adding instances containing change instructions or priority adjustments to the training dataset.

[0085] Please continue reading Figure 5 , Figure 5 is a flowchart of an image processing method shown in an exemplary embodiment of the present application. It should be noted that the method can be applied to Figure 1 The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0086] like Figure 5 As shown, in an exemplary embodiment, the image processing method includes S510-S530, which are described in detail as follows:

[0087] S510, extracting context information of the image to be processed, and obtaining question text and question answer options corresponding to the image to be processed.

[0088] First of all, it should be noted that the image processing method disclosed in this embodiment is a process of applying reasoning using the large model trained in the above embodiments. Therefore, the application reasoning process involved in the image processing method disclosed in this embodiment is consistent with the process of training the large model in the above embodiments.

[0089] In this embodiment, the context information of the image to be processed can be extracted by a trained image context extraction model. The trained image context extraction model can be included in the trained large model itself, or can be an additional pre-trained model, which is not limited here.

[0090] The question text corresponding to the image to be processed is usually a user question input by a user. In some exemplary embodiments, the question text corresponding to the image to be processed may also be a preset question, which is not limited here either.

[0091] The answer options can be generated based on the analysis results obtained after the question text is parsed. The question parsing of the question text may include extracting keywords from the question text. For example, the answer options can be generated based on all candidate answers corresponding to the question. For example, for the question "What season is it now?", the keyword "season" can be obtained through question parsing. Spring, summer, ball season and winter are all candidate answers, so four answer options can be generated accordingly.

[0092] In other exemplary embodiments, considering that too many question answer options may affect the accuracy of the large model to a certain extent, a certain number of target candidate answers may be selected from all candidate answers, and question answer options may be generated based on the target candidate answers. For example, the target candidate answers may be selected based on the context information of the image to be processed. For example, if the key information of "lotus blooms" is obtained from the context information of the image to be processed, and it can be determined based on empirical knowledge that lotus does not bloom in winter and spring, summer and autumn may be determined as target candidate answers.

[0093] S520, based on the context information, the question text and the answer options, generate a first input text, and input the first input text and the image to be processed into the trained multimodal model to obtain the intermediate reasoning logic output by the multimodal model.

[0094] This embodiment generates a first input text based on the context information, question text and question answer options according to the logic template, and then inputs the first input text and the image to be processed into the trained multimodal model to obtain the intermediate reasoning logic output by the multimodal model. The multimodal model has a high prediction accuracy based on the training mentioned in the above embodiment, so the accuracy of the intermediate reasoning logic output by the trained multimodal model is also high, which is reflected in the degree of matching between the intermediate reasoning logic and the image content, question text, and question answer options.

[0095] S530, based on the intermediate reasoning logic, the question text and the question answer options, generate a second input text, and input the second input text into the trained language model to obtain a prediction result output by the language model, which is an answer text in the question answer options.

[0096] After obtaining the intermediate reasoning logic, a second input text is generated according to the reasoning template based on the intermediate reasoning logic, the question text and the question answer options. The second input text is input into the trained language model so that the trained language model outputs a prediction result accordingly. The prediction result is an answer text in the question answer options. Similarly, the language model has a high prediction accuracy based on the training mentioned in the above embodiment. Therefore, the accuracy of the prediction result output by the trained language model is also high, and the accuracy is reflected in the degree of matching between the prediction result and the true answer.

[0097] Therefore, the image processing method disclosed in this embodiment introduces the technical means of thought chain reasoning in the process of processing data in both visual and language modalities. Specifically, the intermediate reasoning logic of the trained multimodal model about the question text is obtained through the context information of the image to be processed, and then the predicted answer of the trained language model about the question text is obtained through the intermediate reasoning logic. This not only improves the interpretability of the multimodal model, but also explicitly generates the intermediate reasoning logic, which can not only improve the reasoning accuracy, but also visualize the reasoning logic, solving the black box characteristics of the large model.

[0098] Figure 6 is a block diagram of a model training device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The implementation environment shown, for example, can be deployed on the server 120, can also be deployed on the terminal 110, or can be deployed on both the terminal 110 and the server 120. Of course, the device can also be applied to other implementation environments, and can be deployed on a terminal or server in other implementation environments, or can be deployed on a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0099] like Figure 6As shown, in an exemplary embodiment, the model training device includes:

[0100] A first processing module 610 is configured to generate a first training text based on context information, questions, and answer options corresponding to the training image, and input the first training text and the training image into the multimodal model to obtain an intermediate reasoning logic output by the multimodal model;

[0101] The second processing module 620 is configured to generate a second training text based on the intermediate reasoning logic, the training question and the question answer options, and input the second training text into the language model to obtain a predicted answer output by the language model, where the predicted answer is the answer text in the question answer options;

[0102] A loss determination module 630, configured to determine a training loss value based on the intermediate reasoning logic, the predicted answer, and the true answer corresponding to the question;

[0103] The model optimization module 640 is configured to optimize the parameters of the multimodal model and the language model based on the training loss value.

[0104] In another exemplary embodiment, the loss determination module 630 is further configured to perform the following steps:

[0105] The first cross entropy loss value is calculated based on the intermediate reasoning logic and the true answer, and the second cross entropy loss value is calculated based on the predicted answer and the true answer;

[0106] A training loss value is determined based on the first cross entropy loss value and the second cross entropy loss value.

[0107] In another exemplary embodiment, the question answer options include answer texts of at least two options, and one of the answer texts is a true answer corresponding to the training image.

[0108] In another exemplary embodiment, the model training device further includes:

[0109] an extraction module configured to input the training image into the image context extraction model to obtain context information output by the image context extraction model;

[0110] The model optimization module 640 is further configured to optimize parameters of the image context extraction model, the multimodal model, and the language model based on the training loss value.

[0111] Figure 7 is a block diagram of an image processing device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1The implementation environment shown, for example, can be deployed on the server 120, can also be deployed on the terminal 110, or can be deployed on both the terminal 110 and the server 120. Of course, the device can also be applied to other implementation environments, and can be deployed on a terminal or server in other implementation environments, or can be deployed on a terminal and a server in other implementation environments, and this embodiment is not limited thereto.

[0112] like Figure 7 As shown, in an exemplary embodiment, the image processing device includes:

[0113] The acquisition module 710 is configured to extract context information of the image to be processed, and obtain the question text and question answer options corresponding to the image to be processed;

[0114] The reasoning module 720 is configured to generate a first input text based on the context information, the question text and the answer options, and input the first input text and the image to be processed into the trained multimodal model to obtain the intermediate reasoning logic output by the multimodal model;

[0115] The prediction module 730 is configured to generate a second input text based on the intermediate reasoning logic, the question text and the question answer options, and input the second input text into the trained language model to obtain a prediction result output by the language model, where the prediction result is an answer text in the question answer options.

[0116] In another exemplary embodiment, the image processing apparatus further includes:

[0117] The context extraction module is configured to input the image to be processed into the trained image context extraction model to obtain the context information output by the image context extraction model.

[0118] It should be noted that the model training device and image processing device provided in the above embodiment belong to the same concept as the model training method and image processing method provided in the above embodiment, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0119] The device provided in the above embodiment introduces the technical means of thought chain reasoning in the process of processing data in both visual and language modalities. Specifically, the intermediate reasoning logic of the trained multimodal model about the question text is obtained through the context information of the image to be processed, and then the predicted answer of the trained language model about the question text is obtained through the intermediate reasoning logic. This not only improves the interpretability of the multimodal model, but also explicitly generates the intermediate reasoning logic, which can not only improve the reasoning accuracy, but also visualize the reasoning logic, solving the black box characteristics of the large model.

[0120] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the model training method or image processing method provided in the above-mentioned embodiments.

[0121] Figure 8 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0122] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the method described in the above embodiment. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, ROM 802 and RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0123] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0124] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.

[0125] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0126] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0128] Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0129] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in each of the above embodiments.

[0130] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A model training method, characterized in that: The method comprises: Generate a first training text based on context information, questions, and answer options corresponding to the training image, and input the first training text and the training image into a multimodal model to obtain an intermediate reasoning logic output by the multimodal model; Based on the intermediate reasoning logic, the training question and the question answer options, a second training text is generated, and the second training text is input into a language model to obtain a predicted answer output by the language model, wherein the predicted answer is an answer text in the question answer options; Determine a training loss value according to the intermediate reasoning logic, the predicted answer, and the true answer corresponding to the question; Parameters of the multimodal model and the language model are optimized based on the training loss value.

2. The method according to claim 1, characterized in that The determining of the training loss value according to the intermediate reasoning logic, the predicted answer and the true answer corresponding to the question includes: Calculating a first cross entropy loss value according to the intermediate reasoning logic and the true answer, and calculating a second cross entropy loss value according to the predicted answer and the true answer; The training loss value is determined based on the first cross entropy loss value and the second cross entropy loss value.

3. The method according to claim 1, characterized in that The question answer options include answer texts of at least two options, and one of the answer texts is a true answer corresponding to the training image.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Inputting the training image into an image context extraction model to obtain the context information output by the image context extraction model; The performing parameter optimization on the multimodal model and the language model based on the training loss value includes: Based on the training loss value, parameters of the image context extraction model, the multimodal model and the language model are optimized.

5. An image processing method, characterized in that: The method comprises: Extracting context information of the image to be processed, and obtaining question text and question answer options corresponding to the image to be processed; Based on the context information, the question text and the answer options, a first input text is generated, and the first input text and the image to be processed are input into a trained multimodal model to obtain an intermediate reasoning logic output by the multimodal model; Based on the intermediate reasoning logic, the question text and the question answer options, a second input text is generated, and the second input text is input into the trained language model to obtain a prediction result output by the language model, wherein the prediction result is an answer text in the question answer options.

6. The method according to claim 5, characterized in that The method further comprises: The image to be processed is input into a trained image context extraction model to obtain the context information output by the image context extraction model.

7. A model training device, characterized in that: The device comprises: A first processing module is configured to generate a first training text based on context information, questions, and answer options corresponding to the training image, and input the first training text and the training image into a multimodal model to obtain an intermediate reasoning logic output by the multimodal model; A second processing module is configured to generate a second training text based on the intermediate reasoning logic, the training question and the question answer options, and input the second training text into the language model to obtain a predicted answer output by the language model, wherein the predicted answer is an answer text in the question answer options; a loss determination module, configured to determine a training loss value according to the intermediate reasoning logic, the predicted answer, and a true answer corresponding to the question; A model optimization module is configured to optimize parameters of the multimodal model and the language model based on the training loss value.

8. An image processing device, characterized in that: The device comprises: An acquisition module is configured to extract context information of the image to be processed, and obtain question text and question answer options corresponding to the image to be processed; A reasoning module, configured to generate a first input text based on the context information, the question text and the answer options, and input the first input text and the image to be processed into a trained multimodal model to obtain an intermediate reasoning logic output by the multimodal model; The prediction module is configured to generate a second input text based on the intermediate reasoning logic, the question text and the question answer options, and input the second input text into the trained language model to obtain a prediction result output by the language model, wherein the prediction result is an answer text in the question answer options.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images