Reconstruction-guided multi-modal large model training method, device, application and equipment

Abstract

The invention provides a reconstruction-guided multi-modal large model training method and device, application, equipment, a medium and a program product. The method comprises the following steps: reconstructing a sample image to obtain reconstructed image features; the sample visual features for the sample image and the sample text features for the sample text are input into a multi-modal large model, predicted visual features and predicted text features are output, and the sample text is associated with the sample image; determining a visual loss value of reconstruction guidance based on the reconstructed image features and the predicted visual features; determining a text loss value based on the predicted text features and the sample text features; and training the multi-modal large model based on the visual loss value and the text loss value of the reconstruction guidance to obtain a trained multi-modal large model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to deep learning, computer vision, pattern recognition, multimodal learning, and large language models, and more specifically to reconstruction-guided multimodal large model training methods, devices, applications, and equipment. Background Art

[0002] The success of large language models in the style of Generative Pre-trained Transformer (GPT) has led researchers to try to make these models understand multiple types of data, such as images and videos. Traditional multimodal large models usually adopt a plug-in structure, in which a pre-trained visual-language base model, such as CLIP, is responsible for converting images into visual features for multimodal understanding tasks. However, this design causes the understanding of images to depend on the performance of the selected visual model, resulting in poor fine-grained understanding capabilities. Summary of the invention

[0003] In view of this, the present invention provides a reconstruction-guided multi-modal large model training method, apparatus, application, equipment, medium and program product.

[0004] One aspect of the present invention provides a reconstruction-guided multimodal large model training method, comprising: reconstructing a sample image to obtain reconstructed image features; inputting sample visual features for the sample image and sample text features for the sample text into the multimodal large model, and outputting predicted visual features and predicted text features, wherein the sample text is associated with the sample image; determining a reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features; determining a text loss value based on the predicted text features and the sample text features; and training the multimodal large model based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

[0005] According to an embodiment of the present invention, a reconstruction-guided visual loss value is determined based on reconstructed image features and predicted visual features, including: combining the reconstructed image features and random noise to obtain mixed features; inputting the mixed features into a denoiser so that the denoiser outputs predicted noise using the predicted visual features as denoising conditions; and determining the reconstruction-guided visual loss value based on the predicted noise and the random noise.

[0006] According to an embodiment of the present invention, determining a reconstruction-guided visual loss value based on predicted noise and random noise includes: obtaining the reconstruction-guided visual loss value according to the square of the difference between the predicted noise and the random noise.

[0007] According to an embodiment of the present invention, the sample text includes: a sample question text and a sample answer text matching the sample question text, the sample question text is used to represent the question asked for the sample image, and the sample answer text is used to represent the answer to the question based on the sample image;

[0008] The reconstruction-guided multimodal large model training method also includes: inputting the sample image into a visual encoder to output sample visual features; and inputting the sample question text and the sample answer text matching the sample question text into a language segmentor to output sample text features.

[0009] According to an embodiment of the present invention, a multimodal large model is trained based on a reconstruction-guided visual loss value and a text loss value to obtain a trained multimodal large model, including: summing the reconstruction-guided visual loss value and the text loss value to obtain a total loss value; and based on the total loss value, adjusting the parameters of the multimodal large model and the visual encoder until the number of training times reaches a maximum number of loss iterations to obtain a trained multimodal large model.

[0010] According to an embodiment of the present invention, the sample text features include features corresponding to each word in the sample text and a label for predicting the next word, and the sample visual features include visual features corresponding to each pixel in the sample image and a label for predicting the visual features;

[0011] Based on the predicted text features and the sample text features, a text loss value is determined, including: determining a predicted question text in the predicted text features and a predicted answer text matching the predicted question text; and determining a text loss value based on the predicted answer text and the sample answer text.

[0012] According to an embodiment of the present invention, a sample image is reconstructed to obtain reconstructed image features, including: determining the color value of each pixel in the sample image as a reconstruction target, reconstructing the sample image, and obtaining reconstructed image features; or inputting the sample image into a self-supervised visual model or a visual model with reconstruction capabilities, and outputting reconstructed image features.

[0013] A second aspect of the present invention provides an application of a multimodal large model trained by a reconstruction-guided multimodal large model training method in image reconstruction or understanding of image content.

[0014] The third aspect of the present invention provides a reconstruction-guided multimodal large model training device, comprising: a reconstruction module, used to reconstruct a sample image to obtain reconstructed image features; a prediction module, used to input sample visual features for the sample image and sample text features for the sample text into the multimodal large model, and output predicted visual features and predicted text features, wherein the sample text is associated with the sample image; a first determination module, used to determine a reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features; a second determination module, used to determine a text loss value based on the predicted text features and the sample text features; and a training module, used to train the multimodal large model based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

[0015] The fourth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the reconstruction-guided multimodal large model training method.

[0016] The fifth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implements the steps of the reconstruction-guided multimodal large model training method.

[0017] The sixth aspect of the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the reconstruction-guided multimodal large model training method.

[0018] According to an embodiment of the present invention, by reconstructing a sample image to obtain reconstructed image features, based on the reconstructed image features and the predicted visual features, a reconstruction-guided visual loss is determined, and the reconstruction-guided visual loss is combined with the text loss to train a multimodal large model. Since training a multimodal large model based on reconstruction-guided visual loss enables the multimodal large model to learn the fine-grained features of the sample image, and training a multimodal large model based on text loss enables the multimodal large model to learn the text description features of the sample image, and the fine-grained features and the text description features have a correlation, therefore, combining the reconstruction-guided visual loss with the text loss to train the multimodal large model enables the multimodal large model to focus on the details of the image and enhance the fine-grained understanding ability of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1A flowchart of a reconstruction-guided multi-modal large model training method according to an embodiment of the present invention is schematically shown;

[0021] Figure 2 A schematic diagram of a reconstruction-guided multi-modal large model training method according to another embodiment of the present invention is schematically shown;

[0022] Figure 3 A schematic diagram of a reconstruction-guided multi-modal large model training method according to yet another embodiment of the present invention is schematically shown;

[0023] Figure 4 A structural block diagram of a reconstruction-guided multi-modal large model training device according to an embodiment of the present invention is schematically shown; and

[0024] Figure 5 A block diagram of an electronic device suitable for implementing a reconstruction-guided multi-modal large model training method according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0025] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a method having at least one of A, B and C" should include but is not limited to a method having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] In the process of implementing the concept of the present invention, the inventors found that the traditional multimodal large model usually adopts a plug-in structure, in which a pre-trained visual-language basic model, such as CLIP, is responsible for converting images into visual features for multimodal understanding tasks. However, this design causes the understanding of the image to depend on the performance of the selected visual model, resulting in poor fine-grained understanding ability.

[0030] For example, in the related examples, although the images are publicly converted into visual features, the visual features are used to assist the large model in understanding the text. Since it relies on the performance of the selected visual model, there is still the problem of poor fine-grained understanding ability of the multimodal large model.

[0031] Based on this, the embodiments of the present invention provide a reconstruction-guided multimodal large model training method, device, application, equipment, medium and program product. The method includes: reconstructing a sample image to obtain reconstructed image features; inputting sample visual features for the sample image and sample text features for the sample text into the multimodal large model, outputting predicted visual features and predicted text features, wherein the sample text is associated with the sample image; determining a reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features; determining a text loss value based on the predicted text features and the sample text features; and training the multimodal large model based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

[0032] The following will be passed Figure 1~Figure 3 The reconstruction-guided multimodal large model training method of an embodiment of the invention is described in detail.

[0033] Figure 1 The flowchart of the reconstruction-guided multi-modal large model training method according to an embodiment of the present invention is schematically shown.

[0034] like Figure 1 As shown, the reconstruction-guided multimodal large model training method includes operations S110 to S150.

[0035] In operation S110, the sample image is reconstructed to obtain reconstructed image features.

[0036] In operation S120 , sample visual features for the sample image and sample text features for the sample text are input into the multimodal macro model, and predicted visual features and predicted text features are output.

[0037] In operation S130 , a reconstruction-guided visual loss value is determined based on the reconstructed image features and the predicted visual features.

[0038] In operation S140 , a text loss value is determined based on the predicted text features and the sample text features.

[0039] In operation S150, the multimodal large model is trained based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

[0040] According to an embodiment of the present invention, the sample text is associated with the sample image. For example, the sample text can be used to describe the semantic information of the sample image. The sample image can be an image associated with the actual application determined according to the actual application.

[0041] According to an embodiment of the present invention, the reconstruction may be a reconstruction using an existing reconstruction model capable of fine-grained characterization, or a reconstruction through a mapping relationship. The reconstructed image features may be used to characterize the fine-grained features of the reconstructed image. The reconstructed image may be completely consistent with the original sample image.

[0042] According to an embodiment of the present invention, both the sample visual features and the sample text features may be obtained by encoding the sample image and the sample text respectively through an encoder.

[0043] According to an embodiment of the present invention, the multimodal large model may be able to infer sample visual features of sample images after pre-annotation, output predicted visual features, infer sample text features, output predicted text features, or infer sample visual features and sample text features, and output predicted text features.

[0044] According to an embodiment of the present invention, the visual loss value of the reconstruction guidance can be determined according to the difference between the reconstructed image feature and the predicted visual feature. The text loss value can be determined according to the difference between the predicted text feature and the sample text feature.

[0045] According to an embodiment of the present invention, the parameters of the multimodal large model can be iteratively adjusted according to the cross entropy loss of the reconstruction-guided visual loss value and the text loss value to achieve training of the multimodal large model.

[0046] According to an embodiment of the present invention, by reconstructing a sample image to obtain reconstructed image features, based on the reconstructed image features and the predicted visual features, a reconstruction-guided visual loss is determined, and the reconstruction-guided visual loss is combined with the text loss to train a multimodal large model. Since training a multimodal large model based on reconstruction-guided visual loss enables the multimodal large model to learn the fine-grained features of the sample image, and training a multimodal large model based on text loss enables the multimodal large model to learn the text description features of the sample image, and the fine-grained features and the text description features have an associated relationship, therefore, combining the reconstruction-guided visual loss with the text loss to train the multimodal large model assists the training of the conventional multimodal large model through the reconstruction-guided visual loss, and therefore enables the multimodal large model to focus on the details of the image and enhance the fine-grained understanding ability of the multimodal large model.

[0047] According to an embodiment of the present invention, for the above Figure 1 In operation S110, the sample image is reconstructed to obtain the reconstructed image features, which may include the following operations: determining the color value of each pixel in the sample image as a reconstruction target, reconstructing the sample image, and obtaining the reconstructed image features; or inputting the sample image into a self-supervised visual model or a visual model with reconstruction capability, and outputting the reconstructed image features.

[0048] According to an embodiment of the present invention, the self-supervised visual model may include, but is not limited to, DINOv2, etc. The visual model with reconstruction capability may include, but is not limited to, VQGAN, etc.

[0049] According to an embodiment of the present invention, the RGB value of the sample image may be used as a reconstruction target and directly mapped to obtain the reconstructed image features.

[0050] According to an embodiment of the present invention, by introducing additional visual supervision signals, namely, reconstructing images, the ability of a multimodal large model to understand image details is enhanced.

[0051] Figure 2 A schematic diagram of a reconstruction-guided multimodal large model training method according to another embodiment of the present invention is schematically shown.

[0052] According to an embodiment of the present invention, the sample text may include: a sample question text and a sample answer text matching the sample question text, the sample question text may be used to characterize the question asked for the sample image, and the sample answer text may be used to characterize the answer to the question based on the sample image.

[0053] Reconstruction-guided multimodal large model training methods can include Figure 1In addition to the operations S110 to S150 shown, the operations may also include: inputting a sample image into a visual encoder to output sample visual features; and inputting a sample question text and a sample answer text matching the sample question text into a language segmentor to output sample text features.

[0054] According to the embodiment of the present invention, the visual encoder and the language segmenter are not specifically limited in the present invention.

[0055] For example, Figure 2 As shown, for a sample image showing an airplane, the sample question text may be "what is in this picture" issued by the user, and the sample answer text matching the sample question text may be "this picture shows a passenger plane" answered by the model. The sample image showing an airplane can be input into the visual encoder to be trained, and the sample visual features are output. "What is in this picture" and "this picture shows a passenger plane" are input into a frozen language segmenter, that is, a language segmenter that does not participate in training, to obtain sample text features. The sample text features may be composed of sample question text features and sample answer text features.

[0056] The sample visual features and sample text features are input into the multimodal large model to be trained. The multimodal large model outputs predicted visual features for the sample visual features and outputs predicted text features by predicting the next word for the sample text features. Based on the sample image, the visual loss guided by the reconstruction of the predicted visual features can be determined. Based on the sample text features, determine the text loss of the predicted text features .based on and , iteratively optimize the parameters of the visual encoder and the multimodal large model.

[0057] According to an embodiment of the present invention, by encoding the sample image, then determining the reconstruction-guided visual loss, and then assisting the text loss to train the multimodal large model, compared to the related examples of converting the image into visual features and using the visual features to assist the large model in understanding the text, the present invention does not rely on the performance of the selected visual model, and enhances the fine-grained understanding ability of the multimodal large model. In addition, since the multimodal large model is trained based on sample images and sample texts, the multimodal large model trained by the present invention can be independent of the form of image data or text data, and can be used for image reconstruction or understanding image content.

[0058] Figure 3 A schematic diagram of a reconstruction-guided multi-modal large model training method according to yet another embodiment of the present invention is schematically shown.

[0059] In the process of implementing the embodiments of the present invention, it is found that by designing a loss function, the model complexity can be controlled to prevent overfitting, while dealing with the problem of data imbalance and improving the performance of the model on a minority of categories. In addition, the loss function also helps to evaluate the performance of the model, ensuring that the model not only fits the training data during the training process, but also performs well on new, unseen data. Therefore, reasonable design and adjustment of the loss is crucial to improving the optimization effect of the feature model parameters.

[0060] Based on this, for the above Figure 1 Operation S130, determining the reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features, may include the operations of: combining the reconstructed image features and random noise to obtain a mixed feature; inputting the mixed feature into a denoiser so that the denoiser outputs a predicted noise with the predicted visual features as a denoising condition; and determining the reconstruction-guided visual loss value based on the predicted noise and the random noise.

[0061] According to an embodiment of the present invention, the random noise may be obtained by random sampling from a noise library or multiple noises. The reconstructed image features may be obtained by inputting the sample image into a fine-grained visual feature extractor to obtain clean latent features of the sample image.

[0062] According to an embodiment of the present invention, the reconstructed image features and random noise may be randomly combined to obtain mixed features.

[0063] According to an embodiment of the present invention, a reconstruction-guided visual loss value may be obtained according to the difference between the predicted noise and the random noise.

[0064] According to an embodiment of the present invention, by adding noise to the reconstructed image features as the input of the denoiser, the visual output of the multimodal large model, i.e., the predicted visual features, is used as the condition of the denoiser, so that the denoiser predicts the added noise, and the loss of the predicted noise is determined as the visual loss guided by the reconstruction. Since the visual output of the multimodal large model is used as the condition of the denoiser during the prediction process of the denoiser, the denoiser can refer to the visual output of the multimodal large model to predict the added noise, so when the prediction accuracy is higher, the output of the multimodal large model is closer to the sample image, that is, the multimodal large model can pay more attention to the fine-grained features of the image, and the fine-grained understanding ability is enhanced.

[0065] According to an embodiment of the present invention, determining a reconstruction-guided visual loss value based on predicted noise and random noise may include an operation of obtaining a reconstruction-guided visual loss value according to a square of a difference between the predicted noise and the random noise.

[0066] For example, Figure 3 As shown, the sample image can be Input into a frozen fine-grained representation model to obtain the fine-grained features of the image, that is, the reconstructed image feature z0; randomly sample a noise, that is, random noise ,Will Combined with z0, we get a fine-grained feature with noise, namely the mixed feature z t ; The visual output of the multimodal large model is the predicted visual features , N is the length of the sample visual feature, as the condition of the denoiser, so that the denoiser takes z t As input, predict the added noise, that is, predict the noise , and supervise it. The supervised objective function can include the square loss of prediction noise and random noise, as shown in the following formula (1):

[0067] (1)

[0068] Input the sample image into the visual encoder to be trained, and the sample visual features can be output , D is the feature dimension. Input the sample text into the frozen language segmenter, that is, the language segmenter that does not participate in the training, and obtain the sample text feature , T is the total length of the feature. Sample visual features and sample text features are input into the multimodal large model, and the output is the predicted visual features and text output, that is, the predicted text features .

[0069] It should be noted that the objective function can also be other reasonable loss functions.

[0070] According to an embodiment of the present invention, by calculating the square loss of random noise and prediction noise, it is possible to guide the direction of multimodal large model training, and optimize model parameters through visual loss tuning guided by denoising reconstruction, thereby affecting the generalization ability and robustness of the model.

[0071] According to an embodiment of the present invention, the sample text features may include features corresponding to each word in the sample text and a label for predicting the next word, and the sample visual features include visual features corresponding to each pixel in the sample image and a label for predicting the visual features.

[0072] According to an embodiment of the present invention, determining a text loss value based on predicted text features and sample text features may include the following operations: determining a predicted question text in the predicted text features and a predicted answer text matching the predicted question text; and determining a text loss value based on the predicted answer text and the sample answer text.

[0073] According to another embodiment of the present invention, for the above Figure 1Operation S130, determining a reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features, may include the following operations: constructing a linear regression loss function based on the reconstructed image features and the predicted visual features, and then calculating the linear regression loss value, and determining the linear regression loss value as the reconstruction-guided visual loss value.

[0074] According to an embodiment of the present invention, training a multimodal large model based on a reconstruction-guided visual loss value and a text loss value to obtain a trained multimodal large model may include the following operations: summing the reconstruction-guided visual loss value and the text loss value to obtain a total loss value; and adjusting the parameters of the multimodal large model and the visual encoder based on the total loss value until the number of training times reaches a maximum number of loss iterations to obtain a trained multimodal large model.

[0075] According to an embodiment of the present invention, for the text output part, the next word is predicted to be trained in the answer text part, and for the image output part, the reconstruction-guided visual loss proposed by the present invention is used for training. In the reasoning stage, the normal reasoning process of the multimodal large model is retained, and the next text word is continuously output based on the visual features. Since the reconstruction-guided visual loss is proposed, the input image is reconstructed to assist the training of the multimodal large model, so at least part of the problem of insufficient fine-grained understanding ability caused by the multimodal large model's over-reliance on the visual encoder can be solved.

[0076] According to an embodiment of the present invention, a multimodal large model trained by a reconstruction-guided multimodal large model training method can be applied in image reconstruction or understanding image content.

[0077] For example, the image to be reconstructed can be input into a trained multimodal large model, the predicted visual features can be output, and the image can be rendered based on the predicted visual features to obtain a reconstructed and rendered image. The image to be analyzed and the instruction to be analyzed, that is, the question text to be analyzed, can also be input into a trained multimodal large model, the predicted answer text features can be output, and the analysis results of the content of the image to be analyzed can be obtained based on the predicted answer text features.

[0078] Figure 4 A block diagram of a reconstruction-guided multi-modal large model training apparatus according to an embodiment of the present invention is shown.

[0079] like Figure 4 As shown, the reconstruction-guided multimodal large model training device 400 includes a reconstruction module 410, a prediction module 420, a first determination module 430, a second determination module 440 and a training module 450.

[0080] The reconstruction module 410 is used to reconstruct the sample image to obtain reconstructed image features.

[0081] The prediction module 420 is used to input sample visual features for sample images and sample text features for sample texts into the multimodal macro model, and output predicted visual features and predicted text features, wherein the sample texts are associated with the sample images.

[0082] The first determination module 430 is used to determine the reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features.

[0083] The second determination module 440 is used to determine the text loss value based on the predicted text features and the sample text features.

[0084] The training module 450 is used to train the multimodal large model based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

[0085] According to an embodiment of the present invention, the sample text may include: a sample question text and a sample answer text matching the sample question text, the sample question text is used to characterize the question posed for the sample image, and the sample answer text is used to characterize the answer to the question based on the sample image. The reconstruction-guided multimodal large model training device 400 may also include: a visual encoding module and a text encoding module.

[0086] The visual encoding module is used to input sample images into the visual encoder and output sample visual features.

[0087] The text encoding module is used to input the sample question text and the sample answer text matching the sample question text into the language segmentor, and output the sample text features.

[0088] According to an embodiment of the present disclosure, any multiple modules of the reconstruction module 410, the prediction module 420, the first determination module 430, the second determination module 440 and the training module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the reconstruction module 410, the prediction module 420, the first determination module 430, the second determination module 440 and the training module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), an on-chip method, a method on a substrate, a method on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the reconstruction module 410, the prediction module 420, the first determination module 430, the second determination module 440 and the training module 450 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0089] It should be noted that the reconstruction-guided multimodal large model training device part in the embodiment of the present invention corresponds to the reconstruction-guided multimodal large model training method part in the embodiment of the present invention. The description of the reconstruction-guided multimodal large model training device part specifically refers to the reconstruction-guided multimodal large model training method part, which will not be repeated here.

[0090] Figure 5 A block diagram of an electronic device suitable for implementing a reconstruction-guided multi-modal large model training method according to an embodiment of the present invention is schematically shown.

[0091] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0092] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0093] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage portion 508 as needed.

[0094] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0095] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0096] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0097] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0098] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0099] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0100] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0101] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or couplings fall within the scope of the present invention.

[0103] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A reconstruction-guided multimodal large model training method, characterized in that: The method comprises: Reconstruct the sample image to obtain the reconstructed image features; Inputting sample visual features for the sample image and sample text features for the sample text into a multimodal macro model, and outputting predicted visual features and predicted text features, wherein the sample text is associated with the sample image; Determining a reconstruction-guided visual loss value based on the reconstructed image features and the predicted visual features; Determining a text loss value based on the predicted text features and the sample text features; and Based on the reconstruction-guided visual loss value and the text loss value, the multimodal large model is trained to obtain a trained multimodal large model.

2. The method according to claim 1, characterized in that: The step of determining a reconstruction-guided visual loss value based on the reconstructed image feature and the predicted visual feature comprises: Combining the reconstructed image features with random noise to obtain mixed features; Inputting the mixed features into a denoiser, so that the denoiser uses the predicted visual features as denoising conditions and outputs predicted noise; and Based on the predicted noise and the random noise, a visual loss value of the reconstruction guidance is determined.

3. The method according to claim 2, characterized in that The step of determining the visual loss value of the reconstruction guidance based on the predicted noise and the random noise comprises: The reconstruction-guided visual loss value is obtained according to the square of the difference between the predicted noise and the random noise.

4. The method according to claim 1, characterized in that: The sample text includes: a sample question text and a sample answer text matching the sample question text, wherein the sample question text is used to represent the question posed for the sample image, and the sample answer text is used to represent the answer to the question based on the sample image; The method further comprises: Input the sample image into a visual encoder and output the sample visual features; and The sample question text and the sample answer text matched with the sample question text are input into a language segmenter, and the sample text features are output.

5. The method according to claim 4, characterized in that The step of training the multimodal large model based on the visual loss value and the text loss value guided by the reconstruction to obtain a trained multimodal large model comprises: Summing the reconstruction-guided visual loss value and the text loss value to obtain a total loss value; and Based on the total loss value, the parameters of the multimodal large model and the visual encoder are adjusted until the number of training times reaches the maximum number of loss iterations, thereby obtaining the trained multimodal large model.

6. The method according to claim 4, characterized in that The sample text features include features corresponding to each word in the sample text and a label used to predict the next word, and the sample visual features include visual features corresponding to each pixel in the sample image and a label used to predict the visual features; The determining of the text loss value based on the predicted text feature and the sample text feature includes: Determining a predicted question text in the predicted text feature and a predicted answer text matching the predicted question text; and The text loss value is determined according to the predicted answer text and the sample answer text.

7. The method according to claim 1, characterized in that The reconstructing the sample image to obtain the reconstructed image features includes: Determine the color value of each pixel in the sample image as a reconstruction target, reconstruct the sample image, and obtain the reconstructed image feature; or The sample image is input into a self-supervised visual model or a visual model with reconstruction capability, and the reconstructed image features are output.

8. An application of a large multimodal model trained using the method described in any one of claims 1 to 7 in image reconstruction or understanding of image content.

9. A reconstruction-guided multi-modal large model training device, characterized in that: The device comprises: A reconstruction module, used to reconstruct the sample image and obtain the reconstructed image features; A prediction module, configured to input a sample visual feature for the sample image and a sample text feature for the sample text into the multimodal macro model, and output a predicted visual feature and a predicted text feature, wherein the sample text is associated with the sample image; A first determination module, configured to determine a reconstruction-guided visual loss value based on the reconstructed image feature and the predicted visual feature; A second determination module is used to determine a text loss value based on the predicted text feature and the sample text feature; and A training module is used to train the multimodal large model based on the reconstruction-guided visual loss value and the text loss value to obtain a trained multimodal large model.

10. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images