Training method and device of unified multi-modal model, electronic equipment and storage medium

Abstract

The application discloses a training method and device of a unified multimodal model, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; generating a first predicted visual feature unit sequence based on the first sample description text by using a generation expert model; determining a target loss by using the first predicted visual feature unit sequence; and adjusting parameters of the generation expert model by using the target loss. In the foregoing manner, the application can integrate visual understanding and image generation into a unified multimodal model, and simultaneously realize visual understanding and image generation in one model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, electronic device, and storage medium for a unified multimodal model. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal models, capable of simultaneously processing various types of modal data such as text and images, have been widely applied and extensively researched in fields such as image generation, visual question answering, and cross-modal retrieval. Unified multimodal models, as an important development direction of multimodal technology, aim to achieve deep fusion and collaborative work between different modal data. Among these, the collaborative training of generative expert models and understanding expert models is the key to improving the performance of unified multimodal models.

[0003] However, current training methods for unified multimodal models suffer from technical problems such as low accuracy of visual feature unit sequences generated by generative expert models, ineffective alignment with the feature space of understanding expert models, and insufficient ability to generate visual feature representations due to single training constraints. There is an urgent need for a unified multimodal model training method that can solve these problems. Summary of the Invention

[0004] This application provides at least one training method, apparatus, electronic device, and storage medium for a unified multimodal model.

[0005] This application provides a training method for a unified multimodal model, which includes a generative expert model and a trained understanding expert model, wherein the trained understanding expert model is capable of answering questions based on an input image. The method includes: acquiring a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; wherein the first sample question is labeled with a first true answer; generating a first predicted visual feature unit sequence based on the first sample description text using the generative expert model, the first predicted visual feature unit sequence including a plurality of first visual feature units, each first visual feature unit representing features of different regions in the image described by the first sample description text; determining a target loss using the first predicted visual feature unit sequence; wherein the target loss includes at least one of a first loss and a second loss, the first loss being obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the target visual feature unit sequence being extracted from the first sample image and aligned with a feature space recognizable by the understanding expert model, and the second loss being obtained based on the difference between the first predicted answer and the first true answer, the first predicted answer being the answer about the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence; and adjusting the parameters of the generative expert model using the target loss.

[0006] Therefore, by using the first loss obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the parameters of the generative expert model can be adjusted so that the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text can approximate the target visual feature unit sequence. Since the target visual feature unit sequence can accurately represent the features of different regions in the first sample image, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text can accurately represent the features of different regions in the first sample image. In other words, it drives the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to be as accurate as possible, thereby improving the accuracy of the visual feature unit sequence generated by the generative expert model based on the descriptive text.

[0007] Furthermore, by using the first loss obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the parameters of the generative expert model are adjusted, which enables the feature space of the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to approximate the feature space of the target visual feature unit sequence. Since the target visual feature unit sequence is aligned with the feature space that the understanding expert model can recognize, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text is aligned with the feature space that the understanding expert model can recognize. In other words, the generative expert model is driven to align the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text as closely as possible with the feature space that the understanding expert model can recognize.

[0008] By utilizing the second loss obtained from the difference between the first predicted answer generated by the understanding expert model based on the first predicted visual feature unit sequence and the first true answer, the parameters of the generative expert model are adjusted. This enables the understanding expert model to generate a first predicted answer that approximates the first true answer based on the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text. Therefore, the generative expert model generates a first predicted visual feature unit sequence based on the first sample descriptive text that can be interpreted as having correct semantics by the understanding expert model. In other words, it drives the generative expert model to generate a first predicted visual feature unit sequence based on the first sample descriptive text that can be accurately interpreted by the understanding expert model, thereby enhancing the representational ability of the visual feature unit sequence generated by the generative expert model based on the descriptive text.

[0009] A second aspect of this application provides a training apparatus for a unified multimodal model, the unified multimodal model including a generative expert model and a trained understanding expert model, the trained understanding expert model being able to answer based on an input image; the apparatus includes an acquisition module, a generation module, a determination module, and an adjustment module; the acquisition module is used to acquire a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; wherein, the first sample question is labeled with a first true answer; the generation module is used to use the generative expert model to generate a first predicted visual feature unit sequence based on the first sample description text, the first predicted visual feature unit sequence including a plurality of first visual feature units, each first visual feature unit representing a first... Features of different regions in an image described by a sample description text; a determination module is used to determine a target loss using a first predicted visual feature unit sequence; wherein the target loss includes at least one of a first loss and a second loss, the first loss being obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the target visual feature unit sequence being extracted from the first sample image and aligned with the feature space that the understanding expert model can recognize, and the second loss being obtained based on the difference between the first predicted answer and the first true answer, the first predicted answer being the answer to the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence; an adjustment module is used to adjust the parameters of the generating expert model using the target loss.

[0010] A third aspect of this application provides an electronic device including a memory and a processor, wherein the memory is used to store program instructions and the processor is used to execute the program instructions to implement the above-described training method for a unified multimodal model.

[0011] A fourth aspect of this application provides a computer-readable storage medium for storing program instructions that can be executed to implement the above-described training method for a unified multimodal model.

[0012] The above technical solution utilizes a first loss derived from the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence to adjust the parameters of the generative expert model. This enables the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to approximate the target visual feature unit sequence. Since the target visual feature unit sequence can accurately characterize the features of different regions in the first sample image, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text can accurately characterize the features of different regions in the first sample image. In other words, it drives the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to be as accurate as possible, thereby improving the accuracy of the visual feature unit sequence generated by the generative expert model based on the descriptive text.

[0013] Furthermore, by using the first loss obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the parameters of the generative expert model are adjusted, which enables the feature space of the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to approximate the feature space of the target visual feature unit sequence. Since the target visual feature unit sequence is aligned with the feature space that the understanding expert model can recognize, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text is aligned with the feature space that the understanding expert model can recognize. In other words, the generative expert model is driven to align the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text as closely as possible with the feature space that the understanding expert model can recognize.

[0014] Furthermore, by using the second loss obtained from the difference between the first predicted answer generated by the understanding expert model based on the first predicted visual feature unit sequence and the first true answer, the parameters of the generative expert model are adjusted. This enables the understanding expert model to generate a first predicted answer that approximates the first true answer based on the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text. Therefore, the generative expert model generates a first predicted visual feature unit sequence based on the first sample descriptive text that can be interpreted as having correct semantics by the understanding expert model. In other words, it drives the generative expert model to generate a first predicted visual feature unit sequence based on the first sample descriptive text that can be accurately interpreted by the understanding expert model, thereby enhancing the representational ability of the visual feature unit sequence generated by the generative expert model based on the descriptive text. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an embodiment of the training method for the unified multimodal model provided in this application; Figure 2 This is a schematic diagram of an embodiment of the training framework for the unified multimodal model provided in this application; Figure 3 This is a flowchart illustrating an embodiment of the training method for a pre-trained semantic visual word segmenter provided in this application; Figure 4 This is a flowchart illustrating an embodiment of the pre-training method for the semantic visual word segmenter provided in this application; Figure 5 This is a flowchart illustrating another embodiment of the training method for the unified multimodal model provided in this application; Figure 6 This is a flowchart illustrating another embodiment of the training method for the unified multimodal model provided in this application; Figure 7 This is a schematic diagram of the structure of an embodiment of the training device for the unified multimodal model provided in this application; Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0016] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0017] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0018] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0019] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the training method for the unified multimodal model provided in this application. It should be noted that if substantially the same result is obtained, this embodiment is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the unified multimodal model includes a generative expert model and a trained understanding expert model. The trained understanding expert model is able to provide answers based on the input image. This embodiment includes: Step S11: Obtain the first sample description text, the first sample image corresponding to the first sample description text, and the first sample question about the first sample image.

[0020] Unified Multimodal Models (UMMs) comprise trained understanding expert models capable of providing responses based on input images. These expert models are pre-trained on massive amounts of image and text data, thus exhibiting good performance in general visual understanding tasks and providing accurate responses based on input images. It's important to note that these understanding expert models are Multimodal Large Language Models (MLLMs), which are also pre-trained on massive amounts of image and text data and demonstrate good performance in general visual understanding tasks. Therefore, Unified Multimodal Models possess powerful multimodal understanding capabilities, or more specifically, visual understanding capabilities.

[0021] The unified multimodal model also includes a generative expert model. Untrained generative expert models perform poorly in tasks such as text-to-image generation; that is, they perform poorly in image generation tasks. Therefore, the method in this embodiment is used to train the generative expert model within the unified multimodal model to improve its image generation capabilities.

[0022] The understanding expert model in the unified multimodal model has powerful visual understanding capabilities, while the generative expert model in the unified multimodal model has powerful image generation capabilities. Therefore, the unified multimodal model possesses both powerful visual understanding and image generation capabilities. Essentially, it integrates visual understanding and image generation into a single model, achieving both visual understanding and image generation within a single framework.

[0023] In this embodiment, a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image are obtained; wherein, the first sample question is labeled with a first true answer. It should be noted that the first sample description text is textual information describing the content, features, and / or scene of the first sample image, and is a semantic expression of the first sample image.

[0024] In one embodiment, the first sample description text, the first sample image corresponding to the first sample description text, and the first sample question about the first sample image can be obtained from local storage or cloud storage.

[0025] For example, such as Figure 2 As shown, Figure 2 This is a schematic diagram of an embodiment of the training framework for the unified multimodal model provided in this application. The first sample description text is "A blue umbrella, a brown teddy bear, and a greenvase." The first sample question about the first sample image is "Identify the main objects in this image."

[0026] Step S12: Using a generative expert model, generate a first predicted visual feature unit sequence based on the first sample description text.

[0027] In this embodiment, a generative expert model is used to generate a first predicted visual feature unit sequence based on the first sample descriptive text. The first predicted visual feature unit sequence includes several first visual feature units, each representing the features of a different region in the image described by the first sample descriptive text. Specifically, the generative expert model generates a series of discrete first visual feature units (visual tokens) based on the first sample descriptive text.

[0028] In one implementation, the generating expert model predicts a series of discrete first visual feature units based on the autoregressive prediction of the first sample descriptive text.

[0029] Step S13: Determine the target loss using the first predicted visual feature unit sequence.

[0030] In this embodiment, a target loss is determined using a first predicted visual feature unit sequence. The target loss includes at least one of a first loss and a second loss. The first loss is obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence. The target visual feature unit sequence is extracted from the first sample image and aligned with the feature space that the understanding expert model can recognize. The second loss is obtained based on the difference between the first predicted answer and the first true answer. The first predicted answer is the answer to the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence.

[0031] It should be noted that feature space alignment can be understood as ensuring that feature vectors generated by different models have the same dimension and distribution, and that their semantics remain consistent.

[0032] In one implementation, supervised fine-tuning (SFT) data with diverse problem styles is introduced to avoid the understanding expert model overfitting to the task description. It is important to note that the understanding expert model is required to directly understand the data generated by the generative expert model; therefore, only supervised fine-tuning data generated by the generative expert model and high-quality images are used.

[0033] The target visual feature unit sequence is extracted from the first sample image, therefore, the target visual feature unit sequence can accurately represent the features of different regions in the first sample image. Furthermore, the target visual feature unit sequence is aligned with the feature space that the understanding expert model can recognize. Therefore, in one embodiment, the first loss includes a feature alignment loss that enables the generative expert model to generate a first predicted visual feature unit sequence based on the first sample descriptive text, and that aligns with the feature space that the understanding expert model can recognize. Additionally, the first loss also includes the visual feature unit prediction loss that enables the generative expert model to accurately predict the first predicted visual feature unit sequence based on the first sample descriptive text. ( Figure 2 (The loss for predicting the word corresponding to the blue square). It should be noted that the loss is for predicting visual feature units. It is the prediction loss of the next visual feature unit after the visual feature unit answers.

[0034] In one implementation, the second loss is the text lexical prediction loss. ( Figure 2 The term prediction loss corresponding to the green square is the negative log-likelihood loss, which is the probability of predicting the next term.

[0035] In one embodiment, before generating a first predicted answer to the first sample question using the understanding expert model based on the first predicted visual feature unit sequence, the first sample description text is masked. That is, when using the understanding expert model to answer the first sample question, the first sample description text is masked, so that the understanding expert model can only see the first predicted visual feature unit sequence generated by the generating expert model. This forces the understanding expert model to obtain information from the first predicted visual feature unit sequence, thereby enabling the understanding expert model to generate a first predicted answer to the first sample question based on the first predicted visual feature units.

[0036] In one specific implementation, an attention mask matrix can be used when calculating attention to mask the descriptive text of the first sample.

[0037] In one implementation, some elements in the first predicted visual feature unit sequence are also randomly replaced / discarded. Specifically, a reference visual feature unit sequence is defined as a learnable parameter with the same shape as the first predicted visual feature unit; then, 50% and 100% of the elements in the first predicted visual feature unit sequence are replaced with the reference visual feature unit sequence with probabilities m1 and m2, respectively.

[0038] Step S14: Use the target loss to adjust the parameters of the generated expert model.

[0039] In this embodiment, the parameters of the generated expert model are adjusted using the target loss.

[0040] The target loss includes a first loss, which is obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, extracted from the first sample image. Since the target visual feature unit sequence is extracted from the first sample image, it accurately represents the features of different regions in the first sample image. Therefore, by using the first loss obtained from the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the parameters of the generative expert model can be adjusted so that the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text approximates the target visual feature unit sequence. Because the target visual feature unit sequence accurately represents the features of different regions in the first sample image, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text can accurately represent the features of different regions in the first sample image. This drives the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text to be as accurate as possible, thus improving the accuracy of the visual feature unit sequence generated by the generative expert model based on the descriptive text.

[0041] Furthermore, since the target visual feature unit sequence is aligned with the feature space that the understanding expert model can recognize, by using the first loss obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the parameters of the generative expert model can be adjusted so that the feature space of the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text approximates the feature space of the target visual feature unit sequence. Since the target visual feature unit sequence is aligned with the feature space that the understanding expert model can recognize, the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text is aligned with the feature space that the understanding expert model can recognize, that is, the generative expert model is driven to align the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text as closely as possible with the feature space that the understanding expert model can recognize.

[0042] Furthermore, the generative expert model aligns the sequence of visual feature units describing text generation with the feature space that the understanding expert model can recognize, thereby matching the generative feature space of the generative expert model with the visual feature space of the understanding expert model, or in other words, narrowing the distance between them. By aligning the generative feature space of the generative expert model with the visual feature space of the understanding expert model, that is, by aligning their representational capabilities, a unified visual understanding and image generation is achieved, effectively addressing the problem of coordinating visual understanding and image generation capabilities.

[0043] The target loss includes a second loss, which is derived from the difference between the first predicted answer and the first true answer. The first predicted answer is the answer to the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence. By using this second loss, derived from the difference between the first predicted answer generated by the understanding expert model based on the first predicted visual feature unit sequence and the first true answer, the parameters of the generative expert model are adjusted. This allows the understanding expert model to generate a first predicted answer that approximates the first true answer based on the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text. Therefore, this ensures that the generative expert model generates a first predicted visual feature unit sequence based on the first sample descriptive text that can be interpreted correctly by the understanding expert model. In other words, it drives the generative expert model to generate a first predicted visual feature unit sequence based on the first sample descriptive text that can be accurately interpreted by the understanding expert model, thus enhancing the representational ability of the visual feature unit sequence generated by the generative expert model based on the descriptive text. In other words, by adjusting the parameters of the generative expert model using the second loss, the unified multimodal model gains the ability to understand its own generated visual feature unit sequences.

[0044] In one implementation, during the training of the generative expert model in the unified multimodal model, the relevant model parameters of the understanding expert model in the unified multimodal model are frozen (not participating in gradient updates) to preserve the powerful general visual understanding capabilities of the understanding expert model, or the unified multimodal model. Therefore, training the generative expert model in the unified multimodal model is equivalent to adding a new set of parameters to the unified multimodal model as a generative expert model, specifically for learning image generation capabilities.

[0045] In one specific implementation, the parameters of the generative expert model (GEM) are adjusted using a first loss and a second loss. The second loss is used to adjust the GEM parameters, enabling the unified multimodal model to understand the sequences of visual feature units it generates. The first loss is used to adjust the GEM parameters, improving the accuracy of the GEM in generating visual feature unit sequences based on descriptive text and ensuring a match with the visual feature space of the understanding expert model. Therefore, by adjusting the GEM parameters using the first and second losses, the unified multimodal model gains image generation capabilities without compromising its original visual understanding capabilities, and can accurately understand the images it generates; that is, it can generate high-quality or accurate images while maintaining and utilizing strong visual understanding capabilities.

[0046] Please see Figure 3 , Figure 3 This is a flowchart illustrating an embodiment of the training method for a pre-trained semantic visual word segmenter provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 3The illustrated process sequence is limited. For example... Figure 3 As shown, the extraction of the target visual feature unit sequence from the first sample image is performed using a pre-trained semantic visual word segmenter. This embodiment includes: Step S31: Use an understanding expert model to extract target image generation features from the first sample image.

[0047] In this embodiment, the target image generation features are extracted from the first sample image using an understanding expert model; wherein the target image generation features are aligned with the feature space that the understanding expert model can recognize.

[0048] In one implementation, such as Figure 2 As shown, the visual encoder of the understanding expert model extracts continuous image features from the first sample image; the continuous image features are mapped to obtain the target image generation features.

[0049] In one specific implementation, such as Figure 2 As shown, the visual encoder for understanding expert models can be Intern-ViT. Intern-ViT has a large number of parameters and is one of the largest visual encoders in the open-source field, capable of handling complex visual inputs. In addition, Intern-ViT supports dynamic high resolution, which can adaptively adjust the processing resolution according to the aspect ratio and content of the image, optimizing computational efficiency while ensuring detail preservation. It integrates cutting-edge technologies such as RMSnorm, QKNorm, LayerScale, DropPath, and FlashAttention to improve the processing speed of the visual encoder while reducing memory consumption.

[0050] In one specific implementation, such as Figure 2 As shown, the mapping layer MLP1 can be used to map continuous image features to obtain the target image generation features.

[0051] Step S32: Use a pre-trained semantic visual segmenter to extract the sample visual feature unit sequence from the first sample image, and map the sample visual feature unit sequence to obtain the first predicted image generation feature.

[0052] In this embodiment, a pre-trained semantic visual word segmenter is used to extract a sequence of sample visual feature units from the first sample image, and the sequence of sample visual feature units is mapped to obtain the first predicted image generation features.

[0053] In one embodiment, a mapping layer can be used to map the sequence of visual feature units of the samples to obtain the first predicted image generation features.

[0054] In one specific implementation, such as Figure 2As shown, the mapping layer that maps the sample visual feature unit sequence is MLP1, which is the same as the mapping layer in the understanding expert model. Of course, in other specific embodiments, the mapping layer that maps the sample visual feature unit sequence can also have the same architecture as the mapping layer in the understanding expert model.

[0055] In one specific implementation, the formula for obtaining the first predicted image generation features is as follows:

[0056]

[0057] In the formula: This represents the features generated from the initial first predicted image; The projection representing the generative expert model, and the visual projection representing the understanding of the expert model. Same architecture; Represents a sequence of visual feature units of a sample; Represents the codebook; This represents the index sequence or position sequence of the sample visual feature unit sequence in the codebook; This represents a function that interpolates the input length to the maximum scale Ns; Represents learnable scale embeddings; The projection representing the feature alignment loss; The first predicted image generation feature represents the target image, which is used subsequently to calculate the difference between the first predicted image generation feature and the target image generation feature.

[0058] It should be noted that there is no restriction on the execution order of steps S31 and S32. Step S31 can be executed first, followed by step S32; step S32 can be executed first, followed by step S31; or steps S31 and S32 can be executed simultaneously.

[0059] Step S33: Adjust the parameters of the pre-trained semantic visual word segmenter based on the difference between the generated features of the target image and the generated features of the first predicted image.

[0060] In this embodiment, the parameters of the pre-trained semantic visual word segmenter are adjusted based on the difference between the generated features of the target image and the generated features of the first predicted image. The generated features of the target image are aligned with the feature space that the understanding expert model can recognize. Since the generated features of the target image are aligned with the feature space that the understanding expert model can recognize, adjusting the parameters of the pre-trained semantic visual word segmenter using the difference between the generated features of the target image and the generated features of the first predicted image can make the feature space of the generated features of the first predicted image obtained by the pre-trained semantic visual word segmenter from the first sample image approximate the feature space of the generated features of the target image. Since the generated features of the target image are aligned with the feature space that the understanding expert model can recognize, the generated features of the first predicted image obtained by the pre-trained semantic visual word segmenter from the first sample image are aligned with the feature space that the understanding expert model can recognize, that is, driving the generated features of the first predicted image obtained by the pre-trained semantic visual word segmenter from the first sample image to be aligned as closely as possible with the feature space that the understanding expert model can recognize.

[0061] Furthermore, since the features generated by the first predicted image obtained by the pre-trained semantic visual segmenter from the first sample image are aligned with the feature space that the understanding expert model can recognize, the sequence of sample visual feature units extracted by the pre-trained semantic visual segmenter from the first sample image is aligned with the feature space that the understanding expert model can recognize.

[0062] The parameters of the pre-trained semantic visual word segmenter are adjusted based on the difference between the generated features of the target image and the generated features of the first predicted image. This is to align the generated features of the first predicted image obtained by the semantic visual word segmenter from the first sample image with the feature space that the understanding expert model can recognize. Therefore, the feature alignment loss is determined based on the difference between the generated image features and the generated features of the first predicted image. The specific feature alignment loss is as follows:

[0063] in, Indicates feature alignment loss; Represents the visual projection of the understanding expert model; This represents the continuous image features extracted from the first sample image by the visual encoder of the understanding expert model; This represents the features generated by the first predicted image.

[0064] In other implementations, the difference between the generated image features and the first predicted image features can be calculated using cosine distance to determine the feature alignment loss.

[0065] Please see Figure 4 , Figure 4This is a flowchart illustrating an embodiment of the pre-training method for the semantic visual word segmenter provided in this application. It should be noted that if substantially the same result is obtained, this embodiment does not necessarily reflect that result. Figure 4 The illustrated process sequence is limited. For example... Figure 4 As shown, this embodiment includes: Step S41: Obtain the second sample image.

[0066] In this embodiment, a second sample image is acquired. To align the feature space of the understanding expert model, a pre-trained discrete semantic visual segmenter aligned with the visual encoder is proposed to reconstruct the sequence of visual feature units.

[0067] Step S42: Using the teacher model, extract the target visual semantic features from the second sample image.

[0068] In this embodiment, a teacher model is used to extract target visual semantic features from the second sample image. Specifically, the second sample image is input into the teacher model, which then extracts the target visual semantic features.

[0069] In one implementation, a pre-trained semantic visual word segmenter ( Figure 2 When using the word segmenter in the model, a pre-trained visual encoder is introduced as the teacher model.

[0070] In one specific implementation, such as Figure 2 As shown, the teacher model can be Intern-ViT ( Figure 2 Teacher Model 1), SigLIP (Sigmoid Loss for Language-Image Pre-Training) Figure 2 Teacher models in [the class of] SigLIP (2) etc. Compared to CLIP, SigLIP trains faster and has higher inference efficiency with the same number of parameters, making it very suitable as a teacher model for pre-training semantic visual word segmenters. In addition, SigLIP is adaptable to dynamic resolution, which can adaptively adjust the processing resolution according to the image content and aspect ratio, optimizing computational efficiency while preserving details. Furthermore, SigLIP has strong multilingual generalization ability and can understand and align the semantics of text and images in different languages.

[0071] In one specific implementation, SigLIP is selected as the teacher model to ensure good image-text alignment, and a lightweight decoder is instantiated, containing two branches for reconstructing features of SigLIP and Intern-ViT respectively, with each branch containing three visual Transformer blocks.

[0072] In one implementation, when pre-training the semantic visual word segmenter, the teacher model can be 1, 2, 5, etc., and there is no limitation here.

[0073] Step S43: Use a semantic visual word segmenter to extract continuous image features from the second sample image, perform feature quantization on the continuous image features to obtain the feature quantization results, and perform feature reconstruction based on the feature quantization results to obtain the predicted visual semantic features.

[0074] In this embodiment, a semantic visual word segmenter is used to extract sample continuous image features from the second sample image, the sample continuous image features are quantized to obtain the feature quantization result, and feature reconstruction is performed based on the feature quantization result to obtain the predicted visual semantic features; wherein, the feature quantization result includes a sample code sequence, which is the position of each code feature in the codebook that matches the sample continuous image features.

[0075] In one embodiment, the semantic visual segmenter includes an encoder that receives a second sample image and extracts sample continuous image features from the second sample image.

[0076] In one embodiment, the semantic visual word segmenter further includes a quantizer module, which is used to quantize the features of continuous image samples to obtain the feature quantization result.

[0077] In one embodiment, the semantic visual segmenter further includes a decoder, which is used to reconstruct features based on the feature quantization results to obtain predicted visual semantic features.

[0078] In one specific implementation, the feature quantization performed on the continuous image features of the sample is standard vector quantization.

[0079] In one specific implementation, feature quantization is performed on the continuous image features of the sample to obtain the feature quantization result. Specifically, this can be done by performing multiple rounds of quantization operations. In each round of quantization, the target feature residual of the current round of quantization is obtained. The target feature residual of the first round of quantization is the continuous feature of the sample, and the target feature residual of the non-first round of quantization is the feature residual between the target feature residual of the previous round of quantization and at least one target code feature of the previous round of quantization. In the codebook (used to map continuous features to discrete features), at least one target code feature that matches the target feature residual is found, and the position of each target code feature in the codebook is used as the sub-code sequence of the current round of quantization. The feature residual between the target feature residual and at least one target code feature is obtained as the new target feature residual of the current round of quantization, and the next round of quantization operation is performed. That is, in the new round, the above steps of obtaining the target feature residual of the current round of quantization and subsequent steps are repeated until the number of quantization rounds reaches the round number threshold or the new target feature residual of the current round of quantization reaches the residual threshold. Each round of quantization operation corresponds to a sub-code sequence and feature residual of one scale. After the quantization operation stops, the sub-code sequences obtained from each round of quantization constitute the sample code sequence. Optionally, the sub-code sequences from each round of quantization can be weighted and summed to obtain the final sample image features. In other words, multi-scale quantization is performed on the continuous image features of the samples. Through multi-scale quantization, the original continuous image features of the samples are gradually approximated, thereby improving the feature representation ability while maintaining a small number of visual feature units.

[0080] There are no restrictions on the size of the round number threshold and the residual threshold; they can be set according to actual usage needs.

[0081] It should be noted that the quantization module included in the semantic visual word segmenter at this time is the multi-scale residual quantization (MSRQ) module.

[0082] In one specific implementation, the multi-scale residual quantization formula is as follows:

[0083]

[0084] in, Indicates the first Round quantification operation; This represents the subcode sequence quantized in the current round. ; This represents the target code feature sequence corresponding to the sub-code sequence quantized in the previous round; Represents the codebook; This indicates that the input length is interpolated to the maximum scale N. s The function; This represents the residual of the target feature in the current round of quantization; This represents the residual of the target feature from the previous round of quantization.

[0085] It should be noted that there is no restriction on the execution order of steps S42 and S43. Step S42 can be executed first and then step S43, or step S43 can be executed first and then step S42, or steps S42 and S43 can be executed simultaneously.

[0086] Step S44: Adjust the parameters of the semantic visual segmenter based on the difference between the predicted visual semantic features and the target visual semantic features, and adjust the codebook based on the difference between the codebook and the sample continuous image features.

[0087] In this embodiment, the parameters of the semantic visual segmenter are adjusted based on the difference between the predicted visual semantic features and the target visual semantic features, and the codebook is adjusted based on the difference between the codebook and the features of the sample continuous images.

[0088] Since the target visual semantic features are extracted from the second sample image, they can accurately represent the feature information of the second sample image. Therefore, by utilizing the difference between the predicted and target visual semantic features, adjusting the parameters of the semantic visual segmenter can make the predicted visual semantic features generated by the semantic visual segmenter approximate the target visual semantic features. Because the target visual semantic features can accurately represent the feature information of the second sample image, the predicted visual semantic features generated by the semantic visual segmenter can accurately represent the feature information in the second sample image. This drives the predicted visual semantic features generated by the semantic visual segmenter to be as accurate as possible, improving the accuracy of the predicted visual semantic features generated by the semantic visual segmenter. This allows the semantic visual segmenter to extract predicted visual semantic features that accurately represent the feature information of the second sample image. The optimization objective is to make the direction vector of the predicted visual semantic features coincide as closely as possible with the direction vector of the target visual semantic features.

[0089] Furthermore, since the semantic visual word segmenter obtains the feature quantization result by performing feature quantization on the continuous image features of the sample, and then performs feature reconstruction to obtain the predicted visual semantic features, the feature quantization result obtained by the semantic visual word segmenter from the feature quantization of the continuous image features of the sample can accurately represent the feature information of the second sample image.

[0090] In one implementation, there are two teacher models: Intern-ViT and SigLIP. The reconstruction loss for each teacher model is determined based on the difference between the predicted visual semantic features and the target visual semantic features corresponding to each teacher model. The reconstruction losses for Intern-ViT and SigLIP are shown below:

[0091] In the formula, Let th represent the reconstruction loss corresponding to the teacher model, where th∈(SigLIP,Intern-ViT); This represents the target visual semantic features corresponding to the teacher model; This indicates the prediction of visual semantic features.

[0092] The codebook is adjusted based on the differences between it and the continuous image features of the samples. Vector quantization training is prone to codebook collapse, where some code features are frequently used while others are never used, leading to low codebook utilization and reduced reconstruction capability. By adjusting the codebook based on the differences between it and the continuous image features of the samples, the distance between the quantized discrete codes and the continuous image features is minimized, making the codebook features and the continuous image features of the samples closely approximate each other, thus ensuring the stability and utilization of the codebook.

[0093] In one implementation, the codebook loss, determined based on the difference between the codebook and the continuous image features of the samples, is as follows:

[0094] In the formula, Indicates codebook loss; This indicates that the gradient operation has been stopped. This indicates that the codebook C is frozen, and the continuous image features of the samples are optimized. To make the continuous image features of the sample Approaching the direction of codebook C; Representing continuous image features of samples Freeze and optimize codebook C to make codebook C more like sample continuous image features. The direction approximation is achieved by using bidirectional gradient-free backpropagation constraints to fit the code features in the codebook with the continuous image features of the samples, thus avoiding codebook collapse while ensuring the discreteness and stability of the continuous image features of the samples.

[0095] In one specific implementation, there are two teacher models, Intern-ViT and SigLIP. The reconstruction loss is determined based on the difference between the target visual semantic features extracted by these two models from the second sample image and the predicted visual semantic features. Therefore, the total loss for training the semantic visual word segmenter is as follows:

[0096] In the formula, This represents the total loss during the training of the semantic visual word segmenter; Indicates codebook loss; This represents the reconstruction loss corresponding to the SigLIP teacher model; This represents the reconstruction loss corresponding to the teacher model Intern-ViT.

[0097] It should be noted that the semantic visual segmenter can be removed when the unified multimodal model is subsequently applied to inference.

[0098] Please see Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the training method for the unified multimodal model provided in this application. It should be noted that if substantially the same result is obtained, this embodiment is not necessarily identical. Figure 5 The illustrated process sequence is limited. For example... Figure 5 As shown, the generative expert model is an image generation encoder, and the unified multimodal model also includes an image generation decoder. This embodiment includes: Step S51: Obtain the first sample description text, the first sample image corresponding to the first sample description text, and the first sample question about the first sample image.

[0099] Step S51 is similar to step S11, and will not be described again here.

[0100] Step S52: Using a generative expert model, generate a first predicted visual feature unit sequence based on the first sample description text.

[0101] Step S52 is similar to step S12, and will not be described again here.

[0102] Step S53: Determine the target loss using the first predicted visual feature unit sequence.

[0103] Step S53 is similar to step S13, and will not be described again here.

[0104] Step S54: Use the target loss to adjust the parameters of the generated expert model.

[0105] Step S54 is similar to step S14, and will not be described again here.

[0106] Step S55: Map the first predicted visual feature unit sequence to obtain the second predicted image generation features.

[0107] In this embodiment, the first predicted visual feature unit sequence is mapped to obtain the second predicted image generation feature.

[0108] In one implementation, the last hidden state of the second predicted image generation feature is processed by a lightweight Transformer connector and passed to the image generation decoder via a cross-attention module.

[0109] Step S56: Generate a first predicted image using the image generation decoder based on the second predicted image generation features.

[0110] In this embodiment, an image generation decoder is used to generate a first predicted image based on a second predicted image generation feature. Specifically, the image generation decoder converts the second predicted image generation feature into the first predicted image.

[0111] In one embodiment, the image generation decoder can be a decoder for a diffusion model.

[0112] Step S57: Adjust the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image.

[0113] In this embodiment, the parameters of the image generation decoder are adjusted based on the difference between the first predicted image and the first sample image. Since the first sample description text corresponds to the first sample image, the first sample image is an accurate image corresponding to the first sample description text. Therefore, by adjusting the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image, the image generation decoder can generate a first predicted image that approximates the first sample image based on the second predicted image generation features corresponding to the first sample description text. Since the first sample image is an accurate image corresponding to the first sample description text, the image generation decoder generates a first predicted image that approximates the accurate image corresponding to the first sample description text based on the second predicted image generation features corresponding to the first sample description text. In other words, the first predicted image generated by the image generation decoder based on the second predicted image generation features corresponding to the first sample description text is driven to be as accurate as possible, thereby improving the image generation capability of the image generation decoder and enabling the image generation decoder to generate high-fidelity images.

[0114] Adjusting the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image is to enable the image generation decoder to generate high-fidelity images and improve its image generation capability. Therefore, the image generation loss is determined based on the difference between the first predicted image and the first sample image. ( Figure 2 (diffusion loss in the process).

[0115] In one embodiment, the first loss includes a feature alignment loss that aligns the first predicted visual feature unit sequence generated by the generative expert model based on the first sample descriptive text with the feature space that the understanding expert model can recognize. Additionally, the first loss also includes the visual feature unit prediction loss that enables the generative expert model to accurately predict the first predicted visual feature unit sequence based on the first sample descriptive text. The second loss is the text lexical prediction loss. That is, the negative log-likelihood loss for predicting the probability of the next word. The image generation loss is determined based on the difference between the first predicted image and the first sample image. Therefore, the total loss for training the unified multimodal model is as follows:

[0116] in, Indicates the total loss; This represents the image generation loss; Indicates feature alignment loss; This represents the prediction loss, including the prediction loss for visual feature units. Text lexical prediction loss ; , This represents the loss weight.

[0117] Please see Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of the training method for the unified multimodal model provided in this application. It should be noted that if substantially the same result is obtained, this embodiment is not necessarily identical. Figure 6 The illustrated process sequence is limited. For example... Figure 6 As shown, this embodiment includes: Step S611: Obtain the first sample description text, the first sample image corresponding to the first sample description text, and the first sample question about the first sample image.

[0118] Step S611 is similar to step S11, and will not be described again here.

[0119] Step S612: Generate a first predicted visual feature unit sequence based on the first sample description text using a generative expert model.

[0120] Step S612 is similar to step S12, and will not be described again here.

[0121] Step S613: Determine the target loss using the first predicted visual feature unit sequence.

[0122] Step S613 is similar to step S13, and will not be described again here.

[0123] Step S614: Use the target loss to adjust the parameters of the generated expert model.

[0124] Step S614 is similar to step S14, and will not be described again here.

[0125] Step S615: Map the first predicted visual feature unit sequence to obtain the second predicted image generation features.

[0126] Step S615 is similar to step S55, and will not be described again here.

[0127] Step S616: Generate a first predicted image using the image generation decoder based on the second predicted image generation features.

[0128] Step S616 is similar to step S56, and will not be described again here.

[0129] Step S617: Adjust the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image.

[0130] Step S617 is similar to step S57, and will not be described again here.

[0131] Step S618: Obtain the second sample description text and the second sample question.

[0132] In this embodiment, a second sample description text and a second sample question are obtained; wherein, the second sample question is generated based on the second sample description text. The second sample description text and the second sample question are obtained for subsequent post-training of the generative expert model and / or the understanding expert model, to further improve the model performance of the generative expert model and / or the understanding expert model.

[0133] For example, such as Figure 2 As shown, the description text for the second sample is "A blue umbrella, a brown teddybear, and a green vase." The question for the second sample is "What is the color of the vase?".

[0134] In one implementation, the second sample problem is randomly generated based on the second sample description text.

[0135] Step S619: Using a generative expert model to generate a second predicted visual feature unit sequence based on the second sample description text, and using an image generation decoder to generate a second predicted image based on the second predicted visual feature unit sequence.

[0136] In this embodiment, a generative expert model is used to generate a second predicted visual feature unit sequence based on the second sample description text, and an image generation decoder is used to generate a second predicted image based on the second predicted visual feature unit sequence. Specifically, given the second sample description text, the generative expert model generates a second predicted visual feature unit sequence based on the second sample description text, and then the image generation decoder generates a second predicted image corresponding to the second sample description text based on the second predicted visual feature unit sequence.

[0137] In one implementation, such as Figure 2As shown, the image generation decoder generates several second predicted images based on the second predicted visual feature unit sequence.

[0138] Step S620: Using the understanding expert model based on the second predicted image, generate the second predicted answer to the corresponding second sample question.

[0139] In this embodiment, an understanding expert model is used to generate a second predicted answer to the second sample question based on the second predicted image. Specifically, the understanding expert model answers the second sample question based on the second predicted image corresponding to the generated second sample description text, thus generating a second predicted answer to the second sample question.

[0140] In one implementation, such as Figure 2 As shown, the image generation decoder generates several second prediction images based on the second prediction visual feature unit sequence; then, the understanding expert model generates the second prediction answer corresponding to each second prediction image based on each second prediction image, forming trajectory samples O1, O2, ..., O G .

[0141] Step S621: Based on the second predicted image and the second predicted answer, fine-tune the generated expert model, the understanding expert model, and / or the understanding expert model.

[0142] In this embodiment, the expert model for generating, understanding, and / or understanding is fine-tuned based on the second predicted image and the second predicted answer.

[0143] In one embodiment, fine-tuning the generative expert model and / or comprehension expert model can be achieved by: using a referee model to generate a second true answer to the second sample question based on a second predicted image, and then fine-tuning the generative expert model and / or comprehension expert model based on the difference between the second true answer and the second predicted answer. The referee model generates the answer corresponding to the second sample question online, i.e., the second true answer, and the generative expert model and / or comprehension expert model are fine-tuned based on the difference between the answer generated by the referee model and the second predicted answer, thereby improving the model performance of the generative expert model and / or comprehension expert model.

[0144] In one specific implementation, the referee model is a general multimodal large language model.

[0145] In other embodiments, fine-tuning the generative expert model and / or the comprehension expert model can also involve: obtaining a comprehensive score based on the image generation score and the image comprehension score, and fine-tuning the generative expert model and / or the comprehension expert model in response to the comprehensive score being less than or equal to a first scoring threshold; wherein the image generation score is determined using a referee model based on the content consistency between the second predicted image and the second sample descriptive text, and the image comprehension score is determined using a referee model based on the correctness of the second predicted answer. The comprehensive score is determined online using a referee model, and the generative expert model and / or the comprehension expert model are fine-tuned based on the comprehensive score to improve the model performance of the generative expert model and / or the comprehension expert model. The size of the first scoring threshold is not limited.

[0146] In one specific implementation, the image generation score and the image understanding score can be weighted and summed to obtain a comprehensive score.

[0147] In one specific implementation, the image generation decoder generates several second predicted images based on the second predicted visual feature unit sequence; then, the understanding expert model generates a second predicted answer corresponding to each second predicted image based on each second predicted image, forming trajectory samples O1, O2, ..., O G At this point, the image generation score of the second predicted image and the image understanding score of the corresponding second predicted answer constitute a scoring group. Sub-scores corresponding to each scoring group are determined, and the sub-scores are weighted and summed to obtain a comprehensive score. The calculation formula for determining the sub-scores corresponding to each scoring group is shown below:

[0148] in, This represents the i-th rating group; This indicates that the image in the i-th rating group generates a rating; This represents the image understanding score in the i-th rating group; , Indicates the weight.

[0149] In one specific implementation, the image generation decoder generates several second predicted images based on the second predicted visual feature unit sequence; then, the understanding expert model generates a second predicted answer corresponding to each second predicted image based on each second predicted image, forming trajectory samples O1, O2, ..., O G Based on each trajectory sample, the generated expert model and / or the understanding expert model are fine-tuned.

[0150] In one specific implementation, the total loss for post-training of the generative expert model and / or the understanding expert model is as follows:

[0151] in, This represents the total loss after training; The supervised fine-tuning loss is defined as the loss between the second true answer and the next predicted answer, which is generated by the referee model based on the second predicted image to produce the second true answer to the second sample question. This indicates an amplified loss; Indicates the weight.

[0152] In one embodiment, fine-tuning the image generation decoder can involve: obtaining an image generation score, and fine-tuning the image generation decoder in response to the image generation score being less than or equal to a second score threshold; wherein the image generation score is determined using a referee model based on the content consistency between the second predicted image and the second sample description text. The size of the second score threshold is not limited. The image generation score is determined online using a referee model, and the image generation decoder is fine-tuned based on the image generation score to improve the performance of the image generation decoder.

[0153] Please see Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the training apparatus for the unified multimodal model provided in this application. The training apparatus 70 for the unified multimodal model includes an acquisition module 71, a generation module 72, a determination module 73, and an adjustment module 74. The acquisition module 71 is used to acquire a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; wherein, the first sample question is labeled with a first true answer; the generation module 72 is used to generate a first predicted visual feature unit sequence based on the first sample description text using a generative expert model, the first predicted visual feature unit sequence including several first visual feature units, each first visual feature unit representing the features of different regions in the image described by the first sample description text; the determination module 73 is used to determine a target loss using the first predicted visual feature unit sequence; wherein, the target loss includes at least one of a first loss and a second loss, the first loss being obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the target visual feature unit sequence being extracted from the first sample image and aligned with the feature space that the understanding expert model can recognize, and the second loss being obtained based on the difference between the first predicted answer and the first true answer, the first predicted answer being the answer about the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence; the adjustment module 74 is used to adjust the parameters of the generative expert model using the target loss.

[0154] The extraction of the target visual feature unit sequence from the first sample image is performed using a pre-trained semantic visual segmenter. The training device 70 of the unified multimodal model also includes an extraction module 75, which is used to train the pre-trained semantic visual segmenter. The extraction module 75 includes: extracting target image generation features from the first sample image using an understanding expert model; wherein the target image generation features are aligned with the feature space that the understanding expert model can recognize; extracting sample visual feature unit sequences from the first sample image using the pre-trained semantic visual segmenter and mapping the sample visual feature unit sequences to obtain first predicted image generation features; and adjusting the parameters of the pre-trained semantic visual segmenter based on the difference between the target image generation features and the first predicted image generation features.

[0155] The training device 70 for the unified multimodal model further includes a pre-training module 76, which is used for pre-training the semantic visual segmenter. The pre-training module 76 includes: acquiring a second sample image; extracting target visual semantic features from the second sample image using a teacher model; extracting sample continuous image features from the second sample image using the semantic visual segmenter; quantizing the sample continuous image features to obtain the feature quantization result; and reconstructing features based on the feature quantization result to obtain predicted visual semantic features. The feature quantization result includes a sample code sequence, which represents the position of each code feature in the codebook that matches the sample continuous image features. The module also adjusts the parameters of the semantic visual segmenter based on the difference between the predicted visual semantic features and the target visual semantic features, and adjusts the codebook based on the difference between the codebook and the sample continuous image features.

[0156] The pre-training module 76 is used to perform feature quantization on continuous image features of samples to obtain feature quantization results, including: obtaining the target feature residual of the current round of quantization; wherein, the target feature residual of the first round of quantization is the continuous feature of the sample, and the target feature residual of non-first round of quantization is the feature residual between the target feature residual of the previous round of quantization and at least one target code feature of the previous round of quantization; in the codebook, finding at least one target code feature that matches the target feature residual, and taking the position of each target code feature in the codebook as the sub-code sequence of the current round of quantization; obtaining the feature residual between the target feature residual and at least one target code feature as the new target feature residual of the current round of quantization, and re-executing the steps of obtaining the target feature residual of the current round of quantization and subsequent steps until the number of quantization rounds reaches the round number threshold or the new target feature residual of the current round of quantization reaches the residual threshold, wherein the sub-code sequences of each round of quantization constitute the sample code sequence.

[0157] The generation module 72 is used to mask the first sample description text before generating a first predicted answer to the first sample question based on the first predicted visual feature unit sequence using the understanding expert model.

[0158] The aforementioned generative expert model is an image generation encoder, and the unified multimodal model also includes an image generation decoder; the adjustment module 74 is used to map the first predicted visual feature unit sequence to obtain the second predicted image generation feature; the image generation decoder is used to generate the first predicted image based on the second predicted image generation feature; and the parameters of the image generation decoder are adjusted based on the difference between the first predicted image and the first sample image.

[0159] The aforementioned image generation decoder includes a decoder for the diffusion model.

[0160] The training device 70 for the unified multimodal model further includes a fine-tuning module 77. The fine-tuning module 77, after adjusting the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image, includes: obtaining a second sample description text and a second sample question; wherein the second sample question is generated based on the second sample description text; using a generative expert model to generate a second predicted visual feature unit sequence based on the second sample description text, and using the image generation decoder to generate a second predicted image based on the second predicted visual feature unit sequence; using an understanding expert model to generate a second predicted answer corresponding to the second sample question based on the second predicted image; and fine-tuning the generative expert model, the understanding expert model, and / or the image generation decoder based on the second predicted image and the second predicted answer.

[0161] The fine-tuning module 77 is used to fine-tune the generator expert model and / or the understanding expert model, including at least one of the following: using the referee model to generate a second true answer to the second sample question based on the second predicted image, and fine-tuning the generator expert model and / or the understanding expert model based on the difference between the second true answer and the second predicted answer; obtaining a comprehensive score based on the image generation score and the image understanding score, and fine-tuning the generator expert model and / or the understanding expert model in response to the comprehensive score being less than or equal to a first score threshold; wherein the image generation score is determined by the referee model based on the consistency of the content between the second predicted image and the second sample description text, and the image understanding score is determined by the referee model based on the correctness of the second predicted answer.

[0162] The fine-tuning module 77 is used to fine-tune the image generation decoder by: obtaining an image generation score and fine-tuning the image generation decoder in response to the image generation score being less than or equal to a second score threshold; wherein the image generation score is determined by the referee model based on the consistency of the content between the second predicted image and the second sample description text.

[0163] Please see Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described unified multimodal model training method embodiments. In a specific implementation scenario, the electronic device 80 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0164] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described unified multimodal model training method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0165] Please see Figure 9 , Figure 9This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 90 of this application embodiment stores program instructions 91. When executed, these program instructions 91 implement the methods provided by any embodiment and any non-conflicting combination of the training method for the unified multimodal model of this application. The program instructions 91 can form a program file and be stored in the aforementioned computer-readable storage medium 90 in the form of a software product, so that a computer device (which may be a personal computer, server, or network device, etc.) can execute all or part of the steps of the methods of various embodiments of this application. The aforementioned computer-readable storage medium 90 includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0166] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the user through pop-up information or by asking the user to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0167] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A training method for a unified multimodal model, characterized in that, The unified multimodal model includes a generative expert model and a trained understanding expert model, wherein the trained understanding expert model is capable of responding based on an input image; the method includes: Obtain a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; wherein, the first sample question is labeled with a first true answer; Based on the first sample description text, the generative expert model generates a first predicted visual feature unit sequence. The first predicted visual feature unit sequence includes a plurality of first visual feature units, and each first visual feature unit represents the features of different regions in the image described by the first sample description text. Using the first predicted visual feature unit sequence, a target loss is determined; wherein the target loss includes at least one of a first loss and a second loss, the first loss is obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the target visual feature unit sequence being extracted from the first sample image and aligned with the feature space that the understanding expert model can recognize, and the second loss is obtained based on the difference between the first predicted answer and the first true answer, the first predicted answer being the answer to the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence; The parameters of the generative expert model are adjusted using the target loss.

2. The method according to claim 1, characterized in that, Extracting the target visual feature unit sequence from the first sample image is performed using a pre-trained semantic visual word segmenter. The training steps for the pre-trained semantic visual word segmenter include: The understanding expert model is used to extract target image generation features from the first sample image; wherein the target image generation features are aligned with the feature space that the understanding expert model can recognize. Furthermore, a sequence of sample visual feature units is extracted from the first sample image using a pre-trained semantic visual word segmenter, and the sequence of sample visual feature units is mapped to obtain the first predicted image generation features. Based on the difference between the target image generation features and the first predicted image generation features, the parameters of the pre-trained semantic visual word segmenter are adjusted.

3. The method according to claim 2, characterized in that, The pre-training steps of the semantic visual word segmenter include: Obtain the second sample image; Using the teacher model, target visual semantic features are extracted from the second sample image; Furthermore, the semantic visual word segmenter is used to extract sample continuous image features from the second sample image, the sample continuous image features are quantized to obtain feature quantization results, and feature reconstruction is performed based on the feature quantization results to obtain predicted visual semantic features; wherein, the feature quantization results include sample code sequences, the sample code sequences being the positions of each code feature in the codebook that matches the sample continuous image features in the codebook; The parameters of the semantic visual segmenter are adjusted based on the difference between the predicted visual semantic features and the target visual semantic features, and the codebook is adjusted based on the difference between the codebook and the sample continuous image features.

4. The method according to claim 3, characterized in that, The step of performing feature quantization on the continuous image features of the sample to obtain the feature quantization result includes: Obtain the target feature residual for the current round of quantization; wherein, the target feature residual for the first round of quantization is the continuous feature of the sample, and the target feature residual for non-first rounds of quantization is the feature residual between the target feature residual of the previous round of quantization and at least one target code feature of the previous round of quantization; In the codebook, at least one target code feature that matches the target feature residual is found, and the position of each target code feature in the codebook is used as the sub-code sequence of the current round of quantization; The feature residual between the target feature residual and the at least one target code feature is obtained as the new target feature residual for the current round of quantization. The steps of obtaining the target feature residual for the current round of quantization and subsequent steps are repeated until the number of quantization rounds reaches the round number threshold or the new target feature residual for the current round of quantization reaches the residual threshold. The sub-code sequences of each round of quantization constitute the sample code sequence.

5. The method according to claim 1, characterized in that, Before generating a first predicted answer to the first sample question based on the first predicted visual feature unit sequence using the understanding expert model, the method further includes: The description text of the first sample is masked.

6. The method according to claim 1, characterized in that, The generative expert model is an image generation encoder, and the unified multimodal model further includes an image generation decoder; the method further includes: The first predicted visual feature unit sequence is mapped to obtain the second predicted image generation feature; The image generation decoder generates a first predicted image based on the second predicted image generation features; The parameters of the image generation decoder are adjusted based on the difference between the first predicted image and the first sample image.

7. The method according to claim 6, characterized in that, The image generation decoder includes a decoder for the diffusion model.

8. The method according to claim 6, characterized in that, After adjusting the parameters of the image generation decoder based on the difference between the first predicted image and the first sample image, the method further includes: Obtain the second sample description text and the second sample question; wherein the second sample question is generated based on the second sample description text; The generative expert model is used to generate a second predicted visual feature unit sequence based on the second sample description text, and the image generation decoder is used to generate a second predicted image based on the second predicted visual feature unit sequence. Using the aforementioned expert model, a second predicted answer corresponding to the second sample question is generated based on the second predicted image; Based on the second predicted image and the second predicted answer, the generative expert model, the understanding expert model, and / or the image generation decoder are fine-tuned.

9. The method according to claim 8, characterized in that, The steps of fine-tuning the generated expert model and / or the understanding expert model include at least one of the following: Using the referee model based on the second predicted image, a second true answer corresponding to the second sample question is generated, and the generator expert model and / or the understanding expert model are fine-tuned based on the difference between the second true answer and the second predicted answer; A comprehensive score is obtained based on the image generation score and the image understanding score. In response to the comprehensive score being less than or equal to a first score threshold, the generation expert model and / or the understanding expert model are fine-tuned. The image generation score is determined by the referee model based on the consistency of the content between the second predicted image and the second sample description text, and the image understanding score is determined by the referee model based on the correctness of the second predicted answer.

10. The method according to claim 8, characterized in that, The steps for fine-tuning the image generation decoder include: An image generation score is obtained, and the image generation decoder is fine-tuned in response to the image generation score being less than or equal to a second score threshold; wherein the image generation score is determined by the referee model based on the content consistency between the second predicted image and the second sample description text.

11. A training device for a unified multimodal model, characterized in that, The unified multimodal model includes a generative expert model and a trained understanding expert model, the trained understanding expert model being able to respond based on an input image; the apparatus includes: The acquisition module is used to acquire a first sample description text, a first sample image corresponding to the first sample description text, and a first sample question about the first sample image; wherein, the first sample question is marked with a first true answer; The generation module is used to generate a first predicted visual feature unit sequence based on the first sample description text using the generative expert model. The first predicted visual feature unit sequence includes a plurality of first visual feature units, and each first visual feature unit represents the features of different regions in the image described by the first sample description text. A determination module is used to determine a target loss using the first predicted visual feature unit sequence; wherein the target loss includes at least one of a first loss and a second loss, the first loss is obtained based on the difference between the first predicted visual feature unit sequence and the target visual feature unit sequence, the target visual feature unit sequence being extracted from the first sample image and aligned with the feature space that the understanding expert model can recognize, and the second loss is obtained based on the difference between a first predicted answer and the first true answer, the first predicted answer being the answer to the first sample question generated by the understanding expert model based on the first predicted visual feature unit sequence; An adjustment module is used to adjust the parameters of the generative expert model using the target loss.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store program instructions, and the processor being used to execute the program instructions to implement the training method for the unified multimodal model as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program instructions that can be executed to implement the training method for the unified multimodal model as described in any one of claims 1-10.

Images