Image-text processing method and device based on multi-modal pre-training model, electronic equipment and storage medium

Abstract

The invention provides an image-text processing method and device based on a multi-modal pre-training model, electronic equipment and a storage medium, and relates to the technical field of deep learning. The method comprises the following steps: acquiring to-be-processed image data and text data; the image data and the text data are input into a multi-mode pre-training model, an image-text processing result output by the pre-training model is obtained, an encoder of the pre-training model comprises any one of a single-mode encoder and a multi-mode encoder, and the pre-training model further comprises a multi-mode decoder. According to the invention, a new VLP framework is provided, the encoder and the multi-mode decoder are fused, the image-text processing efficiency is improved, and the image-text processing effect is enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and in particular to a method, apparatus, electronic device, and storage medium for image and text processing based on a multimodal pre-trained model. Background Technology

[0002] Currently, in the field of multimodal computing, the most common mainstream approach is to improve upon the dual-encoding model. This involves encoding data from different modalities using corresponding encoders to obtain their features. Contrastive learning is often used for model training. For example, SimVLM, ALIGN, and CLIP have achieved excellent results in cross-modal tasks such as image and text retrieval by training transformer-based models on a large number of paired image-text corpora.

[0003] However, due to their lack of fusion, they typically perform poorly in tasks involving visual reasoning and visual recognition. Summary of the Invention

[0004] In view of this, the purpose of this disclosure is to propose a method, apparatus, electronic device and storage medium for image and text processing based on a multimodal pre-trained model, which can specifically solve existing problems.

[0005] Based on the above objectives, in a first aspect, this disclosure proposes a text-image processing method based on a multimodal pre-trained model, comprising: acquiring image data and text data to be processed; inputting the image data and the text data into a multimodal pre-trained model to obtain a text-image result output from the pre-trained model, wherein the encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder, and the pre-trained model further includes a multimodal decoder.

[0006] Secondly, an image and text processing apparatus based on a multimodal pre-trained model is also provided, comprising: an acquisition unit configured to acquire image data and text data to be processed; and an output unit configured to input the image data and the text data into a multimodal pre-trained model to obtain an image and text result output from the pre-trained model, wherein the encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder, and the pre-trained model further includes a multimodal decoder.

[0007] Thirdly, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor running the computer program to implement the method of the first aspect.

[0008] Fourthly, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being executed by a processor to implement the method described in any one of the first aspects.

[0009] Fifthly, a computer program product is also provided, comprising a computer program that is executed by a processor to implement the method described in any one of the first aspects.

[0010] In summary, this disclosure has at least the following beneficial effects: it provides a novel VLP (Vision-Language Pre-training) framework that integrates encoders and multimodal decoders, thereby improving the efficiency and effectiveness of image and text processing. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this disclosure and should not be construed as limiting the scope of this disclosure.

[0012] Figure 1 A flowchart of a text and image processing method based on a multimodal pre-trained model according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A schematic diagram of a text processing apparatus based on a multimodal pre-trained model according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure is shown;

[0015] Figure 4 A schematic diagram of a storage medium provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0016] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] Figure 1This disclosure illustrates a text and image processing method based on a multimodal pre-trained model. In embodiments of this disclosure, the method includes:

[0019] Step S101: Obtain the image data and text data to be processed.

[0020] Step S102: Input the image data and the text data into a multimodal pre-trained model to obtain the image and text processing result output from the pre-trained model. The encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder. The pre-trained model also includes a multimodal decoder.

[0021] In this embodiment, the execution entity of the image and text processing method based on the multimodal pre-trained model can achieve unified visual language understanding and generation.

[0022] Specifically, this disclosure proposes a multimodal pre-trained model based on understanding and generation: guided language image pre-training to achieve unified visual language understanding and generation. The proposed method is a novel VLP framework that enables a wider range of downstream tasks compared to existing methods. The proposed multimodal encoder-decoder hybrid model is a novel model architecture for efficient multi-task pre-training and flexible transfer learning. It can function as a unimodal encoder, an image-based text encoder, or an image-based text decoder. The model is jointly pre-trained with three visual language objectives: image-text contrast, image-text matching, and image conditional language modeling (caption generation and object detection).

[0023] This invention uses a Visual Transformer (ViT) as an image encoder, which divides the input image into sub-region patches and encodes them into embedding sequences with additional image identifiers [CLS] to represent global image features. Using ViT is more computationally efficient than using pre-trained object detectors for visual feature extraction and has been adopted by newer methods.

[0024] This disclosure provides a novel VLP framework that fuses an encoder and a multimodal decoder, thereby improving the efficiency and effectiveness of image and text processing.

[0025] In some optional implementations of any embodiment of this disclosure, the encoder of the pre-trained model includes the unimodal encoder; the unimodal encoder encodes the image data and the text data respectively, and adds a preset identifier to the starting position of the text corresponding to the text data, the preset identifier being used to enable the unimodal encoder to capture the overall semantic information of the text.

[0026] To pre-train a unified model with both understanding and generative capabilities, this invention proposes a multimodal hybrid encoder-decoder, a multi-task model. If a single-modal encoder is included, images and text can be encoded separately using an encoder based on a transformer block. The text encoder is identical to BERT, where a predefined identifier [CLS] is appended to the beginning of the text input to summarize the sentence.

[0027] In some optional implementations of any embodiment of this disclosure, the encoder of the pre-trained model includes the multimodal encoder; the multimodal encoder includes a text encoder and an image encoder, and there is a cross-attention layer between the self-attention layer and the feedforward network of each converter block of the text encoder, and the cross-attention layer contains information corresponding to the image data; the output of the text encoder is embedded as multimodal representation information of the image semantic information corresponding to the image data, and multimodal representation information of the text semantic information corresponding to the text data.

[0028] Optionally, the step of inputting the image data and the text data into a multimodal pre-trained model to obtain the image-text processing result output from the pre-trained model includes: inputting the image data and the text data into the multimodal encoder, using the text encoder to generate multimodal representation information including the image semantic information and the text semantic information, to obtain the output embedding; extracting entities from the text in the output embedding to obtain entity information, detecting objects in the image data through an object detection head to obtain object information; and fusing the entities and the objects through a cross-attention layer to obtain fused multimodal representation information.

[0029] First, visual information is injected by inserting an additional cross-attention layer between the self-attention layer and the feedforward network in each transformer block of the text encoder. A task-specific text encoder, Encode, is then appended to the text, and the output embedding of Encode is used as a multimodal representation of the image-text pair. Next, entities are extracted from the text via syntactic dependency extraction, corresponding to object detection in the image using an object detection head. A cross-attention layer then fuses the entity and object features, followed by adaptive weight selection for soft matching of entities and objects, outputting a fused target representation. Finally, the multimodal and target representations are concatenated, fed into a self-attention layer for information fusion, and outputting a fused multimodal representation.

[0030] Image encoders are used for feature extraction from images. This feature extraction can be performed by first performing object detection and then extracting features from the bounding boxes, or by directly extracting features from image blocks. The image encoder itself can be a ResNet-like structure or a VIT-like structure. Multimodal encoders can choose a BERT structure. In a single-stream model, text features and image features are concatenated and then input into the multimodal encoder. In a two-stream structure, text features and image features are input into the multimodal encoder separately. Furthermore, the multimodal encoder includes both single-modal self-attention and cross-modal cross-attention computation.

[0031] In some optional implementations of any embodiment of this disclosure, the multimodal decoder includes a causal self-attention layer, and the multimodal decoder includes an image decoder and a text decoder; the text encoder and the text decoder share non-self-attention parameters, the encoder uses bidirectional self-attention to generate multimodal representation information of the input word, and the multimodal decoder uses the causal self-attention layer to predict the multimodal representation information of the next word.

[0032] This disclosure also considers the issue of training and prediction efficiency. To perform efficient pre-training while leveraging multi-task learning, the text encoder and text decoder share all parameters except for the self-attention mechanism layer. This is because the differences between the encoding and decoding tasks are best captured by the self-attention mechanism layer. Specifically, the encoder employs bidirectional self-attention to construct a representation of the current input word token, while the decoder employs causal self-attention to predict the next token. On the other hand, the embedding layer and FFN play similar roles between the encoding and decoding tasks, so sharing these layers improves training efficiency while benefiting from multi-task learning.

[0033] In some optional implementations of any embodiment of this disclosure, the loss function used by the pre-trained model during training includes at least one of the following: image-text contrast loss function, image-text matching loss function, and title generation and object detection loss function.

[0034] Optionally, the image-text contrast loss function is used to align the transformation results of image data features and text data features; the image-text matching loss function is used to learn multimodal representation information and to align the features of the image data and the features of the text data, and the pre-trained model uses the linear layer to predict whether the image and the text match; the title generation and object detection loss function is used to generate text descriptions of the image and reconstruct the target, and the training objective of this loss function includes minimizing the difference between the predicted text and the source text to maximize the retention of the features of the text data and to ensure that the features are correlated and specific compared to the features of the image data.

[0035] Specifically, the image-text contrastive loss function aims to align the feature spaces of the visual transformation block transformer and the text transformation block transformer by encouraging positive image-text pairs to have similar representations. This has proven to be an effective loss for improving visual and language understanding.

[0036] Image-text matching loss function. Its purpose is to learn the multimodal representation information of both image and text, capturing fine-grained alignment information between vision and language. Image-text matching is a binary classification task; given its multimodal features, the model uses linear layers to predict whether image-text pairs match.

[0037] The title generation and object detection loss function aims to generate textual descriptions of a given image and reconstruct the target. A generative model is employed to generate the predicted text. By minimizing the difference between the predicted and source texts, the semantics of the text features are largely preserved after layers of encoding and fusion, maintaining their relevance and specificity with image features. Specifically, the decoder used in this part consists of two parts. First, GPT-J is used as the decoder for text features. Then, a cross-attention mechanism is used to fuse the decoded text features and multimodal features. After multiple layers of GPT-J and cross-attention mechanisms, the final fused text features are output. Finally, an MLP is used to map the fused text features to probabilities in the vocabulary dimension. The object detection head reuses the object detection head from the encoding process to ensure that the object detection head does not shift during the entire training process, ensuring the accuracy of object detection and thus strengthening focus.

[0038] This disclosure provides an image and text processing apparatus based on a multimodal pre-trained model. This apparatus is used to execute the image and text processing method based on a multimodal pre-trained model described in the above embodiments, such as... Figure 2As shown, the device includes: an acquisition unit 201 configured to acquire image data and text data to be processed; and an output unit 202 configured to input the image data and the text data into a multimodal pre-trained model to obtain an image and text result output from the pre-trained model. The encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder. The pre-trained model also includes a multimodal decoder.

[0039] The image processing apparatus based on a multimodal pre-trained model provided in the above embodiments of this disclosure and the image processing method based on a multimodal pre-trained model provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0040] This disclosure also provides an electronic device corresponding to the image processing method based on a multimodal pre-trained model provided in the foregoing embodiments, for executing the aforementioned image processing method based on a multimodal pre-trained model. This disclosure does not limit the scope of the embodiments.

[0041] Please refer to Figure 3 This illustrates a schematic diagram of an electronic device provided by some embodiments of the present disclosure. For example... Figure 3 As shown, the electronic device 30 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the method provided in any of the foregoing embodiments of this disclosure.

[0042] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0043] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The image and text processing method based on a multimodal pre-trained model disclosed in any of the foregoing embodiments of this disclosure can be applied to the processor 300, or implemented by the processor 300.

[0044] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0045] The electronic device provided in this disclosure and the image processing method based on a multimodal pre-trained model provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0046] This disclosure also provides a computer-readable storage medium corresponding to the image processing method based on a multimodal pre-trained model provided in the foregoing embodiments. Please refer to [link / reference]. Figure 4 The computer-readable storage medium shown is an optical disc 40, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the image and text processing method based on a multimodal pre-trained model provided in any of the foregoing embodiments.

[0047] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0048] The computer-readable storage medium provided in the above embodiments of this disclosure and the image and text processing method based on a multimodal pre-trained model provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0049] It should be noted that:

[0050] In the foregoing text, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in this disclosure is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0051] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0052] The embodiments of this disclosure have been described above with reference to the accompanying drawings. These are merely specific implementations of this disclosure, but this disclosure is not limited to the specific implementations described above. The specific implementations described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this disclosure without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this disclosure.

Claims

1. A text and image processing method based on a multimodal pre-trained model, characterized in that, include: Acquire the image and text data to be processed; The image data and the text data are input into a multimodal pre-trained model to obtain the image and text processing results output from the pre-trained model. The encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder. The pre-trained model also includes a multimodal decoder.

2. The method according to claim 1, characterized in that, The encoder of the pre-trained model includes the unimodal encoder; the unimodal encoder encodes the image data and the text data respectively, and adds a preset identifier to the starting position of the text corresponding to the text data, the preset identifier being used to enable the unimodal encoder to capture the overall semantic information of the text.

3. The method according to claim 1, characterized in that, The encoder of the pre-trained model includes the multimodal encoder; the multimodal encoder includes a text encoder and an image encoder, and there is a cross-attention layer between the self-attention layer and the feedforward network of each converter block of the text encoder, and the cross-attention layer contains information corresponding to the image data; The output of the text encoder is embedded as multimodal representation information of the image semantic information corresponding to the image data, and multimodal representation information of the text semantic information corresponding to the text data.

4. The method according to claim 3, characterized in that, The step of inputting the image data and the text data into a multimodal pre-trained model to obtain the image and text processing results output from the pre-trained model includes: The image data and the text data are input into the multimodal encoder, and the text encoder is used to generate multimodal representation information including the image semantic information and the text semantic information to obtain the output embedding; Entities are extracted from the text in the output embedding to obtain entity information, and objects in the image data are detected by the object detection head to obtain object information; By using a cross-attention layer, the entity and the object are fused to obtain fused multimodal representation information.

5. The method according to claim 3, characterized in that, The multimodal decoder includes a causal self-attention layer and includes an image decoder and a text decoder. The text encoder and the text decoder share non-self-attention parameters. The encoder uses bidirectional self-attention to generate multimodal representation information of the input word, and the multimodal decoder uses the causal self-attention layer to predict the multimodal representation information of the next word.

6. The method according to any one of claims 1-5, characterized in that, The loss function used during training of the pre-trained model includes at least one of the following: image-text comparison loss function, image-text matching loss function, and title generation and object detection loss function.

7. The method according to claim 6, characterized in that, The image-text contrast loss function is used to align the transformation results of image data features with the transformation results of text data features; The image-text matching loss function is used to learn multimodal representation information and to align the features of the image data and the features of the text data. The pre-trained model uses the linear layer to predict whether the image and the text match. The title generation and object detection loss function is used to generate text descriptions of images and reconstruct objects. The training objective of the loss function includes minimizing the difference between the predicted text and the source text to maximize the retention of features of the text data and to ensure that the features are correlated and specific compared to the features of the image data.

8. A text and image processing device based on a multimodal pre-trained model, characterized in that, include: The acquisition unit is configured to acquire image data and text data to be processed; The output unit is configured to input the image data and the text data into a multimodal pre-trained model to obtain an image and text result output from the pre-trained model. The encoder of the pre-trained model includes any one of the following: a unimodal encoder and a multimodal encoder. The pre-trained model also includes a multimodal decoder.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-7.

Images