Small-scale Data Visual Language Pretraining Method

Abstract

The present invention proposes a small-scale data visual language pre-training method. For the image side, the U-Net network is adopted. After encoding the missing image, channel attention is performed between each corresponding downsampling and upsampling stage and the text features additionally input, and the text information is used to complete the image. The encoded features and the text description features are fused and input into the decoder to decode into a normal image. For the text side, a bilinear attention network is used. Among them, the text features are denoted as X, the image features are Y, and the attention weight map w is calculated according to the affinity between different modality features. With the assistance of the attention weight map, feature fusion between X and Y is performed, and finally the prediction result of the missing word is output. This technical solution reduces the scale of the pre-training data while improving the performance of downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing and machine learning, and particularly relates to a visual language pre-training method for small-scale data. Background Art

[0002] A pre-trained model obtains knowledge from a large amount of labeled and unlabeled data, and learns a general model therefrom to be applicable to different downstream tasks. In recent years, continuously emerging large-scale pre-trained models have achieved great breakthroughs in many fields, which proves that training a general model and using a specific data set for fine-tuning when dealing with a specific task can achieve better results on the task while saving a certain amount of computing power. Pre-training in multiple modalities expects to learn the correlation relationships between two or more modalities. Compared with pre-training in a single modality, multiple types of information can be comprehensively obtained, making the pre-trained model have better generalization ability. Although the understanding of a single modality such as vision or language is indispensable in vision or language tasks, the mutual relationships between each modality are equally important. However, most of the existing visual language pre-training tasks rely on masked image / text completion or object detection methods to extract the features of a certain region and perform fine-grained local alignment on the extracted features. The design of most pre-training tasks lacks the interaction between different modalities.

[0003] With the continuous growth of the data scale on the Internet, the data types are increasingly showing diverse characteristics. The data modalities that users are interested in are no longer single, and the retrieval requirements of users are increasingly showing a development trend from single modality to cross-modal. A modality refers to the expression form of data, including text, image, video, audio, etc. Multimodal data refers to data obtained for the same description object from different fields or perspectives, and each field or perspective describing these data is called a modality. The role of pre-training is to extract as many common features as possible from a large amount of training data, making the learning burden of the model for a specific task lighter. Pre-train the model on a large amount of unlabeled data and fine-tune the downstream tasks using task-specific labeled data [1] .

[0004] For image-text pre-training models (Pre-Training Model, PTM), most of the current work is based on the architecture of Vision-Language BERT [2] . The main challenge lies in the alignment of visual and text contents in a unified semantic space. Therefore, two model architecture designs have been developed: two-stream and single-stream.

[0005] In terms of two-stream models, the ViLBERT model was proposed by Lu et al. in 2019 [3] , which first extended the BERT [2] structure to a multimodal two-stream model and used a similar BERT[2] The architecture learning is for the joint representation of images and texts. However, since both vision and language have separate Transformers [4] structures, the number of parameters has increased significantly. Similar to ViLBERT [3] , the LXMERT [5] model also applies two Transformers [4] to images and texts and fuses them through a third Transformer [4] . In 2021, the CLIP (Contrastive Language-Image Pre-Training) model proposed by Radford et al. used 400 million image-text data pairs from the web to train the text as image labels. Two encoders were used to process texts and pictures respectively, achieving remarkable performance in the image-text retrieval task but performing poorly in other vision-language tasks. To address the above problems, the ALBEF (ALign BEfore Fuse) model proposed by Li et al. later introduced an intermediate image-text contrastive loss. First, it aligned the unimodal image representation with the text representation and then fused it with the multimodal encoder to guide the learning of visual and language representations, obtaining a faster inference speed in multiple downstream tasks. [6] In the single-stream model, Sun et al. proposed VideoBERT in 2019 [7] . As a single-stream model, it uses a stacked Transformer

[0006] structure in its architecture and uses clustering techniques to process video frames and audio languages. VisualBERT [8] , compared with VideoBERT [4] , has a simpler architecture and can establish connections between speech and images under unsupervised conditions, but has not applied this model to pure image tasks. Subsequently, the Unicoder-VL proposed by Li et al. in 2020 [9] , as a pre-trained model in the image-text field, continues to use a stacked Transformer [8] structure. Compared with the above three models, it is trained with a large number of image-text pairs, can learn common cross-modal knowledge and apply it to a wider range of downstream tasks, but cannot extract information from a single image modality.

[10] As a pre-trained model in the image-text field, it continues to use a stacked Transformer [4] structure. Compared with the above three models, it is trained with a large number of image-text pairs, can learn common cross-modal knowledge and apply it to a wider range of downstream tasks, but cannot extract information from a single image modality.

[0007] Since most existing vision-language pre-training tasks rely on masked modality data completion or object detection methods to extract features of a certain region and perform fine-grained local alignment on the extracted features, there is also less interaction between modalities in most pre-training tasks. The masked completion task mainly relies on single-modal features, and during training, it mainly improves the single-modal feature extraction ability without learning the semantic associations between different modal features. In addition, previous pre-training models often use data on the scale of millions or even billions to perform pre-training tasks, which requires a large amount of computing power.

[0008] The existing technical literature for reference includes:

[0009] [1] Luo H, Ji L, Shi B, et al. Univl: A unified video and language pre-training model for multimodal understanding and generation[J]. arXiv preprint arXiv:2002.06353, 2020.

[0010] [2] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.

[0011] [3] Lu J, Batra D, Parikh D, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[J]. Advances in neural information processing systems, 2019, 32.

[0012] [4] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30.

[0013] [5]Tan H, Bansal M. Lxmert: Learning cross-modality encoder representations from transformers[J]. arXiv preprint arXiv:1908.07490, 2019.

[0014] [6]Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C] / / International Conference on Machine Learning. PMLR, 2021:8748 - 8763.

[0015] [7]Li J, Selvaraju R, Gotmare A, et al. Align before fuse: Vision and language representation learning with momentum distillation[J]. Advances in neural information processing systems, 2021, 34:9694 - 9705.

[0016] [8]Sun C, Myers A, Vondrick C, et al. Videobert: A joint model for video and language representation learning[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019:7464 - 7473.

[0017] [9]Li L H, Yatskar M, Yin D, et al. Visualbert: A simple and performant baseline for vision and language[J]. arXiv preprint arXiv:1908.03557, 2019.

[0018]

[10] Li G, Duan N, Fang Y, et al. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 11336-11344. Summary of the Invention

[0019] To bridge the gaps and deficiencies in the existing technologies, and to learn better models in small-scale data pre-training by designing a better pre-training framework, the present invention proposes a visual language pre-training method for small-scale data, which includes a neural network-based visual language pre-training framework. Based on the neural network, during the pre-training process, by reconstructing the missing images / texts, a better feature extraction model can be learned, and high-performance cross-modal tasks can be achieved with relatively low pre-training costs.

[0020] Different from most previous pre-training tasks that only consider intra-modal data, the present invention proposes a new solution to perform "fine-grained visual language pre-training". The key to learning fine-grained alignment is to use the data of the other modality to complete the data after a given missing image / text. On this basis, in order to increase the task difficulty and improve the performance of the pre-trained model, the present invention further proposes the idea of partial assistance.

[0021] In this solution, for the missing images, the U-Net network is used to perform channel attention with the text features additionally input between each corresponding down-sampling and up-sampling stage, and the text information is used to complete the images; for the missing texts, the bilinear attention network is used to mix the missing text information and the additionally input image information to complete the texts. In addition, the present invention proposes the idea of partial completion on this basis. Using a simple RNN network, according to the missing modality data and the complete data of the other modality, the data of the other modality that are as complementary as possible are selected, the semantic overlap between different modality data is reduced, and thus the model error is reduced and the model performance is improved. This technical solution improves the performance of downstream tasks such as cross-modal retrieval and image caption generation while reducing the scale of the pre-training data.

[0022] Specifically, for the image side, the present invention uses the U-Net network that is frequently used in the field of semantic segmentation recently. After encoding the missing image, the encoded features and text description features are fused and then input into the decoder to decode into a normal image. For the text side, the present invention uses the bilinear attention network that is frequently used in the field of visual question answering. The text features can be extracted from the text through the LSTM model. Denote the text features as X, the image features as Y, and the attention weight map w is calculated based on the affinity between different modality features. With the assistance of the attention weight map, the model realizes the feature fusion between X and Y. The result output by the last layer of the BAN network is input into a classifier constructed by a multi-layer perceptron, and finally the prediction result of the missing word is output. On this basis, the present invention further proposes the idea of partial auxiliary completion, using a simple selector to complementarily select another modality data according to the missing modality data and replace the original complete another modality data.

[0023] The present invention specifically adopts the following technical solutions:

[0024] A visual language pre-training method for small-scale data, characterized in that: for the pre-training sub-task on the image side, the U-Net network is adopted. After encoding the missing image, channel attention is performed with the additionally input text features between each corresponding downsampling and upsampling stage, and the text information is used to complete the image. The encoded features and text description features are fused and input into the decoder to decode into a normal image;

[0025] For the pre-training sub-task on the text side, the bilinear attention network is used; wherein, the text features are denoted as X, the image features as Y, the attention weight map w is obtained by calculating the affinity between different modality features, and with the assistance of the attention weight map, the feature fusion between X and Y is performed, and finally the prediction result of the missing word is output.

[0026] Furthermore, in the pre-training sub-task on the image side, a text selector is used for the text; an image selector is used for the image on the text side; the text selector and the image selector adopt a recurrent neural network to output the selection result of each region bit by bit; the input of the text selector is the missing image and the complete text, and the input of the image selector is the missing text and the complete image.

[0027] Furthermore, for the pre-training sub-task on the image side, in the encoding stage, visual features of multiple scales are extracted from the input image; and the text feature f is obtained through feature extraction t and then input into the model and fused with the output of the encoder;

[0028] During the decoding stage, first, the encoded image features and text features are fused through a channel attention mechanism; at the beginning, the feature map is converted into a feature vector through a global pooling layer, which is then concatenated with the text features and input into a simple feed-forward neural network with a softmax layer to generate an attention weight, which is finally used to update the feature values to obtain the finally fused features:[[ID=`1]]

[0029]

[0030] The subscript i ∈ {1, 2, …, c s} represents the index channel, and represents a scalar;

[0031] The decoder of the U-Net network has multiple deconvolution layers; the input of each deconvolution layer is formed by concatenating the fused features obtained by combining the encoded features and text features output by the encoding layer in the same stage and the output of the previous deconvolution layer, and the output of the last layer of the encoder is used to replace the output of the previous deconvolution layer for the first deconvolution layer; the feature map a output by the last layer is used as the final image output after upsampling and convolution.

[0032] Furthermore, during the training process, the overall subtask is regarded as a pixel-level regression problem, where G if represents the image completion model, and the paired missing images I miss and the text features f t are used as inputs, and the finally output completed image I normal ; the original normal image is used as the completion target of the model, and a pixel-level mean square error is used as follows:

[0033]

[0034] Furthermore, for the pre-training subtask of the text end, for the given features X and Y of two modalities, the attention weight is calculated based on the affinity between the features of different modalities:

[0035] [[ID=`36]]

[0036] where U ∈ R N×d and V ∈ R M×d are mapping matrices, is a vector, p h ∈ R d [[ID=`47]]where is the attention map index, is the element-wise product;

[0037] With the assistance of the attention weight map, the model performs feature fusion between X and Y and uses residual connections in the model. In the nth residual block, the model outputs as follows:

[0038]

[0039] P ∈ R d×c is the mapping matrix, and X is the initial input F0 of the model, BAN h is a function for generating intermediate features, defined as:

[0040]

[0041] where U′ ∈ R N×d , V′ ∈ R M×d ,

[0042] After the result output by the last layer of the BAN network is input into a classifier constructed by a multi-layer perceptron, the predicted result of the missing word is finally output.

[0043] Furthermore, regarding the pre-training sub-task at the text end as a visual question answering problem, G tf represents the text completion model, taking the paired missing text T miss and the image feature f i as the model input, and finally outputting the missing word T tar ; the model prediction target is the covered part in the text of the dataset; the loss function uses cross-entropy loss, and the formula is as follows:

[0044]

[0045] Compared with the prior art, the present invention proposes a pre-training strategy of modal-assisted completion, which uses data of another modality to complete the missing modal data, and searches for more general and finer-grained correlation relationships in the pre-training stage. And on this basis, the idea of partial-assisted completion is proposed, further improving the task performance.

[0046] Inspired by the Transformer [4] model, the present invention uses the Transformer [4]The positional embeddings added in the model are transformed into data of another modality and used as additional information to complete the data. Specifically, for the image side, the present invention uses a U-Net network. After encoding the missing image, the encoded features and text description features are fused and then input into the decoder to be decoded into a normal image. For the text side, the present invention uses a bilinear attention network commonly used in the field of visual question answering. The text features can be extracted from the text by passing it through an LSTM model. Denote the text features as X and the image features as Y. The attention weight map w is calculated based on the affinity between different modality features. With the assistance of the attention weight map, the model realizes the feature fusion between X and Y. The result output by the last layer of the BAN network is input into a classifier constructed by a multi-layer perceptron, and finally the prediction result of the missing word is output. On this basis, the present invention further proposes the idea of partial auxiliary completion. A simple selector is used to complementarily select data of another modality according to the missing modality data and replace the original complete data of another modality. The method proposed by the present invention uses the partial modality data selected by the selector to complete the missing data, further reducing the semantic overlap between different modalities and improving the performance of the model.

[0047] Compared with the prior art, the present invention and its preferred solutions have made progress in both the computing power required for training and downstream tasks. The main contributions of the present invention include:

[0048] (1) Two new cross-modal pre-training tasks are proposed to find more general and finer-grained correlation relationships between visual content and text content and learn better unified representations.

[0049] (2) In order to further reduce errors and improve task performance, the present invention proposes the idea of partial assistance to minimize the overlapping part between different modality data as much as possible in order to reduce the model errors caused thereby.

[0050] (3) Through this design, only a small amount of data is used for model pre-training in the pre-training stage. Compared with the prior art of pre-training on a scale of millions or even billions, a large amount of computing power is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of the overall architecture of the visual language pre-training framework in an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of the basic structure of the U-Net network in an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of the basic structure of the BAN network in an embodiment of the present invention;

[0054] Figure 4Visualization diagram of the selection result of the selector and the data completion effect in the embodiment of the present invention. Detailed implementation manners

[0055] To make the features and advantages of this patent more obvious and understandable, specific embodiments are given below and are described in detail as follows:

[0056] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further descriptions of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs.

[0057] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] Embodiment 1:

[0059] Refer to Figure 1 , based on the basic structure design of the vision-language pre-training framework of the neural network, a multi-modal pre-training scheme is proposed in this embodiment in the data communication scenario, which mainly includes two parties: the service provider and the user.

[0060] The specific implementation steps are as follows:

[0061] (1) The service provider trains the neural network using a large number of image-text pairs in a secure computing environment to obtain a pre-trained model that meets the expected requirements. A specific proportion of the training set samples are covered as the missing data set. Since the model is for completing missing data, the data set needs to be pre-processed in advance. Only the data with missing values can be used in the solution of this embodiment.

[0062] (2) The service provider distributes the trained pre-trained model to all users with requirements. The network structures of the users' models vary according to the downstream tasks performed.

[0063] (3) The user fine-tunes the pre-trained model using a specific data set for a specific task.

[0064] (4) The user inputs different modal data into the fine-tuned model to obtain aligned modality-independent high-level abstract features for subsequent specific tasks.

[0065] Just as Figure 1As shown in the figure, during model pre-training, for images, after encoding the missing image, the encoded features and text description features are fused to decode it into a normal image. For text, attention weights are calculated based on the affinity between features from different modalities. With the help of the attention weight map, feature fusion is achieved, and the final output is the missing word prediction result. During the model fine-tuning phase, users retain different sub-models based on different downstream tasks and obtain the fine-tuned model by training a small number of batches using a specific dataset.

[0066] Example 2:

[0067] Assuming that the user does not have sufficient computing power to perform model training in the pre-training phase, the framework of the cross-modal pre-training solution based on a neural network in this embodiment can achieve the requirement of only requiring a small amount of computing power for fine-tuning for downstream tasks after one training through the process of pre-training and fine-tuning.

[0068] In Example 1, this embodiment generally describes the infrastructure required for a cross-modal pre-training solution based on a neural network. Next, this embodiment further describes the specific implementation details based on Example 1. The special features are as follows:

[0069] (1) Data preprocessing and model initialization:

[0070] The quality of network model training depends largely on the training samples and the initial parameters of the network model.

[0071] Therefore, first, the service provider initializes the model parameters using a reasonable initialization parameter setting;

[0072] The MSCOCO dataset is a training set for model pre-training. It contains 133,287 images, each with five descriptions. This example uses 123,287 images as the training set, 5,000 images as the validation set, and 5,000 images as the test set. This example masks a specific proportion of the existing dataset as the missing dataset, and then trains the model of this embodiment.

[0073] When the service provider trains the model, the overall flow chart of the solution is as follows: Figure 1 shown.

[0074] (2) Image completion:

[0075] The image completion module performs image completion based on the missing image and text features. In this subtask, the model understands the text description, discovers useful context semantics, and uses this as additional information for image completion. In this embodiment, it is hoped that during the completion process, the model can learn finer-grained association relationships between different modal data. In this embodiment, the U-Net network commonly used in the field of semantic segmentation is used, and the basic structure of the model is as Figure 2 shown.

[0076] In the encoding stage, the model extracts visual features at multiple scales from the input image. In the text branch, the text feature f t is obtained through the feature extraction model and then input into the model to be fused with the output of the encoder;

[0077] In the decoding stage, it is necessary to first fuse the encoded image features and text features. This step is completed by the channel attention mechanism in this embodiment to achieve text assistance in the image completion process. At the beginning, the feature map is converted into a feature vector through a global pooling layer. After being connected to the text feature, it is input into a simple feedforward neural network with a softmax layer to generate an attention weight, which is finally used to update the feature value to obtain the finally fused feature;

[0078]

[0079] The decoder of the U-Net network has multiple deconvolution layers. The input of each deconvolution layer is composed of the fused feature obtained by combining the encoded feature output by the encoding layer in the same stage and the text feature and the output of the previous deconvolution layer. The output of the last layer of the encoder is used instead of the output of the previous deconvolution layer for the first deconvolution layer. The feature map a output by the last layer is used as the final image output after simple upsampling and convolution.

[0080] During the training process, the subtask can generally be regarded as a pixel-level regression problem. The paired missing images and text features are used as inputs, and the finally completed image is output. The original normal image is used as the completion target of the model, and a pixel-level mean square error is used, as follows: <0,

[0081]

[0082] (3), Text Completion:

[0083] Text completion, which uses image features to complete missing text. The model discovers fine-grained semantic information by comprehensively understanding different regions of the image and the relationships between regions, and uses this as additional information for text completion. Similar to image completion, this embodiment aims to construct more accurate association information between images and text through this task. To solve this subproblem, this embodiment uses a bilinear attention network commonly used in the field of visual question answering. The model details are as Figure 3 shown.

[0084] Among them, the text features are extracted from the text through an LSTM model. Let the text features be written as X and the image features be Y. Given the features X and Y of the two modalities, the attention weights are calculated based on the affinity between the features of different modalities;

[0085]

[0086] With the assistance of the attention weight map, the model realizes the feature fusion between X and Y. This embodiment also uses residual learning of attention to ensure the model performance. This embodiment uses a residual connection in the model. In the nth residual block, the model output is as follows:

[0087]

[0088] BAN is a function that generates intermediate features and is defined as:

[0089]

[0090] The result output by the last layer of the BAN network is input into a classifier constructed by a multi-layer perceptron, and finally the prediction result of the missing word is output.

[0091] This sub-task can be approximately regarded as a visual question answering problem, with the paired missing text and image features as the model input, and finally the missing word is output. The model prediction target is the part covered in the text of the dataset. The loss function uses the typical cross-entropy loss, and the formula is as follows:

[0092]

[0093] (4) Selection strategy:

[0094] In further research, it is found that when the model completes the completion of images or texts, giving the complete data of another modality may not be a good choice. There is semantic overlap between the missing image (text) and the features of the complete text (image), which may mislead the model and cause certain errors in the model. Therefore, based on the two subtasks, this embodiment proposes a partial auxiliary completion strategy. With the help of a pre-selection selector, a part of the auxiliary data that is more complementary to the missing modality data is selected to further improve the performance of the model and form an overall end-to-end model.

[0095] Since the overall network is already relatively complex, this embodiment hopes to implement the function of this selector through a simple model. Therefore, this embodiment uses the recurrent neural network commonly used in natural language processing to output the selection results of each region bit by bit, as Figure 4 shown;

[0096] Denote the image selector as S i , the text selector as S t , the input of S i is the missing text and the complete image for selection, and the input of S t is the missing image and the complete text for selection. Finally, the selection results of the image / text are output respectively, as follows:

[0097] mask i = S i (T miss , I normal ) (7)

[0098] mask t = S t (I miss , T normal ) (8)

[0099] (5) Model training:

[0100] The above subtasks can be combined with some common modality feature alignment methods in the model. As an overall multi-modal pre-training model, this embodiment uses contrastive learning to align image features and text features. This embodiment uses the ResNet50 model and the BERT model to extract image features and text features respectively.

[0101] The overall loss of the model can be listed as:

[0102]

[0103] λ1, λ2, λ3 are adjustable hyperparameters used to balance the importance of each subtask. is the selection ratio of two selection models. In this embodiment, it is hoped that through this constraint, the model can select as little data as possible to better achieve the complementary effect.

[0104] (6), Model fine-tuning:

[0105] After the model is pre-trained, since the dataset used in the pre-training stage may not be the same as the dataset of the downstream task, there is a certain deviation in the model. Before the user uses it for a specific downstream task, a small number of fine-tuning is required. According to the task performed, a specific dataset is selected to fine-tune the deviation of the model.

[0106] For the cross-modal retrieval task, in this embodiment, the ResNet model and the BERT model for extracting image / text features are retained. Since the different-modal features have been aligned through the contrastive loss in the pre-training stage, in the downstream task, this embodiment directly selects to directly output the modality-agnostic common representation through these two networks;

[0107] For the image caption generation task, since the model aligns the different-modal features in the pre-training stage, the image features output by the ResNet network can be considered as modality-agnostic high-level abstract semantic features. Therefore, in this embodiment, only the ResNet network for extracting image features is retained. After the image features are output by the ResNet network, a pre-trained seq2seq network is used to convert the image features into text.

[0108] It can be seen from the above embodiments that the solution of the present invention has the following features: First, the present invention proposes two new cross-modal pre-training tasks to find more general and finer-grained correlation relationships between visual content and text content, and learn better unified representations. Second, in order to further reduce errors and improve task performance, the present invention proposes the idea of partial assistance, minimizing the overlapping part between different-modal data as much as possible to reduce the model errors caused thereby. Third, the present invention only uses a small-scale data for model pre-training. Compared with pre-training on a scale of millions or even billions, the present invention saves a large amount of computing power.

[0109] The above program design solutions related to algorithms provided in this embodiment can be stored in a computer-readable storage medium in a coded form, implemented in the form of a computer program, and input the basic parameter information required for calculation through computer hardware and output the calculation result.

[0110] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0111] The present invention is described with reference to the flowcharts of methods, devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process in the flowchart and the combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one Figure 1 process or multiple processes.

[0112] These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in one Figure 1 process or multiple flowcharts.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one Figure 1 process or multiple processes.

[0114] As described above, it is only the preferred embodiments of the present invention, and the present invention is not limited to other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

[0115] This patent is not limited to the above best implementation manner. Anyone can obtain other various forms of small-scale data visual language pre-training methods under the inspiration of this patent. All equivalent changes and modifications made according to the scope of the patent application of the present invention shall fall within the coverage scope of this patent.

Claims

1. A small-scale visual language pre-training method for completing missing data, characterized by: For the image pre-training subtask, a U-Net network is used. After encoding the missing image, the encoded image features and text features are fused with the additional input text features through a channel attention mechanism between each corresponding downsampling and upsampling stage. The text information is used to complete the image. The fused encoded features and text features are input into the decoder and decoded into a normal image. For the text-side pre-training subtask, a bilinear attention network is used. The text features are denoted as X and the image features are denoted as Y. The attention weight map w is calculated based on the affinity between the features of different modalities. With the help of the attention weight map, the features of X and Y are fused, and the final prediction result of missing words is output. In the pre-training subtask on the image side, a text selector is used for text; on the text side, an image selector is used for images; the text selector and image selector adopt a recurrent neural network to output the selection results of each area bit by bit; the input of the text selector is the missing image and the complete text, and the input of the image selector is the missing text and the complete image.

2. The small-scale data visual language pre-training method according to claim 1, characterized in that: For the image-side pre-training subtask, in the encoding phase, visual features of multiple scales are extracted from the input image; And obtain the text feature f through feature extraction t , which is then fed into the model and fused with the encoder output; During the decoding phase, the encoded image features and text features are first fused through the channel attention mechanism. Initially, the feature map is converted into a feature vector through a global pooling layer. After being concatenated with the text features, it is input into a simple feedforward neural network with a softmax layer to generate an attention weight, which is ultimately used to update the feature value to obtain the final fused feature: Subscript i∈{1,2,…,c s } represents the index channel, represents a scalar; The decoder of the U-Net network has multiple deconvolution layers. The input of each deconvolution layer is composed of the fused features obtained by combining the encoding features and text features output by the encoding layer at the same stage and the output of the previous deconvolution layer. The first deconvolution layer uses the output of the last layer of the encoder instead of the output of the previous deconvolution layer. The feature map a output by the last layer is upsampled and convolved as the final image output.

3. The small-scale data visual language pre-training method according to claim 2, characterized in that: During the training process, the subtask is considered as a pixel-level regression problem. if Represents the image completion model, which completes the paired missing images I miss and text features f t As input, the final output is the completed image I normal ; The original normal image is used as the completion target of the model, using a pixel-level mean square error, as shown below:

4. The small-scale data visual language pre-training method according to claim 1, characterized in that: For the text-side pre-training subtask, given the features X and Y of two modalities, the attention weight is calculated based on the affinity between the features of different modalities: where U∈R N×d and V∈R M×d is the mapping matrix, is a vector, p h ∈R d ,in is the attention graph index, is the element-wise product; With the help of the attention weight map, the model performs feature fusion between X and Y and uses residual connections in the model. In the nth residual block, the model output is as follows: P∈R d×c Is the mapping matrix, X is the initial input F0 of the model, BAN h is a function that generates intermediate features and is defined as: where U′∈R N×d ,V′∈R M×d , The output of the last layer of BAN network is input into a classifier built by a multi-layer perceptron, and finally the prediction result of the missing word is output.

5. The small-scale data visual language pre-training method according to claim 4, characterized in that: Treat the text-side pre-training subtask as a visual question answering problem, G tf Represents a text completion model, with paired missing text T miss and image features f i As the model input, the final output is the missing word T tar The model predicts the masked part of the text in the dataset. The loss function uses cross entropy loss, and the formula is as follows:

Images