An Adversarial Multimodal Pre-training Method Enhanced by Multimodal Scene Graph Knowledge

Abstract

The present invention discloses an adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge. The steps of the present invention are as follows: 1. Construct the feature representation of image regions and image prior knowledge; 2. Construct text and cross-modal prior knowledge; 3. Construct a multi-modal scene graph; 4. Construct an adversarial noise generation network and a multi-modal pre-training network; 5. Adversarial pre-training enhanced by multi-modal scene graph knowledge. The present invention extracts prior knowledge from two modalities, i.e., images and texts, thereby constructing a multi-modal scene graph, and constructing an adversarial noise generation network with a multi-modal self-attention network enhanced by multi-modal scene graph knowledge as the core to generate noise perturbations for the input image-text pair data and perform adversarial multi-modal pre-training with the multi-modal pre-training network, and finally achieving better performance on downstream tasks.

Description

Technical Field

[0001] The present invention proposes an adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge. The core method is to extract prior knowledge from two modalities of images and texts, thereby constructing a multi-modal scene graph, and constructing an adversarial noise generation network with a multi-modal self-attention network enhanced by multi-modal scene graph knowledge as the core to generate noise perturbations for the input image-text pair data and perform adversarial multi-modal pre-training with the multi-modal pre-training network. Background Art

[0002] Inspired by the success of the BERT model based on the "pre-training - fine-tuning" paradigm proposed by Google researchers in natural language understanding tasks, some multi-modal pre-training (Vison-and-Language Pretraining, VLP) models for solving a wide range of vision-and-language (V+L) tasks have been proposed. These methods first pre-train a Transformer-based model in a large image-text corpus, and then fine-tune it in downstream vision-and-language tasks including visual question answering, image-text retrieval, and text-based visual content localization. Different from the previous methods that were only applicable to one vision-and-language task, the multi-modal pre-training models can be generalized to multiple vision-and-language tasks and have achieved significant performance improvements on these tasks.

[0003] Learning fine-grained semantic alignment between image regions and text words is a very crucial point in vision-and-language tasks. However, it is very expensive and unrealistic to perform fine-grained manual annotation of image regions and text words in large-scale scenes. Therefore, most of the existing multi-modal pre-training methods use a weakly-supervised learning strategy to implicitly learn this alignment. These methods take image regions and text words as inputs, adopt a multi-layer Transformer as the backbone network, and learn fine-grained semantic alignment through coarse-grained image-text matching supervision. In addition, intra-modal and inter-modal interference, as well as the widespread noisy data in the image-text pair dataset, also bring more challenges to learning semantic alignment.

[0004] When performing multi-modal pre-training, how to explicitly or implicitly enhance the model to learn more fine-grained cross-modal semantic alignment is a very worthy research issue. This patent intends to start from several key points of enhancing fine-grained semantic alignment learning in multi-modal pre-training to conduct discussions and form a complete set of adversarial multi-modal pre-training methods enhanced by multi-modal scene graph knowledge. Summary of the Invention

[0005] The purpose of the present invention is to provide an adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge for the deficiencies of the prior art. The present invention mainly includes two points:

[0006] Model the image scene graph using the relative position relationship information and relative spatial association information between image regions as prior knowledge within the image modality; model the text scene graph using the semantic association information between text words as prior knowledge within the text modality; model the cross-modal association graph using the semantic similarity information between image regions and text words, and combine the image scene graph and the text scene graph to form a multi-modal scene graph containing intra-modal and inter-modal knowledge.

[0007] Construct a multi-modal self-attention network enhanced with multi-modal scene graph knowledge, and based on this, construct an adversarial noise generation network, enabling it to fully explore the representational associations within and between modalities, generate targeted noise perturbations, and perform adversarial pre-training with the multi-modal pre-trained model, so that the multi-modal pre-trained model can implicitly learn richer cross-modal fine-grained alignment information.

[0008] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0009] Step (1): Construct the feature representation of image regions and image prior knowledge;

[0010] For each image in the dataset, use the trained object detection network to extract a series of objects contained in the image and their corresponding visual features and spatial positions, and use them as the feature representation of image regions. Calculate the relative position information between image regions and the relative spatial association information between image regions through the spatial positions of the objects, and use the relative position information and the relative spatial association information as the prior knowledge of the image.

[0011] Step (2): Construct text and cross-modal prior knowledge;

[0012] For each image-text pair in the dataset, use the trained scene graph generator to extract the set of object words, the set of attribute words, and the set of relationship words in the text description, and record the semantic associations between each object word and its related attribute words and relationship words according to the co-occurrence frequencies statistically calculated on the dataset. These semantic associations constitute the text prior knowledge.

[0013] For each image-text pair in the dataset, calculate the semantic similarity between the category text corresponding to the target category of each image region and each object word, and use the semantic similarity as the semantic association information between the image region and the text word, and use it as the representation of the cross-modal prior knowledge.

[0014] Step (3): Construct a multi-modal scene graph

[0015] The multi-modal scene graph is composed of an image scene graph, a text scene graph, and a cross-modal association graph. Among them, the vertices in the image scene graph are image regions, and the weights of the edges are the quantitative representations of image prior knowledge; the vertices in the text scene graph are text words, and the weights of the edges are the quantitative representations of text prior knowledge; the vertices of the cross-modal association graph are image regions and text words, and the weights of the edges are the quantitative representations of cross-modal prior knowledge. The final multi-modal scene graph will be represented by an adjacency matrix.

[0016] Step (4): Construct an adversarial noise generation network and a multi-modal pre-training network;

[0017] The adversarial noise generation network is composed of a multi-modal self-attention network enhanced by multi-modal scene graph knowledge and a noise generation module. Among them, the multi-modal self-attention network enhanced by multi-modal scene graph knowledge is the core part. Its backbone structure consists of a feature embedding layer containing position embeddings and several stacked Transformer encoders. In the self-attention mechanism part of the Transformer encoder, self-attention enhanced by multi-modal scene graph knowledge is used. Above the multi-modal self-attention network enhanced by multi-modal scene graph knowledge, there is also a noise generation module structure composed of MLP, which is used to generate adversarial noise perturbations.

[0018] The multi-modal pre-training network is a single-stream structure, and its backbone network consists of several stacked Transformer encoders. Above the backbone network, there are also multiple task module structures composed of MLP for different pre-training tasks.

[0019] Step (5): Adversarial pre-training enhanced by multi-modal scene graph knowledge;

[0020] When performing adversarial multi-modal pre-training, the feature representations of image regions and text words are respectively processed through visual embedding and text embedding to obtain image features and text features, and the image features and text features are concatenated to obtain multi-modal input features.

[0021] The multi-modal input features and the multi-modal scene Figure 1 are fed into the adversarial noise generation network to obtain perturbed multi-modal input features. Subsequently, the multi-modal pre-training network respectively receives the perturbed multi-modal input features and the unperturbed multi-modal input features as inputs, and respectively outputs the pre-training task scores after perturbation and the pre-training task scores before perturbation, and calculates the pre-training task loss after perturbation and the pre-training task loss before perturbation.

[0022] A perturbation difference loss is calculated on the pre-training task scores after perturbation and the pre-training scores before perturbation. The perturbation difference loss, the pre-training task loss after perturbation, and the pre-training task loss before perturbation together constitute the total loss of adversarial multi-modal pre-training. Both the adversarial noise generation network and the multi-modal pre-training network will be optimized based on the total loss. The adversarial nature of adversarial multi-modal pre-training is reflected in that the multi-modal pre-training network is optimized in the direction of minimizing the total loss, while the adversarial noise generation network is optimized in the direction of maximizing the total loss.

[0023] Further, constructing the feature representation and prior knowledge of the image region in step (1) of the present invention means using the visual features of the image region combined with the region box coordinates as the feature representation of the image region, and using the relative position information between the image regions and the relative spatial association information between the image regions as the prior knowledge of the image region, specifically as follows:

[0024] For each image in the dataset, use the Faster R-CNN object detection network trained on the Visual Genome dataset to extract at most M objects contained in the image and their corresponding spatial positions, where the spatial position of the m-th object is represented as (where represents a 4-dimensional vector), the visual feature representation is (where represents a d f -dimensional vector, usually 2048), and the predicted object category is represented as (where represents a 1-dimensional vector). Among them, the position coordinates and visual features will be used as the feature representation of the image region.

[0025] Model the relative position relationship information by the relative position between any two region boxes; define the spatial position of the i-th object as (x i , y i , w i , h i ), and these four-dimensional coordinates respectively represent the horizontal and vertical coordinates of the center point of the object, width, and height; the relative spatial relationship feature between the i-th object and the j-th object is represented as The specific formula is as follows:

[0026]

[0027] Represent the relative spatial association information by the IoU between any two region boxes, and the specific formula is as follows:

[0028]

[0029] Furthermore, the construction of text and cross-modal prior knowledge in step (2) of the present invention refers to using the semantic association information between text words as the text prior knowledge, and using the semantic association information between text words and image regions as the cross-modal prior knowledge, as follows:

[0030] For each image-text pair in the dataset, use the trained scene graph generator to extract the object word set W in the text description. obj , attribute word set W attr and the set of relation words W rel , and record each object word w according to the statistical co-occurrence frequency in the entire data set i ∈W obj and its related attribute word w j ∈W attr Semantic association between And each object word w i ∈W obj and the related word w k ∈W rel Semantic association between The specific formula is as follows:

[0031]

[0032]

[0033] Among them, n i 、n j and n k Respectively represent the word w i 、w j and w k The number of times it appears in the data set, n i，j and n i，k Respectively represent the word w i and w j 、w i and w k The number of times they co-occur in the dataset.

[0034] Use the pre-trained word vector model to transform each image region m i Target category The corresponding category text and each object word w j ∈W obj Convert to word vector containing semantic information Use cosine similarity to calculate the relative semantic association information between the two The specific formula is as follows:

[0035]

[0036] Further, the construction of the multimodal scene graph in step (3) of the present invention refers to constructing a multimodal scene graph by using an image scene graph, a text scene graph, and a cross-modal association graph, specifically as follows:

[0037] 3-1. Construct an image scene graph

[0038] For each image in the dataset, the image scene graph G i = <V i , E i , S i > will be used to encode the knowledge within the image modality. The vertex set V i contains all regions in the image, the edge set E i contains the pairwise relationships between vertices, and the similarity set S i represents the similarity corresponding to an edge, which is represented by the relative position information and relative spatial association information between image regions. For a certain edge e i，j with similarity s i，j , the calculation formula is as follows:

[0039]

[0040] where Linear is a linear transformation.

[0041] The image scene graph G i is represented by an M×M adjacency matrix, where M represents the number of vertices of the image scene graph, i.e., the number of image regions, and the element in the i-th row and j-th column of the matrix is the similarity s i，j between vertex i and vertex j.

[0042] 3-2. Construct a text scene graph

[0043] For each text in the dataset, the text scene graph G t = <V t , E t , S t > will be used to encode the knowledge within the text modality. The vertex set V t contains all words in the text, the edge set E t contains the pairwise relationships between vertices, and the similarity set S t represents the similarity corresponding to an edge, which is represented by the semantic association information between text words. For a certain edge e i，j with similarity s i，j , it is represented by the semantic association information . Similarly, the text scene graph G tIt is represented by an N×N adjacency matrix, where N represents the number of vertices of the text scene graph, i.e., the number of text words, and the element in the i-th row and j-th column of the matrix is the similarity s between vertex i and vertex j i，j 。

[0044] 3-3. Construct a cross-modal association graph

[0045] For each image-text pair in the dataset, the cross-modal association graph G c =<V c ,E c ,S c > will be used to encode the knowledge between modalities. The vertex set V c contains all regions in the image and all words in the text. The edge set E c contains the pairwise relationships between vertices where the two endpoints of the vertex belong to different modalities. The similarity set S c represents the similarity represented by the semantic association information between the image region and the text word corresponding to the edge. For a certain edge e i，j with similarity s i，j is represented by the semantic association information Similarly, the cross-modal association graph G c is represented by an (M + N)×(M + N) adjacency matrix, where M represents the number of vertices of the image scene graph, i.e., the number of image regions, N represents the number of vertices of the text scene graph, i.e., the number of text words, and the element in the i-th row and j-th column of the matrix is the similarity s between vertex i and vertex j i，j 。

[0046] 3-4. Compose a multi-modal scene graph

[0047] For each image-text pair in the dataset, after merging the image scene graph, the text scene graph, and the cross-modal association graph, the multi-modal scene graph structure G = <V, E, S> is obtained to encode the intra-modal and inter-modal knowledge simultaneously, where V = V i U V t U V c ,E = E i ∪ E t U E c ,S = S i U S t ∪ S cThe vertex set V contains regions in the image and words in the text, and the edge set E and the similarity set S contain pairwise relationships between vertices and their corresponding similarities respectively. Among them, the subgraph composed of image region vertices comes from the image scene graph, the subgraph composed of text word vertices comes from the text scene graph, and the cross-modal edges connecting image region vertices and text word vertices come from the cross-modal association graph. Since the similarity distributions of the image and text modalities may be very different, the scene graph representations in each modality need to be normalized. Similarly, the multi-modal scene graph G is represented by an (M + N) × (M + N) adjacency matrix, where M represents the number of vertices in the image scene graph, that is, the number of image regions, and N represents the number of vertices in the text scene graph, that is, the number of text words. The element in the i-th row and j-th column of the matrix is the normalized similarity between vertex i and vertex j Specifically, it is as follows:

[0048]

[0049]

[0050] …

[0051]

[0052] where P = M + N.

[0053] Furthermore, the construction of the adversarial noise generation network described in step (4) of the present invention refers to constructing an adversarial noise generation network by using a multi-modal self-attention network enhanced by multi-modal scene graph knowledge and an adversarial noise generation head, specifically as follows:

[0054] 4-1. Constructing image feature representation

[0055] First, the image is processed by a target detection model to extract the m regions with the highest confidence. The i-th region is represented as a visual feature and a location feature Subsequently, two linear mappings fuse these two features into a d-dimensional image feature

[0056]

[0057] where Finally, the image is represented as a feature matrix The specific formula is as follows:

[0058] X = [x1, x2,..., x m (Formula 9)

[0059] 4-2. Constructing text feature representation

[0060] The input text paired with the image is cropped or padded to a sequence containing at most n words. Each word w i and its position i in the sequence are respectively mapped to vectors through two independent embedding layers, and then added to obtain a position-sensitive text feature representation y i , and the specific formula is as follows:

[0061] y i = WordEmbed(w i ) + IdxEmbed(i) (Formula 10)

[0062] where y i is the d-dimensional feature of the word w i in the text. Finally, the text is represented as a feature matrix The specific formula is as follows:

[0063] Y = [y1, y2,..., y n (Formula 11)

[0064] 4-3. Combining Image Feature Representation and Text Feature Representation

[0065] The image feature X = [x1,..., x m and the text feature Y = [y1,..., y n are concatenated before being input into the network. Two special symbols are inserted into the concatenated features to obtain the multimodal input feature Z, and the specific formula is as follows:

[0066] Z = [x1,..., x m , [SEP], y1,..., y n , [CLS]] (Formula 12)

[0067] where the [SEP] symbol marks the boundary between the image feature and the text feature, and the [CLS] symbol is used to predict whether the given image and text match.

[0068] 4-4. Constructing a Multimodal Self-Attention Network Enhanced by Multimodal Scene Graph Knowledge

[0069] The multimodal self-attention network enhanced by multimodal scene graph knowledge is an L-layer single-stream Transformer structure. Each layer in the Transformer consists of a multi-head self-attention KMSA module enhanced by multimodal scene graph knowledge and a feed-forward network (FFN) module. The multimodal input feature Z and the multimodal scene graph G are used as the inputs of the multimodal self-attention network enhanced by multimodal scene graph knowledge, and the output The specific formula is as follows:

[0070]

[0071]

[0072] Among them, Z 0 = Z. After each module, layer normalization (LN) and residual connections are connected. The input of KMSA is Z l-1 and G, and the output is a feature vector rich in multimodal information The specific formula is as follows:

[0073] Q = Linear[Z l-1 (Formula 15)

[0074] K = Linear[Z l-1 (Formula 16)

[0075] V = Linear[Z l-1 (Formula 17)

[0076]

[0077] Among them, Q, K, are respectively obtained by mapping Z l-1 through a fully connected layer;

[0078] ∈ = 1e -6 to avoid underflow problems;

[0079] Among them, MLP is a two-layer perceptron, and the specific formula is as follows:

[0080]

[0081] Among them, ReLU is an activation function, and FC1, is a fully connected layer;

[0082] Among them, the input of the FFN structure is The specific formula is as follows:

[0083]

[0084] 4 - 5. Construct a noise generation module

[0085] The noise generation module receives the output of the multimodal self-attention network enhanced by the multimodal scene graph knowledge, and uses two MLPs to generate adversarial noise perturbations P l for the text modality and adversarial noise perturbations P v for the image modality respectively. The specific formula is as follows:

[0086] Pv = MLP(X L ), P l = MLP(Y L ) (Equation 21)

[0087] where X L and Y L are the multi-modal image representation and multi-modal text representation respectively sliced from the output Z of the multi-modal self-attention network enhanced by multi-modal scene graph knowledge. The specific slicing process is as follows: L

[0088]

[0089] 4-6. Constructing a multi-modal pre-training network

[0090] The multi-modal pre-training network is a single-stream structure, with its backbone network composed of several stacked Transformer encoders. Above the backbone network, there are also multiple MLP task module structures for different pre-training tasks and downstream tasks.

[0091] Furthermore, the adversarial pre-training enhanced by multi-modal scene graph knowledge in step (5) of the present invention is as follows:

[0092] 5-1. Multi-modal pre-training target tasks

[0093] The multi-modal pre-training network is pre-trained on three target tasks commonly used in VLP methods: the Image-Text Matching (ITM) task, the Masked Language Modeling (MLM) task, and the Masked Region Modeling (MRM) task.

[0094] The ITM task needs to judge whether the currently input image-text pair matches. This task uses cross-entropy loss, and the specific formula is as follows:

[0095]

[0096] where θ are the trainable parameters in the multi-modal pre-training network, and each pair is sampled from the entire training dataset D. is the output at the corresponding position of the [CLS] symbol after passing through the network. After passing through the MLP, it predicts whether the input image pair matches.

[0097] MLM and MRM respectively randomly mask the input text words and image regions, and predict the masked content through the context.

[0098] ​The MLM task uses cross - entropy loss, and the specific formula is as follows:

[0099]

[0100] Where M w represents the text words to be masked. The masked input features are fed into the MLM task after passing through the network to reconstruct the masked words represents the remaining unmasked text words in

[0101] Similarly, the MRM task is represented as:

[0102]

[0103] Where f θ (·) represents a certain loss function. In this patent, both regression - based and classification - based loss functions are used here represents the remaining unmasked image regions in

[0104] 5 - 2. Adversarial Multimodal Pre - training

[0105] When performing adversarial multimodal pre - training, as described in formulas (8) to (11), the visual features of each image region and each text word w i are respectively processed through visual embedding and text embedding to obtain the image feature X = [x1,..., x m and the text feature Y = [y1,..., y n .

[0106] After that, the image feature X and the text feature Y are concatenated, and together with the multimodal scenario Figure 1 are fed into an adversarial noise generation network composed of a multimodal self - attention network enhanced by multimodal scene graph knowledge and a noise generation module to obtain image feature perturbations and text feature perturbations, and the perturbations are applied to the feature representations of the original data to obtain the perturbed image feature X′ and the perturbed text feature Y′

[0107] As shown in formula (12), the image feature X and the text feature Y before perturbation will be concatenated after inserting two special symbols to obtain the pre - perturbation multimodal input feature Z = [x1,..., x m , [SEP], y1,..., y n , [CLS]]. The perturbed image feature X′ and the perturbed text feature Y′ are also concatenated to obtain the perturbed multimodal input feature Z′ = [x′1,..., x′ m, [SEP], y′1, ..., y′ n , [CLS], where the [SEP] symbol and the [CLS] symbol do not impose perturbations.

[0108] Subsequently, the multi-modal pre-training network takes the perturbed multi-modal input features Z′ and the unperturbed multi-modal input features Z as inputs respectively, outputs the pre-training task scores after perturbation and the pre-training task scores before perturbation, and further calculates the pre-training task loss after perturbation and the pre-training task loss before perturbation. The pre-training task loss before perturbation is calculated in the form of formulas (23) to (25), and the specific calculation formula for the pre-training task loss after perturbation is as follows:

[0109]

[0110]

[0111]

[0112] where represents the learnable parameters in the adversarial noise generation network, and represent the perturbed text features and image features generated by the adversarial noise generation network respectively.

[0113] In addition, the pre-training task score after perturbation and the pre-training score before perturbation also calculate a perturbation difference loss through a symmetric KL divergence. The specific formula for the perturbation difference loss is as follows:

[0114]

[0115] where h θ (·) represents the pre-training task score output by the multi-modal pre-training network, D KL (p||q) is the KL divergence, and the specific formula is as follows:

[0116]

[0117] where p(x) and q(x) are two probability distributions of discrete random variables.

[0118] Finally, the total loss of adversarial multi-modal pre-training is expressed as:

[0119]

[0120] The optimization objective of adversarial multi-modal pre-training is expressed as:

[0121]

[0122] The adversarial nature of adversarial multi-modal pre-training is reflected in that the multi-modal pre-training network is optimized in the direction of minimizing the pre-training loss after perturbation, the pre-training loss before perturbation, and the perturbation difference, while the adversarial noise generation network is optimized in the direction of maximizing the pre-training loss after perturbation and the perturbation difference.

[0123] In terms of implementation, in order to save the number of forward and backward propagations and thus speed up the training, the adversarial noise generator will not perform separate forward and backward propagations. Instead, like the multi-modal pre-training model, it performs backward propagation based on the pre-training loss after perturbation, the pre-training loss before perturbation, and the perturbation difference. After the backward propagation is completed, the gradients of the parameters in the adversarial noise generation network are separately negated. Since the calculation of the pre-training loss before perturbation does not involve the parameters of the adversarial noise generation network, the results of the backward propagation of this part of the loss will not affect the gradients of these parameters. And negating the gradients obtained by backward propagation based on the pre-training loss after perturbation and the perturbation difference is equivalent to backward propagation based on the negative pre-training loss after perturbation and the negative perturbation difference, thus achieving the effect of adjusting the optimization objective of the adversarial noise generation network to increase the pre-training task loss.

[0124] During adversarial multi-modal pre-training, the adversarial noise generation network and the multi-modal pre-training model are continuously optimized according to the above steps until the multi-modal pre-training model converges.

[0125] The beneficial effects of the present invention are as follows:

[0126] The present invention proposes an adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge. By modeling the multi-modal scene graph, it encodes both intra-modal and inter-modal knowledge, and uses the multi-modal self-attention network enhanced by multi-modal scene graph knowledge to drive the adversarial noise generation network to generate noise perturbations for cross-modal semantics, enabling the multi-modal pre-training model to learn a more fine-grained cross-modal semantic alignment ability in adversarial training, and having a great improvement in performance compared to previous multi-modal pre-training methods that do not target more fine-grained semantic alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 : Multi-modal self-attention module enhanced by multi-modal scene graph knowledge

[0128] Figure 2 : Adversarial multi-modal pre-training framework enhanced by multi-modal scene graph knowledge DETAILED DESCRIPTION OF THE EMBODIMENTS

[0129] The following further specifically describes the detailed parameters of the present invention with reference to the accompanying drawings.

[0130] As Figure 2As shown, the present invention provides an adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge.

[0131] The construction of the feature representation and prior knowledge of the image region in step (1) is specifically as follows:

[0132] For each image in the dataset, use the Faster R-CNN object detection network trained on the Visual Genome dataset to extract at most 36 objects contained in the image and their corresponding spatial positions, where the spatial position of the m-th object is represented as The visual feature representation is The predicted object category is represented as Among them, the position coordinates and visual features will be used as the feature representation of the image region.

[0133] The prior knowledge of the image region includes the relative position information between image regions and the relative spatial association information between image regions.

[0134] For every two candidate boxes m i and m j in each image of the dataset, according to their position coordinates and calculate the relative position relationship Define the spatial position of the i-th object as (x i , y i , w i , h i ), and these four-dimensional coordinates respectively represent the horizontal and vertical coordinates of the center point of the object, width, and height; the relative position information between the i-th object and the j-th object is represented as

[0135] For every two candidate boxes m i and m j in each image of the dataset, calculate the IoU between the two region boxes according to their position coordinates and and further calculate the relative spatial association information

[0136] The relative position information and relative spatial association information between image regions constitute the prior knowledge of the image region.

[0137] The construction of the text and cross-modal prior knowledge in step (2) is specifically as follows:

[0138] The text prior knowledge is represented by the semantic association information between text words.

[0139] For each group of image-text pairs in the dataset, use the trained scene graph generator to extract the set of object words \(W\) in the text description obj , the set of attribute words \(W\) attr and the set of relationship words \(W\) rel , and record the co-occurrence frequency on the entire dataset for each object word \(w\) i \(\in W\) obj and the semantic association between the associated attribute word \(w\) j \(\in W\) attr [[ID=?]] and the semantic association between each object word \(w\) i \(\in W\) obj and the associated relationship word \(w\) k \(\in W\) rel between

[0140] Cross-modal prior knowledge is represented by the semantic association information between image regions and text words.

[0141] For each group of image-text pairs in the dataset, use the pre-trained Glove word vector model to convert the category text corresponding to the target category of each image region \(m\) i into a word vector containing semantic information respectively for each object word \(w\) \(\in W\) j \(\in W\) obj and calculate the relative semantic association information between the two using cosine similarity as the semantic association information between the image region and the text word. <00%0580>

[0142] The construction of the multi-modal scene graph described in step (3) is as follows:

[0143] The multi-modal scene graph consists of an image scene graph, a text scene graph, and a cross-modal association graph.

[0144] 3-1. Construct the image scene graph

[0145] For each image in the dataset, the image scene graph \(G\) i = <V i , E i , S i > will be used to encode the knowledge within the image modality. The vertex set \(V\) i contains all regions in the image, and the edge set \(E\) i contains the pairwise relationships between the vertices. The similarity set \(S\) i represents the similarity corresponding to the edge, which is represented by the relative position information and relative spatial association information between the image regions where Linear is a linear transformation. The image scene graph \(G\) i ​It is represented by an M×M adjacency matrix, where M represents the number of vertices of the image scene graph, i.e., the number of image regions, and the element in the i-th row and j-th column of the matrix is the similarity s between vertex i and vertex j i，j 。

[0146] 3-2. Construct the text scene graph

[0147] For each text in the dataset, the text scene graph G t = <V t , E t , S t > will be used to encode the knowledge within the text modality. The vertex set V t contains all the words in the text, and the edge set E t contains the pairwise relationships between the vertices. The similarity set S t represents the similarity represented by the semantic association information between the text words corresponding to the edge. For the similarity s i，j of a certain edge e i，j , it is represented by the semantic association information . Similarly, the text scene graph G t is represented by an N×N adjacency matrix, where N represents the number of vertices of the text scene graph, i.e., the number of text words, and the element in the i-th row and j-th column of the matrix is the similarity s i，j 。

[0148] 3-3. Construct the cross-modal association graph

[0149] For each image-text pair in the dataset, the cross-modal association graph G c = <V c , E c , S c > will be used to encode the knowledge between modalities. The vertex set V c contains all the regions in the image and all the words in the text. The edge set E c contains the pairwise relationships between the vertices where the two endpoints of the edge belong to different modalities. The similarity set S c represents the similarity represented by the semantic association information between the image regions and the text words corresponding to the edge. For the similarity s i，j of a certain edge e i，j , it is represented by the semantic association information . Similarly, the cross-modal association graph G c is represented by an (M + N)×(M + N) adjacency matrix, where M represents the number of vertices of the image scene graph, i.e., the number of image regions, and N represents the number of vertices of the text scene graph, i.e., the number of text words. The element in the i-th row and j-th column of the matrix is the similarity s i，j 。

[0150] 3-4. Constructing a Multimodal Scene Graph

[0151] For each group of image-text pairs in the dataset, after merging the image scene graph, the text scene graph, and the cross-modal association graph, a multimodal scene graph structure G = <V, E, S> is obtained to encode the intra-modal and inter-modal knowledge simultaneously, where V = V i ∪ V t ∪ V c ,E = E i ∪ E t ∪ E c ,S = S i ∪ S t ∪ S c 。The vertex set V contains the regions in the image and the words in the text, and the edge set E and the similarity set S contain the pairwise relationships between the vertices and their corresponding similarities respectively. The subgraph composed of the image region vertices comes from the image scene graph, the subgraph composed of the text word vertices comes from the text scene graph, and the cross-modal edges connecting the image region vertices and the text word vertices come from the cross-modal association graph. Since the similarity distributions of the image and text modalities may be very different, the scene graph representations in each modality need to be normalized. Similarly, the multimodal scene graph G is represented by an (M + N) × (M + N) adjacency matrix, where M represents the number of vertices in the image scene graph, i.e., the number of image regions, N represents the number of vertices in the text scene graph, i.e., the number of text words, and the element in the i-th row and j-th column of the matrix is the normalized similarity between vertex i and vertex j

[0152] The construction of the adversarial noise generation network and the multimodal pre-training network described in step (4) is as follows:

[0153] 4-1. Constructing Image Feature Representations

[0154] The image first extracts the m regions with the highest confidence by the object detection model, and the i-th region is represented as a visual feature and a location feature Subsequently, two linear mappings fuse these two features into a d-dimensional image feature where Finally, the image is represented as a feature matrix X = [x1, x2,..., m .

[0155] 4-2. Constructing Text Feature Representations

[0156] The input text paired with the image is cropped or padded to a sequence containing at most n words. Each word w iIt and its position i in the sequence are respectively mapped to vectors through two independent embedding layers, and then added together to obtain a position-sensitive text feature representation y i = WordEmbed(w i ) + IdxEmbed(i), where y i is a d-dimensional feature that matches the image feature. Finally, the text is represented as a feature matrix Y = [y1, y2,..., y m .

[0157] 4-3. Combining Image Feature and Text Feature Representations

[0158] The image feature X = [x1,..., x m and the text feature Y = [y1,..., y n are concatenated before being input into the network. We insert two special symbols into the concatenated features to obtain the multimodal input feature Z = [x1,..., x m , [SEP], y1,..., y n , [CLS]], where the [SEP] symbol marks the boundary between the image feature and the text feature, and the [CLS] symbol is used to predict whether the given image and text match.

[0159] 4-4. Constructing a Multimodal Self-Attention Network Enhanced by Multimodal Scene Graph Knowledge

[0160] The multimodal self-attention network enhanced by multimodal scene graph knowledge is an L-layer single-stream Transformer structure. Each layer in the Transformer consists of a multi-head self-attention KMSA module enhanced by multimodal scene graph knowledge and a feed-forward network (FFN) module. The multimodal input feature Z and the multimodal scene graph G are used as the inputs of the multimodal self-attention network enhanced by multimodal scene graph knowledge, and the output where Z 0 = Z. After each module, layer normalization (LN) and residual connection are connected. The input of KMSA is Z l-1 and G, and the output is a feature vector rich in multimodal information

[0161] 4-4. Constructing a Noise Generation Module

[0162] The noise generation module receives the output of the multimodal self-attention network enhanced by multimodal scene graph knowledge and uses two MLPs to generate adversarial noise perturbations P l for the text modality and adversarial noise perturbations P v = MLP(XL ), Pl = MLP(Y L ), where X L and Y L are the multi-modal image representation and multi-modal text representation respectively sliced from the output Z of the multi-modal self-attention network enhanced by multi-modal scene graph knowledge L .

[0163] 4-5. Constructing a multi-modal pre-training network

[0164] The multi-modal pre-training network is a single-stream structure, with its backbone network composed of several stacked Transformer encoders. Above the backbone network, there are also multiple MLP task module structures for different pre-training tasks and downstream tasks.

[0165] The final adversarial noise generation network is as shown in the dashed box in Figure 2 , and the multi-modal pre-training network is as shown in the dashed box in Figure 2 .

[0166] The adversarial pre-training enhanced by multi-modal scene graph knowledge described in step (5) is as follows:

[0167] 5-1. Multi-modal pre-training target tasks

[0168] The multi-modal pre-training network is pre-trained on three target tasks commonly used in VLP methods: the Image-Text Matching (ITM) task, the Masked Language Modeling (MLM) task, and the Masked Region Modeling (MRM) task.

[0169] The ITM task needs to judge whether the current input image-text pair is matched, and this task uses the cross-entropy loss. MLM and MRM respectively randomly mask the input text words and image regions, and predict the masked content through the context environment. The MLM task uses the cross-entropy loss, while the MRM task uses both regression-based and classification-based loss functions.

[0170] 5-2. Adversarial multi-modal pre-training

[0171] When performing adversarial multi-modal pre-training, as described in formulas (8) to (11), the visual features of each image region and each text word w i are respectively obtained as image feature X = [x1,..., x m and text feature Y = [y1,..., y n after visual embedding processing and text embedding processing.

[0172] Subsequently, the image feature X and the text feature Y are concatenated, and sent together into an adversarial noise generation network composed of a multi-modal self-attention network enhanced by multi-modal scene graph knowledge and a noise generation module to obtain the image feature perturbation and the text feature perturbation, and the perturbations are applied to the feature representations of the original data to obtain the perturbed image feature X' and the perturbed text feature Y'. Figure 1 As shown in formula (12), the image feature X and the text feature Y before perturbation will be concatenated after inserting two special symbols to obtain the multi-modal input feature Z before perturbation = [x1,..., x

[0173] ,[SEP], y1,..., y m , [SEP], y1,..., y n , [CLS]]. The perturbed image feature X' and the perturbed text feature Y' are also concatenated to obtain the perturbed multi-modal input feature Z' = [x'1,..., x' m , [SEP], y'1,..., y' n , [CLS]], where the [SEP] symbol and the [CLS] symbol are not perturbed.

[0174] Subsequently, the multi-modal pre-training network takes the perturbed multi-modal input feature Z' and the multi-modal input feature Z before perturbation as inputs respectively, and outputs the pre-training task score after perturbation and the pre-training task score before perturbation respectively, and further calculates the pre-training task loss after perturbation and the pre-training task loss before perturbation. The pre-training task loss before perturbation is calculated in the form of formulas (23) to (25), and the pre-training task loss after perturbation is calculated in the form of formulas (26) to (28).

[0175] In addition, the pre-training task score after perturbation and the pre-training score before perturbation also calculate a perturbation difference loss through a symmetric KL divergence calculation.

[0176] Finally, the total loss of adversarial multi-modal pre-training is the sum of the pre-training task loss before perturbation, the pre-training task loss after perturbation, and the perturbation difference loss.

[0177] The adversarial nature of adversarial multi-modal pre-training is reflected in that the multi-modal pre-training network is optimized in the direction of minimizing the pre-training loss after perturbation, the pre-training loss before perturbation, and the perturbation difference, while the adversarial noise generation network is optimized in the direction of maximizing the pre-training loss after perturbation and the perturbation difference.

[0178] In implementation, in order to save the number of forward and backward propagations and thus accelerate the training speed, the adversarial noise generator will not perform separate forward and backward propagations. Instead, like the multi-modal pre-training model, it will perform backward propagation based on the pre-training loss after perturbation, the pre-training loss before perturbation, and the perturbation difference. After the backward propagation ends, the gradients of the parameters in the adversarial noise generation network will be inverted separately. Since the calculation of the pre-training loss before perturbation does not involve the parameters of the adversarial noise generation network, the results of the backward propagation of this part of the loss will not affect the gradients of these parameters. And inverting the gradients obtained by backward propagation based on the pre-training loss after perturbation and the perturbation difference is equivalent to performing backward propagation based on the negative pre-training loss after perturbation and the negative perturbation difference, thus achieving the effect of adjusting the optimization objective of the adversarial noise generation network to increase the pre-training task loss.

[0179] During adversarial multi-modal pre-training, the adversarial noise generation network and the multi-modal pre-training model are continuously optimized according to the above steps until the multi-modal pre-training model converges.

Claims

1. An adversarial multi-modal pre-training method enhanced by multi-modal scene graph knowledge, characterized in that It includes the following steps: Step (1): Construct the feature representation of the image region and the image prior knowledge; Step (2): Construct the text and cross-modal prior knowledge; Step (3): Construct the multi-modal scene graph; Step (4): Construct the adversarial noise generation network and the multi-modal pre-training network; Step (5): Adversarial pre-training enhanced by multi-modal scene graph knowledge; Step (1) is further implemented as follows: For each image in the dataset, use the Faster R-CNN object detection network trained on the Visual Genome dataset to extract at most M objects contained in the image and their corresponding spatial locations, where the spatial location of the m-th object is represented as The visual feature is represented as The predicted object category is represented as where the position coordinates and visual features will be used as the feature representation of the image region; Generate the relative position relationship information by modeling the relative positions between any two region boxes; Define the spatial position of the i-th object as (x i , y i , w i , h i ). These four-dimensional coordinates respectively represent the horizontal and vertical coordinates of the center point of the object, width, and height; the relative spatial relationship feature between the i-th object and the j-th object is expressed as The specific formula is as follows: Represent the relative spatial association information by the IoU between any two region boxes. The specific formula is as follows: Step (2) is specifically implemented as follows: For each group of image-text pairs in the dataset, use the trained scene graph generator to extract the set of object words $W$ in the text description obj , the set of attribute words $W$ attr and the set of relationship words $W$ rel , and record the semantic association between each object word $w$ i ∈ $W$ obj and its related attribute word $w$ j ∈ $W$ attr according to the co-occurrence frequency statistics on the entire dataset and the semantic association between each object word $w$ i ∈ $W$ obj and its related relationship word $w$ k ∈ $W$ rel according to the co-occurrence frequency statistics on the entire dataset The specific formula is as follows: where n i , n j and n k represent the number of times the words w i , w j and w k appear in the dataset respectively, and n i,j and n i,k represent the number of times the words w i and w j , w i and w k appear together in the dataset; Use the pre-trained word vector model to convert the target category i of each image region m and the category text corresponding to each object word w j ∈W obj into word vectors containing semantic information Calculate the relative semantic association information between the two using cosine similarity The specific formula is as follows: Step (3) is specifically implemented as follows: 3-1. Construct the image scene graph For each image in the dataset, the image scene graph G i = <V i , E i , S i > will be used to encode the knowledge within the image modality. The vertex set V i contains all regions in the image, and the edge set E i contains pairwise relationships between vertices. The similarity set S i represents the similarity corresponding to an edge, which is represented by the relative position information and relative spatial association information between image regions; for a certain edge e i,j with similarity s i,j , the calculation formula is as follows: where Linear is a linear transformation; the image scene graph G i is represented by an M×M adjacency matrix, where M represents the number of vertices of the image scene graph, i.e., the number of image regions, and the element in the i-th row and j-th column of the matrix is the similarity s between vertex i and vertex j i,j ; 3-2. Construct the text scene graph For each piece of text in the dataset, the text scene graph G t = <V t , E t , S t > will be used to encode the knowledge within the text modality. The vertex set V t contains all the words in the text, and the edge set E t contains the pairwise relationships between the vertices. The similarity set S t represents the similarity represented by the semantic association information between the text words corresponding to the edges; for an edge e i,j with similarity s i,j , it is represented by the semantic association information ; Similarly, the text scene graph G t is represented using an N×N adjacency matrix, where N represents the number of vertices in the text scene graph, i.e., the number of text words, and the element in the i-th row and j-th column of the matrix is the similarity s i,j between vertex i and vertex j; 3-3. Construct the cross-modal association graph For each group of image-text pairs in the dataset, the cross-modal association graph G c = <V c , E c , S c > will be used to encode the knowledge between modalities. The vertex set V c contains all regions in the image and all words in the text. The edge set E c contains the pairwise relationships between vertices, where the two endpoints of the edge belong to different modalities. The similarity set S c represents the similarity represented by the semantic association information between the image region and the text word corresponding to the edge; for an edge e i,j with similarity s i,j , it is represented by the semantic association information ; Similarly, the cross-modal association graph G c is represented by an (M + N) × (M + N) adjacency matrix, where M represents the number of vertices in the image scene graph, i.e., the number of image regions, and N represents the number of vertices in the text scene graph, i.e., the number of text words. The element in the i-th row and j-th column of the matrix is the similarity s i,j between vertex i and vertex j; 3-4. Compose the multi-modal scene graph For each group of image-text pairs in the dataset, after merging the image scene graph, the text scene graph, and the cross-modal association graph, a multi-modal scene graph structure G = <V, E, S> is obtained to simultaneously encode intra-modal and inter-modal knowledge, where V = V i ∪ V t ∪ V c , E = E i ∪ E t ∪ E c , S = S i ∪ S t ∪ S c ; The vertex set V contains regions in the image and words in the text. The edge set E and the similarity set S contain pairwise relationships between vertices and their corresponding similarities, respectively. The subgraph composed of image region vertices comes from the image scene graph, the subgraph composed of text word vertices comes from the text scene graph, and the cross-modal edges connecting image region vertices and text word vertices come from the cross-modal association graph. Since the similarity distributions of the image and text modalities may be very different, the scene graph representations in each modality need to be normalized. Similarly, the multi-modal scene graph G is represented by an (M + N) × (M + N) adjacency matrix, where M represents the number of vertices in the image scene graph, i.e., the number of image regions, and N represents the number of vertices in the text scene graph, i.e., the number of text words. The element in the i-th row and j-th column of the matrix is the normalized similarity between vertex i and vertex j Specifically, it is represented as follows: where P = M + N; Step (4) is specifically implemented as follows: 4-1. Construct the image feature representation The image first extracts the m regions with the highest confidence by the object detection model, and the i-th region among them is represented as a visual feature and a location feature Subsequently, two linear mappings fuse these two features into a d-dimensional image feature Among them Finally, the image is represented as a feature matrix The specific formula is as follows: X = [x1, x2, …, x m (Equation 9) 4-2. Construct the text feature representation The input text paired with the image is cropped or padded into a sequence containing at most n words; each word w i and its position i in the sequence are respectively mapped to vectors through two independent embedding layers, and then added to obtain a position-sensitive text feature representation y i , and the specific formula is as follows: y i = WordEmbed(w i ) + IdxEmbed(i) (Equation 10) where y i is the d-dimensional feature of the word w i in the text; finally, the text is represented as a feature matrix The specific formula is as follows: Y = [y1, y2, …, y n (Formula 11) 4-3. Combine the image feature representation and the text feature representation The image feature X = [x1, …, x m and the text feature Y = [y1, …, y n are concatenated before being input into the network; two special symbols are inserted into the concatenated feature to obtain the multi-modal input feature Z, and the specific formula is as follows: Z = [x1, …, x m , …, y1, …, y n , ] (Equation 12) where the [SEP] symbol marks the boundary between the image feature and the text feature, and the [CLS] symbol is used to predict whether the given image and text match; 4-4. Construct the multi-modal self-attention network enhanced by multi-modal scene graph knowledge The multi-modal self-attention network enhanced by multi-modal scene graph knowledge is an L-layer single-stream Transformer structure. Each layer in the Transformer consists of a multi-head self-attention KMSA module enhanced by multi-modal scene graph knowledge and a feed-forward networks (FFN) module. Taking the multi-modal input feature Z and the multi-modal scene graph G as the inputs of the multi-modal self-attention network enhanced by multi-modal scene graph knowledge, the output is The specific formula is as follows: where Z 0 = Z; after each module, layer normalization and residual connections are connected, and the input of KMSE is Z l-1 and G, and the output is a feature vector rich in multimodal information The specific formula is as follows: Q = Linear[Z l-1 (Equation 15) K = Linear[Z l-1 (Formula 16) V = Linear[Z l-1 (Formula 17) Among them, Q, K, are respectively mapped by Z l-1 through a fully connected layer; ∈ = 1e -6 , to avoid the underflow problem; where MLP is a two-layer perceptron. The specific formula is as follows: Among them, ReLU is the activation function, and FC1, is the fully connected layer; Among them, the input of the FFN structure is The specific formula is as follows: 4-5. Construct the noise generation module The noise generation module receives the output of the multi-modal self-attention network enhanced by the multi-modal scene graph knowledge, and uses two MLPs to generate adversarial noise perturbations \(P\) for the text modality l and adversarial noise perturbations \(P\) for the image modality v , and the specific formula is as follows: P v = MLP(X L ), P l = MLP(Y L ) (Equation 21) Where X L and Y L are respectively the multi-modal image representation and multi-modal text representation sliced from the output Z L of the multi-modal self-attention network enhanced by multi-modal scene graph knowledge. The specific slicing process is as follows: [x1 L ,…,x m L ,[[SEP] L ,y1 L ,…,y n L ,[CLS] L =X L ,Y L = Split[Z L (Formula 22) = Split[x1 L ,…,x m L ,[SEP] L ,y1 L ,…,y n L ,[CLS] L ​ 4-6. Construct the multi-modal pre-training network The multi-modal pre-training network is a single-stream structure, with several stacked Transformer encoders forming its backbone network, and there are also multiple MLP task module structures on top of the backbone network; Step (5) is specifically implemented as follows: 5-1. Multi-modal pre-training target tasks The multi-modal pre-training network is pre-trained on three target tasks commonly used in VLP methods: the image-text matching task, the masked language modeling task, and the masked region modeling task; the ITM task needs to judge whether the currently input image-text pair matches; MLM and MRM respectively randomly mask the input text words and image regions, and predict the masked content through the context; 5-2. Adversarial multi-modal pre-training When performing adversarial multi-modal pre-training, the visual features of each image region and each text word w i After visual embedding processing and text embedding processing respectively, the image feature X = [x1, …, x m and the text feature Y = [y1, …, y n ; After that, the image feature X and the text feature Y are concatenated, and together with the multi-modal scene graph, they are fed into the adversarial noise generation network composed of the multi-modal self-attention network enhanced by multi-modal scene graph knowledge and the noise generation module to obtain the image feature perturbation and the text feature perturbation, and the perturbations are applied to the feature representation of the original data to obtain the perturbed image feature X' and the perturbed text feature Y'; As shown in formula (12), the image feature X and text feature Y before perturbation will be concatenated after inserting two special symbols, resulting in the multi-modal input feature Z before perturbation = [x1, …, x m , [SEP], y1, …, y n , [CLS]]; the image feature X' and text feature Y' after perturbation are also concatenated to obtain the multi-modal input feature Z' after perturbation = [x'1, …, x' m , [SEP], y'1, …, y' n , [CLS]], where the [SEP] symbol and [CLS] symbol are not perturbed; Subsequently, the multi-modal pre-training network takes the perturbed multi-modal input feature Z' and the unperturbed multi-modal input feature Z as inputs respectively, outputs the perturbed pre-training task score and the unperturbed pre-training task score respectively, and further calculates the perturbed pre-training task loss and the unperturbed pre-training task loss; in addition, the perturbed pre-training task score and the unperturbed pre-training score also calculate the perturbation difference loss through a symmetric KL divergence calculation; Finally, the optimization objective of adversarial multi-modal pre-training is expressed as: Among them, is the total loss of adversarial multi-modal pre-training.

2. The adversarial multi-modal pre-training method with knowledge enhancement of multi-modal scene graph according to claim 1, wherein The construction of the feature representation of the image region and the image prior knowledge described in Step (1) is implemented as follows: For each image in the dataset, use the trained object detection network to extract a series of objects contained in the image and their corresponding visual features and spatial positions, which are used as the feature representation of the image regions; calculate the relative position information between the image regions and the relative spatial association information between the image regions through the spatial positions of the objects, and use the relative position information and the relative spatial association information as the prior knowledge of the image.

3. A multi-modal scene graph knowledge enhanced adversarial multi-modal pre-training method according to claim 1, characterized in that The construction of the text and cross-modal prior knowledge described in step (2) is achieved as follows: For each text-image pair in the dataset, use the trained scene graph generator to extract the set of object words, the set of attribute words, and the set of relationship words in the text description, and record the semantic associations between each object word and its related attribute words and relationship words according to the co-occurrence frequencies statistically calculated on the dataset. These semantic associations constitute the text prior knowledge; For each text-image pair in the dataset, calculate the semantic similarity between the category text corresponding to the target category of each image region and each object word, and use the semantic similarity as the semantic association information between the image region and the text word, which is used as the representation of the cross-modal prior knowledge.

4. A method for adversarial multi-modal pre-training with knowledge enhancement of multi-modal scene graphs according to claim 1, characterized in that The construction of the multi-modal scene graph described in step (3) is achieved as follows: The multi-modal scene graph consists of an image scene graph, a text scene graph, and a cross-modal association graph; where the vertices in the image scene graph are image regions, and the weights of the edges are the quantitative representations of the image prior knowledge; the vertices in the text scene graph are text words, and the weights of the edges are the quantitative representations of the text prior knowledge; The vertices of the cross-modal association graph are image regions and text words, and the weights of the edges are the quantitative representations of the cross-modal prior knowledge; the final multi-modal scene graph will be represented by an adjacency matrix.

Images