A multi-modal adaptation network learning method based on adversarial training

Abstract

The application discloses a kind of multi-modal adaptation network learning methods based on adversarial training, comprising the following steps: S1, constructing adversarial multi-modal adaptation network, including the construction of multi-modal model and the construction of adversarial disturbance generator;S2, setting loss function, including original loss, loss after disturbance and disturbance difference loss;S3, the multi-modal model constructed is optimized towards the loss direction of minimizing the loss function set, and the adversarial disturbance generator constructed is optimized towards the loss direction of maximizing the loss function set.Less method builds adapter for adapting downstream task and disturbance generator for generating adversarial noise disturbance on existing multi-modal model, and adapter and disturbance generator are unified in an adversarial training framework for end-to-end training, so that multi-modal model can obtain better robustness and generalization in adapting downstream task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer deep learning, and particularly relates to a multi-modal adaptation network learning method based on adversarial training. BACKGROUND

[0002] When BERT [1] , GPT [2] representative pre-training technology sweeps the natural language processing field, the general multi-modal learning represented by multi-modal pre-training technology spreads in the multi-modal learning field, and quickly replaces the original way of separately designing and training models for different multi-modal tasks in the "pre-training-fine-tuning" paradigm. The core of the multi-modal pre-training method is to train a general pre-training model on the basis of multi-modal tasks using a large number of image-text pair data composed of images and texts describing the content of the images, so as to improve the model's general language understanding, general visual understanding and general cross-modal correlation understanding. Subsequently, the pre-trained model will be fine-tuned on a specific downstream task, so that the model can convert the general ability learned in the pre-training phase into the ability to solve specific tasks. The general multi-modal learning using the "pre-training-fine-tuning" method can greatly improve the model's performance on downstream tasks, which is the reason why this method quickly occupies a dominant position in the multi-modal field.

[0003] However, this method also has some problems to be improved. For example, when fine-tuning on each downstream task, all the parameters participating in the calculation in the multi-modal model need to be optimized, which makes the model still have a large time overhead when deployed to different downstream tasks, and the reuse rate of model parameters is very low, which to some extent makes the multi-modal model lose the original intention of generalization. Therefore, methods of using adapter structure on multi-modal model for rapid adaptation of downstream tasks begin to appear, which have performance comparable to complete fine-tuning, and at the same time have higher flexibility and convenience.

[0004] However, these methods also face some problems, for example, the amount of parameters participating in optimization in the pre-training phase is very large, while the amount of parameters participating in optimization in the adaptation training is very small, which makes the model have the risk of overfitting when performing adaptation training. When performing multi-modal adaptation training, how to explicitly or implicitly suppress the risk of overfitting is a very worthy problem to study. This patent intends to discuss from the key point of enhancing the robustness of the adapter in multi-modal adaptation training, and form a complete multi-modal model training method based on adversarial training. SUMMARY

[0005] The application aims at solving the problems of the prior art, and provides a multi-modal adapter network learning method based on adversarial training.

[0006] To solve the above technical problems, the technical scheme of the application is as follows:

[0007] A multi-modal adapter network learning method based on adversarial training, comprising the following steps:

[0008] S1, constructing an adversarial multi-modal adapter network

[0009] A multi-modal model is constructed, an adapter is added in a Transformer block, the adapter is composed of two full connection layers, the adapter accepts the output of a self-attention module in the Transformer block as input, and the output of the adapter is added to the output of a feedforward network in the Transformer block and then sent to layer normalization for subsequent residual calculation.

[0010] An adversarial noise generator is constructed, the adversarial noise generator is composed of two full connection layers, the output of the multi-modal model after calculation is sent to the adversarial noise generator, adversarial noise disturbance is generated and applied to the original data to generate disturbed data.

[0011] S2, setting a loss function, the loss function includes an original loss calculated by the multi-modal model receiving original data, a disturbed loss calculated by the multi-modal model receiving disturbed data, and a disturbance difference loss for the difference between the two outputs, and the adversarial multi-modal adapter network is optimized according to the three loss functions;

[0012] S3, training of the adversarial multi-modal adapter network

[0013] The constructed multi-modal model is optimized towards the direction of minimizing the loss of the set loss function, and the constructed adversarial noise generator is optimized towards the direction of maximizing the loss of the set loss function.

[0014] Preferably, the adapter is a two-layer linear mapping layer, and a gelu activation function is used in the middle.

[0015] Preferably, the adversarial noise generator is divided into an image noise generator and a text noise generator, both of which are two-layer linear mapping layers, and layer normalization and a gelu activation function are used in the middle.

[0016] As preferred, the image perturbation generator receives the image corresponding part in the multi-modal model output as input; the text perturbation generator receives the text corresponding part in the multi-modal model output as input.

[0017] As preferred, the original loss is an instance-level binary cross-entropy loss function.

[0018] The specific implementation is as follows:

[0019]

[0020] where θ is the trainable parameter of the adapter part in the multi-modal model, each image-text pair is sampled from the entire training data set D, C is the number of categories, and the model needs to make a 2-classification judgment on each category, represents the process of calculating the visual question answering task score according to the input data.

[0021] As preferred, the specific calculation method of the perturbation loss is as follows:

[0022]

[0023] where θ is the trainable parameter of the adapter part in the multi-modal model, represents the learnable parameters in the adversarial perturbation generator, D is the data set for training, image-text pairs sampled from the entire training data set D, C is the number of categories, represents the process of generating perturbed data after the input image-text pair data passes through the multi-modal model and the adversarial perturbation generator.

[0024] As preferred, the input and output of the perturbation loss are calculated by a KL divergence to calculate the perturbation difference loss, and the specific calculation method is as follows:

[0025]

[0026] where SD KL is a symmetric KL divergence calculation, and the specific formula is as follows:

[0027]

[0028] where p(x) and q(x) are two probability distributions of discrete random variables.

[0029] As preferred, the specific training method in step S3 is as follows:

[0030] S3-1, construct image feature representation and text feature representation;

[0031] S3-2, a joint image feature representation and a text feature representation obtain a multi-modal input feature;

[0032] S3-3, the multi-modal input feature is taken as input, and a multi-modal model is used to calculate an output feature vector containing multi-modal information;

[0033] S3-4, the output feature vector of the multi-modal model can be split into a multi-modal image output and a multi-modal text output according to the corresponding positions of the image and the text, and is used for an image perturbation generator and a text perturbation generator to generate a perturbed image feature and a perturbed text feature, respectively, and the perturbed image feature and the perturbed text feature are spliced to obtain a perturbed multi-modal input feature;

[0034] S3-5, adversarial multi-modal adaptation training

[0035] The total loss function is obtained by steps S3-1 to S3-4, and the expression is as follows:

[0036]

[0037] wherein, is an original loss, is a perturbed loss, is a perturbation difference loss, and λ is a coefficient of the perturbation difference loss .

[0038] The optimization objective of the adversarial multi-modal adaptation training is expressed as the following formula:

[0039]

[0040] The present application has the following characteristics and beneficial effects:

[0041] By using the above technical solution, the multi-modal model module is constructed to replace the complete fine-tuning training mode, and the adversarial noise perturbation is generated by the adversarial perturbation generator driven by the output of the multi-modal model, so that the multi-modal model with the adapter can better adapt to the downstream task in the adversarial training, and the generalization and robustness are greatly improved compared with the previous multi-modal adaptation training method without adversarial training. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 : Adversarial multi-modal adaptation training framework.

[0044] Figure 2 : Multi-modal model module.

[0045] Figure 3 : Adversarial perturbation generator module. DETAILED DESCRIPTION

[0046] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0048] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0049] The present application provides an adversarial multi-modal adaptation network learning method based on adversarial training, as shown in Figure 1 The method comprises the following steps:

[0050] Step (1): constructing an adversarial multi-modal adaptation network

[0051] Constructing a multi-modal model;

[0052] As shown in Figure 2As shown, a multi-modal model is constructed on the basis of an existing Transformer block, and an adapter is added in the Transformer block structure in the UNITER structure. The adapter is composed of two fully connected layers, which accepts the output of the self-attention module as input, and the output result is added to the output result of the feedforward network and then sent to the layer normalization for subsequent calculation.

[0053] Specifically, the multi-modal model constructed in the embodiment is to add a calculation structure parallel to the feedforward network after the self-attention module in the Transformer block, which is specifically as follows:

[0054] The structure of the adapter is two linear mapping layers, and a gelu activation function is used in the middle. The specific formula is as follows:

[0055]

[0056] Wherein represents the output of the self-attention module in the lth Transformer block structure, Linear i represents linear mapping, and the specific formula is as follows:

[0057] Linear i (x)=W i x+b i (2)

[0058] Wherein W i and b i are learnable parameters. The output of the adapter is added to the output of the feedforward network, and then the subsequent layer normalization and residual connection calculation are continued.

[0059] An adversarial noise generator is constructed.

[0060] The adversarial noise generator is used to generate adversarial noise disturbance in the framework of adversarial training, which is composed of two fully connected layers. The output of the original data after the calculation of the multi-modal model will be sent to the adversarial noise generator to generate adversarial noise disturbance and apply it to the disturbed data after the original data is disturbed.

[0061] Specifically, the adversarial noise generator constructed in the embodiment is to add an adversarial noise generator on the output of the multi-modal model, and the result will be used to generate disturbed data and then sent to the multi-modal model again, which is specifically as follows:

[0062] As Figure 3As shown, the adversarial perturbation generator is divided into an image perturbation generator and a text perturbation generator, both of which are two-layer linear mapping layers with layer normalization and gelu activation functions in between. The image perturbation generator receives the image corresponding part in the multimodal model output as input, and the specific formula is as follows:

[0063] ADV img (X L )=Linear img2 (gelu(LN(Linea img1 (X L ))) (3)

[0064] Where X L represents the image corresponding part in the multimodal model output, L represents the total number of Transformer blocks in the multimodal model, and LN represents layer normalization. The output of the image perturbation generator is the image perturbation applied to the image features. The text perturbation generator receives the text corresponding part in the multimodal model output as input, and the specific formula is as follows:

[0065] ADV txt (Y L )=Linear txt2 (gelu(LN(Linear txt1 (Y L ))) (4)

[0066] Where Y L represents the text corresponding part in the multimodal model output, and the output of the text perturbation generator is the text perturbation applied to the text features.

[0067] Step (2): loss function setting;

[0068] Specifically, the adversarial training framework in the embodiment includes three loss functions, namely the original loss calculated by the multimodal model receiving the original data, the perturbed loss calculated by the multimodal model receiving the perturbed data, and the perturbation difference loss for the difference between the two output results. The adversarial training framework will be optimized according to the three loss functions.

[0069] The loss function setting refers to constructing a loss function for adversarial training, including original loss, perturbed loss, and perturbation difference loss, as follows:

[0070] 3-1. Original loss

[0071] In this embodiment, taking the visual question answering task as an example, the multi-modal feature output by the multi-modal backbone network is first subjected to a visual question answering task module to calculate an original loss. The VQA v2.0 dataset used in this embodiment regards visual question answering as a multi-classification task, in which the answer to each question can be regarded as a distribution, and the loss function selected is an instance-level binary cross-entropy loss function, which performs binary classification judgment on each class, as follows:

[0072]

[0073] where θ is the trainable parameter of the adapter part in the multi-modal model, each image-text pair is sampled from the entire training data set D, C is the number of classes, and the model needs to make a 2-classification judgment on each class, represents the process of calculating the visual question answering task score according to the input data.

[0074] 3-2. Loss after perturbation

[0075] The multi-modal model accepts the perturbed multi-modal input feature as input, outputs the perturbed visual question answering task score, and further calculates the perturbed task loss. The specific calculation method of the perturbed task loss is as follows:

[0076]

[0077] where represents the learnable parameters in the adversarial perturbation generator, represents the process of generating perturbed data by the multi-modal model and the adversarial perturbation generator after the input image-text pair data.

[0078] 3-3. Perturbation difference loss

[0079] The perturbed task score and the task score before perturbation are calculated by a symmetric KL divergence to calculate the perturbation difference loss. The perturbation difference loss can measure the degree of influence of the perturbation on the model, and its specific calculation method is as follows:

[0080]

[0081] where SD KL is a symmetric KL divergence calculation, and the specific formula is as follows:

[0082]

[0083] where p(x) and q(x) are two probability distributions of discrete random variables.

[0084] Step (3): training of the multi-modal adapter network based on adversarial training;

[0085] The model parameters in step (1) and step (2) are trained in an adversarial training framework, in which the multi-modal model constructed in step (1) is optimized towards minimizing the loss in step (2), while the adversarial perturbation generator constructed in step (2) is optimized towards maximizing the loss in step (2). The core idea of the multi-modal adaptation network learning method based on adversarial training is that the adapter of the multi-modal model tries to eliminate the influence of the adversarial perturbation on the input data during the learning process, so that the model can have better adaptation effect and be more robust in actual application.

[0086] Specifically, as shown in the following: Figure 1

[0087] 4-1. Constructing image feature representation

[0088] The image is first represented by the m regions with the highest confidence extracted by the target detection model, where the i-th region is represented as a visual feature and a position feature Two linear mappings are then used to fuse the two features into a d-dimensional image feature

[0089]

[0090] where Finally, the image is represented as a feature matrix The specific formula is as follows:

[0091] X = [x1, x2, …, x m ] (10)

[0092] 4-2. Constructing text feature representation

[0093] The input text paired with the image is cropped or padded to a sequence containing at most n words. Each word w i and its position i in the sequence are mapped to vectors by two independent embedding layers, respectively, and then added to obtain a position-sensitive text feature representation y i The specific formula is as follows:

[0094] y i = WordEmbed(w i ) + IdxEmbed(i) (11)

[0095] where WordEmbed represents the learned word embedding layer, IdxEmbed represents the learned position embedding layer, and y i is the text word w i ​d-dimensional features. Finally, the text is represented as a feature matrix The specific formula is as follows:

[0096] Y = [y1, y2,..., y n ] (12)

[0097] 4-3. Joint image feature representation and text feature representation

[0098] The image feature X = [x1,..., x m ] and the text feature Y = [y1,..., y n ] are concatenated before being input into the network. Two special symbols are inserted in the concatenated features to obtain the multi-modal input feature Z, and the specific formula is as follows:

[0099] Z = [x1,..., x m , [SEP], y1,..., y n , [CLS]] (13)

[0100] Where [SEP] marks the boundary between image features and text features, and the output corresponding to [CLS] is used to predict the answer to the given image and text question.

[0101] 4-4. Multi-modal model calculation containing adapter

[0102] The multi-modal model in this method is a single-flow Transformer structure with L layers, and each layer in the Transformer is composed of a self-attention module, an adapter module, and a feedforward network module. The multi-modal input feature Z is input into the multi-modal model, and the output is The specific formula is as follows:

[0103]

[0104]

[0105]

[0106] Where Z 0 = Z, l = 1, 2,..., L, LN represents the layer normalization after each module, MSA is the self-attention module, and its input is the output Z l-1 of the previous Transformer block, and its output is a feature vector rich in multi-modal information ADP represents the adapter module, and FFN represents the feedforward network module. Finally, the output of the multi-modal model is represented as Z L .

[0107] 4-5. Generation and application of adversarial perturbation

[0108] Output of the multi-modal model Z L Splitting the multi-modal image output X according to the corresponding position of image and text L And multi-modal text output Y L , respectively used for image perturbation module and text perturbation module to generate perturbation of corresponding modal, the specific process is as follows:

[0109] P img = ADV img (X L ) (17)

[0110]

[0111] Where ADV img represents the image perturbation generator, ADV txt represents the text perturbation generator, P img represents the image perturbation for applying on image feature, P txt represents the text perturbation for applying on text feature. In order to ensure that the generated adversarial perturbation is small enough, the image perturbation and the text perturbation will be modified according to the size of the image feature and the text feature respectively, and the specific formula is as follows:

[0112] ||clip(X+e img ×P img , min(X), max(X)||2 = ∈ img (19)

[0113] ||clip(Y+e txt ×P txt , min(Y), max(Y))||2 = ∈ txt (20)

[0114] Where e img and e txt are the correction coefficients of image perturbation and text perturbation respectively, their values are determined according to formula 19 and formula 20, min(·) represents taking the minimum value, max(·) represents taking the maximum value, clip(a, b, c) represents the clipping operation, that is, taking the value less than b in a as b, and taking the value greater than c in a as c, ∈ img and ∈ txt represent a fixed coefficient of image modal and text modal respectively. The process of applying perturbation to image feature and text feature is as follows:

[0115]

[0116]

[0117] wherein denotes the perturbed image feature, denotes the perturbed text feature, and the two are concatenated to obtain the perturbed multi-modal input feature The specific results are as follows:

[0118]

[0119] wherein the [SEP] symbol and the [CLS] symbol are not subjected to perturbation.

[0120] 4-6. Adversarial multi-modal adaptation training

[0121] In the adversarial multi-modal adaptation training, as described in formulas (9) to (13), the visual feature of each image region and the text word are subjected to visual embedding processing and text embedding processing respectively and are concatenated to obtain the multi-modal input feature Z, which is sent to the multi-modal model containing the adapter module, and the multi-modal model output Z is obtained through the calculation as described in formulas (14) to (16). L The output at the position corresponding to the [CLS] symbol will be used for the original loss calculation as described in formula (5), and the original loss L is obtained. As described in formulas (17) to (23), the multi-modal model output Z L is sent to the adversarial perturbation generator to generate adversarial perturbation and apply it to the original data, and the perturbed multi-modal input feature Z is obtained. Similarly, the perturbed multi-modal input feature Z is input into the multi-modal model containing the adapter module, and the perturbed multi-modal model output Z is obtained through the same calculation process as described in formulas (14) to (16). The output at the position corresponding to the [CLS] symbol will be used for the perturbed loss calculation as described in formula (6), and the perturbed loss L is obtained. Z L and The output at the position corresponding to the [CLS] symbol in formula (23) will also be subjected to the calculation process as described in formulas (7) to (8) to calculate the perturbation difference loss L The final total loss is represented by the following formula:

[0122]

[0123] wherein λ is the coefficient of the perturbation difference loss L The optimization objective of the adversarial multi-modal adaptation training is represented by the following formula:

[0124]

[0125] The adversarialness of the adversarial multi-modal adaptation training is that the adapter module in the multi-modal model is optimized in the direction of minimizing the total loss, while the adversarial perturbation generator is optimized in the direction of maximizing the total loss. In implementation, in order to save the number of forward propagation and back propagation, and thus speed up the training, the adversarial perturbation generator will not perform separate forward propagation and back propagation, but will perform back propagation based on the total loss together with the adapter module, and then separately take the gradient of the parameters in the adversarial perturbation generator to be negative after the back propagation is completed. Since the calculation of the original loss does not involve the parameters of the adversarial perturbation generator, the result of this part of the loss back propagation will not affect the gradient of these parameters. The gradient taking negative of the back propagation based on the perturbed loss and the perturbation difference loss can be equivalent to the back propagation based on the negative perturbed loss and the negative perturbation difference loss, so as to achieve the effect of adjusting the optimization goal of the adversarial perturbation generator to increase the total loss.

[0126] In the adversarial multi-modal adaptation training, the adversarial perturbation generator and the adapter are continuously optimized according to the above steps until the adapter training converges.

[0127] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of these embodiments including components can be made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.

Claims

1. A learning method for adversarial multimodal adaptive networks based on adversarial training, characterized in that, Includes the following steps: S1. Construct an adversarial multimodal adaptation network To construct a multimodal model, an adapter is added to the Transformer block. The adapter consists of two fully connected layers. The adapter accepts the output of the self-attention module in the Transformer block as input. The output of the adapter is added to the output of the feedforward network in the Transformer block and then fed into the layer for normalization and subsequent residual calculation. An adversarial perturbation generator is constructed, which consists of two fully connected layers. The output of the original data after calculation by the multimodal model is sent to the adversarial perturbation generator to generate adversarial noise perturbation and apply it to the original data to generate perturbed data. S2. Set the loss function, which includes the original loss calculated by the multimodal model receiving the original data, the perturbation loss calculated by the multimodal model receiving the perturbed data, and the perturbation difference loss for the difference between the two output results. The adversarial multimodal adaptation network is optimized based on these three loss functions. Training S3, Adversarial Multimodal Adaptive Network The constructed multimodal model is optimized to minimize the loss of the set loss function, while the constructed adversarial perturbation generator is optimized to maximize the loss of the set loss function. The specific training methods are as follows: S3-1. Constructing image feature representations and text feature representations; S3-2. Combine image feature representation and text feature representation to obtain multimodal input features; S3-3. Using multimodal input features as input, the multimodal model calculates and outputs a feature vector containing multimodal information. S3-4. The output feature vector of the multimodal model can be split into multimodal image output and multimodal text output according to the corresponding positions of the image and text. These are used by the image perturbation generator and the text perturbation generator to generate perturbed image features and perturbed text features, respectively. The perturbed image features and perturbed text features are then concatenated to obtain the perturbed multimodal input features. S3-5, Adversarial Multimodal Adaptation Training The total loss function, obtained from steps S3-1 to S3-4, is expressed as follows: in, For the original loss, Loss after disturbance Let be the perturbation difference loss, where λ is the perturbation difference loss. The coefficient; The optimization objective of adversarial multimodal adaptation training is expressed by the following formula:

2. The adversarial multimodal adaptive network learning method based on adversarial training according to claim 1, characterized in that, The adapter consists of two linear mapping layers with the Gelu activation function in between.

3. The adversarial multimodal adaptive network learning method based on adversarial training according to claim 1, characterized in that, The adversarial perturbation generator is divided into an image perturbation generator and a text perturbation generator. Both the image perturbation generator and the text perturbation generator have two linear mapping layers with layer normalization and Gelu activation function in between.

4. The adversarial multimodal adaptive network learning method based on adversarial training according to claim 3, characterized in that, The image perturbation generator receives the corresponding portion of the image from the multimodal model output as input; The text perturbation generator receives the corresponding text portion from the multimodal model output as input.

5. The adversarial multimodal adaptive network learning method based on adversarial training according to claim 1, characterized in that, The original loss function used is the instance-level binary cross-entropy loss function.

6. The adversarial multimodal adaptive network learning method based on adversarial training according to claim 1, characterized in that, The specific calculation method for the loss after the disturbance is as follows: Where θ represents the trainable parameters of the adapter component in the multimodal model. Let represent the learnable parameters in the adversarial perturbation generator, and D be the dataset used for training. Image-text pairs sampled from the entire training dataset D, where C is the number of categories. This refers to the process by which the input image and text data is perturbed by a multimodal model and an adversarial perturbation generator.

7. The adversarial multimodal adaptation network learning method based on adversarial training according to claim 1, characterized in that, The input and output of the perturbation loss are calculated using a KL divergence to determine the perturbation difference loss.

Images