Change description system and method fusing cross-view cross-modal double momentum contrast learning

Abstract

The invention belongs to the technical field of image processing, and discloses a change description system and method fusing cross-view mode double momentum contrast learning, and the system comprises an input module which is used for converting two images and text descriptions in a to-be-compared image pair into visual features and word embedding of corresponding images; the single-view-angle Transform encoder is used for carrying out feature extraction on the visual features of each image to obtain the invariance features of each image; the cross-view angle Transformer encoder is used for matching the invariance features of the two images in the image pair and obtaining the change feature representation of the image pair; the single-mode language Transform encoder is used for converting word embedding into language feature representation; the multi-mode Transform decoder is used for fusing the change feature representation and the language feature representation of the image pair to obtain a fused feature; and the output module is used for obtaining change description of the image pair according to the fusion features. According to the method, key visual invariance features can be captured, and change description is enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and relates to image change description technology, and in particular to a change description system and method integrating cross-view modality dual momentum contrast learning. Background Art

[0002] Distinguishing and describing subtle differences between similar visual scenes is a fundamental human skill. In recent years, this unique skill has been simulated by the change description / image difference description computational task in the field of deep learning. The change description task extends the traditional change detection task based on pure computer vision and attempts to simulate the complex interaction between the human visual system and the language system, which brings greater challenges. Specifically, this task requires the complex process of distinguishing and determining subtle but important changes in visual scenes in the presence of a large number of interfering variables, and it is also necessary to effectively use natural language to express these changes. Therefore, a mature change description system is inseparable from a deep understanding of scene changes and the establishment of a close connection between vision and language. This task has broad practical application prospects in the fields of industrial anomaly identification, industrial infrastructure monitoring, and medical diagnosis.

[0003] With the rapid development of deep learning in recent years, a large number of mature methods have emerged in the field of change description tasks. Most of these methods follow the traditional change detection framework, mainly identifying semantic changes in images on the visual side. In these methods, the features that remain unchanged in the two images are generally regarded as potential interference factors. Therefore, these methods usually adopt methods such as: subtracting the "before change" image from the "after change" image, or filtering out the invariant features from the changed features to decouple this potential interference factor. However, human annotators often capture the salient features in the two images, which not only highlight the changes, but also include the invariant features, providing valuable contextual information.

[0004] Based on this, a variation description method that is more in line with human habits should expand its capabilities from simple variation feature description to include key invariant elements in the image; this includes: 1) salient background elements that provide contextual information between the two images; 2) invariant features related to understanding variation targets, especially invariant features when images lack alignment and are taken from different angles.

[0005] Therefore, insightful change descriptions must detect and consider both changes and key invariant features. In these cases, identifying these consistent visual cues and effectively associating them with linguistic elements is crucial for generating informative change descriptions. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention aims to provide a change description system that can capture key visual invariance features and enhance change description.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions to achieve it.

[0008] The present invention provides a change description system, which includes:

[0009] An input module, used to convert two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images;

[0010] The single-view Transformer encoder is used to extract the visual features of each image and obtain the invariant features of each image;

[0011] A cross-view Transformer encoder that matches the invariant features of two images in an image pair and obtains a representation of the changing features of the image pair;

[0012] A unimodal language Transformer encoder, which converts word embeddings into language feature representations;

[0013] The multimodal Transformer decoder is used to fuse the change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder to obtain a fused feature;

[0014] The output module is used to obtain the change description of the image pair based on the fused features.

[0015] The above input module uses a shared pre-trained visual backbone network to linearly project each image in the image pair into a number of grid visual features. Specifically, the visual backbone network can be ResNet, Vision Transformer, etc. The input module also labels each word in the text description to obtain word embedding.

[0016] The above-mentioned single-view Transformer encoder is used for information interaction within a single image and obtaining invariant features within each individual image. intra , the visual features of each image are taken as input. Single-view Transformer encoder T intra Including N intra Layer, by N intra The multi-head self-attention mechanism is used to fuse the visual features of the blocks (i.e., the local context information in the two images B and A to be compared).i+1 and A i+1 represents the output of the i-th Transformer block (i.e. the output of the i-th layer), and the expression is as follows:

[0017] B i+1 =FFN(LN(B i +MHA(B i ,B i ,B i )));

[0018] A i+1 =FFN(LN(A i +MHA(A i ,A i ,A i )));

[0019] Among them, FFN represents feedforward neural network, LN represents layer normalization operation, and MHA represents multi-head attention operation. intra After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and

[0020] The cross-view Transformer encoder is responsible for matching the invariant features in the two images and inferring the representation of the change features of the two images. inter , taking the output of the single-view Transformer encoder as input, the cross-view Transformer encoder T inter Including N inter Layer, by N inter The multi-head cross-attention mechanism is used to fuse and interact with the invariant feature representation across views. and The i′th Transformer block (i.e., i′th layer) in the cross-view encoder operates as follows:

[0021] B i′+1 =FFN(LN(B i′ +MHA(B i′ ,A i′ ,A i′ )));

[0022] A i′+1 =FFN(LN(A i′ +MHA(A i′ ,B i′ ,B i′ )));

[0023] After N inter After layer cross-attention and feed-forward operations, the final inter-image visual representation is obtained: and Where N v =N intra +N inter In this process, key invariant visual features are obtained. Since informative change descriptions usually contain both change features and invariant concepts, we continue the operation by adding the learned invariant representation to the original visual representation. Then, the cross-view representations are concatenated and linearly projected to the same dimension as the word embedding, expressed as:

[0024]

[0025] Among them, B0 and A0 represent the visual features of the two images in the image pair, [·] represents the cross-dimensional splicing operation, and W c and b c denote the weight and bias matrices of the linear projection, respectively. Represents mesh-level features fused across views. This step generates a visual representation that fuses variations and key invariant features.

[0026] The purpose of the above-mentioned unimodal language Transformer encoder is to align the language representation with the matching visual features. In order to encode the contextual information describing the text, in the present invention, the unimodal language Transformer encoder T :ni Convert the description into a high-level language feature representation, which contains N :ni layer, using N :ni The layers are stacked with masked Transformer blocks; each layer consists of two sublayers: a masked self-attention sublayer and a feed-forward sublayer. The computational operation of the language encoder layer is as follows:

[0027]

[0028] in, represents the forward masked multi-head attention operation, which allows each word in the encoder to only focus on all the previous words in the sentence while preventing information leakage from future words. The final language feature is represented as Contains contextual information of the entire sentence and is used as sentence features.

[0029] The above multimodal Transformer decoder promotes the interaction between aligned visual and language features, and then generates a variation description through the output module. and Afterwards, through a multimodal Transformer decoder T 3:I Fusion, interactive cross-modal features, the decoder contains N 3:I Layer, using N 3:I The i′′th layer is composed of stacked Transformer blocks, which include masked self-attention, cross-attention and feed-forward operations. The i′′th layer can be expressed as:

[0030]

[0031] The final output can be expressed as: Where N I =N :ni +N 3:I .

[0032] The above output module linearly transforms the fused features to the same size as the given vocabulary and calculates the probability distribution of the target word:

[0033]

[0034] Among them, W E and b E denote the weight and bias matrices respectively.

[0035] Finally, we search for descriptions with higher probabilities based on the probability distribution of all words. Specifically, we use greedy decoding to take the word with the highest probability at each step as the output to obtain the final description of the changes in the image pair.

[0036] The present invention also provides a training method for the above-mentioned change description system, integrating cross-view modality dual momentum contrast learning, comprising the following steps:

[0037] S1 divides the training data into several training batches, each training sample includes an image pair before and after the change and a text description; for any training batch, the following steps S2-S6 are performed;

[0038] S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrast learning;

[0039] S3 obtains the variation feature representation of the training sample image pairs and the language feature representation of the text description, and constructs an alignment loss function between the image pairs and the text description based on cross-modal contrastive learning;

[0040] S4 constructs a cross entropy loss function for predicting change description;

[0041] S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross entropy loss function of the predicted change description;

[0042] S6 optimizes the parameters of the change description system according to the total loss function;

[0043] Repeat the above steps S2-S6 until the change description system converges.

[0044] In the above step S1, the two images and text descriptions in the training sample image pair are also converted into visual features and word embeddings of the corresponding images through the input module.

[0045] In the above step S2, cross-view contrastive learning is introduced to obtain a change-invariant representation by aligning the internal features of the “before change” and “after change” images across views.

[0046] Step S2 includes the following sub-steps:

[0047] S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder;

[0048] S22 obtains a queue of similar samples of the invariant features of each image in the image pair through a single-view Transformer momentum encoder;

[0049] S23 constructs an alignment loss function for image pairs based on the invariance features of image pairs and similar sample queues.

[0050] In the above step S21, according to the explanation of the single-view Transformer encoder given above, the visual features of each image in the image pair are input into the single-view Transformer encoder to obtain the invariant features of each image in the training sample image pair. and Represents the invariant features of image pairs.

[0051] In the above step S22, the single-view Transformer momentum encoder has the same structure as the single-view Transformer encoder. First, a sliding average single-view Transformer momentum encoder is used to generate queues of the most recent Q1 "before change" and "after change" samples respectively.

[0052] The visual features of the image before the change in the image pair are input into the single-view Transformer momentum encoder to obtain a queue of similar samples of the image before the change in Q1 image pairs; similarly, the visual features of the image after the change in the image pair are input into the single-view Transformer momentum encoder to obtain a queue of similar samples of the image after the change in Q1 image pairs.

[0053] In the above step S23, the first feature extraction module first extracts features of the invariant features of the image pair and the similar sample queue to obtain corresponding image internal feature representations.

[0054] The first feature extraction module uses the invariant features of image pairs ( and ) and linearly project them to obtain the internal feature representation X of the image before and after the change De\ and X a\t :

[0055]

[0056] Among them, W _ represents the weight matrix, b _ represents the bias matrix, and Respectively and The elements in .

[0057] You can also calculate the image I before and after the change according to the following formula: De\ and I a\t The cross-view feature cosine similarity of:

[0058]

[0059] At the same time, the samples in the similar sample queue of the image before the change and the similar sample queue of the image after the change are extracted by the first feature extraction module to obtain the corresponding image internal feature representation, that is: and

[0060] Then, the alignment between the two pairs of matched images is maximized by using the InfoNCE loss, while minimizing the alignment of the unmatched images; for the lth "before" and "after" images in the training batch:

[0061]

[0062] in, represents the internal feature representation of the image of the similar sample queue output by the single-view Transformer momentum encoder, K represents the number of batch samples, τ1 represents the temperature parameter, Q1 represents the number of samples in the similar sample queue, Representation of internal features of images representing similar sample queues after changes Image internal feature representation of the image before and after the change The closest sample (i.e. positive sample), Representation of the internal features of the image before the change, representing the queue of similar samples Image internal feature representation of the transformed image The closest sample (i.e., positive sample), sim(·) represents the cosine similarity function. L |}| It aims to learn invariant feature representations across views to align cross-view images, thereby achieving better cross-view feature fusion interaction and filtering out variable features.

[0063] In the above step S3, in order to better integrate and interact visual and language representations, this step introduces a cross-modal contrastive learning objective.

[0064] Step S3 includes the following sub-steps:

[0065] S31 matches the invariant features of the two images in the image pair obtained in step S21 through a cross-view Transformer encoder to obtain a change feature representation of the image pair;

[0066] S32 matches similar samples of the invariant features of the two images in the image pair obtained in step S22 through a cross-view Transformer momentum encoder to obtain a queue of similar samples represented by the change features of the image pair;

[0067] S33 converts the word embeddings corresponding to the text descriptions in the training samples into language feature representations through a unimodal language Transformer encoder;

[0068] S34 obtains a similar sample queue represented by the language feature obtained in step S33 through a unimodal language Transformer momentum encoder;

[0069] S35 constructs an alignment loss function between image pairs and text descriptions based on the changing feature representation of image pairs and similar sample queues, combined with the language feature representation and similar sample queues.

[0070] In the above step S31, according to the explanation of the cross-view Transformer encoder given above, the invariant features of the images before and after the change output by the single-view Transformer encoder are used as input to obtain the change feature representation of the two images in the training sample image pair.

[0071] In the above step S32, the cross-view Transformer momentum encoder has the same structure as the cross-view Transformer encoder.

[0072] Taking the similar samples of the image before and after the change output by the single-view Transformer momentum encoder in the previous step S22 as input, the sliding average cross-view Transformer momentum encoder is used to obtain Q k The changing features of the image pairs represent similar sample queues.

[0073] In the above step S33, according to the explanation given above, the word embedding of the text description obtained in step S1 is used as input and converted into a language feature representation through a unimodal language Transformer encoder.

[0074] In the above step S34, the unimodal language Transformer momentum encoder has the same structure as the unimodal language Transformer encoder.

[0075] Taking the word embedding of the text description obtained in step S1 as input, Q is obtained through the sliding average unimodal language Transformer momentum encoder k Language feature representation A similar sample cohort.

[0076] In the above step S35, the second feature extraction module and the third feature extraction module are first used to linearly project the change feature representation and the similar sample queue of the image pair and the language feature representation and the similar sample queue to the same dimension to obtain the corresponding change feature update representation and the language feature update representation.

[0077] The second feature extraction module uses the variation feature representation of the image pair The mean pooling feature of the corresponding change feature update representation (Y viF ):

[0078]

[0079] Among them, W v represents the weight matrix, b v represents the bias matrix.

[0080] Similarly, samples in the similar sample queue represented by the change feature of the image pair are obtained through the second feature extraction module to obtain the corresponding change feature update representation, that is:

[0081] The third feature extraction module uses language feature representation (specifically using middle ) to obtain the corresponding language feature update representation (Y Ian ):

[0082]

[0083] Among them, W I represents the weight matrix, b I represents the bias matrix.

[0084] Similarly, samples in the similar sample queue represented by the language features of the text description are updated by obtaining corresponding language features through the third feature extraction module, that is:

[0085] The cosine similarity between the updated representation of the change feature of the image pair and the updated representation of the language feature can also be calculated according to the following formula, sim(Y vis ,Y lan ):

[0086]

[0087] Subsequently, the InfoNCE loss is used to maximize the cross-modal alignment between the matched variation features and the language features (i.e., the positive sample pairs), while minimizing the alignment between the unmatched variation features and the language features:

[0088]

[0089] in, Represents the updated representation of the changing features of the queue of similar samples output by the cross-view Transformer momentum encoder, Represents the updated representation of the language features of the similar sample queue output by the unimodal language Transformer momentum encoder, K represents the number of batch samples, τ k represents the temperature parameter, Representing the updated representation of language features of similar sample queues Update the representation of the changes in the image pair The closest sample (i.e. positive sample), Represents the updated representation of the changing features of similar sample queues Update representation with language features The closest sample (i.e., positive sample), sim(·) represents the cosine similarity function.

[0090] In the above step S4, the overall change description system is trained by minimizing the negative log-likelihood of the predicted word sequence and the true change description S, and the cross entropy loss function of the predicted change description is:

[0091]

[0092] Among them, p θrepresents the probability distribution, which is calculated by the probability distribution O of the previous target word (that is, the output result of the output module), and θ represents all the parameters in the change description system network.

[0093] In the above step S5, during the training phase, the total loss function is obtained by comparing the two learning objectives L |}| , L |‰| and cross entropy L _E Integration for collaborative learning:

[0094] L 总 =L _E +λ1L |}| +λ k L |‰| ;

[0095] Among them, λ1 and λ k is a hyperparameter that adjusts the importance weight.

[0096] In the above step S6, according to the total loss function, the parameters of the change description system (including a single-view Transformer encoder, a cross-view Transformer encoder, a single-modal language Transformer encoder, a multimodal Transformer decoder, a first feature extraction module, a second feature extraction module and a third feature extraction module) are optimized through a gradient descent optimization algorithm (such as an Adam optimization algorithm).

[0097] For momentum models (including single-view Transformer momentum encoder, cross-view Transformer encoder, and unimodal language Transformer momentum encoder), their parameters are updated in a sliding average manner. Instead of back-propagating the gradient down, the parameter update process can be expressed as:

[0098]

[0099] in, represents the momentum model parameters updated at the tth iteration, represents the momentum model parameters updated at the t-1th iteration, θ represents the change description system parameters updated at the tth iteration, and M represents the weight.

[0100] Whether the change description coefficient converges can be judged by the change trend of the total loss function (for example, tending to the set threshold), reaching the minimum, or reaching the upper limit of the number of iterations. When the above requirements are met, the change description system converges.

[0101] The present invention also provides a change description method, which uses the change description system given above to perform the following steps:

[0102] Step 1: The two images and text descriptions in the image pair to be compared are converted into visual features and word embeddings of the corresponding images through the input module;

[0103] Step 2: extract the visual features of each image through a single-view Transformer encoder to obtain the invariant features of each image;

[0104] Step 3, match the invariant features of the two images in the image pair through the cross-view Transformer encoder and obtain the change feature representation of the image pair;

[0105] Step 4: Convert word embeddings into language feature representations through a unimodal language Transformer encoder;

[0106] Step 5: The change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder are fused through the multimodal Transformer decoder to obtain a fused feature;

[0107] Step 6: Based on the fused features, the change description of the image pair is obtained through the output module.

[0108] Compared with the prior art, the change description system and method for integrating cross-view modality dual momentum contrastive learning provided by the present invention has the following beneficial effects:

[0109] (1) The change description system of the present invention can match the invariant features in two images and infer the change representation of the two images through the cross-view Transformer encoder provided;

[0110] (2) The change description system of the present invention can promote the interaction between the change feature representation and the language feature representation of the aligned image pair through the provided multimodal Transformer decoder, thereby generating a change description;

[0111] (3) The present invention introduces a dual momentum contrastive learning in the training of the variation description system; the introduced cross-view contrastive learning objective helps to align cross-view visual features and effectively model the invariant representations between cross-view images to generate information-rich descriptions; the introduced cross-modal contrastive learning objective helps to use language to guide the learning of variational visual features and invariant visual features, and generate strongly correlated visual and linguistic cross-modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 To describe the system structure and principle block diagram for the change of dual momentum contrastive learning that integrates cross-view modalities;

[0113] Figure 2 A training method for a variation description system for fusing dual-momentum contrastive learning across viewpoint modalities;

[0114] Figure 3 This is a schematic diagram of the learning process based on cross-perspective comparison;

[0115] Figure 4 This is a schematic diagram of the cross-modal contrast learning process;

[0116] Figure 5 Describe the method flow diagram for the change;

[0117] Figure 6 The change description results of image pairs under different change types; among them, (a) corresponds to color change, (b) corresponds to texture change, (c) corresponds to object addition, (d) corresponds to object removal, (e) corresponds to object movement, and (f) only perspective change; in the figure, "Before" and "After" respectively represent the perspective before and after the change, and "Unchange" and "Change" respectively represent the background objects that have not changed and the subject that has changed. DETAILED DESCRIPTION

[0118] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0119] Example 1

[0120] This embodiment provides a change description system, such as Figure 1 As shown, it includes an input module, a single-view Transformer encoder, a cross-view Transformer encoder, a single-modal language Transformer encoder, a multi-modal Transformer decoder and an output module.

[0121] 1. Input module

[0122] The input module is used to convert the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images.

[0123] The variation description system processes input in the form of image pairs and text descriptions, represented as (I De\ ,I a\t ,S), where I De\ and I a\t, respectively represent the images before and after the change, and S represents their corresponding descriptive sentences. The input module uses a shared pre-trained visual backbone network to linearly project each image in the image pair into visual features. Specifically, the visual backbone network can be resnet-101; these can be represented as and Where P is the number of grid cells, represents the grid features extracted from the pre-trained grid. The input module also tags each word in the text description to obtain word embeddings. For the input sentence S, special bos and eos stems are inserted at the beginning and end as the start and end symbols of the sentence. Subsequently, it is segmented and converted into word embeddings, represented as Where N represents the length of the sentence, In addition, in order to clearly represent the position of the grid and words in each modality, we also ％ , A ％ and C ％ Position encoding has been added.

[0124] 2. Single-view Transformer encoder

[0125] The single-view Transformer encoder is used to extract the visual features of each image separately and obtain the invariant features of each image.

[0126] The single-view Transformer encoder T provided in this embodiment intra , taking the visual features of each image as input. Single-view Transformer encoder T intra Including N intra Layer, by N intra The multi-head self-attention mechanism is used to fuse the visual features of the blocks (i.e., the local context information in the two images B and A to be compared). i+1 and A i+1 represents the output of the i-th Transformer block (i.e. the output of the i-th layer), and the expression is as follows:

[0127] B i+1 =FFN(LN(B i +MHA(B i ,B i ,B i )));

[0128] A i+1 =FFN(LN(A i +MHA(A i ,A i ,A i )));

[0129] Among them, FFN represents feedforward neural network, LN represents layer normalization operation, and MHA represents multi-head attention operation. intra After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and

[0130] 3. Cross-view Transformer Encoder

[0131] A cross-view Transformer encoder is used to match the invariant features of two images in an image pair and obtain a representation of the changing features of the image pair.

[0132] The cross-view Transformer encoder T introduced in this embodiment inter , taking the output of the single-view Transformer encoder as input, the cross-view Transformer encoder T inter Including N inter Layer, by N inter The multi-head cross-attention mechanism is used to fuse and interact with the invariant feature representation across views. and The i′th Transformer block (i.e., i′th layer) in the cross-view encoder operates as follows:

[0133] B i′+1 =FFN(LN(B i′ +MHA(B i′ ,A i′ ,A i′ )));

[0134] A i′+1 =FFN(LN(A i′ +MHA(A i′ ,B i′ ,B i′ )));

[0135] After N inter After layer cross-attention and feed-forward operations, the final inter-image visual representation is obtained: and Where N v =N intra +N interIn this process, key invariant visual features are obtained. Since informative change descriptions usually contain both change features and invariant concepts, we continue the operation by adding the learned invariant representation to the original visual representation. Then, the cross-view representations are concatenated and linearly projected to the same dimension as the word embedding, expressed as:

[0136]

[0137] Among them, B ％ , A ％ represents the visual features of the two images in the image pair, [·] represents the cross-dimensional splicing operation, and W ( and b ( denote the weight and bias matrices of the linear projection, respectively. Represents mesh-level features fused across views. This step generates a visual representation that fuses variations and key invariant features.

[0138] 4. Unimodal Language Transformer Encoder

[0139] A unimodal language Transformer encoder that converts word embeddings into language feature representations.

[0140] In order to encode the contextual information of the description text, in this embodiment, the unimodal language Transformer encoder T :ni Convert the description into a high-level language feature representation, which contains N :ni layer, using N :ni The layers are stacked with masked Transformer blocks; each layer consists of two sublayers: a masked self-attention sublayer and a feed-forward sublayer. The computational operation of the language encoder layer is as follows:

[0141]

[0142] in, represents the forward masked multi-head attention operation, which allows each word in the encoder to only focus on all the previous words in the sentence while preventing information leakage from future words. The final language feature is represented as Contains contextual information of the entire sentence and is used as sentence features.

[0143] 5. Multimodal Transformer Decoder

[0144] The multimodal Transformer decoder is used to fuse the change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder to obtain a fused feature.

[0145] The multimodal Transformer decoder promotes the interaction between aligned visual and language features to generate variational descriptions through the output module. and Afterwards, through a multimodal Transformer decoder T 3:I Fusion, interactive cross-modal features, the decoder contains N 3:I Layer, using N 3:I The i′′th layer is composed of stacked Transformer blocks, which include masked self-attention, cross-attention and feed-forward operations. The i′′th layer can be expressed as:

[0146]

[0147] The final output can be expressed as: Where N I =N :ni +N 3:I .

[0148] 6. Output module

[0149] The output module is used to obtain the change description of the image pair based on the fused features.

[0150] The above output module linearly transforms the fused features to the same size as the given vocabulary and calculates the probability distribution of the target word:

[0151]

[0152] Among them, W E and b E denote the weight and bias matrices respectively.

[0153] Finally, we search for descriptions with higher probabilities based on the probability distribution of all words. Specifically, we use greedy decoding to take the word with the highest probability at each step as the output to obtain the final description of the changes in the image pair.

[0154] This embodiment also provides a training method for the above-mentioned change description system, such as Figure 2 As shown in Figure 2, the fusion cross-view modality dual momentum contrastive learning includes the following steps:

[0155] S1 divides the training data into several training batches, and each training sample includes an image pair before and after the change and a text description.

[0156] This step also converts the two images and text descriptions in the training sample image pairs into visual features and word embeddings of the corresponding images through the input module.

[0157] The change description system is then trained based on the training samples in each training batch.

[0158] For any training batch, the following steps S2-S6 are performed.

[0159] S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrast learning.

[0160] This step introduces cross-view contrastive learning to obtain a representation that is invariant to changes by aligning the internal features of the “before change” and “after change” images across views.

[0161] like Figure 3 As shown, this step includes the following sub-steps:

[0162] S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder.

[0163] In this step, according to the explanation of the single-view Transformer encoder given above, the visual features of each image in the image pair are input into the single-view Transformer encoder to obtain the invariant features of each image in the training sample image pair. Here, and Represents the invariant features of image pairs.

[0164] S22 obtains a queue of similar samples of the invariant features of each image in the image pair through a single-view Transformer momentum encoder.

[0165] The single-view Transformer momentum encoder has the same structure as the single-view Transformer encoder. First, a sliding average single-view Transformer momentum encoder is used to maintain a queue that stores the most recent Q1 "before change" and "after change" samples respectively.

[0166] The visual features of the image before the change in the image pair are input into the single-view Transformer momentum encoder to obtain Q1 queues of samples similar to the image before the change in the image pair; similarly, the visual features of the image after the change in the image pair are input into the single-view Transformer momentum encoder to obtain Q1 queues of samples similar to the image after the change in the image pair.

[0167] S23 constructs an alignment loss function for image pairs based on the invariance features of the image pairs and a queue of similar negative samples.

[0168] In this step, the first feature extraction module is used to extract the invariant features of the image pair, as well as the positive sample and the similar negative sample queue, to obtain the corresponding image internal feature representation.

[0169] The first feature extraction module uses the invariant features of image pairs ( and ) and linearly project them to obtain the internal feature representation X of the image before and after the change De\ and X a\t :

[0170]

[0171] Among them, W _ represents the weight matrix, b _ represents the bias matrix, and Respectively and The elements in .

[0172] You can also calculate the image I before and after the change according to the following formula: De\ and I a\t The cross-view feature cosine similarity of:

[0173]

[0174] At the same time, the samples in the similar sample queue of the image before the change and the similar sample queue of the image after the change are extracted by the first feature extraction module to obtain the corresponding image internal feature representation, which is: and

[0175] Then, the alignment between the two pairs of matched images is maximized by using the InfoNCE loss, while minimizing the alignment of the unmatched images; for the lth "before" and "after" images in the training batch:

[0176]

[0177] in, represents the internal feature representation of the image of the similar sample queue output by the single-view Transformer momentum encoder, K represents the number of batch samples, τ1 represents the temperature parameter, Q1 represents the number of samples in the similar sample queue, Representation of internal features of images representing similar sample queues after changes Image internal feature representation of the image before and after the change The closest sample (i.e. positive sample), Representation of the internal features of the image before the change, representing the queue of similar samples Image internal feature representation of the image after the change The most similar sample (i.e., positive sample), sim(·) represents the cosine similarity function, and the above log function uses e as the base. L |}| It aims to learn invariant feature representations across views to align cross-view images, thereby achieving better cross-view feature fusion interaction and filtering out variable features.

[0178] S3 obtains the variation feature representation of the sample image pairs for training and the language feature representation of the text description, and constructs an alignment loss function between the image pairs and the text description based on cross-modal contrastive learning.

[0179] In order to better integrate and interact visual and language representations, this step introduces a cross-modal contrastive learning objective.

[0180] like Figure 4 As shown, this step includes the following sub-steps:

[0181] S31 matches the invariant features of the two images in the image pair obtained in step S21 through a cross-view Transformer encoder to obtain a change feature representation of the image pair.

[0182] This step follows the explanation of the cross-view Transformer encoder given above, and uses the output of the single-view Transformer encoder as input to obtain the change feature representation of the two images in the training sample image pair.

[0183] S32 matches similar samples of the invariant features of the two images in the image pair obtained in step S22 through a cross-view Transformer momentum encoder to obtain a queue of similar samples represented by the changing features of the image pair.

[0184] In this step, the cross-view Transformer momentum encoder has the same structure as the cross-view Transformer encoder.

[0185] Taking the similar samples of the image before and after the change output by the single-view Transformer momentum encoder in the previous step S22 as input, the sliding average cross-view Transformer momentum encoder is used to obtain Q k The changing features of the image pairs represent similar sample queues.

[0186] S33 converts the word embeddings corresponding to the text descriptions in the training samples into language feature representations through a unimodal language Transformer encoder.

[0187] According to the explanation given above, the word embedding of the text description obtained in step S1 is used as input and converted into a language feature representation through a unimodal language Transformer encoder

[0188] S34 obtains a similar sample queue represented by the language features obtained in step S33 through a unimodal language Transformer momentum encoder.

[0189] In this step, the unimodal language Transformer momentum encoder has the same structure as the unimodal language Transformer encoder.

[0190] Taking the word embedding of the text description obtained in step S1 as input, it is converted into language feature representation through the sliding average unimodal language Transformer momentum encoder A similar sample cohort.

[0191] S35 constructs an alignment loss function between image pairs and text descriptions based on the changing feature representation of image pairs and similar sample queues, combined with the language feature representation and similar sample queues.

[0192] In this step, the second feature extraction module and the third feature extraction module first linearly project the change feature representation of the image pair and the corresponding positive sample and similar negative sample queues, and the language feature representation and the corresponding positive sample and similar negative sample queues to the same dimension to obtain the corresponding change feature update representation and language feature update representation.

[0193] The second feature extraction module uses the variation feature representation of the image pair The mean pooling feature of the corresponding change feature update representation (Y viF ):

[0194]

[0195] Among them, W v represents the weight matrix, b v represents the bias matrix.

[0196] Similarly, samples in the similar sample queue represented by the change feature of the image pair are passed through the second feature extraction module to obtain the corresponding change feature update representation, that is:

[0197] The third feature extraction module uses language feature representation (specifically using middle ) to obtain the corresponding language feature update representation (Y Ian ):

[0198]

[0199] Among them, W I represents the weight matrix, b I represents the bias matrix.

[0200] Similarly, samples in the similar sample queue represented by the language features of the text description are updated by obtaining the corresponding language features through the third feature extraction module, that is:

[0201] The cosine similarity between the updated representation of the change feature of the image pair and the updated representation of the language feature can also be calculated according to the following formula, sim(Y viF ,Y Ian ):

[0202]

[0203] Then, the cross-modal contrastive alignment of positive sample pairs is maximized through the InfoNCE loss:

[0204]

[0205] in, Represents the updated representation of the changing features of the queue of similar samples output by the cross-view Transformer momentum encoder, Represents the updated representation of the language features of the similar sample queue output by the unimodal language Transformer momentum encoder, K represents the number of batch samples, τ k represents the temperature parameter, Representing the updated representation of language features of similar sample queues Update the representation of the changes in the image pair The closest sample (i.e. positive sample), Represents the updated representation of the changing features of similar sample queues Update representation with language features The most similar sample (i.e., positive sample), sim(·) represents the cosine similarity function, and the above log function uses e as the base.

[0206] S4 constructs a cross entropy loss function for predicting change description;

[0207] The overall change description system is trained by minimizing the negative log-likelihood of the predicted word sequence and the true change description S. The accuracy loss function of the predicted change description uses the cross entropy loss function:

[0208]

[0209] Among them, p θrepresents the probability distribution, which is calculated by the probability distribution O of the previous target word (that is, the output result of the output module), n = 1, 2, ..., N, N represents the sentence length, and θ represents all parameters in the variation description system network.

[0210] S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross entropy loss function of the predicted change description;

[0211] In the above step S5, during the training phase, the total loss function is obtained by comparing the two learning objectives L |}| , L |‰| and cross entropy L _E Integration for collaborative learning:

[0212] L 总 =L_ E +λ1L |}| +λ k L |‰| ;

[0213] Among them, λ1 and λ k is a hyperparameter that adjusts the importance weight.

[0214] S6 optimizes the parameters of the change description system based on the total loss function.

[0215] In the above step S6, according to the total loss function, the parameters of the change description system (including a single-view Transformer encoder, a cross-view Transformer encoder, a single-modal language Transformer encoder, a multimodal Transformer decoder, a first feature extraction module, a second feature extraction module and a third feature extraction module) are optimized through a gradient descent optimization algorithm (such as an Adam optimization algorithm).

[0216] For momentum models (including single-view Transformer momentum encoder, cross-view Transformer encoder, and unimodal language Transformer momentum encoder), their parameters are updated in a sliding average manner. Instead of back-propagating the gradient down, the parameter update process can be expressed as:

[0217]

[0218] in, represents the momentum model parameters updated at the tth iteration, represents the momentum model parameters updated at the t-1th iteration, θ represents the change description system parameters updated at the tth iteration, and M represents the weight.

[0219] Repeat the above steps S2-S6 until the change description system converges.

[0220] When the number of training iterations reaches the upper limit of the number of iterations, the changes describe the convergence of the system.

[0221] Example 2

[0222] like Figure 5 As shown, this embodiment provides a change description method, which uses the change description system given above to perform the following steps:

[0223] Step 1: The two images and text descriptions in the image pair to be compared are converted into visual features and word embeddings of the corresponding images through the input module.

[0224] This step converts the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images through the input module.

[0225] Step 2: Extract the visual features of each image through the single-view Transformer encoder to obtain the invariant features of each image.

[0226] In this step, according to the explanation of the single-view Transformer encoder given above, the visual features of each image in the image pair are input into the single-view Transformer encoder to obtain the invariant features of each image in the image pair to be compared. and Represents the invariant features of image pairs.

[0227] Step 3: Match the invariant features of the two images in the image pair through the cross-view Transformer encoder and obtain the change feature representation of the image pair.

[0228] This step follows the explanation of the cross-view Transformer encoder given above. The output of the single-view Transformer encoder is the invariant features of each image in the image pair to be compared. and As input, the cross-view Transformer encoder is used to obtain the change feature representation of the two images in the image pair to be compared.

[0229] Step 4: Convert word embeddings into language feature representations through a unimodal language Transformer encoder.

[0230] According to the explanation of the unimodal language Transformer encoder given above, the word embedding of the text description obtained in step 1 is used as input and converted into a language feature representation through the unimodal language Transformer encoder

[0231] Step 5: The change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder are fused through the multimodal Transformer decoder to obtain a fused feature.

[0232] According to the explanation of the multimodal Transformer decoder given above, the change feature representation of the two images in the image pair to be compared is and language feature representation As input, the fusion features are obtained through the multimodal Transformer decoder

[0233] Step 6: Based on the fused features, the change description of the image pair is obtained through the output module.

[0234] This step uses the output module to fusion feature The linear transformation is performed to the same size as the given vocabulary, and then the probability distribution of the target word is calculated. The result of greedy decoding is used as the description of the change of the final image pair according to the probability distribution.

[0235] Application Examples

[0236] The present invention is evaluated on the publicly available CLEVR-Change dataset. During the application, five evaluation indicators are used to measure the performance of the method, namely: BLEU-4, METEOR, ROUGE, CIDEr and SPICE.

[0237] The CLEVR-Change dataset contains 79,606 pairs of images and 493,735 descriptions. The image pairs are taken from different angles, with certain deviations in viewing angles, and involve five types of object changes: "color", "texture", "addition", "removal" and "movement". At the same time, it also contains descriptions of images with only perspective changes and no object-level changes. The present invention uses the official division, including 67,660 training images (constituting the training set), 3,976 verification images (constituting the verification set), and 7,970 test images (constituting the test set).

[0238] First, the training set and the validation set are used to train and validate the change description system according to the training method provided in the previous embodiment 1, and then the test set is used to test the change description system according to the change description method provided in the previous embodiment 2. The specific indicators are shown in Table 1.

[0239] Table 1 Test results of the variation description system

[0240]

[0241] Note: Semantic level changes, not only the perspective changes but also the object level changes occur in the image.

[0242] It can be seen from Table 1 that the change description system provided by the present invention can generate effective change descriptions between different graphics.

[0243] On the above datasets, the results of (a) color change, (b) texture change, (c) object addition, (d) object removal, (e) object movement, and (f) only viewpoint change are described as follows: Figure 6 For each pair of images, the variation description generated by the present invention and the real description provided by the annotator are shown as follows: Figure 6 As shown in . The changed and unchanged visual elements in the description are highlighted in green and red, respectively. As shown in the figure, the present invention is able to filter out irrelevant background elements and locate key unchanged visual elements. At the same time, it has the ability to dynamically locate the changed area in the unchanged visual features, and successfully locate the changed objects without the need for bounding box information. By fusing the changed / non-changed visual features, the present invention can produce a detailed description of the concepts of changed and unchanged visual elements, which is very close to the real labels provided by humans.

[0244] Those skilled in the art will appreciate that the embodiments herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A change description system, characterized in that: include: An input module, used to convert two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images; The single-view Transformer encoder is used to extract the visual features of each image and obtain the invariant features of each image; A cross-view Transformer encoder that matches the invariant features of two images in an image pair and obtains a representation of the changing features of the image pair; A unimodal language Transformer encoder, which converts word embeddings into language feature representations; The multimodal Transformer decoder is used to fuse the change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder to obtain a fused feature; The output module is used to obtain the change description of the image pair based on the fused features.

2. The change description system according to claim 1, characterized in that: The input module uses a pre-trained visual backbone network to linearly project each image in the image pair into visual features; at the same time, each word in the text description is labeled to obtain word embedding.

3. The change description system according to claim 1, characterized in that: Single-view Transformer encoder T intra Contains N intra Layer, through the multi-head self-attention mechanism to fuse the local context information in the two images to be compared, B i+1 and A i+1 Represents the output of the i-th layer, and the expression is as follows: B i+1 =FFN(LN(B i +MHA(B i ,B i ,B i ))); A i+1 =FFN(LN(A i +MHA(A i ,A i ,A i ))); Among them, FFN represents feedforward neural network, LN represents layer normalization operation, and MHA represents multi-head attention operation; after N intra After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and 4. The change description system according to claim 3, characterized in that: Cross-view Transformer encoder T inter Contains N inter Layer, through the multi-head cross-attention mechanism, the invariant feature representation of cross-view is fused and interacted and The i′th layer in the cross-view encoder operates as follows: B i′+1 =FFN(LN(B i′ +MHA(B i′ ,A i′ ,A i′ ))); A i′+1 =FFn(LN(A i′ +MHA(A i′ ,B i′ ,B i′ ))); After N inter After layer cross-attention and feed-forward operations, the final inter-image visual representation is obtained: and Where N v =N intra +N inter ; Connect and linearly project the cross-view representations, expressed as: Among them, B0 and A0 represent the visual features of the two images in the image pair, [·] represents the cross-dimensional splicing operation, and W c and b c denote the weight and bias matrices of the linear projection, respectively. Represents the network features after cross-view fusion.

5. The change description system according to claim 1, characterized in that: Unimodal language Transformer encoder T uni Contains N uni Layer; each layer consists of two sub-layers: a masked self-attention sub-layer and a feed-forward sub-layer; the computational operation of the language encoder layer is as follows: in, represents the multi-head attention operation of the forward mask; the final language feature is represented as 6. The change description system according to claim 1, characterized in that: The multimodal Transformer decoder contains N mul layer, including masked self-attention, cross-attention and feed-forward operations; the i′′th layer is represented as: The final multimodal Transformer decoder output fusion feature is expressed as: Where N l =N uni +N mul .

7. The change description system according to claim 1, characterized in that: The output module calculates the probability distribution of the target word: Among them, W o and b o denote the weight and bias matrices respectively; Represents the fused features output by the multimodal Transformer decoder.

8. A training method for a change description system according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1 divides the training data into several training batches, each training sample includes an image pair before and after the change and a text description; for any training batch, the following steps S2-S6 are performed; S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrast learning; S3 obtains the variation feature representation of the training sample image pairs and the language feature representation of the text description, and constructs an alignment loss function between the image pairs and the text description based on cross-modal contrastive learning; S4 constructs a cross entropy loss function for predicting change description; S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross entropy loss function of the predicted change description; S6 optimizes the parameters of the change description system according to the total loss function; Repeat the above steps S2-S6 until the change description system converges.

9. The training method of the change description system according to claim 8, characterized in that: In step S1, the two images and text descriptions in the training sample image pair are also converted into visual features and word embeddings of the corresponding images through the input module.

10. The training method of the change description system according to claim 8, characterized in that: Step S2 includes the following sub-steps: S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder; S22 obtains a queue of similar samples of the invariant features of each image in the image pair through a single-view Transformer momentum encoder; S23 constructs an alignment loss function for image pairs based on the invariance features of the image pairs and similar sample queues.

11. The training method of the change description system according to claim 10, characterized in that: In step S23, firstly, the first feature extraction module extracts features of the invariant features of the image pair and the similar sample queue to obtain corresponding image internal feature representations; The first feature extraction module uses the invariant features of image pairs and The mean pooling features are linearly projected to obtain the internal feature representation X of the image before and after the change. bef and X aft : Among them, W X represents the weight matrix, b X represents the bias matrix, and Respectively and Elements in At the same time, the samples in the similar sample queue of the image before the change and the similar sample queue of the image after the change are extracted by the first feature extraction module to obtain the corresponding image internal feature representation, that is: and Then, the alignment between two pairs of matched images is maximized by using the InfoNCE loss, while minimizing the alignment of unmatched images: in, represents the internal feature representation of the image of the similar sample queue output by the single-view Transformer momentum encoder, K represents the number of batch samples, τ1 represents the temperature parameter, Q1 represents the number of samples in the similar sample queue, Representation of internal features of images representing similar sample queues after changes Image internal feature representation of the image before and after the change The closest sample, Representation of the internal features of the image before the change, representing the queue of similar samples Image internal feature representation of the transformed image The closest sample.

12. The training method of the change description system according to claim 10, characterized in that: Step S3 includes the following sub-steps: S31 matches the invariant features of the two images in the image pair obtained in step S21 through a cross-view Transformer encoder to obtain a change feature representation of the image pair; S32 matches similar samples of the invariant features of the two images in the image pair obtained in step S22 through a cross-view Transformer momentum encoder to obtain a queue of similar samples represented by the change features of the image pair; S33 converts the word embeddings corresponding to the text descriptions in the training samples into language feature representations through a unimodal language Transformer encoder; S34 obtains a similar sample queue represented by the language feature obtained in step S33 through a unimodal language Transformer momentum encoder; S35 constructs an alignment loss function between image pairs and text descriptions based on the changing feature representation of image pairs and similar sample queues, combined with the language feature representation and similar sample queues.

13. The training method of the change description system according to claim 12, characterized in that: In step S35, the second feature extraction module and the third feature extraction module are firstly used to linearly project the change feature representation and the similar sample queue of the image pair, and the language feature representation and the similar sample queue to the same dimension to obtain the corresponding change feature update representation and the language feature update representation; The second feature extraction module uses the mean pooling feature of the change feature representation of the image pair to obtain the corresponding change feature update representation: Among them, W v represents the weight matrix, b v represents the bias matrix; Similarly, samples in the similar sample queue represented by the change feature of the image pair are obtained through the second feature extraction module to obtain the corresponding change feature update representation, that is: The third feature extraction module uses the mean pooling feature of the language feature representation to obtain the corresponding language feature update representation: Among them, W l represents the weight matrix, b l represents the bias matrix; Similarly, samples in the similar sample queue represented by the language features of the text description are updated by obtaining corresponding language features through the third feature extraction module, that is: Subsequently, the cross-modal alignment between the matched variation features and the language features is maximized through the InfoNCE loss: in, Represents the updated representation of the changing features of the queue of similar samples output by the cross-view Transformer momentum encoder, represents the updated representation of the language features of the similar sample queue output by the unimodal language Transformer momentum encoder, K represents the number of batch samples, τ2 represents the temperature parameter, Representing the updated representation of language features of similar sample queues Update the representation of the changes in the image pair The closest sample, Represents the updated representation of the changing features of similar sample queues Update representation with language features The closest sample.

14. The training method of the change description system according to claim 8, characterized in that: In step S4, the cross entropy loss function of the predicted change description is expressed as: Among them, p θ represents the probability distribution, that is, the output result of the output module, n = 1, 2, ..., N, N represents the sentence length, and θ represents all parameters in the change description system network.

15. The training method of the change description system according to claim 14, characterized in that: In step S5, during the training phase, the total loss function is obtained by comparing the two learning objectives L CVC , L CMC and cross entropy L XE Integration for collaborative learning: L 总 =L XE +λ1L CVC +λ2L CMC ; Among them, λ1 and λ2 are hyperparameters for adjusting the importance weights; In step S6, according to the total loss function, the parameters of the change description system are optimized by a gradient descent optimization algorithm; For the momentum model single-view Transformer momentum encoder, cross-view Transformer encoder and unimodal language Transformer momentum encoder, their parameters are updated in a sliding average manner. The update process is expressed as: in, represents the momentum model parameters updated at the tth iteration, represents the momentum model parameters updated at the t-1th iteration, θ represents the change description system parameters updated at the tth iteration, and M represents the weight.

16. A change description method, using the change description system according to any one of claims 1 to 7, and performing the following steps: Step 1: The two images and text descriptions in the image pair to be compared are converted into visual features and word embeddings of the corresponding images through the input module; Step 2: extract the visual features of each image through a single-view Transformer encoder to obtain the invariant features of each image; Step 3, match the invariant features of the two images in the image pair through the cross-view Transformer encoder and obtain the change feature representation of the image pair; Step 4: Convert word embeddings into language feature representations through a unimodal language Transformer encoder; Step 5: The change feature representation of the image pair obtained by the cross-view Transformer encoder and the language feature representation obtained by the unimodal language Transformer encoder are fused through the multimodal Transformer decoder to obtain a fused feature; Step 6: Based on the fused features, the change description of the image pair is obtained through the output module.

Images