Change description system and method fusing cross-view cross-modal dual momentum contrast learning

Abstract

This invention, in the field of image processing technology, discloses a change description system and method integrating cross-view modality dual-momentum contrastive learning. The change description system includes an input module for converting two images and a text description in a pair of images to be compared into visual features and word embeddings for the corresponding images; a single-view Transformer encoder for extracting visual features from each image to obtain invariant features; a cross-view Transformer encoder for matching the invariant features of the two images in the pair and obtaining a change feature representation of the image pair; a single-modality language Transformer encoder for converting word embeddings into linguistic feature representations; a multimodal Transformer decoder for fusing the change feature representations and linguistic feature representations of the image pair to obtain fused features; and an output module for obtaining a change description of the image pair based on the fused features. This invention can capture key visual invariant features and enhance change description.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and relates to image change description technology, and in particular to a change description system and method that integrates cross-view modal dual momentum contrastive learning. Background Technology

[0002] Distinguishing and describing subtle differences between similar visual scenes is a fundamental human skill. In recent years, this unique skill has been simulated in deep learning tasks related to change description / image difference description. Change description tasks extend beyond traditional change detection tasks based on pure computer vision, attempting to simulate the complex interaction between the human visual and language systems, thus presenting greater challenges. Specifically, this task requires distinguishing and identifying subtle but important changes in a visual scene amidst numerous interfering variables, and also necessitates effectively utilizing natural language to express these changes. Therefore, a mature change description system relies on a deep understanding of scene changes and the establishment of a close connection between vision and language. This task has broad practical application prospects in fields ranging from industrial anomaly detection and industrial infrastructure monitoring to medical diagnostics.

[0003] With the rapid development of deep learning in recent years, a large number of mature methods have emerged in the field of change description tasks. Most of these methods follow the traditional change detection framework, primarily identifying semantic changes in images from the visual perspective. In these methods, invariant features between two images are generally considered potential interfering factors. Therefore, these methods often employ techniques such as subtracting the "before change" image from the "after change" image, or filtering out invariant features from the changed features, thereby decoupling this potential interfering factor. However, human annotators often capture salient features between two images that not only highlight the change but also include invariant features, providing valuable contextual information.

[0004] Based on this, a more human-friendly method for describing change should expand its capabilities from simply describing variability features to include key invariant elements in an image; this includes: 1) salient background elements that provide contextual information between two images; and 2) invariant features relevant to understanding variability targets, especially in cases where images are not aligned and are taken from different angles.

[0005] Therefore, insightful descriptions of change must detect and consider both changing and key invariant features. In these cases, recognizing these consistent visual cues and effectively associating them with linguistic elements is crucial for generating informative descriptions of change. Summary of the Invention

[0006] In view of the above-mentioned shortcomings in the prior art, the present invention aims to provide a change description system that can capture key visual invariance features and enhance change description.

[0007] To achieve the above objectives, the present invention adopts the following technical solutions.

[0008] This invention provides a change description system, comprising:

[0009] The input module is used to convert the two images and text descriptions in the image pair to be compared into the visual features and word embeddings of the corresponding images;

[0010] A single-view Transformer encoder is used to extract visual features from each image separately and obtain the invariant features of each image.

[0011] A cross-view Transformer encoder is used to match the invariant features of two images in an image pair and obtain a representation of the changes in the image pair.

[0012] A single-modal language Transformer encoder is used to convert word embeddings into linguistic feature representations;

[0013] A multimodal Transformer decoder is used to fuse the variation feature representations of image pairs obtained from a cross-view Transformer encoder and the linguistic feature representations obtained from a single-modal language Transformer encoder to obtain fused features;

[0014] The output module is used to obtain a description of the changes in the image pair based on the fused features.

[0015] The input module described above uses a shared pre-trained visual backbone network to linearly project each image in the image pair into several grid-like visual features. Specifically, the visual backbone network can be ResNet, Vision Transformer, etc. The input module also tags each word in the text description to obtain word embeddings.

[0016] The aforementioned single-view Transformer encoder is used for information exchange within a single image and to obtain invariant features within each individual image. The single-view Transformer encoder T provided by this invention... intra The visual features of each image are used as input. The single-view Transformer encoder T... intra Including N intra Layer, consisting of N intra It consists of several Transformer blocks, which fuse the visual features of the blocks (i.e., local contextual information within the two images B and A to be compared) through a multi-head self-attention mechanism. For example: Bi+1 and A i+1 The output of the i-th Transformer block (i.e., the output of the i-th layer) is expressed as follows:

[0017] B i+1 =FFN(LN(B i +MHA(B i B i B i )));

[0018] A i+1 =FFN(LN(A i +MHA(A i A i A i )));

[0019] Here, FFN represents a feedforward neural network, LN represents layer normalization, and MHA represents multi-head attention. After N... intra After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and

[0020] The aforementioned cross-view Transformer encoder is responsible for matching invariant features in two images and inferring the representation of changes in the two images. The cross-view Transformer encoder T introduced in this invention... inter Using the output of a single-view Transformer encoder as input, this cross-view Transformer encoder T... inter Including N inter Layer, consisting of N inter Composed of several Transformer blocks, it integrates and interacts with cross-view invariant feature representations through a multi-head cross-attention mechanism. and The operation of the i′-th Transformer block (i.e., the i′-th layer) in the cross-view encoder is as follows:

[0021] B i′+1 =FFN(LN(B i′ +MHA(B i′ A i′ A i′ )));

[0022] A i′+1 =FFN(LN(A i′ +MHA(A i′ B i′ B i′ )));

[0023] After N inter After layer cross-attention and feedforward operations, the final visual representation between images is obtained: and Where N v =N intra +N inter In this process, key invariant visual features were obtained. Since information-rich descriptions of change typically contain both changing features and invariant concepts, the operation continues by adding the learned invariant representation to the original visual representation. Then, the cross-view representations are concatenated and linearly projected to the same dimension as the word embeddings, represented as:

[0024]

[0025] Where B0 and A0 represent the visual features of the two images in the image pair, [·] represents the cross-dimensional stitching operation, and W c and b c These represent the weight and bias matrices of the linear projection, respectively. This represents the grid-level features fused across different viewpoints. This step generates a visual representation that incorporates both variable and key invariant features.

[0026] The aforementioned unimodal language Transformer encoder aims to align language representations with matching visual features. To encode the contextual information describing the text, the unimodal language Transformer encoder T in this invention... uni The description is transformed into a high-level language feature representation, which contains N uni Layers, using N uni The layers are composed of stacked masked Transformer blocks; each layer consists of two sub-layers: a masked self-attention sub-layer and a feedforward sub-layer. The computational operations of the language encoder layer are as follows:

[0027]

[0028] in, This represents a multi-head attention operation for forward masking, which allows each word in the encoder to focus only on all the preceding words in the sentence, while preventing information leakage from future words. The final linguistic features are represented as follows: Contextual information encompassing the entire sentence is used as sentence features.

[0029] The aforementioned multimodal Transformer decoder facilitates the interaction between aligned visual and linguistic features, thereby generating a description of changes through the output module. After learning... and Then, through a multimodal Transformer decoder T mul The decoder contains N features that integrate and interact across modalities. mul Layers, using N mul The layers are composed of stacked Transformer blocks, containing masked self-attention, cross-attention, and feedforward operations. The i″′ layer can be represented as:

[0030]

[0031] The final output can be represented as: Where N I =N uni +N mul .

[0032] The output module described above linearly transforms the fused features to the same size as the given vocabulary and calculates the probability distribution of the target words:

[0033]

[0034] Among them, W E and b E These represent the weight and bias matrices, respectively.

[0035] Finally, based on the probability distribution of all words, the description with the higher probability is searched. Specifically, greedy decoding is used, taking the word with the highest probability at each step as the output, to obtain the final description of the changes in the image pair.

[0036] This invention also provides a training method for the aforementioned change description system, which integrates cross-view modal dual momentum contrastive learning, including the following steps:

[0037] S1 divides the training data into several training batches. Each training sample includes image pairs before and after the change, as well as text descriptions. For any training batch, the following steps S2-S6 are performed.

[0038] S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrastive learning;

[0039] S3 acquires the variation feature representation of training sample image pairs and the linguistic feature representation of text descriptions, and constructs an alignment loss function between image pairs and text descriptions based on cross-modal contrastive learning;

[0040] S4 constructs the cross-entropy loss function to predict changes;

[0041] S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross-entropy loss function for predicting the change description;

[0042] S6 optimizes the parameters of the change description system based on the total loss function;

[0043] Repeat steps S2-S6 above until the change description system converges.

[0044] In step S1 above, the input module also converts the two images and text descriptions in the training sample image pair into the visual features and word embeddings of the corresponding images.

[0045] In step S2 above, cross-view contrastive learning is introduced to obtain a representation that remains unchanged by aligning the internal features of the cross-view "before change" and "after change" images.

[0046] Step S2 includes the following sub-steps:

[0047] S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder;

[0048] S22 obtains a queue of similar samples of invariant features for each image in an image pair through a single-view Transformer momentum encoder.

[0049] S23 constructs an alignment loss function for image pairs based on the invariant characteristics of image pairs and the queue of similar samples.

[0050] In step S21 above, following the explanation of the single-view Transformer encoder given earlier, the visual features of each image in the image pair are input into the single-view Transformer encoder to obtain the invariant features of each image in the training sample image pair. Here, we take... and This represents the invariant features of image pairs.

[0051] In step S22 above, the single-view Transformer momentum encoder has the same structure as the single-view Transformer encoder. First, a moving average single-view Transformer momentum encoder is used to generate queues of the most recent Q1 “before change” and “after change” samples, respectively.

[0052] The visual features of the image before the image centering change are input into the single-view Transformer momentum encoder to obtain a queue of Q1 similar samples of the image before the image centering change; similarly, the visual features of the image after the image centering change are input into the single-view Transformer momentum encoder to obtain a queue of Q1 similar samples of the image after the image centering change.

[0053] In step S23 above, the invariant features of the image pair and the similar sample queue are first extracted by the first feature extraction module to obtain the corresponding image internal feature representation.

[0054] The first feature extraction module uses invariant features of image pairs ( and The mean pooling features of the image are linearly projected to obtain the image's internal feature representation X before and after the change. bef

[0055] and X aft :

[0056]

[0057] Among them, W x Let b represent the weight matrix. x Represents the bias matrix. and They represent and The elements in.

[0058] The image I before and after the change can also be calculated using the following formula. bef and I aft Cross-view feature cosine similarity:

[0059]

[0060] Simultaneously, the samples from the similar sample queues of the image before and after the change are processed by the first feature extraction module to obtain the corresponding internal image feature representations, namely: and

[0061] Then, the alignment between the two pairs of matched images is maximized using InfoNCE loss, while the alignment of the unmatched images is minimized; for the l-th “before” and “after” images in the training batch:

[0062]

[0063] in, This represents the image internal feature representation of a similar sample queue output by a single-view Transformer momentum encoder, where K represents the batch size, τ1 represents the temperature parameter, and Q1 represents the number of samples in the similar sample queue. Image internal feature representation of a queue of similar samples after image transformation Image internal feature representation of the image before and after the change The closest sample (i.e., the positive sample), Image internal feature representation of a cohort of similar samples before the change Image internal feature representation of the transformed image The most similar sample (i.e., the positive sample), sim(·) represents the cosine similarity function. L CVC The goal is to learn invariant feature representations across viewpoints to align cross-view images, thereby enabling better cross-view feature fusion interactions and filtering out variable features.

[0064] In step S3 above, in order to better integrate and interact with visual and linguistic representations, a cross-modal contrastive learning objective is introduced.

[0065] Step S3 includes the following sub-steps:

[0066] S31 obtains the change feature representation of the image pair by matching the invariant features of the two images in the image pair obtained in step S21 through the cross-view Transformer encoder;

[0067] S32 uses the cross-view Transformer momentum encoder to match similar samples of invariant features between two images in the image pair obtained in step S22, and obtains a queue of similar samples representing the change features of the image pair.

[0068] S33 uses a single-modal language Transformer encoder to convert the word embeddings corresponding to the text descriptions in the training samples into language feature representations;

[0069] S34 obtains a queue of similar samples with language feature representations obtained in step S33 through a single-modal language Transformer momentum encoder;

[0070] S35 constructs an alignment loss function between image pairs and text descriptions based on the variation feature representation of image pairs and the similar sample queue, combined with the language feature representation and the similar sample queue.

[0071] In step S31 above, according to the explanation of the cross-view Transformer encoder given above, the invariance features of the images before and after the change in the output of the single-view Transformer encoder are used as input to obtain the change feature representations of the two images in the training sample image pair.

[0072] In step S32 above, the cross-view Transformer momentum encoder has the same structure as the cross-view Transformer encoder.

[0073] Using the similar samples of the image before and after the change, which are the final outputs of the single-view Transformer momentum encoder in step S22, as input, Q is obtained through a moving average cross-view Transformer momentum encoder. k The variation features of a pair of images represent a queue of similar samples.

[0074] In step S33 above, as explained above, the word embeddings of the text description obtained in step S1 are used as input and converted into language feature representations by a single-modal language Transformer encoder.

[0075] In step S34 above, the single-modal language Transformer momentum encoder has the same structure as the single-modal language Transformer encoder.

[0076] Using the word embeddings of the text description obtained in step S1 as input, Q is obtained through a sliding average single-modal language Transformer momentum encoder. k Individual and linguistic feature representation A queue of similar samples.

[0077] In step S35 above, the image pair's change feature representation and similar sample queue are first linearly projected onto the same dimension by the second feature extraction module and the third feature extraction module, along with the language feature representation and similar sample queue, to obtain the corresponding change feature update representation and language feature update representation.

[0078] The second feature extraction module uses the change features of image pairs to represent... The mean pooling feature is used to obtain the corresponding change feature update representation (Y). viF ):

[0079]

[0080] Among them, W v Let b represent the weight matrix. v This represents the bias matrix.

[0081] Similarly, the samples in the similar sample queue representing the change features of the image pair are updated with corresponding change features by the second feature extraction module, which is:

[0082] The third feature extraction module uses language feature representation (specifically, it uses...). middle The mean pooling features of ) are used to obtain the corresponding language feature update representation (Y). Ian ):

[0083]

[0084] Among them, W I Let b represent the weight matrix. I This represents the bias matrix.

[0085] Similarly, the samples in the similar sample queue representing the language features of the text description are updated with corresponding language features through the third feature extraction module, which is:

[0086] The cosine similarity between the updated representations of the changed features and the updated representations of the language features of an image pair can also be calculated using the following formula: sim(Y vis ,Y lan ):

[0087]

[0088] Subsequently, the InfoNCE loss is used to maximize the cross-modal alignment between matched variable features and linguistic features (i.e., positive sample pairs), while minimizing the alignment between unmatched variable features and linguistic features.

[0089]

[0090] in, This represents the feature update representation of the changing features of a similar sample queue output by the cross-view Transformer momentum encoder. This represents the language feature update representation of a similar sample queue output by a single-modal language Transformer momentum encoder, where K represents the batch size and τ is the number of samples in the batch. k Indicates temperature parameter, Language feature update representation of similar sample queues Image feature update representation The closest sample (i.e., the positive sample), The feature update representation of changes in similar sample queues Chinese and language feature update representation The most similar sample (i.e., the positive sample), sim(·) represents the cosine similarity function.

[0091] In step S4 above, the overall change description system is trained by minimizing the negative log-likelihood between the predicted word sequence and the true change description S, and the cross-entropy loss function for predicting the change description is:

[0092]

[0093] Where, p θθ represents the probability distribution, calculated from the probability distribution O of the target word (i.e., the output of the output module), and θ represents the variation of all parameters in the system network.

[0094] In step S5 above, during the training phase, the total loss function is learned through two contrastive learning objectives L. CVC L |CMC and cross-entropy L XE Integration for collaborative learning:

[0095] L 总 =L XE +λ1L CVC +λ2L CMC ;

[0096] Here, λ1 and λ2 are hyperparameters that adjust the importance weights.

[0097] In step S6 above, based on the total loss function, the parameters of the change description system (including single-view Transformer encoder, cross-view Transformer encoder, single-modal language Transformer encoder, multimodal Transformer decoder, first feature extraction module, second feature extraction module and third feature extraction module) are optimized by gradient descent optimization algorithm (e.g. Adam optimization algorithm).

[0098] For momentum models (including single-view Transformer momentum encoders, cross-view Transformer encoders, and single-modal language Transformer momentum encoders), their parameters are updated using a moving average method. Without using backpropagation gradient descent, the parameter update process can be expressed as:

[0099]

[0100] in, Let represent the momentum model parameters obtained in the t-th iteration update. Let θ represent the momentum model parameters obtained in the (t-1)th iteration update, θ represent the change description system parameters obtained in the tth iteration update, and M represent the weights.

[0101] Whether the change description coefficients have converged can be determined by the trend of the total loss function (e.g., tending towards a set threshold), reaching a minimum, or reaching the upper limit of the number of iterations. When the above requirements are met, the change description system has converged.

[0102] The present invention also provides a change description method, which uses the change description system given above and follows these steps:

[0103] Step 1: The input module converts the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images.

[0104] Step 2: Extract visual features from each image using a single-view Transformer encoder to obtain the invariant features of each image.

[0105] Step 3: Match the invariant features of the two images in the image pair using a cross-view Transformer encoder and obtain the change feature representation of the image pair;

[0106] Step 4: Convert word embeddings into linguistic feature representations using a single-modal language Transformer encoder;

[0107] Step 5: The multimodal Transformer decoder fuses the change feature representations of the image pairs obtained from the cross-view Transformer encoder and the language feature representations obtained from the single-modal language Transformer encoder to obtain fused features;

[0108] Step 6: Based on the fusion features, obtain the change description of the image pair through the output module.

[0109] Compared with existing technologies, the change description system and method for integrating cross-view modal dual momentum contrastive learning provided by this invention have the following beneficial effects:

[0110] (1) The change description system of the present invention can match the invariant features in two images and infer the change representation of the two images by providing a cross-view Transformer encoder;

[0111] (2) The change description system of the present invention can facilitate the interaction between the change feature representation and the language feature representation of aligned image pairs through the provided multimodal Transformer decoder, thereby generating change descriptions;

[0112] (3) In the training of the change description system, the present invention introduces a dual momentum contrastive learning; the introduced cross-view contrastive learning objective helps to align cross-view visual features and effectively model the invariant representation between cross-view images to generate information-rich descriptions; the introduced cross-modal contrastive learning objective helps to use language to guide the learning of variable visual features and invariant visual features, and generate strongly correlated visual and linguistic cross-modal features. Attached Figure Description

[0113] Figure 1 The system structure and principle block diagram for the variation description of cross-view modal dual momentum contrastive learning;

[0114] Figure 2 A training method for a change description system that integrates cross-view modal dual momentum contrastive learning;

[0115] Figure 3 This is a schematic diagram of a cross-perspective comparative learning process;

[0116] Figure 4 This is a schematic diagram of the cross-modal contrastive learning process;

[0117] Figure 5 A flowchart illustrating the method for describing changes;

[0118] Figure 6 The image pairs are described under different change types; (a) corresponds to color change, (b) corresponds to texture change, (c) corresponds to object addition, (d) corresponds to object removal, (e) corresponds to object movement, and (f) corresponds to only viewpoint change; in the figure, "Before" and "After" represent the viewpoint before and after the change, respectively, and "Unchange" and "Change" represent the background object that has not changed and the subject that has changed, respectively. Detailed Implementation

[0119] The technical solutions of various embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0120] Example 1

[0121] This embodiment provides a change description system, such as Figure 1 As shown, it includes an input module, a single-view Transformer encoder, a cross-view Transformer encoder, a single-modal language Transformer encoder, a multimodal Transformer decoder, and an output module.

[0122] 1. Input module

[0123] The input module is used to convert the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images.

[0124] This change description system processes input in the form of image pairs and text descriptions, represented as (I bef ,I aft ,S), where I bef and I aft, represent the images before and after the transformation, respectively, and S represents their corresponding descriptive sentences. The input module uses a shared pre-trained visual backbone network to linearly project each image in the image pair into visual features. Specifically, the visual backbone network can be ResNet-101; these can be represented as... and Where P is the number of grid cells. This represents the grid features extracted from the pre-trained grid. The input module also tags each word in the text description to obtain word embeddings. For the input sentence S, special bos and eos stems are inserted at the beginning and end as the sentence's start and end symbols. Subsequently, it is segmented and converted into word embeddings, represented as... Where N represents the length of the sentence. Word embeddings are represented. Furthermore, to explicitly characterize the positions of grids and words in each modality, positional encoding is added to B0, A0, and C0.

[0125] 2. Single-view Transformer encoder

[0126] A single-view Transformer encoder is used to extract visual features from each image separately, obtaining invariant features for each image.

[0127] The single-view Transformer encoder T provided in this embodiment intra It takes the visual features of each image as input. Single-view Transformer encoder T intra Including N intra Layer, consisting of N intra It consists of several Transformer blocks, which fuse the visual features of the blocks (i.e., local contextual information within the two images B and A to be compared) through a multi-head self-attention mechanism. For example: B i+1 and A i+1 The output of the i-th Transformer block (i.e., the output of the i-th layer) is expressed as follows:

[0128] B i+1 =FFN(LN(B i +MHA(B i B i B i )));

[0129] A i+1 =FFN(LN(A i +MHA(A i A i A i )));

[0130] Here, FFN represents a feedforward neural network, LN represents layer normalization, and MHA represents multi-head attention. After N... intra After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and

[0131] 3. Cross-view Transformer encoder

[0132] A cross-view Transformer encoder is used to match the invariant features of two images in an image pair and obtain a representation of the changes in the image pair.

[0133] This embodiment introduces a cross-view Transformer encoder T. inter Using the output of a single-view Transformer encoder as input, this cross-view Transformer encoder T... inter Including N inter Layer, consisting of N inter Composed of several Transformer blocks, it integrates and interacts with cross-view invariant feature representations through a multi-head cross-attention mechanism. and The operation of the i′-th Transformer block (i.e., the i′-th layer) in the cross-view encoder is as follows:

[0134] B i′+1 =FFN(LN(B i′ +MHA(B i′ A i′ A i′ )));

[0135] A i′+1 =FFN(LN(A i′ +MHA(A i′ B i′ B i′ )));

[0136] After N inter After layer cross-attention and feedforward operations, the final visual representation between images is obtained: and Where N v =N intra +N inter In this process, key invariant visual features were obtained. Since information-rich descriptions of change typically contain both changing features and invariant concepts, the operation continues by adding the learned invariant representation to the original visual representation. Then, the cross-view representations are concatenated and linearly projected to the same dimension as the word embeddings, represented as:

[0137]

[0138] Where B0 and A0 represent the visual features of the two images in the image pair, [·] represents the cross-dimensional stitching operation, and W c and b c These represent the weight and bias matrices of the linear projection, respectively. This represents the grid-level features fused across different viewpoints. This step generates a visual representation that incorporates both variable and key invariant features.

[0139] 4. Single-modal language Transformer encoder

[0140] A single-modal language Transformer encoder is used to convert word embeddings into language feature representations.

[0141] In order to encode the contextual information of the descriptive text, in this embodiment, the single-modal language Transformer encoder T... uni The description is transformed into a high-level language feature representation, which contains N uni Layers, using N uni The layers are composed of stacked masked Transformer blocks; each layer consists of two sub-layers: a masked self-attention sub-layer and a feedforward sub-layer. The computational operations of the language encoder layer are as follows:

[0142]

[0143] in, This represents a multi-head attention operation for forward masking, which allows each word in the encoder to focus only on all the preceding words in the sentence, while preventing information leakage from future words. The final linguistic features are represented as follows: Contextual information encompassing the entire sentence is used as sentence features.

[0144] 5. Multimodal Transformer Decoder

[0145] The multimodal Transformer decoder is used to fuse the variation feature representations of image pairs obtained from the cross-view Transformer encoder and the language feature representations obtained from the single-modal language Transformer encoder to obtain fused features.

[0146] A multimodal Transformer decoder facilitates the interaction between aligned visual and linguistic features, thereby generating a description of changes through the output module. After learning... and Then, through a multimodal Transformer decoder Tmul The decoder contains N features that integrate and interact across modalities. mul Layers, using N mul The layers are composed of stacked Transformer blocks, containing masked self-attention, cross-attention, and feedforward operations. The i″′ layer can be represented as:

[0147]

[0148] The final output can be represented as: Where N I =N uni +N mul .

[0149] 6. Output Module

[0150] The output module is used to obtain a description of the changes in the image pair based on the fused features.

[0151] The output module described above linearly transforms the fused features to the same size as the given vocabulary and calculates the probability distribution of the target words:

[0152]

[0153] Among them, W E and b E These represent the weight and bias matrices, respectively.

[0154] Finally, based on the probability distribution of all words, the description with the higher probability is searched. Specifically, greedy decoding is used, taking the word with the highest probability at each step as the output, to obtain the final description of the changes in the image pair.

[0155] This embodiment also provides a training method for the above-mentioned change description system, such as... Figure 2 As shown, the fusion of cross-view modal dual momentum contrastive learning includes the following steps:

[0156] S1 divides the training data into several training batches, and each training sample includes image pairs before and after the change, as well as text descriptions.

[0157] This step also uses the input module to convert the two images and text descriptions in the training sample image pair into the visual features and word embeddings of the corresponding images.

[0158] The change description system is then trained based on the training samples in each training batch.

[0159] For any training batch, perform the operation according to the following steps S2-S6.

[0160] S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrastive learning.

[0161] This step introduces cross-view contrastive learning, which obtains a change-invariant representation by aligning the internal features of the cross-view "before change" and "after change" images.

[0162] like Figure 3 As shown, this step includes the following sub-steps:

[0163] S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder.

[0164] Following the explanation of the single-view Transformer encoder given earlier, this step inputs the visual features of each image in the image pair into the single-view Transformer encoder to obtain the invariant features of each image in the training sample image pair. Here, we take... and This represents the invariant features of image pairs.

[0165] S22 obtains a queue of similar samples of invariant features for each image in an image pair through a single-view Transformer momentum encoder.

[0166] The single-view Transformer momentum encoder has the same structure as the single-view Transformer encoder. First, a moving average single-view Transformer momentum encoder is used to maintain queues that store the most recent Q1 "before" and "after" samples, respectively.

[0167] The visual features of the image before the image centering change are input into the single-view Transformer momentum encoder to obtain Q1 similar sample queues to the image before the image centering change; similarly, the visual features of the image after the image centering change are input into the single-view Transformer momentum encoder to obtain Q1 similar sample queues to the image after the image centering change.

[0168] S23 constructs an alignment loss function for image pairs based on the invariance characteristics of image pairs and a queue of similar negative samples.

[0169] This step first uses the first feature extraction module to extract the invariant features of the image pair, as well as the positive sample and similar negative sample queues, to obtain the corresponding internal feature representation of the image.

[0170] The first feature extraction module uses invariant features of image pairs ( and The mean pooling features of the image are linearly projected to obtain the image's internal feature representation X before and after the change.bef and X aft :

[0171]

[0172] Among them, W _ Let b represent the weight matrix. _ Represents the bias matrix. and They represent and The elements in.

[0173] The image I before and after the change can also be calculated using the following formula. bef and I aft Cross-view feature cosine similarity:

[0174]

[0175] Simultaneously, the samples from the similar sample queues of the image before and after the change are processed by the first feature extraction module to obtain the corresponding internal image feature representations, namely: and

[0176] Then, the alignment between the two pairs of matched images is maximized using InfoNCE loss, while the alignment of the unmatched images is minimized; for the l-th “before” and “after” images in the training batch:

[0177]

[0178] in, This represents the image internal feature representation of a similar sample queue output by a single-view Transformer momentum encoder, where K represents the batch size, τ1 represents the temperature parameter, and Q1 represents the number of samples in the similar sample queue. Image internal feature representation of a queue of similar samples after image transformation Image internal feature representation of the image before and after the change The closest sample (i.e., the positive sample), Image internal feature representation of a cohort of similar samples before the change Image internal feature representation of the transformed image The most similar sample (i.e., the positive sample), sim(·) represents the cosine similarity function, and the above log function is based on e. L CVC The goal is to learn invariant feature representations across viewpoints to align cross-view images, thereby enabling better cross-view feature fusion interactions and filtering out variable features.

[0179] S3 acquires the variation feature representations of training sample image pairs and the linguistic feature representations of text descriptions, and constructs an alignment loss function between image pairs and text descriptions based on cross-modal contrastive learning.

[0180] To better integrate and interact with visual and linguistic representations, this step introduces a cross-modal contrastive learning objective.

[0181] like Figure 4 As shown, this step includes the following sub-steps:

[0182] S31 obtains the change feature representation of the image pair by matching the invariant features of the two images in the image pair obtained in step S21 through the cross-view Transformer encoder.

[0183] Following the explanation of the cross-view Transformer encoder given earlier, this step uses the output of the single-view Transformer encoder as input to obtain the variation feature representation of the two images in the training sample image pair.

[0184] S32 uses a cross-view Transformer momentum encoder to match similar samples of invariant features between two images in the image pair obtained in step S22, thus obtaining a queue of similar samples representing the change features of the image pair.

[0185] In this step, the cross-view Transformer momentum encoder has the same structure as the cross-view Transformer encoder.

[0186] Using the similar samples of the image before and after the change, which are the final outputs of the single-view Transformer momentum encoder in step S22, as input, Q is obtained through a moving average cross-view Transformer momentum encoder. k The variation features of a pair of images represent a queue of similar samples.

[0187] S33 uses a single-modal language Transformer encoder to convert the word embeddings corresponding to the text descriptions in the training samples into language feature representations.

[0188] As explained above, the word embeddings of the text description obtained in step S1 are used as input and converted into language feature representations by a single-modal language Transformer encoder.

[0189] S34 obtains a queue of similar samples whose language feature representations were obtained in step S33 by using a single-modal language Transformer momentum encoder.

[0190] In this step, the single-modal language Transformer momentum encoder has the same structure as the single-modal language Transformer encoder.

[0191] Using the word embeddings of the text description obtained in step S1 as input, the text is converted into language feature representations through a sliding average single-modal language Transformer momentum encoder. A queue of similar samples.

[0192] S35 constructs an alignment loss function between image pairs and text descriptions based on the variation feature representation of image pairs and the similar sample queue, combined with the language feature representation and the similar sample queue.

[0193] This step first uses the second feature extraction module and the third feature extraction module to linearly project the image pair's change feature representation and the corresponding positive sample and similar negative sample queues onto the same dimension as the language feature representation and the corresponding positive sample and similar negative sample queues, so as to obtain the corresponding change feature update representation and language feature update representation.

[0194] The second feature extraction module uses the change features of image pairs to represent... The mean pooling feature is used to obtain the corresponding change feature update representation (Y). viF ):

[0195]

[0196] Among them, W v Let b represent the weight matrix. v This represents the bias matrix.

[0197] Similarly, the samples in the similar sample queue representing the change features of the image pair are updated with corresponding change features through the second feature extraction module, which is:

[0198] The third feature extraction module uses language feature representation (specifically, it uses...). middle The mean pooling features of ) are used to obtain the corresponding language feature update representation (Y). Ian ):

[0199]

[0200] Among them, W I Let b represent the weight matrix. I This represents the bias matrix.

[0201] Similarly, the samples in the similar sample queue representing the language features of the text description are updated with corresponding language features through the third feature extraction module, which is:

[0202] The cosine similarity between the updated representations of the changed features and the updated representations of the language features of an image pair can also be calculated using the following formula: sim(Y viF ,Y Ian ):

[0203]

[0204] Subsequently, cross-modal contrastive alignment of positive sample pairs is maximized using InfoNCE loss:

[0205]

[0206] in, This represents the feature update representation of the changing features of a similar sample queue output by the cross-view Transformer momentum encoder. This represents the language feature update representation of a similar sample queue output by a single-modal language Transformer momentum encoder, where K represents the batch size and τ is the number of samples in the batch. k Indicates temperature parameter, Language feature update representation of similar sample queues Image feature update representation The closest sample (i.e., the positive sample), The feature update representation of changes in similar sample queues Chinese and language feature update representation The most similar sample (i.e., the positive sample), sim(·) represents the cosine similarity function, and the above log function is based on e.

[0207] S4 constructs the cross-entropy loss function to predict changes;

[0208] The overall change description system is trained by minimizing the negative log-likelihood between the predicted word sequence and the true change description S. The accuracy loss function for predicting the change description uses the cross-entropy loss function.

[0209]

[0210] Where, p θ The probability distribution is given by the probability distribution O of the target word (i.e., the output of the output module), n = 1, 2, ..., N, where N represents the sentence length, and θ represents all parameters in the variation description system network.

[0211] S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross-entropy loss function for predicting the change description;

[0212] In step S5 above, during the training phase, the total loss function is learned through two contrastive learning objectives L. CVC L CMC and cross-entropy L XE Integration for collaborative learning:

[0213] L 总 =L XE +λ1L CVC +λ2L CMC ;

[0214] Here, λ1 and λ2 are hyperparameters that adjust the importance weights.

[0215] S6 optimizes the parameters of the change description system based on the total loss function.

[0216] In step S6 above, based on the total loss function, the parameters of the change description system (including single-view Transformer encoder, cross-view Transformer encoder, single-modal language Transformer encoder, multimodal Transformer decoder, first feature extraction module, second feature extraction module and third feature extraction module) are optimized by gradient descent optimization algorithm (e.g. Adam optimization algorithm).

[0217] For momentum models (including single-view Transformer momentum encoders, cross-view Transformer encoders, and single-modal language Transformer momentum encoders), their parameters are updated using a moving average method. Without using backpropagation gradient descent, the parameter update process can be expressed as:

[0218]

[0219] in, Let represent the momentum model parameters obtained in the t-th iteration update. Let θ represent the momentum model parameters obtained in the (t-1)th iteration update, θ represent the change description system parameters obtained in the tth iteration update, and M represent the weights.

[0220] Repeat steps S2-S6 above until the change description system converges.

[0221] The change description system converges when the number of training iterations reaches the upper limit.

[0222] Example 2

[0223] like Figure 5 As shown, this embodiment provides a change description method, which uses the change description system given above and follows these steps:

[0224] Step 1: The input module converts the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images.

[0225] This step uses the input module to convert the two images and text descriptions in the image pair to be compared into the visual features and word embeddings of the corresponding images.

[0226] Step 2: Extract visual features from each image using a single-view Transformer encoder to obtain the invariant features of each image.

[0227] Following the explanation of the single-view Transformer encoder given earlier, this step inputs the visual features of each image in the image pair into the single-view Transformer encoder to obtain the invariant features of each image in the image pair to be compared. Here, we take... and This represents the invariant features of image pairs.

[0228] Step 3: Match the invariant features of the two images in the image pair using a cross-view Transformer encoder and obtain the change feature representation of the image pair.

[0229] Following the explanation of the cross-view Transformer encoder given earlier, this step involves the single-view Transformer encoder outputting the invariant features of each image in the image pair to be compared. and As input, the cross-view Transformer encoder obtains the change feature representations of the two images in the image pair to be compared.

[0230] Step 4: Convert word embeddings into language feature representations using a single-modal language Transformer encoder.

[0231] Based on the explanation of the unimodal language Transformer encoder given above, the word embeddings of the text description obtained in step 1 are used as input and converted into language feature representations by the unimodal language Transformer encoder.

[0232] Step 5: The multimodal Transformer decoder fuses the variation feature representation of the image pair obtained from the cross-view Transformer encoder and the language feature representation obtained from the single-modal language Transformer encoder to obtain fused features.

[0233] Based on the explanation of the multimodal Transformer decoder given above, the change feature representation of the two images in the image pair to be compared is... and language feature representation As input, fused features are obtained through a multimodal Transformer decoder.

[0234] Step 6: Based on the fusion features, obtain the change description of the image pair through the output module.

[0235] This step uses the output module to fuse the features. The algorithm is linearly transformed to the same size as the given vocabulary, and then the probability distribution of the target words is calculated. The result of greedy decoding based on the probability distribution is used as the description of the changes in the final image pair.

[0236] Application examples

[0237] This invention was evaluated on the publicly available CLEVR-Change dataset. During the application, five evaluation metrics were used to measure the performance of the method: BLEU-4, METEOR, ROUGE, CIDEr, and SPICE.

[0238] The CLEVR-Change dataset contains 79,606 image pairs and 493,735 descriptions. The image pairs were taken from different angles, resulting in some viewpoint deviation, and involve five types of object changes: "color," "texture," "addition," "removal," and "movement." It also includes descriptions of images where only the viewpoint changed, without any object-level changes. This invention uses the official partitioning, comprising 67,660 training images (forming the training set), 3,976 validation images (forming the validation set), and 7,970 test images (forming the test set).

[0239] First, the change description system is trained and validated using the training set and validation set according to the training method provided in Example 1 above. Then, the change description system is tested using the test set according to the change description method provided in Example 2 above. Specific metrics are shown in Table 1.

[0240] Table 1 Test results of the change description system

[0241]

[0242] Note: Semantic-level changes refer to changes in not only the viewpoint but also the object level in the image.

[0243] As can be seen from Table 1, the change description system provided by the present invention can generate effective change descriptions between different graphics.

[0244] On the above dataset, the results for (a) color change, (b) texture change, (c) object addition, (d) object removal, (e) object movement, and (f) viewpoint change only are described as follows: Figure 6 As shown. For each pair of images, the variation description generated by this invention, and the actual description provided by the annotator, are as follows: Figure 6 As shown in the figure, changed and unchanged visual elements are highlighted in green and red, respectively. As illustrated, this invention can filter out irrelevant background elements and locate key unchanged visual elements. Simultaneously, it possesses the ability to dynamically locate changed regions within unchanged visual features, successfully locating changed objects without requiring bounding box information. By fusing changed / unchanged visual features, this invention can produce a detailed description containing both changed and unchanged visual element concepts, closely resembling human-provided labels.

[0245] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A change description system, characterized in that, include: The input module is used to convert the two images and text descriptions in the image pair to be compared into the visual features and word embeddings of the corresponding images; A single-view Transformer encoder is used to extract visual features from each image separately and obtain the invariant features of each image. A cross-view Transformer encoder is used to match the invariant features of two images in an image pair and obtain a representation of the changes in the image pair. A single-modal language Transformer encoder is used to convert word embeddings into linguistic feature representations; A multimodal Transformer decoder is used to fuse the variation feature representations of image pairs obtained from a cross-view Transformer encoder and the linguistic feature representations obtained from a single-modal language Transformer encoder to obtain fused features; The output module is used to obtain a description of the changes in the image pair based on the fused features.

2. The change description system according to claim 1, characterized in that, The input module uses a pre-trained visual backbone network to linearly project each image in the image pair into visual features; at the same time, it tags each word in the text description to obtain word embeddings.

3. The change description system according to claim 1, characterized in that, Single-view Transformer encoder Include The layer fuses local contextual information within the two images to be compared through a multi-head self-attention mechanism. and Indicates the first The layer's output, expressed as follows: ； ； in, This represents a feedforward neural network. Representative layer normalization operation, This indicates multi-head attention operation; after After layer self-attention and feedforward neural network operations, the final invariant feature representation of each image can be obtained: and .

4. The change description system according to claim 3, characterized in that, Cross-view Transformer encoder Include Layers, through multi-head cross-attention mechanism fusion and interactive cross-view invariant feature representation. and The operation of the i′ layer in the cross-view encoder is as follows: ； ； go through After layer cross-attention and feedforward operations, the final visual representation between images is obtained: and ,in Connecting and linearly projecting the cross-view representations, we get: ； in, , This represents the visual features of the two images in an image pair. This indicates a cross-dimensional splicing operation. and These represent the weight and bias matrices of the linear projection, respectively. This represents the network features fused across different perspectives.

5. The change description system according to claim 1, characterized in that, Transformer encoder for single-modal language Include Each layer consists of two sublayers: a masked self-attention sublayer and a feedforward sublayer; the computational operations of the language encoder layer are as follows: ； in, This represents a feedforward neural network; Representative layer normalization operation; The multi-head attention operation represents the forward mask; the final linguistic features are represented as follows: .

6. The change description system according to claim 1, characterized in that, The multimodal Transformer decoder includes The first layer contains masked self-attention, cross-attention, and feedforward operations; the second layer... Layers are represented as: ； ； The final multimodal Transformer decoder output fused features are represented as follows: ,in ; in, This represents a feedforward neural network; Representative layer normalization operation; This represents a multi-head attention operation in a forward mask; This indicates multi-head attention operation.

7. The change description system according to claim 1, characterized in that, The output module calculates the probability distribution of the target word: ； in, and These represent the weight and bias matrices, respectively. This represents the fused features output by the multimodal Transformer decoder.

8. A training method for a change description system according to any one of claims 1 to 7, characterized in that, Includes the following steps: S1 divides the training data into several training batches. Each training sample includes image pairs before and after the change, as well as text descriptions. For any training batch, the following steps S2-S6 are performed. S2 obtains the invariant features of each image in the training sample image pair and constructs the alignment loss function of the image pair based on cross-view contrastive learning; S3 acquires the variation feature representation of training sample image pairs and the linguistic feature representation of text descriptions, and constructs an alignment loss function between image pairs and text descriptions based on cross-modal contrastive learning; S4 constructs the cross-entropy loss function to predict changes; S5 generates a total loss function based on the alignment loss function of the image pair, the alignment loss function of the image pair and the text description, and the cross-entropy loss function for predicting the change description; S6 optimizes the parameters of the change description system based on the total loss function; Repeat steps S2-S6 above until the change description system converges.

9. The training method for the change description system according to claim 8, characterized in that, In step S1, the input module also converts the two images and text descriptions in the training sample image pair into visual features and word embeddings of the corresponding images.

10. The training method for the change description system according to claim 8, characterized in that, Step S2 includes the following sub-steps: S21 obtains the invariant features of each image in the training sample image pair through a single-view Transformer encoder; S22 obtains a queue of similar samples of invariant features for each image in an image pair through a single-view Transformer momentum encoder. S23 constructs an alignment loss function for image pairs based on the invariant characteristics of image pairs and the queue of similar samples.

11. The training method for the change description system according to claim 10, characterized in that, In step S23, the invariant features of the image pair and the similar sample queue are first extracted by the first feature extraction module to obtain the corresponding internal feature representation of the image; The first feature extraction module uses invariant features of image pairs. and The mean pooling features are linearly projected to obtain the image's internal feature representation before and after the change. and : ； ； in, Represents the weight matrix. Represents the bias matrix. and They represent and Elements in; Simultaneously, the samples from the similar sample queues of the image before and after the change are processed by the first feature extraction module to obtain the corresponding internal image feature representations, namely: and ; Then, the alignment between the two pairs of matched images is maximized by using the InfoNCE loss, while minimizing the alignment of the unmatched images: ； ； ； in, This represents the image internal feature representation of a similar sample queue output by a single-view Transformer momentum encoder. Indicates the number of samples in the batch. Indicates temperature parameter, This indicates the number of samples in the similar sample queue. Image internal feature representation of a queue of similar samples after image transformation Image internal feature representation of the image before and after the change The most similar sample Image internal feature representation of a cohort of similar samples before the change Image internal feature representation of the transformed image The most similar sample.

12. The training method for the change description system according to claim 10, characterized in that, Step S3 includes the following sub-steps: S31 obtains the change feature representation of the image pair by matching the invariant features of the two images in the image pair obtained in step S21 through the cross-view Transformer encoder; S32 uses the cross-view Transformer momentum encoder to match similar samples of invariant features between two images in the image pair obtained in step S22, and obtains a queue of similar samples representing the change features of the image pair. S33 uses a single-modal language Transformer encoder to convert the word embeddings corresponding to the text descriptions in the training samples into language feature representations; S34 obtains a queue of similar samples with language feature representations obtained in step S33 through a single-modal language Transformer momentum encoder; S35 constructs an alignment loss function between image pairs and text descriptions based on the variation feature representation of image pairs and the similar sample queue, combined with the language feature representation and the similar sample queue.

13. The training method for the change description system according to claim 12, characterized in that, In step S35, the change feature representation and similar sample queue of the image pair are first linearly projected onto the same dimension by the second feature extraction module and the third feature extraction module, along with the language feature representation and similar sample queue, to obtain the corresponding change feature update representation and language feature update representation. The second feature extraction module uses mean pooling features represented by the change features of the image pairs to obtain the corresponding change feature update representation: ； in, Represents the weight matrix. Represents the bias matrix; Similarly, the samples in the similar sample queue representing the change features of the image pair are updated with corresponding change features by the second feature extraction module, which is: ; The third feature extraction module uses mean pooling features from the language feature representation to obtain the corresponding updated language feature representation: ； in, Represents the weight matrix. Represents the bias matrix; Similarly, the samples in the similar sample queue representing the language features of the text description are updated with corresponding language features through the third feature extraction module, which is: ; Subsequently, the cross-modal alignment between matched variation features and linguistic features is maximized using InfoNCE loss: ； ； ； in, This represents the feature update representation of the changing features of a similar sample queue output by the cross-view Transformer momentum encoder. This represents the language feature update representation of a similar sample queue output by a Transformer momentum encoder for a unimodal language. Indicates the number of samples in the batch. Indicates temperature parameter, Language feature update representation of similar sample queues Image feature update representation The most similar sample The feature update representation of changes in similar sample queues Chinese and language feature update representation The most similar sample.

14. The training method for the change description system according to claim 8, characterized in that, In step S4, the cross-entropy loss function describing the predicted change is expressed as: ； in, and These represent the images before and after the change; This represents the probability distribution, which is also the output of the output module, where n = 1, 2, ..., N, and N represents the sentence length. This indicates changes in all parameters within the system's network.

15. The training method for the change description system according to claim 14, characterized in that, In step S5, during the training phase, the total loss function is learned through two contrastive learning objectives. , and cross-entropy Integration for collaborative learning: ； in, and It is a hyperparameter that adjusts the importance weights; In step S6, the parameters of the change description system are optimized using the gradient descent optimization algorithm based on the total loss function. For momentum model single-view Transformer momentum encoders, cross-view Transformer encoders, and single-modal language Transformer momentum encoders, their parameters are updated using a moving average method. The update process is represented as follows: ； in, Let represent the momentum model parameters obtained in the t-th iteration update. This represents the momentum model parameters obtained in the (t-1)th iteration update. Let M represent the system parameters whose changes are described by the t-th iteration update, and M represent the weights.

16. A change description method, comprising the steps described in any one of claims 1 to 7, using the change description system of claims 1 to 7: Step 1: The input module converts the two images and text descriptions in the image pair to be compared into visual features and word embeddings of the corresponding images. Step 2: Extract visual features from each image using a single-view Transformer encoder to obtain the invariant features of each image. Step 3: Match the invariant features of the two images in the image pair using a cross-view Transformer encoder and obtain the change feature representation of the image pair; Step 4: Convert word embeddings into linguistic feature representations using a single-modal language Transformer encoder; Step 5: The multimodal Transformer decoder fuses the change feature representations of the image pairs obtained from the cross-view Transformer encoder and the language feature representations obtained from the single-modal language Transformer encoder to obtain fused features; Step 6: Based on the fusion features, obtain the change description of the image pair through the output module.

Images