Retrieval method based on cross-modal semantics and hybrid counterfactual training

Abstract

The present invention discloses a retrieval method based on cross-modal semantics and mixed counterfactual training, comprising the following steps: A. obtaining a reference image I R , target image I T and query text T Q The present invention addresses the following challenges: (1) modeling the feature representation of the image and text; (2) establishing a cross-modal representation modification module and a representation absorption and synthesis module to model visual language representations in a three-level cascade reasoning process; (3) constructing mixed counterfactual samples; (4) modeling global-local combinations to capture local-global information across different scales and modalities; and (5) deriving a final composite representation from the bottom-up hierarchical combination to capture implicit visual modifications and preservation in the reference image. (6) learning matching for combined retrieval of multiple modalities, then incorporating a θ-parameterized excitation. Finally, the learned composite image-text representation is uniquely aligned with the visual representation of the target ground-truth image. (7) evaluating the retrieval results using a loss function.

Description

Technical Field

[0001] The present invention relates to the technical field of video image retrieval in intelligent security systems, and in particular to a retrieval method based on cross-modal semantics and hybrid counterfactual training. Background Art

[0002] To obtain combined representations for multimodal retrieval, existing methods mainly rely on cross-modal interaction and fusion of global semantics and local features, or rely on simple separation / cascade feature learning schemes of reference images and query text from their corresponding encoders. However, this combination approach fails to capture the intrinsic relationship between the underlying informative features and the top-level abstract semantics. From the perspective of deep understanding:

[0003] (i) Detailed local features of image regions with descriptive phrases but without global semantic understanding of the query.

[0004] (ii) Abstract global semantics are learned from the encoder’s hierarchical sequence with increasing abstraction, but lack specific properties rooted in different local locations.

[0005] Simply considering the above feature utilization without properly modeling the global-local combination may lead to degenerate solutions. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a retrieval method based on cross-modal semantics and hybrid counterfactual training, which can solve the shortcomings of the existing technology and improve the accuracy of image retrieval.

[0007] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0008] A retrieval method based on cross-modal semantics and hybrid counterfactual training includes the following steps:

[0009] A. Establish a multi-granular visual representation module and a global-local text embedding module to obtain the reference image I R , target image I T and query texts T at different levels Q Feature representation of

[0010] B. Establish a cross-modal representation modification module and a representation absorption and synthesis module to form a bottom-up cross-modal semantic synthesis module, and model visual language representation in a three-level cascade reasoning;

[0011] C. Construct mixed counterfactual samples;

[0012] D. Establish a local feature module, a global-local feature absorption module, and a global semantic module; use the representation absorption synthesis module to model the global-local combination, so that the designed transformer-based modification and absorption blocks can capture local-global information of different scales and modalities across layers, from local features at the bottom level to global semantics at the top level; the final composite representation derived from the bottom-up hierarchical combination captures the implicit visual modifications and preservation in the reference image according to the different text modifiers;

[0013] E. Learning to match M((I R , T Q ), I T )Combined retrieval of multiple modalities, and then adding θ parameterized excitation, the final learned image-text composite representation is uniquely aligned with the visual representation of the target ground truth image, as shown in the following formula,

[0014]

[0015] Where θ represents the parameterized excitation, k(·,·) represents the similarity solution, Φ(·) and Ψ(·) are the synthesis encoder and image encoder, respectively;

[0016] F. Use loss function to judge the retrieval results.

[0017] Preferably, in step A,

[0018] The multi-granular visual representation module uses visual transformers to generate discriminative representations of the image's visual content. Shallow layers of the visual transformers extract basic syntactic information, while deeper layers extract more complex semantic information. To further improve feature quality, linear projection is used to map the extracted features to a global semantic representation.

[0019] The global local text embedding module uses BERT to tokenize the query text TQ into a sequence of M subword tokens, and then prepends the special token positions [cls] and [end] to the text subword token sequence to divide the query text into local word embedding and global sentence embedding; for local word embedding, the word embedding output of the first layer is As BERT progresses, context labeling and self-attention interaction are performed in multiple steps. The last layer of BERT represents the global information of the embedding of the labeled words in the given text, and these labeled word embeddings are connected to form the global sentence embedding.

[0020] Preferably, in step B,

[0021] The cross-modal representation modification module consists of a self-attention layer, a bidirectional cross-attention layer, and a soft-attention layer; in order to self-discover the latent region-to-region relations necessary for learning transformations, the embedding RR ∈R Nk×d and the embedding R of the query modality Q Q ∈R Mk×d , this module learns a combined embedding conditioned on the reference representation R and the query representation Q, which is obtained by selectively suppressing and highlighting the query pattern R R , so as to achieve more effective learning for image retrieval, the above is input into a self-attention layer with layer normalization and residual links, as shown in the following formula,

[0022]

[0023] Among them, L n and It is represented by layer normalization and residual connection, PSA represents pyramid pooling attention operation, MSA represents multi-head self-attention operation, J(·) determines whether the input is a visualization modal function v, and the self-attention query is referenced. and query modal Self-participation representation of ;

[0024] MSA self-attention captures the non-local correlation used for feature transformation. On the basis of self-attention, pyramid pooling cross attention CSA is introduced, and bidirectional cross co-attention of layer normalization and residual connection is obtained. and As shown in the following formula,

[0025]

[0026]

[0027] Using soft attention, After processing mining learning and The potential relationships in image conversion and preservation, the final cross-modal composition is:

[0028]

[0029] in, SOA Q→R (·) is the soft attention operation;

[0030] The representation absorption synthesis module consists of a self-attention layer and a residual attention layer, which are constructed in a hierarchical order; it is used to absorb meaningful information from the local feature space and then create an information combination to enhance the robustness of subsequent query target matching; the local and global level representations are R L ∈R Nc×dc and R G ∈R Nc×dc , characterize the absorption synthesis module to achieve from RL absorb meaningful and discriminative information from the G The prior knowledge guidance for generating synthetic representations; the absorption process is as follows,

[0031]

[0032] After self-attention modeling, we use and Generate intermediate embeddings, the global semantic representation gets a higher weight, and As a priori guidance for combination generation, the residual attention mechanism is then used to know that the demand side asks for meaningful information, as shown in the following formula:

[0033]

[0034] Among them, [·,·] represents the serial operation, T l represents a nonlinear transformation;

[0035] The fusion embedding is normalized at the residual connection layer and input to the feedforward layer to obtain the final absorption feature representation R AC , this combination represents R AC Absorbing useful knowledge from local features improves the accuracy of query matching targets.

[0036]

[0037] in,

[0038] Preferably, in step D,

[0039] The local feature module uses the transmembrane state representation correction module to and Learning local feature combinations

[0040] The global-local feature absorption module combines local features Absorb meaningful and discriminative information, which plays a priori guidance role from the local feature layer to the robustness of subsequent synthesis-target matching, and derives the global-local absorption combination using the representation absorption synthesis module potential embeddings;

[0041] The global semantic combination module models the final combination from the visual domain and the language domain Aggregate Intermediate Representation Basic semantics and query text global semantic latent vector to update the output of the stream.

[0042] Preferably, in step F, the loss function is composed of a bidirectional ternary loss, a reconstruction loss, and a domain alignment loss;

[0043] The bidirectional ternary loss constructs a fine-grained query-target correspondence between the input query and the target image from the secondary extracted contrast negative samples, ensuring semantic matching of the components and target representation with high similarity. The bidirectional ternary loss is defined as follows,

[0044] L tri (X, Y, m) = max(0, || X + -Y||2-||X - -Y||2+m)

[0045] L bid (C que , C tar ,m,m a )=λ q L tri (C que , C tar ,m)+λ i L tri (C tar , C que , m a )

[0046] Among them, X + and X - are positive samples and negative samples, λ q and λ i is the weight hyperparameter, ||·||2 is the L2 distance, and s qt Represents C que and C tar Semantic similarity, α adaptive edge value m a The hyperparameter, s qt When it approaches 0, m a Otherwise, the maximum value is obtained, otherwise the minimum value is obtained, thereby achieving the adaptive effect of counterfactual training;

[0047] Reconstruction loss L res Constrained C que Mapping vision and language, by R img and R text Represented by the latent embedding C tar and Alignment aims to achieve a balanced utilization of text and images in combined embeddings by refactoring norms and strengthening them.

[0048]

[0049] Among them, λ img and λ textare pre-training hyperparameters;

[0050] The domain alignment loss is to further learn the fine-grained semantic correspondence between the composite domain and the target image domain. Here, the optimal transfer OT is used to align the representation distributions of different domains to bridge the gap between them. First, the cost matrix c between the feature distributions of the composite domain and the target domain is calculated. m , then each feature is classified into features from another modality with different weights to complete the prediction of the corresponding relationship, and the synthetic domain and the target domain are matched using Wasserstein Distance. The Wasserstein Distance and alignment loss are shown in the following formula:

[0051]

[0052] L ali =λ a W d (C, T)

[0053] The beneficial effects brought about by adopting the above technical solution are:

[0054] (i) We propose a modality representation modification module and a representation absorption synthesis module to implicitly model bottom-up semantic composition. The key idea of this step is to achieve complementary synergy between bottom-up visual representations by leveraging complementary global and local representations from different encoder layers, thereby selectively modifying relevant image features while ensuring that unchanged features are preserved, which is crucial for accurate retrieval methods.

[0055] (ii) A plug-and-play hybrid counterfactual training strategy (Note: hybrid counterfactuals are proposed here, not counterfactuals). This strategy aims to facilitate the retrieval model to build fine-grained query-target correspondences for robust image retrieval. This strategy can be used as a plug-and-play component to improve the query-sensitivity of the retrieval model. Specifically, three new different types of counterfactual samples are constructed: image-independent, text-independent, and context-preserving. This hybrid sample implements an explicit bidirectional correspondence learning mechanism, which helps to establish a one-to-one match between the combined query and the expected image, reducing the model's prediction uncertainty for similar queries.

[0056] (iii) We design a bottom-up cross-modal semantic synthesis and hierarchical compositional reasoning approach, incorporating cross-granular semantic updates to learn and understand composite image-text representations. Starting from two new perspectives: implicit bottom-up visual representation synthesis and explicit fine-grained query-target structure correspondence, we gradually digest the information flows from vision and language to address the challenging task of content-based image retrieval. This combination effectively captures and preserves implicit visual modifications based on text modifiers, thereby outperforming existing image retrieval techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an example of multimodal retrieval;

[0058] Figure 2 It is a content-based image retrieval framework. DETAILED DESCRIPTION

[0059] A specific embodiment of the present invention comprises the following steps:

[0060] A. Establish a multi-granular visual representation module and a global-local text embedding module to obtain the reference image I R , target image I T and query texts T at different levels Q Feature representation of

[0061] The multi-granular visual representation module uses visual transformers to generate discriminative representations of the image's visual content. Shallow layers of the visual transformers extract basic syntactic information, while deeper layers extract more complex semantic information. To further improve feature quality, linear projection is used to map the extracted features to a global semantic representation.

[0062] The global local text embedding module uses BERT to embed the query text T Q Mark it into M subword token sequences, then prepend the special token positions [cls] and [end] to the text subword token sequence, and divide the query text into local word embedding and global sentence embedding; for local word embedding, the word embedding output of the first layer is As BERT progresses, context labeling and self-attention interaction are performed in multiple steps. The last layer of BERT represents the global information of the embedding of the labeled words in the given text, and these labeled word embeddings are connected to form the global sentence embedding.

[0063] B. Establish a cross-modal representation modification module and a representation absorption and synthesis module to form a bottom-up cross-modal semantic synthesis module, and model visual language representation in a three-level cascade reasoning;

[0064] The cross-modal representation modification module consists of a self-attention layer, a bidirectional cross-attention layer, and a soft-attention layer; in order to self-discover the latent region-to-region relations necessary for learning transformations, the embedding R R ∈R Nk×d and the embedding R of the query modality Q Q ∈R Mk×d , this module learns a combined embedding conditioned on the reference representation R and the query representation Q, which is obtained by selectively suppressing and highlighting the query pattern R R, so as to achieve more effective learning for image retrieval, the above is input into a self-attention layer with layer normalization and residual links, as shown in the following formula,

[0065]

[0066] Among them, L n and It is represented by layer normalization and residual connection, PSA represents pyramid pooling attention operation, MSA represents multi-head self-attention operation, J(·) determines whether the input is a visualization modal function v, and the self-attention query is referenced. and query modal Self-participation representation of ;

[0067] MSA self-attention captures the non-local correlation used for feature transformation. On the basis of self-attention, pyramid pooling cross attention CSA is introduced, and bidirectional cross co-attention of layer normalization and residual connection is obtained. and As shown in the following formula,

[0068]

[0069]

[0070] Using soft attention, After processing mining learning and The potential relationships in image conversion and preservation, the final cross-modal composition is:

[0071]

[0072] in, SOA Q→R (·) is the soft attention operation;

[0073] The representation absorption synthesis module consists of a self-attention layer and a residual attention layer, which are constructed in a hierarchical order; it is used to absorb meaningful information from the local feature space and then create an information combination to enhance the robustness of subsequent query target matching; the local and global level representations are R L ∈R Nc×dc and R G ∈R Nc×dc , characterize the absorption synthesis module to achieve from R L absorb meaningful and discriminative information from the G The prior knowledge guidance for generating synthetic representations; the absorption process is as follows,

[0074]

[0075] After self-attention modeling, we use and Generate intermediate embeddings, the global semantic representation gets a higher weight, and As a priori guidance for combination generation, the residual attention mechanism is then used to know that the demand side asks for meaningful information, as shown in the following formula:

[0076]

[0077] Among them, [·,·] represents the serial operation, T l represents a nonlinear transformation;

[0078] The fusion embedding is normalized at the residual connection layer and input to the feedforward layer to obtain the final absorption feature representation R AC , this combination represents R AC Absorbing useful knowledge from local features improves the accuracy of query matching targets.

[0079]

[0080] in,

[0081] C. Constructing mixed counterfactual samples; including the following steps,

[0082] C1. Construct image-independent and text-independent counterfactual samples;

[0083] Given a reference image I R and the corresponding query text T Q Use BERT as a pre-trained bidirectional language model to find the least relevant text based on language similarity and its corresponding image Therefore, these texts and images combine the original reference image and the query text to form an image-independent Not related to the text counterfactual samples;

[0084] C2, construct text-retained counterfactual samples;

[0085] First, the query word T is masked based on the prior known attributes. Q Attribute words, generate preliminary candidate words, and replace them with random words to obtain k1 samples, and then replace the original T Q The semantic similarity is calculated in BERT at the input point, and finally the top k2 texts with the reference image are selected as the context-preserving query, as shown in the following formula,

[0086]

[0087] in, represents the selected context-preserving negative samples, P s represents the probability of measuring BERT, Indicates the number selected in the second stage;

[0088] D. Establish a local feature module, a global-local feature absorption module, and a global semantic module; use the representation absorption synthesis module to model the global-local combination, so that the designed transformer-based modification and absorption blocks can capture local-global information of different scales and modalities across layers, from local features at the bottom level to global semantics at the top level; the final composite representation derived from the bottom-up hierarchical combination captures the implicit visual modifications and preservation in the reference image according to the different text modifiers;

[0089] The local feature module uses the transmembrane state representation correction module to and Learning local feature combinations

[0090] The global-local feature absorption module combines local features Absorb meaningful and discriminative information, which plays a priori guidance role from the local feature layer to the robustness of subsequent synthesis-target matching, and derives the global-local absorption combination using the representation absorption synthesis module potential embeddings;

[0091] The global semantic combination module models the final combination from the visual domain and the language domain Aggregate Intermediate Representation Basic semantics and query text global semantic latent vector To update the output of the stream;

[0092] E. Learning to match M((I R , T Q ), I T )Combined retrieval of multiple modalities, and then adding θ parameterized excitation, the final learned image-text composite representation is uniquely aligned with the visual representation of the target ground truth image, as shown in the following formula,

[0093]

[0094] Where θ represents the parameterized excitation, k(·,·) represents the similarity solution, Φ(·) and Ψ(·) are the synthesis encoder and image encoder, respectively;

[0095] F. Use loss function to judge the retrieval results;

[0096] The loss function consists of bidirectional triplet loss, reconstruction loss, and domain alignment loss;

[0097] The bidirectional ternary loss constructs a fine-grained query-target correspondence between the input query and the target image from the secondary extracted contrast negative samples, ensuring semantic matching of the components and target representation with high similarity. The bidirectional ternary loss is defined as follows,

[0098] L tri (X, Y, m) = max(0, || X + -Y||2-||X - -Y||2+m)

[0099] L bid (C que , C tar ,m,m a )=λ q L tri (C que , C tar ,m)+λ i L tri (C tar , C que , m a )

[0100] Among them, X + and X - are positive samples and negative samples, λ q and λ t is the weight hyperparameter, ||·||2 is the L2 distance, and s qt Represents C que and C tar Semantic similarity, α adaptive edge value m a The hyperparameter, s qt When it approaches 0, m a Otherwise, the maximum value is obtained, otherwise the minimum value is obtained, thereby achieving the adaptive effect of counterfactual training;

[0101] Reconstruction loss L res Constrained C que Mapping vision and language, by R img and R text Represented by the latent embedding C tar and Alignment aims to achieve a balanced utilization of text and images in combined embeddings by refactoring norms and strengthening them.

[0102]

[0103] Among them, λ img and λ text are pre-training hyperparameters;

[0104] The domain alignment loss is to further learn the fine-grained semantic correspondence between the composite domain and the target image domain. Here, the optimal transfer OT is used to align the representation distributions of different domains to bridge the gap between them. First, the cost matrix c between the feature distributions of the composite domain and the target domain is calculated. m , then each feature is classified into features from another modality with different weights to complete the prediction of the corresponding relationship, and the synthetic domain and the target domain are matched using Wasserstein Distance. Wasserstein Distance and alignment loss are shown in the following formula:

[0105]

[0106] L ali =λ a W d (C, T).

[0107] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0108] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A content-image retrieval method based on cross-modal semantics and hybrid counterfactual training, characterized by The following steps are involved: A. Establish a multi-granular visual representation module and a global-local text embedding module to obtain the reference image I R , target image I T and query texts T at different levels Q Feature representation of B. Establish a cross-modal representation modification module and a representation absorption and synthesis module to form a bottom-up cross-modal semantic synthesis module, and model visual language representation in a three-level cascade reasoning; The cross-modal representation modification module consists of a self-attention layer, a bidirectional cross-attention layer, and a soft-attention layer; the representation absorption synthesis module contains a self-attention layer and a residual attention layer, constructed in a hierarchical order; C. Constructing mixed counterfactual samples, including the following steps, C1. Construct image-independent and text-independent counterfactual samples; Given a reference image and the corresponding query text Use BERT as a pre-trained bidirectional language model to find the least relevant text based on language similarity and its corresponding image , so these texts and images combine the original reference image and query text to form an image-independent Not related to the text counterfactual samples; C2, construct text-retained counterfactual samples; First, mask the query terms based on the a priori known attributes Attribute words, generate preliminary candidate words, and replace them with random words to obtain samples, and then the original Calculate semantic similarity in the input point BERT and finally select the top one with the reference image The text is retained as a context query, as shown below, in, represents the selected context-keeping negative samples, represents the probability of measuring BERT, Indicates the number selected in the second stage; D. Establish a local feature module, a global-local feature absorption module, and a global semantic module; use the representation absorption synthesis module to model the global-local combination, so that the designed transformer-based modification and absorption blocks can capture local-global information of different scales and modalities across layers, from local features at the bottom level to global semantics at the top level; the final composite representation derived from the bottom-up hierarchical combination captures the implicit visual modifications and preservation in the reference image according to the different text modifiers; E. Learning matching pairs Combined retrieval of multiple modalities, then adding Parameterized excitation, the final learned image-text composite representation is uniquely aligned with the visual representation of the target ground truth image, as shown in the following formula, in, represents parameterized excitation, Indicates the similarity solution. and They are synthesis encoder and image encoder respectively; F. Use loss function to judge the retrieval results.

2. The content-image retrieval method based on cross-modal semantics and hybrid counterfactual training according to claim 1, characterized in that: In step A, The multi-granular visual representation module uses visual transformers to generate discriminative representations of the image's visual content. Shallow layers of the visual transformers extract basic syntactic information, while deeper layers extract more complex semantic information. To further improve feature quality, linear projection is used to map the extracted features to a global semantic representation. The global local text embedding module uses BERT to embed the query text Mark it into M subword token sequences, then prepend the special token positions [cls] and [end] to the text subword token sequence, and divide the query text into local word embedding and global sentence embedding; For local word embedding, the word embedding output of the first layer is ; As BERT advances, context labeling and self-attention interaction are performed in multiple steps. The last layer of BERT represents the global information of the embedding of the labeled words in the given text, and these labeled word embeddings are connected to form the global sentence embedding .

3. The content-image retrieval method based on cross-modal semantics and hybrid counterfactual training according to claim 2, characterized in that: In step B, The cross-modal representation modification module embeds R R ∈R Nk×d and the embedding R of the query modality Q Q ∈R Mk×d , this module learns a combined embedding conditioned on the reference representation R and the query representation Q, which is obtained by selectively suppressing and highlighting the query pattern R R , so as to achieve more effective learning for image retrieval, the above is input into a self-attention layer with layer normalization and residual links, as shown in the following formula, in, and It is represented by layer normalization and residual connection, PSA represents pyramid pooling attention operation, MSA represents multi-head self-attention operation, Then determine whether the input is a visualization modal function , the self-attention query gets the reference and query modal Self-participation representation of ; MSA self-attention captures the non-local correlation used for feature transformation. On the basis of self-attention, pyramid pooling cross attention CSA is introduced, and bidirectional cross co-attention of layer normalization and residual connection is obtained. and , as shown below, ； Using soft attention, After processing mining learning and The potential relationships in image conversion and preservation, the final cross-modal composition is: in, , It is a soft attention operation; The representation absorption and synthesis module is used to absorb meaningful information from the local feature space and then create an information combination to enhance the robustness of subsequent query target matching; the local and global level representations are R L ∈ R Nc×dc and R G ∈ R Nc ×dc , characterize the absorption synthesis module to achieve from R L absorb meaningful and discriminative information from the G The prior knowledge guidance for generating synthetic representations; the absorption process is as follows, ； After self-attention modeling, we use Generate intermediate embeddings, the global semantic representation gets a higher weight, and As a priori guidance for combination generation, the residual attention mechanism is then used to know that the demand side asks for meaningful information, as shown in the following formula: in, represents the concatenation operation, represents a nonlinear transformation; The fusion embedding is normalized at the residual connection layer and input into the feedforward layer to obtain the final absorption feature representation , this combination indicates Absorbing useful knowledge from local features improves the accuracy of query matching targets. in, .

4. The content-image retrieval method based on cross-modal semantics and hybrid counterfactual training according to claim 3, characterized in that: In step D, The local feature module uses the transmembrane state representation correction module to Learning local feature combinations ; The global-local feature absorption module combines local features Absorb meaningful and discriminative information, which plays a priori guidance role from the local feature layer to the robustness of subsequent synthesis-target matching, and derives the global-local absorption combination using the representation absorption synthesis module potential embeddings; The global semantic combination module models the final combination from the visual domain and the language domain , by aggregating intermediate representations Basic semantics and query text global semantic latent vector to update the output of the stream.

5. The content-image retrieval method based on cross-modal semantics and hybrid counterfactual training according to claim 4, characterized in that: In step F, the loss function consists of bidirectional triplet loss, reconstruction loss, and domain alignment loss; The bidirectional ternary loss constructs a fine-grained query-target correspondence between the input query and the target image from the secondary extracted contrast negative samples, ensuring semantic matching of the components and target representation with high similarity. The bidirectional ternary loss is defined as follows, in, are positive samples and negative samples, is the weight hyperparameter, for distance, and Respectively and The semantic similarity of Adaptive edge value The hyperparameters of Near 0 o'clock, Otherwise, the maximum value is obtained, otherwise the minimum value is obtained, thereby achieving the adaptive effect of counterfactual training; Reconstruction loss Constrained The mapping of vision and language, by Represented by latent embedding Alignment aims to achieve a balanced utilization of text and images in combined embeddings by refactoring norms and strengthening them. in, are pre-training hyperparameters; The domain alignment loss is to further learn the fine-grained semantic correspondence between the combined domain and the target image domain. Here, the optimal transfer OT is used to align the representation distributions of different domains to bridge the gap between them. First, the cost matrix between the feature distributions of the synthetic domain and the target domain is calculated. , then each feature is classified into features from another modality with different weights to complete the prediction of the corresponding relationship, and the synthetic domain and the target domain are matched using Wasserstein Distance. Wasserstein Distance and alignment loss are shown in the following formula: 。

Images