A remote sensing image scene classification method based on uncertainty visual language alignment

Abstract

This invention relates to a remote sensing image scene classification method based on uncertain visual-language alignment, comprising the following steps: acquiring remote sensing images as training and testing sets; using a CLIP dual encoder to map the original image and text to a shared point representation space; the uncertain encoder predicting each embedding vector along two paths; internally, the uncertain encoder performing two-level interactions on the input embedding vectors; globally modeling the image and text as two Gaussian distributions based on the mean and variance vectors, maximizing the image-text similarity that should be correctly matched in the entire batch; acquiring new remote sensing images and inputting them into the trained model, which then classifies the scenes in the remote sensing images. This application captures fuzzy mappings between modalities in the probability space, providing a more generalized representation for downstream remote sensing image scene classification tasks.

Description

Technical Field

[0001] This invention relates to a remote sensing image scene classification method based on uncertain visual language alignment. Background Technology

[0002] In existing remote sensing image-text alignment paradigms, models typically map images and text as "deterministic points" in an embedding space. Accurate cross-modal semantic alignment remains a core challenge in the field of remote sensing image parsing. With the rise of Vision-Language Pre-training (VLP) techniques, such as the RemoteCLIP method, the scarcity of pre-training data in remote sensing has been greatly alleviated by transforming heterogeneous annotations such as object detection and semantic segmentation into unified "image-text" pairs. However, while this data construction method increases the data scale, it also introduces significant modal noise and semantic ambiguity.

[0003] In existing remote sensing image-text alignment paradigms, models typically map images and text as "deterministic points" in the embedding space. However, remote sensing images possess extremely high visual semantic richness; a single image often contains complex geographical features (such as the interweaving of vegetation, buildings, and roads), while text descriptions generated through templates or rules are often semantically simplistic and coarse. This inherent contradiction between the richness of visual information and the limitations of text description leads to pre-training data being filled with noise (irrelevant details of the image relative to the text) and ambiguity (underexpression of the text relative to the image). Therefore, traditional deterministic representation faces a severe challenge in handling this complex relationship with intermodal uncertainty; that is, a perfect one-to-one correspondence often cannot be formed between image features and text descriptions, resulting in an imbalance in "point-to-point" metrics such as Euclidean distance or cosine similarity in the embedding space. Summary of the Invention

[0004] To more robustly adapt to noise and ambiguity in large-scale remote sensing image-text pre-training data, this application proposes an uncertainty-based visual-language alignment framework. Unlike previous methods that ignore uncertainty and only express semantic spatial location, this application designs an uncertainty encoder module. This module implicitly models the correlation between features, extending the semantic representations of both images and text from a point representation space to a probability distribution representation space. Specifically, this application connects the uncertainty encoder module after the CLIP encoder (the CLIP encoder is a core component of the contrastive language-image pre-training model proposed by OpenAI, consisting of two independent modules: an image encoder and a text encoder. Through contrastive learning, it maps images and text to a unified high-dimensional feature space, achieving cross-modal semantic alignment and supporting zero-shot classification). Using point representations of different modalities as input features, it simultaneously predicts the mean vector and variance vector, extending the original point representation space of CLIP to a distribution representation space. The original cosine similarity is replaced with 2-Wasserstein distance as the logistic score to construct the contrastive loss function. In this representation space, the mean of the distribution represents the core semantics, while the variance intuitively measures the degree of uncertainty of the sample. In this way, the model can accommodate background noise in remote sensing imagery and capture semantic ambiguity in text descriptions, thereby achieving more accurate and diverse semantic alignment in noisy environments.

[0005] In zero-shot scenarios, the model needs to handle unseen categories, including new categories arising from differences in data representation or data distribution. For the former, uncertainty encoding maps the input to a distribution region through the variance of the distribution, forming a "semantic buffer" that allows features to fluctuate within a certain range, naturally endowing the model with the ability to cover multiple near-synonymous text category label distributions. For the latter, uncertainty encoding broadens the dimension of the representation space by introducing variance, thus widening the boundary of the model's semantic expression and making the model more inclusive of difficult, unseen samples. Therefore, in remote sensing image scene classification, uncertainty modeling not only improves the model's robustness to noise and ambiguity during the training phase but also enhances its zero-shot generalization ability during the inference phase.

[0006] To achieve the above objectives, this application discloses a remote sensing image scene classification method based on uncertainty visual-linguistic alignment. This method extends multimodal remote sensing image data from a point representation space to a high-dimensional probability distribution representation space, performing contrastive learning and semantic alignment. The method includes the following steps: Acquire remote sensing images as training and testing sets; The backbone network uses CLIP's original dual encoders to map the original image and text to a shared point representation space, respectively. To model the image and text features extracted by the backbone network as probability distributions, the uncertainty encoder predicts each embedding vector along two paths: the mean vector and the variance vector. Internally, the uncertainty encoder performs two levels of interaction on the input embedding vectors: interaction at the feature dimension level and interaction at the sequence level. The interaction at the feature dimension level is achieved by a single feedforward layer (FFN), while the interaction at the sequence level is guided by a multi-head self-attention operation. Uncertainty-based contrastive learning pre-training includes: modeling images and text globally as two Gaussian distributions based on mean and variance vectors, respectively, and using the 2-Wasserstein distance between the two Gaussian distributions as a metric; constructing bidirectional cross-entropy loss for image-to-text retrieval and text-to-image retrieval, and graph-guided dynamic Wasserstein loss; weighting these three losses to maximize the image-to-text similarity that should be correctly matched in the entire batch, thereby bringing whole sample pairs closer together and pushing away negative sample pairs; New remote sensing images are acquired and input into the trained model, which then classifies the scenes in the remote sensing images.

[0007] Furthermore, the backbone network uses CLIP's original dual encoder to map the original image and text to a shared point representation space, including: The image backbone uses ViT-B / 32 as an image encoder to extract visual hidden states. B indicates the base version, and 32 indicates the batch size. The visual encoder consists of one fully connected layer, one patch embedding layer, a 12-layer transformer with 12 attention heads, and a one-layer feedforward network (FFN). The text backbone extracts the hidden text states through the text encoder. The text encoder uses a 12-layer transformer, and its structure is identical to that of the visual encoder, except for the use of 8 attention heads; among them, and These represent visual and text embedding vectors, respectively. and Represents the global embedding vector. This indicates the number of visual patches and the number of text tokens. Indicates the sequence length. This represents the feature dimension of the embedding vector.

[0008] Furthermore, the uncertainty encoder predicts each embedding vector along two paths, including the mean vector. Sum of variance vectors The uncertainty encoder module takes as input the embedding vectors in the representation space of different modal points and outputs the mean vector. The variance vector represents the center position of the embedding vector in the probability distribution representation space. It represents the range or uncertainty of the embedding vector in the probability distribution characterization space.

[0009] Furthermore, the uncertain encoder internally performs two levels of interaction on the input embedding vector: interaction at the feature dimension level and interaction at the sequence level, specifically including: The feedforward layer FFN is formed by the first weight. First bias Second weight Second bias The two fully connected layers perform a linear transformation of the features to realize the input features. Dimensions The expansion and recovery of features, where ReLU is used as the activation function, achieves non-linear enhancement of features: Multi-head self-attention operation first considers the hidden state of the input. Perform split processing and project it onto... There are 1 subspace, and the dimension of each subspace is 1. ; Then, in each head, the hidden state is... Perform path splitting Processing the hidden input state of each path Projected matrix Mapped to the Queries in subspaces ,key Sum : Next, self-attention operations are performed in their respective subspaces to obtain... : ; Then, to The results of self-attention operations obtained from each subspace are concatenated and then projected through a projection matrix. Restore the dimensions and map them to the output space: Finally, calculate the uncertainty loss: It is an uncertainty loss, used to optimize module parameters. This is the batch size, representing the number of data samples processed. It is the first The mean vector of samples, with an embedding dimension of . , It is the first The original embedding vector of each sample. It is the square of the L2 norm. It is the reduced Planck constant, used to simulate the lower bound of uncertainty. It is the first The variance vector of each sample. It is a wave potential vector, in functional form. Using spatial coordinates, the image is discretized into a pixel grid in the remote sensing image. It is the Laplace operator, used to calculate the second derivative of the wave function, simulate wave propagation, and quantitatively capture dynamic uncertainties. It's an points operation. It is the Shannon entropy of the variance vector. It is a Lagrange multiplier, used to balance the loss term.

[0010] Furthermore, uncertainty-based contrastive learning pre-training includes: The mean vector obtained from the uncertainty encoder module Sum of variance vectors ,use global mean vector variance vector ,and The corresponding global mean vector Sum of variance vectors The image and text are globally modeled as Gaussian distributions, respectively. Gaussian distribution of text ,in It is the covariance matrix; The 2-Wasserstein distance between two Gaussian distributions is used as a metric, and it is defined as follows: Part One The second part measures the offset between the semantic center positions of two modalities. Measure the estimation of uncertainty for two modes; for A negative scaling factor and an offset are introduced to represent the similarity between two modalities. : in, and Indicates based on and Corresponding global vector , , and Perform similarity calculation. It is a negative scaling factor. This is the offset; Constructing a bidirectional cross-entropy loss for image-to-text retrieval and text-to-image retrieval. and : in, Indicates the index of the current image or text. Indicates batch size; Then, using graph-guided dynamic Wasserstein loss and fusing path integrals, semantic evolution is obtained: It uses graph-guided dynamic Wasserstein loss, combined with the overall pre-training objective of the InfoNCE framework. It refers to the batch size. This represents summing over all positive / negative sample pairs. It is an adaptive negative scaling factor. It is a distribution and The mean vector, It is the square of the L2 norm. It is a distribution and The covariance matrix, It is the square of the Frobenius norm. It is along the map path The points, The variance gradient at the path point The value on the simulating semantic path represents the uncertain flow, and the path is approximated by Dijkstra's algorithm. It's the offset. It is a similarity score. ,in To extend the Wasserstein distance, It is the InfoNCE cross-entropy term; Therefore, contrastive learning loss based on uncertainty The construction is as follows: The goal of uncertainty-based contrastive learning pre-training is to maximize the image-text similarity that should be correctly matched in the entire batch, so as to bring whole sample pairs closer together and push negative sample pairs further apart.

[0011] The uncertainty-based visual language alignment method proposed in this application aims to implicitly solve the robustness problem in large-scale remote sensing image and text pre-training data. By capturing the fuzzy mapping between modalities in the probability space, it provides a more generalizable representation for downstream remote sensing image scene classification tasks. Attached Figure Description

[0012] Figure 1 This is the overall framework diagram of this application; Figure 2 This is a schematic diagram of multimodal data in a high-dimensional probability distribution representation space. Detailed Implementation

[0013] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0014] This application proposes an uncertainty-based visual language alignment method, which aims to implicitly solve the noise and ambiguity problems in large-scale remote sensing image and text pre-training data. By capturing the fuzzy mapping between modalities in the probability space, it provides a more generalizable representation for downstream remote sensing image scene classification tasks.

[0015] Considering the noise and ambiguity present in large-scale remote sensing image and text pre-training data, this application integrates the powerful representational capabilities of the CLIP feature extraction backbone network with the probability distribution modeling capabilities of the designed uncertainty encoder module (e.g., Figure 1 As shown), it includes a feature extraction backbone network, an uncertainty encoder, and uncertainty-based contrastive learning pre-training, which further extends multimodal data from the traditional point representation space to a high-dimensional probability distribution representation space (such as...). Figure 2 As shown, this enables more robust contrastive learning and semantic alignment.

[0016] In one embodiment, the feature extraction backbone network includes: To extract features, the backbone network uses CLIP's original dual encoders to map the original image and text to a shared point representation space, respectively. Specifically, the image backbone uses ViT-B / 32 as the image encoder to extract visual hidden states. B indicates the base version, and 32 indicates the patch size is [size missing]. The visual encoder consists of one fully connected layer, one patch embedding layer, a 12-layer transformer with 12 attention heads, and a one-layer feedforward network (FFN). The text backbone extracts the hidden text states through the text encoder. The text encoder uses a 12-layer transformer, and its structure is identical to that of the visual encoder, except for employing an 8-attention head. and These represent visual and text embedding vectors, respectively. and Represents the global embedding vector. This indicates the number of visual patches and the number of text tokens. Indicates the sequence length. This represents the feature dimension of the embedding vector.

[0017] In one embodiment, the uncertainty encoder module includes: On the one hand, in order to model the image and text features extracted by the backbone network as probability distributions, the uncertainty encoder module predicts each embedding vector along two paths, including the mean vector. Sum of variance vectors The uncertainty encoder module takes as input the embedding vectors in the representation space of different modal points and outputs the mean vector. The variance vector represents the center position of the embedding vector in the probability distribution representation space. This represents the distribution range or uncertainty of the embedding vector in the probability distribution representation space. On the other hand, to fully model the probability distribution of features, the uncertainty encoder module internally performs two levels of interaction on the input embedding vector: interaction at the feature dimension level and interaction at the sequence level. The interaction at the feature dimension level is implemented by a single feedforward layer (FFN), while the interaction at the sequence level is guided by a multi-head self-attention operation. Specifically: The feedforward layer FFN is achieved through... , , and The two fully connected layers perform a linear transformation of the features to realize the input features. Dimensions The expansion and recovery of features, where ReLU is used as the activation function, achieves non-linear enhancement of features: Multi-head self-attention operations first examine the hidden state of the input. Perform split processing and project it onto... There are 1 subspace, and the dimension of each subspace is 1. ,in Set to 8; Then, in each head, the child hidden state is... Perform path splitting Processing (with) (Taking path prediction as an example), the input hidden state of each path Will be Corresponding projection matrix (Here, we use the same characters) It means that, among them Set as Mapped to the first In each subspace , and : ; Next, self-attention operations are performed in their respective subspaces to obtain... : ; Finally, for The results of self-attention operations obtained from each subspace are concatenated and then projected through a projection matrix. Restore the dimensions and map them to the output space: .

[0018] In addition, the uncertainty encoder module has a mean vector The prediction logic and the variance vector The prediction logic is similar. However, due to the sequence embedding vector obtained from the original CLIP... It already possesses a strong ability to characterize location, and This represents the result after correcting the original position in the probability distribution representation space. Therefore, the uncertainty encoder module in The path incorporates residual connections to learn the "correction" from the original position, ensuring a more stable prediction of the final mean vector. .

[0019] Finally, quantum fluctuations calculate the uncertainty loss: It is an uncertainty loss, used to optimize module parameters. This is the batch size, representing the number of data samples processed. It is the first The mean vector of samples, with an embedding dimension of . The default size is 512. It is the first The original embedding vector of each sample. It is the square of the L2 norm, which measures the mean shift. It is the reduced Planck constant (here defined as constant 1 for simplification) used to simulate the lower bound of uncertainty. It is the first The variance vector of each sample, i.e., the diagonal elements. It is a wave potential vector, in functional form. The spatial coordinates are discretized into a pixel grid in the remote sensing image. It is the Laplace operator, which calculates the second derivative of the wave function, simulates wave propagation, and quantitatively captures dynamic uncertainties. It is an integral operation, which is approximated in practice by discrete convolution. It is the Shannon entropy of the variance vector. To prevent excessive variance. It is a Lagrange multiplier, with a default value of 0.1, used to balance the loss term.

[0020] In one embodiment, uncertainty-based contrastive learning pre-training includes: The mean vector obtained from the uncertainty encoder module Sum of variance vectors ,use and Corresponding global vector , , and The image and text are globally modeled as two Gaussian distributions, respectively. and ,in Let be the covariance matrix. By introducing variance to represent uncertainty, the model can implicitly learn deterministic semantics as well as semantics with ambiguity or vagueness, thereby improving the fault tolerance and accuracy of alignment. To measure the difference between two modalities, uncertainty-based contrastive learning differs from traditional contrastive learning in that it calculates the cosine similarity between two points in the point representation space; it requires introducing additional variance for the pre-training target. Therefore, the model uses the 2-Wasserstein distance between two Gaussian distributions as the metric, which is defined as: ; Because the model assumes a covariance matrix It is a diagonal matrix, meaning that each dimension is independent of the others. Then the distance formula can be simplified to a more intuitive form: Part One The second part measures the offset between the semantic center positions of two modalities. To measure the estimation of uncertainty by two modalities, the distance between the two modalities in the probability distribution representation space is only considered if their modal semantic content is similar and they both have relatively consistent confidence in their modal semantics. It is even smaller. Therefore, it is necessary to... A negative scaling factor and an offset are introduced to represent the similarity between two modalities. : in, and Indicates based on and Corresponding global vector , , and Perform similarity calculation. It is a negative scaling factor. This is the offset. When the batch size of the input image-text pairs is... At times, in batches Matched images and text are considered positive sample pairs, and If mismatched images and texts are considered as negative sample pairs, then a similarity matrix based on 2-Wasserstein distance can be constructed.

[0021] In summary, the uncertainty-based contrastive learning pre-training objective still adopts CLIP's InfoNCE contrastive learning loss framework. Specifically, a bidirectional cross-entropy loss is constructed for image retrieval of text and text retrieval of images. and : in, Indicates the index of the current image or text. Indicates the batch size.

[0022] Then, using the graph-guided dynamic Wasserstein loss formula, the path integral is fused to obtain the semantic evolution: It is a graph-guided dynamic Wasserstein loss combined with the overall pre-training objective of the InfoNCE framework. It refers to the batch size. This represents the summation over all positive / negative sample pairs. The default size of the adaptive negative scaling factor is 0.07. It is a distribution and The mean vector. It is the square of the L2 norm. It is the covariance matrix, which is off-diagonal and has dimensions of . . It is the square of the Frobenius norm. It is along the map path The points, The variance gradient at the path point The value on the simulating semantic path represents the uncertain flow, and the path is approximated by Dijkstra's algorithm. This is the offset, with a default value of 1. It is a similarity score. ,in To extend the Wasserstein distance. It is the InfoNCE cross-entropy term.

[0023] This formula quantifies dynamic semantic alignment through path integral terms.

[0024] Therefore, contrastive learning loss based on uncertainty The construction is as follows: Therefore, the goal of uncertainty-based contrastive learning pre-training is to maximize the image-text similarity that should be correctly matched in the entire batch, so as to bring whole sample pairs closer together and push negative sample pairs further apart.

[0025] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A remote sensing image scene classification method based on uncertain visual language alignment, characterized in that, Extending multimodal remote sensing image data from a point representation space to a high-dimensional probability distribution representation space, and performing contrastive learning and semantic alignment, includes the following steps: Acquire remote sensing images as training and testing sets; The backbone network uses CLIP's original dual encoders to map the original image and text to a shared point representation space, respectively. To model the image and text features extracted by the backbone network as probability distributions, the uncertainty encoder predicts each embedding vector along two paths: the mean vector and the variance vector. Internally, the uncertainty encoder performs two levels of interaction on the input embedding vectors: interaction at the feature dimension level and interaction at the sequence level. The interaction at the feature dimension level is achieved by a single feedforward layer (FFN), while the interaction at the sequence level is guided by a multi-head self-attention operation. Uncertainty-based contrastive learning pre-training includes: modeling images and text globally as two Gaussian distributions based on mean and variance vectors, respectively, and using the 2-Wasserstein distance between the two Gaussian distributions as a metric; constructing bidirectional cross-entropy loss for image-to-text retrieval and text-to-image retrieval, and graph-guided dynamic Wasserstein loss; weighting these three losses to maximize the image-to-text similarity that should be correctly matched in the entire batch, thereby bringing whole sample pairs closer together and pushing away negative sample pairs; New remote sensing images are acquired and input into the trained model, which then classifies the scenes in the remote sensing images.

2. The remote sensing image scene classification method based on uncertain visual language alignment according to claim 1, characterized in that, The backbone network uses CLIP's original dual encoder to map the original image and text to a shared point representation space, including: The image backbone uses ViT-B / 32 as an image encoder to extract visual hidden states. B indicates the base version, and 32 indicates the batch size. The visual encoder consists of one fully connected layer, one patch embedding layer, a 12-layer transformer with 12 attention heads, and a one-layer feedforward network (FFN). The text backbone extracts the hidden text states through the text encoder. The text encoder uses a 12-layer transformer, and its structure is identical to that of the visual encoder, except for the use of 8 attention heads; among them, and These represent visual and text embedding vectors, respectively. and Represents the global embedding vector. This indicates the number of visual patches and the number of text tokens. Indicates the sequence length. This represents the feature dimension of the embedding vector.

3. The remote sensing image scene classification method based on uncertain visual language alignment according to claim 2, characterized in that, The uncertainty encoder predicts each embedding vector along two paths, including the mean vector. Sum of variance vectors The uncertainty encoder module takes as input the embedding vectors in the representation space of different modal points and outputs the mean vector. The variance vector represents the center position of the embedding vector in the probability distribution representation space. It represents the range or uncertainty of the embedding vector in the probability distribution characterization space.

4. The remote sensing image scene classification method based on uncertain visual language alignment according to claim 3, characterized in that, The uncertain encoder internally performs two levels of interaction on the input embedding vector: interaction at the feature dimension level and interaction at the sequence level, specifically including: The feedforward layer FFN is formed by the first weight. First bias Second weight Second bias The two fully connected layers perform a linear transformation of the features to realize the input features. Dimensions The expansion and recovery of features, where ReLU is used as the activation function, achieves non-linear enhancement of features: Multi-head self-attention operation first considers the hidden state of the input. Perform split processing and project it onto... There are 1 subspace, and the dimension of each subspace is 1. ; Then, in each head, the hidden state is... Perform path splitting Processing the hidden input state of each path Projected matrix Mapped to the Queries in subspaces ,key Sum : Next, self-attention operations are performed in their respective subspaces to obtain the i-th self-attention. : ； Then, to The results of self-attention operations obtained from each subspace are concatenated and then projected through a projection matrix. Restore the dimensions and map them to the output space: Finally, calculate the uncertainty loss: It is an uncertainty loss, used to optimize module parameters. This is the batch size, representing the number of data samples processed. It is the first The mean vector of samples, with an embedding dimension of . , It is the first The original embedding vector of each sample. It is the square of the L2 norm. It is the reduced Planck constant, used to simulate the lower bound of uncertainty. It is the first The variance vector of each sample. It is a wave potential vector, in functional form. Using spatial coordinates, the image is discretized into a pixel grid in the remote sensing image. It is the Laplace operator, used to calculate the second derivative of the wave function, simulate wave propagation, and quantitatively capture dynamic uncertainties. It's an points operation. It is the Shannon entropy of the variance vector. It is a Lagrange multiplier, used to balance the loss term.

5. The remote sensing image scene classification method based on uncertain visual language alignment according to claim 4, characterized in that, Uncertainty-based contrastive learning pre-training includes: The mean vector obtained from the uncertainty encoder module Sum of variance vectors ,use global mean vector variance vector ,and The corresponding global mean vector Sum of variance vectors The image and text are globally modeled as Gaussian distributions, respectively. Gaussian distribution of text ,in It is the covariance matrix; The 2-Wasserstein distance between two Gaussian distributions is used as a metric, and it is defined as follows: Part One The second part measures the offset between the semantic center positions of two modalities. Measure the estimation of uncertainty for two modes; for A negative scaling factor and an offset are introduced to represent the similarity between two modalities. : in, and Indicates based on and Corresponding global vector , , and Perform similarity calculation. It is a negative scaling factor. This is the offset; Constructing a bidirectional cross-entropy loss for image-to-text retrieval and text-to-image retrieval. and : in, Indicates the index of the current image or text. Indicates batch size; Then, using graph-guided dynamic Wasserstein loss and fusing path integrals, semantic evolution is obtained: It uses graph-guided dynamic Wasserstein loss, combined with the overall pre-training objective of the InfoNCE framework. It refers to the batch size. This represents summing over all positive / negative sample pairs. It is an adaptive negative scaling factor. It is a distribution and The mean vector, It is the square of the L2 norm. It is a distribution and The covariance matrix, It is the square of the Frobenius norm. It is along the map path The points, The variance gradient at the path point The value on the scalar model simulates uncertain flow along semantic paths, and the path is approximated using Dijkstra's algorithm. It's the offset. It is a similarity score. ,in To extend the Wasserstein distance, It is the InfoNCE cross-entropy term; Therefore, contrastive learning loss based on uncertainty The construction is as follows: The goal of uncertainty-based contrastive learning pre-training is to maximize the image-text similarity that should be correctly matched in the entire batch, so as to bring whole sample pairs closer together and push negative sample pairs further apart.

Images