Contrast learning based remote sensing image cross-domain simulation translation method and device

Abstract

The application discloses a remote sensing image cross-domain simulation translation method and device based on contrast learning, a medium and equipment. The method comprises the following steps: acquiring a source image to be translated and a target image to be translated; inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model comprises a generator and a discriminator, and the generator comprises an encoder and a decoder; inputting the source image to be translated and the target image to be translated into the encoder to obtain source encoding features and target encoding features respectively; inputting the source encoding features and the target encoding features into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; inputting the source pseudo sample image into the discriminator for identification; wherein the image translation model is obtained by training based on a content contrast loss function, a style contrast loss function and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image. The application can avoid the problem of image detail distortion caused by excessive stylization of the image.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a method, apparatus, storage medium and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning. Background Technology

[0002] Deep learning is increasingly widely used in the interpretation of aerial remote sensing images, and it has achieved remarkable research results through large-scale datasets and complex network structures. However, in practical applications, remote sensing images are affected by various factors such as sensor type, band differences, geographical location, and seasonal variations, exhibiting significant heterogeneity. This heterogeneity leads to differences in the performance of the same object under different spectral conditions, as well as similar performance of different materials under the same spectrum, posing a significant challenge to data-driven learning models.

[0003] Due to variations in the spectral distribution of target images, the performance of trained models often degrades significantly. This is especially true when there are significant changes in spectral bands or sensor types, typically requiring model retraining to adapt to the new data distribution. However, trainable datasets matching the target domain are extremely scarce, particularly when dealing with small areas or specific application scenarios, where this data shortage is even more pronounced. Furthermore, while existing publicly labeled datasets provide valuable resources for aerial image interpretation, their utilization and exploration are not yet comprehensive or in-depth. Existing methods often focus only on local features or specific tasks within the dataset, neglecting the correlations and complementarities between datasets. This limits further improvements in the model's cross-domain adaptation and generalization capabilities.

[0004] In the field of image processing, traditional methods such as histogram matching and MKL, which rely on matching with low statistical values, have been applied to image color alignment, but primarily focus on color-level matching, neglecting the importance of the image's semantic logic. With the continuous advancement of computer vision technology, numerous style transfer and image translation methods have emerged in recent years. These methods aim to transform an input image into an output image of a different style while preserving as much of the input image's content and semantics as possible. Among these, mainstream methods such as CycleGAN and its variants utilize adversarial loss and periodic consistency loss to constrain image style and preserve content, respectively. However, when applied to remote sensing images, these methods often struggle to maintain semantic consistency and may even lead to over-stylization, resulting in distortion of image details. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning, which can avoid the problem of image detail distortion caused by excessive image stylization.

[0006] This application provides a cross-domain simulation translation method for remote sensing images based on contrastive learning, including:

[0007] Obtain the source image to be translated from the source image domain and the target image to be translated from the target image domain;

[0008] The source image to be translated and the target image to be translated are input into an image translation model; wherein, the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0009] The step of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0010] The source image to be translated and the target image to be translated are input into the encoder to obtain source coding features and target coding features, respectively;

[0011] The source coding features and the target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0012] The source pseudo-sample image is input into the discriminator for identification, and the identification result is obtained.

[0013] The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from the source pseudo-sample image and the target pseudo-sample image.

[0014] Furthermore, in the cross-domain simulation translation method for remote sensing images based on contrastive learning, the image translation model further includes a content projection network, and the method further includes:

[0015] The source coding features are input into the content projection network to obtain the feature vector of the source coding features;

[0016] The source pseudo-sample image is input into the content projection network to obtain the first query vector of the source pseudo-sample image;

[0017] The feature vectors of the source image to be translated and the source pseudo-sample image at the same position are matched, and the matched source image to be translated is taken as the first positive sample, and the source images to be translated corresponding to the feature vectors at other positions in the source image to be translated are taken as the first negative samples.

[0018] The content comparison loss function is calculated based on the first positive sample, the first negative sample, and the first query vector;

[0019] The encoder is trained based on the content contrast loss function to constrain the content consistency between the source image to be translated and the source pseudo-sample image.

[0020] Furthermore, in the cross-domain simulation translation method for remote sensing images based on contrastive learning, the image translation model further includes a style projection network, and the method further includes:

[0021] The target encoded features are input into the style projection network to obtain the feature vector of the target encoded features;

[0022] The source pseudo-sample image is input into the style projection network to obtain the second query vector of the source pseudo-sample image;

[0023] The feature vectors of the target image to be translated and the source pseudo-sample image at the same position are matched, and the matched target image to be translated is used as the second positive sample. The source images to be translated corresponding to the feature vectors at other positions in the source image to be translated are used as the second negative samples.

[0024] The style contrast loss function is calculated based on the second positive sample, the second negative sample, and the second query vector;

[0025] The encoder is trained based on the style contrast loss function to constrain the content consistency between the target image to be translated and the source pseudo-sample image.

[0026] Furthermore, the cross-domain simulation translation method for remote sensing images based on contrastive learning further includes:

[0027] Construct a semantic segmentation model;

[0028] The stylized image output by the image translation model is input into the trained semantic segmentation model to obtain the semantic segmentation result.

[0029] Furthermore, in the cross-domain simulation translation method for remote sensing images based on contrastive learning, the joint training process of the semantic segmentation model and the image translation model includes:

[0030] If the images in the source image domain used for training have corresponding training labels, then a first loss function is calculated based on the training labels corresponding to the images in the source image domain and the new stylized image, and iterative training is performed based on the first loss function to obtain a trained semantic segmentation model.

[0031] If the images in the target image domain used for training have corresponding training labels, then a second loss function is calculated based on the adversarial loss function, content contrast loss function, style contrast loss function, identity consistency loss function and semantic loss function, and iterative training is performed based on the second loss function to obtain a trained semantic segmentation model;

[0032] If neither the image in the source image domain used for training nor the image in the target image domain used for training has a corresponding training label, then the semantic segmentation model is removed, and a third loss function is calculated based on the adversarial loss function, the content contrast loss function, and the style contrast loss function. The trained image translation model is obtained by iterative training based on the third loss function.

[0033] Furthermore, in the cross-domain simulation translation method for remote sensing images based on contrastive learning, the calculation of the content contrastive loss function based on the first positive sample, the first negative sample, and the first query vector includes:

[0034] The content comparison loss function is calculated using the first formula, which is:

[0035]

[0036] in, This is the first query vector. As the first positive sample, As the first negative sample, This is the cross-entropy loss function.

[0037] Furthermore, in the cross-domain simulation translation method for remote sensing images based on contrastive learning, the cross-entropy loss function is calculated using a second formula, which is:

[0038]

[0039] in, This is the first query vector. As positive example vectors, It is a negative instance vector.

[0040] This application also provides a remote sensing image cross-domain simulation translation device based on contrastive learning, including:

[0041] The acquisition module is used to acquire the source image to be translated in the source image domain and the target image to be translated in the target image domain;

[0042] A translation module is used to input the source image to be translated and the target image to be translated into an image translation model; wherein, the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0043] The step of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0044] The source image to be translated and the target image to be translated are input into the encoder to obtain source coding features and target coding features, respectively;

[0045] The source coding features and the target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0046] The source pseudo-sample image and the target pseudo-sample image are input into a discriminator for discrimination, and a first discrimination result and a second discrimination result are obtained respectively.

[0047] The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from the source pseudo-sample image and the target pseudo-sample image.

[0048] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described methods for cross-domain simulation translation of remote sensing images based on contrastive learning.

[0049] This application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used in the steps of the cross-domain simulation translation method for remote sensing images based on contrastive learning described in any of the above claims.

[0050] The present application provides a method, apparatus, storage medium, and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning. The present application trains the image translation model through content contrastive loss function and style contrastive loss function, which can avoid the problem of image detail distortion caused by excessive image stylization. It achieves rapid alignment of target domain style features under unsupervised conditions, and the model has a small number of parameters, which greatly shortens the model training time. Attached Figure Description

[0051] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0052] Figure 1 A flowchart of a cross-domain simulation translation method for remote sensing images based on contrastive learning, provided in an embodiment of this application.

[0053] Figure 2 This is a schematic diagram of the image translation model provided in an embodiment of this application.

[0054] Figure 3 This is another structural schematic diagram of the image translation model provided in the embodiments of this application.

[0055] Figure 4 The flowcharts for calculating the content contrast loss function and the style contrast loss function provided in the embodiments of this application are shown.

[0056] Figure 5 A flowchart illustrating the joint training of the semantic segmentation model and the image translation model provided in this application.

[0057] Figure 6 A schematic diagram of the structure of the remote sensing image cross-domain simulation translation device based on contrastive learning provided in the embodiments of this application.

[0058] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0059] Figure 8 Another structural schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Among the commonly used traditional image color matching methods, histogram matching is the most classic and commonly used technique. The specific technical approach can be summarized as follows: (1) Obtain source and target images: Select the source and target images to be matched. (2) Calculate image histograms: Calculate the histograms of the source and target images respectively to obtain the distribution of each pixel value in the image. (3) Calculate cumulative histograms: Normalize the histogram of the source image to obtain the cumulative distribution function (CDF), and calculate the cumulative distribution function of the target image based on the CDF. (4) Calculate matching function: Calculate the matching function through the difference of the cumulative distribution function. This function reflects the correspondence between the pixel values ​​of the source image and the target image. (6) Map source image pixel values: Map each pixel value of the source image using the matching function to obtain the pixel values ​​of the target image. (7) Generate matching results: Combine the mapped pixel values ​​into a new image to complete the color matching from the source image to the target image. This method fully utilizes the histogram distribution information of images, adjusting the color distribution of the source image to match the color distribution of the target image through a mapping function, thereby achieving color matching. However, the drawback of this method is that it only considers color matching and ignores the semantic information of the images, which may lead to semantic inconsistencies in the matched images.

[0062] In the field of deep learning, image translation methods based on CycleGAN and its variants are the most commonly used techniques for cross-domain image translation tasks. The technical process is as follows: (1) Constructing a generator G and a discriminator D: The generator G is responsible for converting the source image A into a fake image A' with the style of the target domain B, and the discriminator D is responsible for distinguishing between the real image B and the fake image A'. (2) Adversarial loss: The generator G is trained to generate realistic fake images, making it difficult for the discriminator D to distinguish between real and fake. The discriminator D is trained to improve its discrimination ability. (3) Cycle consistency loss: Ensure that after image A is generated by G into A', it is converted back into A by another generator G', requiring G and G' to have cycle consistency. (4) Joint training: The generator and discriminator are trained together by adversarial loss and cycle consistency loss. However, CycleGAN and its variants have the following disadvantages in image translation: (1) Over-stylization: Adversarial loss can easily cause the generator G to pay too much attention to the image style, resulting in distortion of the generated image details. (2) Cyclic consistency constraint: It is assumed that there is a one-to-one correspondence between the source domain and the target domain, but in practical applications, this assumption is often too strict. (3) Ignoring semantic information: It focuses mainly on image style matching while ignoring semantic information, which may lead to semantic inconsistency. (4) Style loss function constraint: It requires manual design of style loss function, making it difficult to automatically extract image style features. (5) High training difficulty: The adversarial loss function has the gradient vanishing problem, making the training process more difficult. (6) High computational resource consumption: It requires training the generator and discriminator at the same time, resulting in high computational resource consumption.

[0063] To address the aforementioned issues, embodiments of this application provide a method, apparatus, storage medium, and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning. The cross-domain simulation translation apparatus for remote sensing images based on contrastive learning provided in this application can be integrated into an electronic device, which may be a terminal, server, or other device. The terminal may include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0064] Please see Figure 1 , Figure 1 The flowchart illustrates a cross-domain simulation translation method for remote sensing images based on contrastive learning, provided in this application embodiment. This method, applied in electronic devices, includes the following steps:

[0065] S1, obtain the source image to be translated in the source image domain and the target image to be translated in the target image domain.

[0066] S2, input the source image to be translated and the target image to be translated into the image translation model; wherein, the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder. Figure 2 A schematic diagram of the image translation model provided in the embodiments of this application can be found in [reference needed]. Figure 2 .

[0067] The process of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0068] S21, input the source image to be translated and the target image to be translated into the encoder to obtain the source coding features and the target coding features respectively.

[0069] Specifically, the source image to be translated and target image to be translated The input is fed into the encoder, where and They represent domain and Number of images in the domain.

[0070] In one embodiment, the decoder uses ResNet_9blocks as the backbone network, which consists of nine residual blocks, each containing two or three convolutional layers, as well as batch normalization and ReLU activation functions. The encoder aims to extract high-level features from the input image.

[0071] Each residual block consists of two 3x3 convolutional layers (or a first 1x1 convolutional layer followed by a 3x3 convolutional layer), connected by Batch Normalization (BN) layers and ReLU activation functions. Residual connections are used to add the block's input and output to ensure efficient propagation of information within the network.

[0072] S22, the source coding features and target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0073] Specifically, the decoder is a key component whose main function is to transform the feature maps extracted by the encoder back into image space to generate the target image. The decoder in this application can use two convolutional layers to progressively increase the resolution of the feature maps. Unlike the convolutional layers in the encoder, the convolutional layers in the decoder employ upsampling operations, and a non-linear activation function ReLU layer is linked after each convolutional layer. The final layer of the decoder is the output layer, which is responsible for converting the feature maps generated by the decoder into the final output image. The specific structure of the output layer depends on the requirements of the task.

[0074] S23, the source pseudo-sample image is input into the discriminator for discrimination, and the discrimination result is obtained. The discrimination result is the source pseudo-sample image with the style of the target image domain.

[0075] The discriminator D of this invention can use a multi-scale discriminator to discriminate the generated source pseudo-sample image A', thereby evaluating the realism of the generated image at different scales. This structure is designed to capture the details and global structure of the image at different resolutions, thus improving the quality of the generated image. Discriminator D specifically includes 16 feature extraction convolutional layers and fully connected layers, as well as a multi-scale feature fusion module that fuses feature maps from different scales. This can be achieved through concatenation, addition, or other operations. The fused feature map contains information from multiple scales, helping the discriminator to more accurately evaluate the realism of the generated image.

[0076] The image translation model is trained using a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from source and target pseudo-sample images.

[0077] During training, the multi-scale discriminator is trained together with the generator. The generator is responsible for generating images that are as realistic as possible, while the multi-scale discriminator is responsible for evaluating the realism of the generated images. Through backpropagation and gradient descent, the system can continuously optimize the parameters of the generator and discriminator to improve the quality of the generated images.

[0078] Figure 3Another structural schematic diagram of the image translation model provided in the embodiments of this application is shown below. Figure 3 As shown, the image translation model also includes a content projection network. The cross-domain simulation translation method for remote sensing images based on contrastive learning provided in this application further includes the following steps:

[0079] S31, input the source coding features into the content projection network to obtain the feature vector of the source coding features. Figure 4 The flowcharts for calculating the content contrast loss function and style contrast loss function provided in the embodiments of this application can be used to participate in... Figure 4 .

[0080] Specifically, the encoder The output source-encoded features (which are multi-layer features) are input into the content projection network. In the process, feature vectors of selected layers of the source encoding features are obtained. These feature vectors represent the content features of the source image to be translated at each layer, and can be expressed by the following formula:

[0081]

[0082] in, This indicates the image to be translated. Represents the eigenvector. It is a two-layer multilayer perceptron (MLP) with 256 units in each layer to obtain the projected feature vector.

[0083] S32, input the source pseudo-sample image into the content projection network to obtain the first query vector of the source pseudo-sample image.

[0084] The target of contrastive learning is the query point in the source pseudo-sample image. By mapping the query point to a K-dimensional vector through a content projection network, the first query vector can be obtained. .

[0085] S33, match the feature vectors of the source image to be translated and the source pseudo-sample image at the same position, take the matched source image to be translated as the first positive sample, and take the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as the first negative sample.

[0086] Furthermore, the feature vectors at the same position between the matched image to be translated and the source pseudo-sample image are taken as the first positive example vector, and the feature vectors at other positions in the source image to be translated are taken as the first negative example vector.

[0087] S34, calculate the content comparison loss function based on the first positive sample, the first negative sample, and the first query vector.

[0088] Specifically, the content comparison loss function can be calculated using the first formula, which is:

[0089]

[0090] in, This is the first query vector. As the first positive sample, As the first negative sample, This is the cross-entropy loss function.

[0091] Among them, the cross-entropy loss function The definition of is:

[0092]

[0093] in, The first positive example vector, First negative instance vector, , This is the temperature coefficient.

[0094] In practice, constructing meaningful and clever positive and negative samples is currently a major challenge in contrastive learning. This application transforms the problem of maximizing mutual information into an (N+1) classification problem, where the distance between a query and other examples is determined by temperature. = 0.07 for scaling.

[0095] S35, train the encoder based on the content contrast loss function to constrain the content consistency between the source image to be translated and the source pseudo-sample image.

[0096] Furthermore, an identity consistency loss function is used to maintain the consistency of details between the target image to be translated b and the target pseudo-sample image b', which are generated by the generator G.

[0097] The identity consistency loss is defined as follows, which is the L1 distance between the two identities:

[0098]

[0099] Content contrastive loss primarily aims to ensure the consistency of content between the synthesized and original images. It compares the feature vectors at the same locations in the original and translated images, allowing the model to preserve the content details and texture of the image during translation. By applying multi-layered, block-based contrastive learning to individual images, it can more effectively extract positive and negative examples from a single image, which is more efficient than searching for positive and negative examples in other images across the entire dataset.

[0100] Please continue reading. Figure 3The image translation model also includes a style projection network. To ensure that the model can learn the style of the target image dataset in terms of image stylization constraints, unlike previous methods that adaptively normalize the image mean and variance using AdaIN and calculate Gram loss for style transfer, this invention introduces a style contrast loss to encourage the model to learn the target style from an unsupervised training perspective. Similar to the content contrast loss, this invention also chooses a multi-layer, slice-style contrast loss for the loss function design.

[0101] In one embodiment, the remote sensing image cross-domain simulation translation method based on contrastive learning provided in this application further includes the following steps:

[0102] S41, the target encoded features are input into the style projection network to obtain the feature vector of the target encoded features.

[0103] Specifically, the encoder The output target encoded features (which are multi-layer features) are input into the style projection network. In the process, feature vectors of selected layers are obtained for the target encoding features. These feature vectors represent the content features of the target image to be translated at each layer, and can be expressed by the following formula:

[0104]

[0105] in, This represents the target translated image. Represents the eigenvector. It is a two-layer multilayer perceptron (MLP) with 128 units in each layer to obtain the projected feature vector.

[0106] S42, input the source pseudo-sample image into the style projection network to obtain the second query vector of the source pseudo-sample image.

[0107] S43, match the feature vectors of the target image to be translated and the source pseudo-sample image at the same position, take the matched target image to be translated as the second positive sample, and take the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as the second negative sample.

[0108] Furthermore, the feature vectors at the same position between the matched image to be translated and the source pseudo-sample image are used as the second positive example vector, and the feature vectors at other positions in the source image to be translated are used as the second negative example vector.

[0109] S44 calculates the style contrast loss function based on the second positive sample, the second negative sample, and the second query vector.

[0110]

[0111] in, This is the second positive sample. As the second negative sample, This is the second query vector. , in Set the number of patches per layer to 256. The cross-entropy loss function is calculated in the same way as previously mentioned. Similarly, we will not go into too much detail here.

[0112] S45, an encoder is trained based on the style contrast loss function to constrain the content consistency between the target image to be translated and the source pseudo-sample image.

[0113] This application trains the decoder using a hierarchical image slicing-based content contrast loss function, which maintains detail consistency and avoids the problem of image detail distortion caused by excessive image stylization. By training the decoder using a hierarchical image slicing-based style contrast loss function, it achieves rapid alignment of target domain style features under unsupervised conditions, and the model has a small number of parameters, which greatly shortens the model training time.

[0114] Furthermore, this application integrates a semantic segmentation model into an image translation framework, jointly training the semantic segmentation model and the image translation model. Users can choose whether to retrain according to their needs and obtain semantic segmentation results. Specifically, this includes the following steps:

[0115] S51, Construct a semantic segmentation model;

[0116] S52, input the newly stylized image output by the image translation model into the trained semantic segmentation model to obtain the semantic segmentation result.

[0117] If the images in the source image domain used for training have corresponding training labels, or if the images in the target image domain used for training have corresponding training labels, these labels can be used to retrain or pretrain the semantic segmentation model to ensure semantic consistency of the translated images. If neither is labeled, the semantic segmentation network can be removed, but this will lead to performance degradation. Figure 5 A flowchart illustrating the joint training of the semantic segmentation model and the image translation model provided in this application is shown below. Figure 5 As shown, the joint training process specifically includes:

[0118] S61, if the images in the source image domain used for training have corresponding training labels, then calculate the first loss function based on the training labels corresponding to the images in the source image domain and the new stylized image, and perform iterative training based on the first loss function to obtain the trained semantic segmentation model.

[0119] The first loss function is:

[0120]

[0121] in, It is the one-hot encoding of the real label, which is 1 for a certain category and 0 for the rest. It is the probability of that category predicted by the model.

[0122] S62, if the images in the target image domain used for training have corresponding training labels, then calculate the second loss function based on the adversarial loss function, content contrast loss function, style contrast loss function, identity consistency loss function and semantic loss function, and perform iterative training based on the second loss function to obtain the trained semantic segmentation model;

[0123] The second loss function is:

[0124]

[0125] in, It is an adversarial loss function. It is a style contrast loss function. It is a content contrast loss function. It is the identity consistency loss function. It is the semantic loss function (i.e., the first loss function).

[0126] S63. If neither the image in the source image domain used for training nor the image in the target image domain used for training has a corresponding training label, then the semantic segmentation model is removed, and a third loss function is calculated based on the adversarial loss function, the content contrast loss function, and the style contrast loss function. The trained image translation model is obtained by iterative training based on the third loss function.

[0127] The third loss function is:

[0128]

[0129] in, , , , and Hyperparameters. For example, , , =1, If you need to set it to Then = 0.5, otherwise set to 0.

[0130] This application adds a semantic segmentation model, providing users with the flexibility to choose whether to retrain the semantic segmentation model based on their needs. This design not only serves as a semantic constraint for style transfer but also creates a complementary closed loop, further improving the accuracy and reliability of image translation.

[0131] Based on the method described in the above embodiments, this embodiment will further describe the remote sensing image cross-domain simulation translation device based on contrastive learning. The remote sensing image cross-domain simulation translation device based on contrastive learning can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other devices. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0132] Please see Figure 6 , Figure 6 This application provides a detailed description of a remote sensing image cross-domain simulation translation device based on contrastive learning, which is applied in electronic devices. The device may include:

[0133] The acquisition module is used to acquire the source image to be translated in the source image domain and the target image to be translated in the target image domain;

[0134] The translation module is used to input the source image to be translated and the target image to be translated into the image translation model; the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0135] The process of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0136] The source image to be translated and the target image to be translated are input into the encoder to obtain the source coding features and the target coding features, respectively;

[0137] The source coding features and target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0138] The source pseudo-sample image and the target pseudo-sample image are input into the discriminator for discrimination, and the first discrimination result and the second discrimination result are obtained respectively.

[0139] The image translation model is trained using a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from source and target pseudo-sample images.

[0140] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0141] In addition, embodiments of this application also provide an electronic device, which may be a computer, tablet computer, or other similar device. Figure 7 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 and the memory 402 are electrically connected.

[0142] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device through various interfaces and lines. By running or loading the application program stored in the memory 402 and calling the data stored in the memory 402, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole.

[0143] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to realize various functions:

[0144] Obtain the source image to be translated from the source image domain and the target image to be translated from the target image domain;

[0145] The source image to be translated and the target image to be translated are input into an image translation model; wherein, the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0146] The step of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0147] The source image to be translated and the target image to be translated are input into the encoder to obtain source coding features and target coding features, respectively;

[0148] The source coding features and the target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0149] The source pseudo-sample image is input into the discriminator for identification, and the identification result is obtained.

[0150] The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from the source pseudo-sample image and the target pseudo-sample image.

[0151] This electronic device can implement the steps in any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of this application. Therefore, it can achieve the beneficial effects that any remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of this invention can achieve, as detailed in the preceding embodiments, which will not be repeated here.

[0152] Figure 8 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the cross-domain simulation translation method for remote sensing images based on contrastive learning provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0153] RF circuit 510 is used to receive and transmit electromagnetic waves, converting electromagnetic waves into electrical signals and vice versa, thereby enabling communication with communication networks or other devices. RF circuit 510 may include various existing circuit elements used to perform these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, subscriber identity modules (SIM cards), memory, etc. RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices via wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). The aforementioned wireless networks may use various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messages, and any other suitable communication protocols, including those that have not yet been developed.

[0154] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, such as taking pictures with the front-facing camera, processing the captured images, and switching the display colors of the content displayed on the screen. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0155] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control.

[0156] Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0157] Audio circuitry 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuitry 560 converts received audio data into electrical signals and transmits them to speaker 561, where speaker 561 converts them into sound signals for output. Conversely, microphone 562 converts collected sound signals into electrical signals, which are then received by audio circuitry 560, converted back into audio data, and processed by processor 580. The audio data is then transmitted via RF circuitry 510 to, for example, another terminal, or output to memory 520 for further processing. Audio circuitry 560 may also include an earphone jack to facilitate communication between external headphones and electronic device 500.

[0158] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0159] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0160] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0161] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations:

[0162] Obtain the source image to be translated from the source image domain and the target image to be translated from the target image domain;

[0163] The source image to be translated and the target image to be translated are input into an image translation model; wherein, the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder;

[0164] The step of inputting the source image to be translated and the target image to be translated into the image translation model includes:

[0165] The source image to be translated and the target image to be translated are input into the encoder to obtain source coding features and target coding features, respectively;

[0166] The source coding features and the target coding features are input into the decoder to obtain the source pseudo-sample image and the target pseudo-sample image, respectively.

[0167] The source pseudo-sample image is input into the discriminator for identification, and the identification result is obtained.

[0168] The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed from the source pseudo-sample image and the target pseudo-sample image.

[0169] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0170] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided by the present invention.

[0171] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0172] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present invention, the beneficial effects that any remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0173] The foregoing has provided a detailed description of a method, apparatus, storage medium, and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A contrastive learning based remote sensing image cross-domain simulation translation method, characterized in that, The method comprises: obtaining a source to-be-translated image in a source image domain and a target to-be-translated image in a target image domain; inputting the source to-be-translated image and the target to-be-translated image into an image translation model; wherein the image translation model comprises a generator and a discriminator, and the generator comprises an encoder and a decoder; wherein the inputting the source to-be-translated image and the target to-be-translated image into the image translation model comprises: inputting the source to-be-translated image and the target to-be-translated image into the encoder to obtain source encoded features and target encoded features, respectively; inputting the source encoded features and the target encoded features into the decoder to obtain source pseudo sample images and target pseudo sample images, respectively; inputting the source pseudo sample images into the discriminator for discrimination to obtain a discrimination result; wherein the image translation model is obtained by training based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample images and the target pseudo sample images; the image translation model further comprises a style projection network, and the method further comprises: inputting the target encoded features into the style projection network to obtain a feature vector of the target encoded features; inputting the source pseudo sample images into the style projection network to obtain second query vectors of the source pseudo sample images; matching the feature vectors of the target to-be-translated image and the source pseudo sample images at the same position, taking the matched target to-be-translated image as a second positive sample, and taking the source to-be-translated images corresponding to the feature vectors of the source to-be-translated images at other positions as second negative samples; calculating a style contrast loss function based on the second positive sample, the second negative sample, and the second query vectors; and training the encoder based on the style contrast loss function to constrain the style consistency of the target to-be-translated image and the source pseudo sample images.

2. The contrastive learning based remote sensing image cross-domain simulation translation method according to claim 1, characterized in that, the image translation model further comprises a content projection network, and the method further comprises: inputting the source encoded features into the content projection network to obtain a feature vector of the source encoded features; inputting the source pseudo sample images into the content projection network to obtain first query vectors of the source pseudo sample images; matching the feature vectors of the source to-be-translated image and the source pseudo sample images at the same position, taking the matched source to-be-translated image as a first positive sample, and taking the source to-be-translated images corresponding to the feature vectors of the source to-be-translated images at other positions as first negative samples; calculating a content contrast loss function based on the first positive sample, the first negative sample, and the first query vectors; training the encoder based on the content contrast loss function to constrain the content consistency of the source to-be-translated image and the source pseudo sample images.

3. The contrastive learning based remote sensing image cross-domain simulation translation method according to claim 1, characterized in that, The method further comprises: constructing a semantic segmentation model; inputting a new stylized image output by the image translation model into the trained semantic segmentation model to obtain a semantic segmentation result.

4. The contrastive learning based remote sensing image cross-domain simulation translation method according to claim 3, characterized in that, The joint training process of the semantic segmentation model and the image translation model comprises: If the images in the source image domain used for training have corresponding training labels, a first loss function is calculated based on the corresponding training labels of the images in the source image domain and the new stylized images, and the trained semantic segmentation model is obtained through iterative training based on the first loss function; If the images in the target image domain used for training have corresponding training labels, a second loss function is calculated based on the adversarial loss function, the content contrast loss function, the style contrast loss function, the identity consistency loss function and the semantic loss function, and the trained semantic segmentation model is obtained through iterative training based on the second loss function; If neither the images in the source image domain used for training nor the images in the target image domain used for training have corresponding training labels, the semantic segmentation model is removed, a third loss function is calculated based on the adversarial loss function, the content contrast loss function and the style contrast loss function, and the trained image translation model is obtained through iterative training based on the third loss function.

5. The contrastive learning based remote sensing image cross-domain simulation translation method according to claim 2, characterized in that, The content contrast loss function is calculated based on the first positive sample, the first negative sample and the first query vector, and the content contrast loss function includes: The content contrast loss function is calculated by a first formula, and the first formula is: wherein, is a first query vector, is a first positive sample, is a first negative sample, is a cross-entropy loss function.

6. The contrastive learning based remote sensing image cross-domain simulation translation method according to claim 5, characterized in that, The cross-entropy loss function is calculated by a second formula, and the second formula is: wherein, is a first query vector, is a positive example vector, is a negative example vector.

7. A contrastive learning based remote sensing image cross-domain simulation translation device for implementing the contrastive learning based remote sensing image cross-domain simulation translation method according to claim 1, characterized in that, It includes: An acquisition module is configured to acquire a source image in a source image domain and a target image in a target image domain; A translation module is configured to input the source image and the target image into an image translation model, wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The inputting of the source image and the target image into the image translation model includes: The source image and the target image are input into the encoder to obtain source encoding features and target encoding features, respectively; The source encoding features and the target encoding features are input into the decoder to obtain source pseudo sample images and target pseudo sample images, respectively; The source pseudo sample images and the target pseudo sample images are input into the discriminator for identification to obtain first identification results and second identification results, respectively; The image translation model is trained based on the content contrast loss function, the style contrast loss function and an adversarial loss function constructed based on the source pseudo sample images and the target pseudo sample images.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the contrast learning based remote sensing image cross-domain simulation translation method of any one of claims 1 to 6.

9. An electronic device, comprising: The computer readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the contrast learning based remote sensing image cross-domain simulation translation method of any one of claims 1 to 6.

Images