Remote sensing image cross-domain simulation translation method and device based on comparative learning

Through the cross-domain simulation translation method of remote sensing images based on contrast learning, combined with content and style comparison loss function, the problem of insufficient ability of remote sensing images to adapt and generalize across the domain is solved, and image detail fidelity and style consistency are achieved, which shortens the Model training time.

CN118917997AActive Publication Date: 2024-11-08WUHAN UNIV +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411026449.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-08
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing remote sensing image processing methods have shortcomings in cross-domain adaptation and generalization capabilities, especially when bands or sensors change, model performance will significantly degrade, and image details are prone to distortion, and excessive stylization problems are serious.

Method used

The cross-domain simulation translation method of remote sensing images based on contrast learning is adopted, and the image translation model is trained through content comparison loss function and style comparison loss function, and combined with content projection network and style projection network to ensure content consistency and style consistency during image translation process. And jointly trained through semantic segmentation models to avoid excessive stylization of images.

Benefits of technology

It realizes rapid alignment of target domain style features under unsupervised conditions, avoids image detail distortion, shortens model training time, and improves the cross-domain adaptability and generalization ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917997A_ABST
    Figure CN118917997A_ABST
Patent Text Reader

Abstract

The invention discloses a remote sensing image cross-domain simulation translation method and device based on comparative learning, a medium and equipment. The method comprises the following steps: acquiring a source to-be-translated image and a target to-be-translated image; inputting the source to-be-translated image and the target to-be-translated image into an image translation model; wherein the image translation model comprises a generator and a discriminator, and the generator comprises an encoder and a decoder; inputting the source to-be-translated image and the target to-be-translated image into an encoder to respectively obtain a source coding feature and a target coding feature; inputting the source pseudo sample image and the target pseudo sample image into a decoder to respectively obtain a source pseudo sample image and a target pseudo sample image; inputting the source pseudo sample image into a discriminator for discriminating; wherein the image translation model is obtained by training based on a content comparison loss function, a style comparison loss function and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image. According to the invention, the problem of image detail distortion caused by excessive stylization of the image can be avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a remote sensing image cross-domain simulation translation method, device, storage medium and electronic device based on contrast learning. Background Art

[0002] Deep learning is increasingly used in the field of aerial remote sensing image interpretation. It has achieved remarkable research results through large-scale data sets and complex network structures. However, in practical applications, remote sensing images are affected by multiple factors such as sensor type, band differences, geographical location, and seasonal changes, showing significant heterogeneity. This heterogeneity leads to differences in the performance of the same object under different spectral conditions, as well as similar performance of different materials under the same spectrum, which poses a huge challenge to models that rely on data learning. Due to changes in the spectral distribution of the target image, the performance of the trained model often degrades significantly. In particular, when there are significant changes in the band or sensor, the model usually needs to be retrained to adapt to the new data distribution. However, trainable datasets matching the target domain are extremely scarce, especially when dealing with small areas or specific application scenarios. This problem of insufficient data is more prominent. In addition, although existing publicly labeled datasets provide valuable resources for aerial image interpretation, their utilization and exploration are not comprehensive and in-depth enough. Existing methods often only focus on local features or specific tasks of the dataset, while ignoring the correlation and complementarity between datasets. This limits the further improvement of the model in cross-domain adaptation and generalization capabilities.

[0003] In the field of image processing, traditional histogram matching, MKL and other methods based on matching with lower statistical values, although they have been applied to image color alignment, mainly focus on matching at the color level and ignore the importance of image semantic logic. With the continuous advancement of computer vision technology, many style transfer and image translation methods have emerged in recent years. These methods aim to convert an input image into an output image of another style while trying to maintain the content and semantics of the input image. Among them, mainstream methods such as CycleGAN and its variants use adversarial loss and cycle consistency loss to constrain image style and retain content respectively. However, when these methods are applied to remote sensing images, they often find it difficult to maintain semantic consistency, and may even suffer from over-stylization, resulting in distortion of image details. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning, which can avoid the problem of distortion of image details caused by excessive stylization of images.

[0005] The present application embodiment provides a remote sensing image cross-domain simulation translation method based on contrastive learning, including: Obtaining a source image to be translated in a source image domain and a target image to be translated in a target image domain; Inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image into a discriminator for identification to obtain an identification result; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0006] Furthermore, in the remote sensing image cross-domain simulation translation method based on contrastive learning, the image translation model further includes a content projection network, and the method further includes: Inputting the source coding feature into the content projection network to obtain a feature vector of the source coding feature; Inputting the source pseudo sample image into the content projection network to obtain a first query vector of the source pseudo sample image; Matching feature vectors at the same position of the source image to be translated and the source pseudo sample image, taking the matched source image to be translated as a first positive sample, and taking the source image to be translated corresponding to feature vectors at other positions in the source image to be translated as a first negative sample; Calculating a content contrast loss function based on the first positive sample, the first negative sample, and the first query vector; The encoder is trained based on the content contrast loss function to constrain the content consistency between the source image to be translated and the source pseudo sample image.

[0007] Furthermore, in the remote sensing image cross-domain simulation translation method based on contrastive learning, the image translation model further includes a style projection network, and the method further includes: Inputting the target encoding feature into the style projection network to obtain a feature vector of the target encoding feature; Inputting the source pseudo sample image into the style projection network to obtain a second query vector of the source pseudo sample image; Matching the feature vectors of the target image to be translated and the source pseudo sample image at the same position, taking the matched target image to be translated as a second positive sample, and taking the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as a second negative sample; Calculating a style contrast loss function based on the second positive sample, the second negative sample, and the second query vector; The encoder is trained based on the style contrast loss function to constrain the content consistency between the target image to be translated and the source pseudo sample image.

[0008] Furthermore, the remote sensing image cross-domain simulation translation method based on contrastive learning further includes: Build a semantic segmentation model; The new stylized image output by the image translation model is input into the trained semantic segmentation model to obtain a semantic segmentation result.

[0009] Furthermore, in the remote sensing image cross-domain simulation translation method based on contrastive learning, the joint training process of the semantic segmentation model and the image translation model includes: If the image in the source image domain used for training has a corresponding training label, a first loss function is calculated based on the training label corresponding to the image in the source image domain and the new stylized image, and iterative training is performed based on the first loss function to obtain a trained semantic segmentation model; If the image in the target image domain used for training has a corresponding training label, a second loss function is calculated based on the adversarial loss function, the content contrast loss function, the style contrast loss function, the identity consistency loss function, and the semantic loss function, and iterative training is performed based on the second loss function to obtain a trained semantic segmentation model; If the images in the source image domain used for training and the images in the target image domain used for training have no corresponding training labels, the semantic segmentation model is removed, and a third loss function is calculated based on the adversarial loss function, the content contrast loss function and the style contrast loss function, and iterative training is performed based on the third loss function to obtain a trained image translation model. Furthermore, the remote sensing image cross-domain simulation translation method based on contrastive learning, wherein the content contrast loss function is calculated based on the first positive sample, the first negative sample, and the first query vector, comprises: The content contrast loss function is calculated by the first formula, which is:

[0010] in, is the first query vector, is the first positive sample, is the first negative sample, is the cross entropy loss function.

[0011] Furthermore, in the remote sensing image cross-domain simulation translation method based on contrastive learning, the cross entropy loss function is calculated by a second formula, and the second formula is:

[0012] in, is the first query vector, is the positive example vector, is the negative example vector.

[0013] The embodiment of the present application also provides a remote sensing image cross-domain simulation translation device based on contrastive learning, comprising: An acquisition module, used for acquiring a source image to be translated in a source image domain and a target image to be translated in a target image domain; A translation module, used for inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image and the target pseudo sample image into a discriminator for identification, and obtaining a first identification result and a second identification result respectively; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0014] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor to execute any of the above-mentioned remote sensing image cross-domain simulation translation methods based on contrastive learning.

[0015] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned methods for cross-domain simulation translation of remote sensing images based on contrast learning.

[0016] The present application provides a method, device, storage medium and electronic device for cross-domain simulation translation of remote sensing images based on contrastive learning. The present application trains an image translation model through a content contrast loss function and a style contrast loss function, which can avoid the problem of image detail distortion caused by excessive stylization of the image, and realizes rapid alignment of the target domain style features under unsupervised conditions. In addition, the model has a small number of parameters, which greatly shortens the model training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0018] Figure 1 A flowchart of a remote sensing image cross-domain simulation translation method based on contrastive learning provided in an embodiment of the present application.

[0019] Figure 2 A schematic diagram of the structure of the image translation model provided in an embodiment of the present application.

[0020] Figure 3 Another structural schematic diagram of the image translation model provided in an embodiment of the present application.

[0021] Figure 4 This is a calculation flow chart of the content contrast loss function and the style contrast loss function provided in the embodiments of the present application.

[0022] Figure 5 A flowchart of the joint training of the semantic segmentation model and the image translation model provided in this application.

[0023] Figure 6 A schematic diagram of the structure of a remote sensing image cross-domain simulation translation device based on contrastive learning provided in an embodiment of the present application.

[0024] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] Figure 8 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0027] Among the commonly used traditional image color matching methods, the histogram matching technology process is the most classic and commonly used. The specific technical ideas can be summarized as follows: (1) Obtain the source image and the target image: Select the source image and the target image that need to be color matched. (2) Calculate the image histogram: Calculate the histogram of the source image and the target image respectively to obtain the distribution of each pixel value in the image. (3) Calculate the cumulative histogram: Normalize the histogram of the source image to obtain the cumulative distribution function (CDF), and calculate the cumulative distribution function of the target image based on the cumulative distribution function. (4) Calculate the matching function: Calculate the matching function through the difference of the cumulative distribution function. The function reflects the corresponding relationship between the pixel value of the source image and the target image. (6) Map the pixel value of the source image: Use the matching function to map each pixel value of the source image to obtain the pixel value of the target image. (7) Generate the matching result: Combine the mapped pixel values ​​into a new image to complete the color matching from the source image to the target image. This method makes full use of the histogram distribution information of the image and adjusts the color distribution of the source image to the color distribution of the target image through a mapping function, thereby achieving color matching. However, the disadvantage of this method is that it only considers color matching and ignores the semantic information of the image, which may lead to semantic inconsistency of the image after matching.

[0028] In the field of deep learning, the image translation method based on CycleGAN and its variants is the most commonly used technology for cross-domain image translation tasks. Its technical process is as follows: (1) Constructing the generator G and the discriminator D: The generator G is responsible for converting the source image A into a fake image A' with the style of the target domain B, and the discriminator D is responsible for distinguishing between the real image B and the fake image A'. (2) Adversarial loss: Train the generator G to generate realistic fake images, making it difficult for the discriminator D to distinguish between the real and the fake. Training the discriminator D improves its discrimination ability. (3) Cycle consistency loss: Ensure that after image A is generated by G to A', it is converted back to A through another generator G', requiring G and G' to have cycle consistency. (4) Joint training: The generator and the discriminator are jointly trained through adversarial loss and cycle consistency loss. However, CycleGAN and its variants have the following disadvantages in image translation: (1) Over-stylization: The adversarial loss can easily cause the generator G to pay too much attention to the image style, resulting in distortion of the details of the generated image. (2) Cycle consistency limitation: It assumes that there is a one-to-one correspondence between the source domain and the target domain, but in practical applications, this assumption is often too strict. (3) Ignoring semantic information: It focuses on image style matching while ignoring semantic information, which may lead to semantic inconsistency. (4) Style loss function limitation: It is necessary to manually design the style loss function, and it is difficult to automatically extract image style features. (5) Difficulty in training: The adversarial loss function has the problem of gradient vanishing, and the training process is more difficult. (6) High consumption of computing resources: It is necessary to train the generator and the discriminator at the same time, which consumes a lot of computing resources.

[0029] To solve the above problems, the embodiments of the present application provide a remote sensing image cross-domain simulation translation method, device, storage medium and electronic device based on contrastive learning. The remote sensing image cross-domain simulation translation device based on contrastive learning provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0030] See also Figure 1 , Figure 1 A flowchart of a remote sensing image cross-domain simulation translation method based on contrastive learning provided in an embodiment of the present application, which is applied in an electronic device, and the remote sensing image cross-domain simulation translation method based on contrastive learning includes the following steps: S1, obtaining a source image to be translated in a source image domain and a target image to be translated in a target image domain.

[0031] S2, inputting the source image to be translated and the target image to be translated into the image translation model; wherein the image translation model includes a generator and a discriminator, the generator includes an encoder and a decoder, Figure 2For a structural diagram of the image translation model provided in the embodiment of the present application, please refer to Figure 2 .

[0032] The step of inputting the source image to be translated and the target image to be translated into the image translation model includes: S21, inputting the source image to be translated and the target image to be translated into an encoder to obtain source coding features and target coding features respectively.

[0033] Specifically, the source image to be translated and the target image to be translated Input to the encoder, where and Respectively Domain and The number of images in the domain.

[0034] In one embodiment, the decoder uses Resnet_9blocks as the backbone network, which consists of 9 residual blocks, each of which contains two or three convolutional layers, as well as batch normalization and ReLU activation functions. The purpose of the encoder is to extract high-level features from the input image.

[0035] Each residual block consists of two 3x3 convolutional layers (or the first is a 1x1 convolutional layer and the second is a 3x3 convolutional layer), and the convolutional layers are connected through a BN layer and a ReLU activation function. The residual connection is used to add the input and output of the block to ensure the effective propagation of information in the network.

[0036] S22, inputting the source coding features and the target coding features into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively.

[0037] Specifically, the decoder is a key component whose main function is to convert the feature map extracted by the encoder back to the image space to generate the target image. The decoder of the present application can use 2 convolutional layers to gradually increase the resolution of the feature map. Unlike the convolutional layer in the encoder, the convolutional layer in the decoder uses an upsampling operation and links a nonlinear activation function ReLU layer after each convolutional layer. The last layer of the decoder is the output layer, which is responsible for converting the feature map generated by the decoder into the final output image. The specific structure of the output layer depends on the requirements of the task.

[0038] S23, inputting the source pseudo sample image into the discriminator for identification, and obtaining an identification result. The identification result is the source pseudo sample image with the style of the target image domain.

[0039] The discriminator D of the present invention can use a multi-scale discriminator to discriminate the generated source pseudo sample image A', which is used to evaluate the authenticity of the generated image at different scales. The design of this structure is intended to capture the details and global structure of the image at different resolutions, thereby improving the quality of the generated image. The discriminator D specifically includes 16 layers of feature extraction convolutional layers and fully connected layers, as well as a multi-scale feature fusion module, and feature maps of different scales are fused. This can be achieved by splicing, addition or other operations. The fused feature map contains information at multiple scales, which helps the discriminator to more accurately evaluate the authenticity of the generated image.

[0040] Among them, the image translation model is trained based on the content contrast loss function, the style contrast loss function, and the adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0041] During the training process, the multi-scale discriminator is trained together with the generator. The generator is responsible for generating images that are as realistic as possible, while the multi-scale discriminator is responsible for evaluating the authenticity of the generated images. Through the back-propagation algorithm and the gradient descent method, the system can continuously optimize the parameters of the generator and the discriminator to improve the quality of the generated images.

[0042] Figure 3 Another structural diagram of the image translation model provided in the embodiment of the present application is as follows: Figure 3 As shown, the image translation model also includes a content projection network. The remote sensing image cross-domain simulation translation method based on contrastive learning provided by the present application also includes the following steps: S31, input the source encoding feature into the content projection network to obtain the feature vector of the source encoding feature. Figure 4 The calculation flow chart of the content contrast loss function and the style contrast loss function provided in the embodiment of the present application can be participated in Figure 4 .

[0043] Specifically, the encoder The output source encoding features (multi-layer features) are input into the content projection network In the above equation, we get the feature vectors of each selected layer 𝐿 of the source coding feature. These feature vectors represent the content features of the source image to be translated at each level and can be expressed by the following formula:

[0044] in, Represents the image to be translated. represents the feature vector, It is a two-layer multi-layer perceptron (MLP) with 256 units in each layer to obtain the projected feature vector.

[0045] S32, inputting the source pseudo sample image into the content projection network to obtain a first query vector of the source pseudo sample image.

[0046] The goal of contrastive learning is to obtain the query point in the source pseudo sample image. The query point is mapped to a K-dimensional vector through a content projection network to obtain the first query vector .

[0047] S33, matching feature vectors at the same position of the source image to be translated and the source pseudo sample image, taking the matched source image to be translated as the first positive sample, and taking the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as the first negative sample.

[0048] Furthermore, the feature vectors at the same position of the matched image to be translated and the source pseudo sample image are taken as the first positive example vector, and the feature vectors at other positions in the source image to be translated are taken as the first negative example vector.

[0049] S34, calculating a content contrast loss function based on the first positive sample, the first negative sample, and the first query vector.

[0050] Specifically, the content contrast loss function can be calculated by the first formula, which is:

[0051] in, is the first query vector, is the first positive sample, is the first negative sample, is the cross entropy loss function.

[0052] Among them, the cross entropy loss function is defined as:

[0053] in, is the first positive example vector, The first negative example vector, , is the temperature coefficient.

[0054] In practice, constructing meaningful and clever positive and negative samples is the current difficulty of contrastive learning. This application transforms the problem of maximizing mutual information into an (N+1) classification problem, where the distance between the query and other examples is expressed by temperature. = 0.07 for scaling.

[0055] S35, training the encoder based on the content contrast loss function to constrain the content consistency between the source image to be translated and the source pseudo sample image.

[0056] Furthermore, an identity consistency loss function is used to maintain the detail consistency between the target to-be-translated image b and the target pseudo sample image b', which are generated by the generator G.

[0057] The identity consistency loss is defined as follows, which is the L1 distance between the two:

[0058] The content contrast loss is mainly to ensure the consistency of the content of the synthesized image and the original image. It compares the feature vectors of the original image and the translated image at the same position, so that the model can keep the content details and texture of the image unchanged during the translation process. By applying multi-layer, block-wise contrastive learning on a single image itself, positive and negative examples can be extracted more effectively from a single image, which is more effective than looking for positive and negative examples in other images of the entire dataset.

[0059] Please continue reading Figure 3 The image translation model also includes a style projection network. In order to ensure that the model can learn the style of the target image dataset in terms of image stylization constraints, different from the previous style transfer by adaptively normalizing the image mean and variance through AdaIN and calculating the Gram loss, the style contrast loss is introduced to encourage the model to learn the target style from the perspective of unsupervised training. Similar to the content contrast loss, the present invention also selects multi-layer, slice-based contrast loss to design the loss function.

[0060] In one embodiment, the remote sensing image cross-domain simulation translation method based on contrastive learning provided by the present application further includes the following steps: S41, input the target encoding feature into the style projection network to obtain a feature vector of the target encoding feature.

[0061] Specifically, the encoder The output target encoding features (multi-layer features) are input into the style projection network In the above equation, we get the feature vectors of each selected layer 𝐿 of the target encoding feature. These feature vectors represent the content features of the target image to be translated at each level and can be expressed by the following formula:

[0062] in, represents the target translation image, represents the feature vector, It is a two-layer multilayer perceptron (MLP) with 128 units in each layer to obtain the projected feature vector.

[0063] S42, inputting the source pseudo sample image into the style projection network to obtain a second query vector of the source pseudo sample image.

[0064] S43, matching the feature vectors of the target image to be translated and the source pseudo sample image at the same position, taking the matched target image to be translated as the second positive sample, and taking the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as the second negative sample.

[0065] Furthermore, the feature vectors at the same position of the matched image to be translated and the source pseudo sample image are taken as the second positive example vector, and the feature vectors at other positions in the source image to be translated are taken as the second negative example vector.

[0066] S44, calculating a style contrast loss function based on the second positive sample, the second negative sample, and the second query vector.

[0067]

[0068] in, is the second positive sample, is the second negative sample, is the second query vector, , in The number of patches per layer is set to 256. is the cross entropy loss function, calculated in the same way as above Similarly, no further description is given here.

[0069] S45, training the encoder based on the style contrast loss function to constrain the content consistency between the target image to be translated and the source pseudo sample image.

[0070] The present application trains the decoder through a layered image slicing content contrast loss function to maintain detail consistency and avoid the problem of image detail distortion caused by excessive image stylization. By training the decoder through a layered image slicing style contrast loss function, rapid alignment of target domain style features under unsupervised conditions is achieved, and the model parameters are small, which greatly shortens the model training time.

[0071] Furthermore, the present application integrates the semantic segmentation model into the image translation framework, and jointly trains the semantic segmentation model and the image translation model. Users can choose whether to retrain according to their own needs and obtain the semantic segmentation results. Specifically, the following steps are included: S51, build semantic segmentation model; S52, inputting the new stylized image output by the image translation model into the trained semantic segmentation model to obtain a semantic segmentation result.

[0072] If the images in the source image domain used for training have corresponding training labels or the images in the target image domain used for training have corresponding training labels, these labels can be used to retrain or pre-train the semantic segmentation model to ensure the semantic consistency of the translated images. If both lack labels, the semantic segmentation network can be removed, but this will result in a decrease in performance. Figure 5 The flowchart of the joint training of the semantic segmentation model and the image translation model provided in this application is as follows: Figure 5 As shown in Figure 2, the joint training process specifically includes: S61: If the image in the source image domain used for training has a corresponding training label, a first loss function is calculated based on the training label corresponding to the image in the source image domain and the new stylized image, and iterative training is performed based on the first loss function to obtain a trained semantic segmentation model.

[0073] Among them, the first loss function is:

[0074] in, It is a one-hot encoding of the true label, which is 1 for one category and 0 for the rest. is the probability of the class predicted by the model.

[0075] S62, if the image in the target image domain for training has a corresponding training label, a second loss function is calculated based on the adversarial loss function, the content contrast loss function, the style contrast loss function, the identity consistency loss function, and the semantic loss function, and iterative training is performed based on the second loss function to obtain a trained semantic segmentation model; Among them, the second loss function is:

[0076] in, is the adversarial loss function, is the style contrast loss function, is the content contrast loss function, is the identity consistency loss function, is the semantic loss function (i.e. the first loss function).

[0077] S63: If the images in the source image domain used for training and the images in the target image domain used for training have no corresponding training labels, the semantic segmentation model is removed, and a third loss function is calculated based on the adversarial loss function, the content contrast loss function, and the style contrast loss function. Iterative training is performed based on the third loss function to obtain a trained image translation model. Among them, the third loss function is:

[0078] in, , , , and Hyperparameters. For example, , , =1, If necessary, set to , then = 0.5, otherwise set to 0.

[0079] This application adds a semantic segmentation model, providing users with the flexibility to choose whether to retrain the semantic segmentation model according to their needs. This design not only serves as a semantic constraint for style transfer, but also creates a complementary closed loop, further improving the accuracy and reliability of image translation.

[0080] According to the method described in the above embodiments, this embodiment will be further described from the perspective of a cross-domain simulation translation device for remote sensing images based on contrastive learning. The cross-domain simulation translation device for remote sensing images based on contrastive learning can be implemented as an independent entity or integrated in an electronic device, which can be a terminal, server or other device, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0081] See also Figure 6 , Figure 6 The present invention specifically describes a remote sensing image cross-domain simulation translation device based on contrastive learning provided in an embodiment of the present invention, which is applied to an electronic device. The remote sensing image cross-domain simulation translation device based on contrastive learning may include: An acquisition module, used for acquiring a source image to be translated in a source image domain and a target image to be translated in a target image domain; A translation module, used for inputting a source image to be translated and a target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into the image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source encoding features and target encoding features respectively; Input the source coding features and the target coding features into the decoder to obtain the source pseudo sample image and the target pseudo sample image respectively; Inputting the source pseudo sample image and the target pseudo sample image into the discriminator for identification, and obtaining a first identification result and a second identification result respectively; Among them, the image translation model is trained based on the content contrast loss function, the style contrast loss function, and the adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0082] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0083] In addition, the embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Figure 7 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0084] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0085] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions: Obtaining a source image to be translated in a source image domain and a target image to be translated in a target image domain; Inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image into a discriminator for identification to obtain an identification result; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0086] The electronic device can implement the steps in any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present invention. For details, please see the previous embodiments and will not be repeated here.

[0087] Figure 8 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the remote sensing image cross-domain simulation translation method based on contrast learning provided in the above embodiment. The electronic device 500 can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0088] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0089] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0090] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0091] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0092] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0093] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0094] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0095] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: Obtaining a source image to be translated in a source image domain and a target image to be translated in a target image domain; Inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image into a discriminator for identification to obtain an identification result; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

[0096] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0097] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps of any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided in an embodiment of the present invention.

[0098] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0099] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present invention, the beneficial effects that can be achieved by any remote sensing image cross-domain simulation translation method based on contrastive learning provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0100] The above is a detailed introduction to the cross-domain simulation translation method, device, storage medium and electronic device of remote sensing images based on contrast learning provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A remote sensing image cross-domain simulation translation method based on contrastive learning, characterized in that: The method comprises: Obtaining a source image to be translated in a source image domain and a target image to be translated in a target image domain; Inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image into a discriminator for identification to obtain an identification result; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

2. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 1 is characterized in that: The image translation model further includes a content projection network, and the method further includes: Inputting the source coding feature into the content projection network to obtain a feature vector of the source coding feature; Inputting the source pseudo sample image into the content projection network to obtain a first query vector of the source pseudo sample image; Matching feature vectors at the same position of the source image to be translated and the source pseudo sample image, taking the matched source image to be translated as a first positive sample, and taking the source image to be translated corresponding to feature vectors at other positions in the source image to be translated as a first negative sample; Calculating a content contrast loss function based on the first positive sample, the first negative sample, and the first query vector; The encoder is trained based on the content contrast loss function to constrain the content consistency between the source image to be translated and the source pseudo sample image.

3. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 1 is characterized in that: The image translation model further includes a style projection network, and the method further includes: Inputting the target encoding feature into the style projection network to obtain a feature vector of the target encoding feature; Inputting the source pseudo sample image into the style projection network to obtain a second query vector of the source pseudo sample image; Matching the feature vectors of the target image to be translated and the source pseudo sample image at the same position, taking the matched target image to be translated as a second positive sample, and taking the source image to be translated corresponding to the feature vectors at other positions in the source image to be translated as a second negative sample; Calculating a style contrast loss function based on the second positive sample, the second negative sample, and the second query vector; The encoder is trained based on the style contrast loss function to constrain the content consistency between the target image to be translated and the source pseudo sample image.

4. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 1 is characterized in that: The method further comprises: Build a semantic segmentation model; The new stylized image output by the image translation model is input into the trained semantic segmentation model to obtain a semantic segmentation result.

5. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 4 is characterized in that: The joint training process of the semantic segmentation model and the image translation model includes: If the image in the source image domain used for training has a corresponding training label, a first loss function is calculated based on the training label corresponding to the image in the source image domain and the new stylized image, and iterative training is performed based on the first loss function to obtain a trained semantic segmentation model; If the image in the target image domain used for training has a corresponding training label, a second loss function is calculated based on the adversarial loss function, the content contrast loss function, the style contrast loss function, the identity consistency loss function, and the semantic loss function, and iterative training is performed based on the second loss function to obtain a trained semantic segmentation model; If the images in the source image domain used for training and the images in the target image domain used for training have no corresponding training labels, the semantic segmentation model is removed, and a third loss function is calculated based on the adversarial loss function, the content contrast loss function and the style contrast loss function, and iterative training is performed based on the third loss function to obtain a trained image translation model.

6. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 2 is characterized in that: The calculating the content contrast loss function based on the first positive sample, the first negative sample, and the first query vector includes: The content contrast loss function is calculated by the first formula, which is: in, is the first query vector, is the first positive sample, is the first negative sample, is the cross entropy loss function.

7. The remote sensing image cross-domain simulation translation method based on contrastive learning according to claim 6 is characterized in that: The cross entropy loss function is calculated by a second formula, which is: in, is the first query vector, is the positive example vector, is the negative example vector.

8. A remote sensing image cross-domain simulation translation device based on contrastive learning, characterized in that: include: An acquisition module, used for acquiring a source image to be translated in a source image domain and a target image to be translated in a target image domain; A translation module, used for inputting the source image to be translated and the target image to be translated into an image translation model; wherein the image translation model includes a generator and a discriminator, and the generator includes an encoder and a decoder; The step of inputting the source image to be translated and the target image to be translated into an image translation model includes: Inputting the source image to be translated and the target image to be translated into the encoder to obtain source coding features and target coding features respectively; Inputting the source coding feature and the target coding feature into the decoder to obtain a source pseudo sample image and a target pseudo sample image respectively; Inputting the source pseudo sample image and the target pseudo sample image into a discriminator for identification, and obtaining a first identification result and a second identification result respectively; The image translation model is trained based on a content contrast loss function, a style contrast loss function, and an adversarial loss function constructed based on the source pseudo sample image and the target pseudo sample image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the remote sensing image cross-domain simulation translation method based on contrastive learning as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the remote sensing image cross-domain simulation translation method based on contrast learning as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Stylized visible watermark adding method

    CN110796583A

  • Unsupervised unpaired image translation method based on attention generator network

    CN113837290A

  • Image translation method based on comparative learning

    CN115909002A

  • Unsupervised learning face cartoon generation method

    CN117132459A

  • Image conversion method and system, electronic equipment and computer storage medium

    CN117314738A