A method and system for change sample simulation based on cross-phase generative adversarial

Abstract

The application discloses a change sample simulation method and system based on cross-time-phase generative adversarial network, which comprises the following steps: firstly, acquiring single-time-phase image data and its label, and image data of another time phase; combining the single-time-phase image data and its label to synthesize the foreground and background of a random image in the image space, and obtaining a synthesized image; taking the synthesized image and the image of another time phase as inputs of a change sample simulation network based on cross-time-phase generative adversarial network, and simultaneously realizing the training of a synthesized image cross-time-phase generative real image style model generator G and a real image cross-time-phase generative synthesized image style generator F in a feature space; and utilizing a cyclic consistency loss function to realize the similarity discrimination between the generated real image style, the synthesized image and the input image, so that the trained network can output a simulation image with a real image style, and finally, the simulation image and the original single-time-phase image constitute a simulated change sample set.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image simulation technology, and relates to a method for simulating dual-temporal change samples using single-temporal labeled images, and more particularly to a change sample simulation method based on cross-temporal generation adversarial. Background Technology

[0002] Remote sensing (RS) is a non-contact, long-range Earth observation technology. Change detection in remote sensing images is crucial for reflecting land cover changes and human activities. Current mainstream change detection methods use paired, bi-temporally labeled images as rigorous input. However, for large-scale land cover updates and mapping tasks such as national geographic surveys, the practical need is to perform rapid cross-temporal object segmentation and change patch detection based on earlier labeled images of later images. These issues significantly limit the application of deep learning-based change detection methods. With the rapid development of deep learning methods, simulation of change samples in the remote sensing field based on deep learning has become a new research hotspot. For example, IAug (Chen et al. (2021)) trained a building generator to generate building instances and used an untrained color conversion method to transform the building styles. These building instances were then blended into bi-temporal images to enhance the change detection dataset. In the same year, Zheng et al. randomly sampled two images from a single-time dataset as change sample pairs, providing a heuristic idea for single-time supervised object change detection. However, these change sample pairs, which only change content and lack style variation, may lead to negative transfer of the model. Self-Pair (Seo et al. (2023)) uses three methods to obtain pseudo-image pairs from the source domain, including randomly cropping two patches with no overlapping regions, randomly erasing selected instances and repairing the background, and randomly copying an instance from one patch and pasting them into another patch. Although Self-Pair considers the generation of change samples more comprehensively, it also lacks the style of the target domain. Summary of the Invention

[0003] This invention addresses the challenges of acquiring change samples and the high cost of annotation. It proposes a cross-temporal generative adversarial change sample simulation method capable of effectively generating change sample pairs from single-temporal labeled images. After reviewing existing sample simulation methods for change detection, this invention presents an unsupervised cross-temporal generative adversarial change sample simulation method.

[0004] The above-mentioned objective of this invention can be achieved through the following technical solution: a method for simulating changing samples based on cross-temporal generative adversarial methods, comprising the following steps: Step S1: Synthesize a single-temporal image set; Step S2: Construct a simulation dataset of changing samples; Step S3: Construct a cross-temporal generative adversarial network model; Step S4: Train a cross-temporal generative adversarial network model using an unsupervised learning mode; Step S5: Use the trained cross-temporal generative adversarial network model to perform style transfer from synthetic images to real images, and output simulated images with the style of real images. Finally, the simulated images and the original single-temporal images form a set of variation samples for simulation.

[0005] Furthermore, in step S1, a single-temporal image set is obtained by random sampling using the image set and semantic annotation data set of a single temporal phase.

[0006] Furthermore, in step S2, for a single-temporal image set, a non-repeating ordered image set and a label set are generated based on the random sequence value k. For each ordered arrangement, based on the information provided by the label, and combined with a batch parallel strategy, the foreground of one image and the background of another image are randomly synthesized in the image space to generate a new set of synthetic images. The synthetic image set and another set of real-time images with geometric consistency are cropped into image patches of the same size to obtain the variation sample simulation dataset.

[0007] Furthermore, in step S3, the synthetic image and the real image are used as a generator to generate a style model of the synthetic image across time to create a real image. The input, through the generator Generate realistic image style ; Through the discriminator Identify the style of the generated target image Similarity to the style of the original authentic image; target image style As a style model generator for cross-temporal generation of real images and composite images The input, through the generator Generate reconstructed synthetic images ; Through the discriminator Identify the generated reconstructed synthetic image Similarity to the original synthetic image; Using real and composite images as a cross-temporal generation technique for real images to create composite image style. The input, through the generator Generate target synthetic image ; Through the discriminator Identify the generated synthetic target image Similarity to the style of the original synthetic image; Composite image of the target As a generator for generating realistic image style models across time and space in synthetic images The input, through the generator Generate reconstructed realistic images ; Through the discriminator Identify the generated reconstructed real image Similarity to the original, authentic image; A transtemporal generative adversarial network model is constructed by a bidirectional loop of generating realistic image styles from synthetic images across time and generating synthetic image styles from real images across time.

[0008] Furthermore, a generator for generating realistic image style models across temporal phases of synthetic images. The encoder and decoder structure consists of an encoder composed of convolution, regularization, and ReLU activation function modules, and a decoder composed of deconvolution, regularization, and ReLU activation function modules. The encoder and decoder are connected by several repeated residual blocks. Realistic image transtemporal generation composite image style generator It also uses an encoder and decoder structure, where the encoder consists of convolution, regularization and ReLU activation function modules, and the decoder consists of deconvolution, regularization and ReLU activation function modules. The encoder and decoder are connected by several repeated residual blocks. Discriminator and discriminator All are patchGAN structures.

[0009] Furthermore, adopting adversarial losses Cyclic consistency loss and reconstruction loss Training a cross-temporal generative adversarial network;

[0010] in, For the total loss function, As hyperparameters, the weighting of the three loss functions is adjusted to balance their proportions. These represent the phases at different times.

[0011] Furthermore, adversarial losses It consists of two parts: one is the generation loss of converting the style of the synthetic image to the style of the real image, and the other is the generation loss of inverse transformation of the style of the real image to the style of the synthetic image. Both losses are calculated using the L1 norm.

[0012]

[0013] in, and All are L1 norm.

[0014] Furthermore, the loss of cycle consistency It includes the process loss of converting the style of a synthetic image to the style of a real image and then reconstructing it back into the style of a synthetic image, and the process loss of converting the style of a real image to the style of a synthetic image and then reconstructing it back into the style of a real image. Both of these losses are calculated using the L1 norm.

[0015] in, and All are L1 norm.

[0016] Furthermore, reconstruction loss The reconstruction loss includes the style of the generated synthetic image and the style of the original synthetic image, as well as the style of the generated real image and the style of the original real image. The L1 norm is used to calculate the loss.

[0017] in, and All are L1 norm.

[0018] This invention also provides a simulation system for changing samples based on cross-temporal generative adversarial methods, comprising the following modules: The single-temporal image synthesis module is used to synthesize a set of single-temporal images; The dataset building module is used to build simulation datasets for changing samples; The model building module is used to build cross-temporal generative adversarial network models. The model training module is used to train a cross-temporal generative adversarial network model using an unsupervised learning model. The simulation generation module is used to perform style transfer from synthetic images to real images using a trained cross-temporal generative adversarial network model, and output simulated images with the style of real images. Finally, the simulated images and the original single-temporal images form a set of variation samples for simulation.

[0019] The specific implementation methods of each module are the same as those of each step, and will not be described in this invention.

[0020] The advantages of this invention compared to the prior art are as follows: (1) A cross-temporal generative adversarial network was used to generate a simulated image of another temporal phase from a single-temporal labeled image, thus obtaining a set of simulated change samples. Compared with traditional change sample simulation methods, this method not only synthesizes the foreground and background of random images in the single-temporal image space to create pseudo-changes in objects and backgrounds, but also performs unsupervised style transfer in the feature space to simulate the seasonal, illumination, and phenological information of the real post-temporal image in the synthesized image, thus satisfying the requirement that the data distribution of the pseudo dual-temporal image pair should be as close as possible to the real change detection dataset.

[0021] (2) An unsupervised cross-temporal generative adversarial network model is adopted. The unsupervised approach allows the model to dynamically carry out change detection sample simulation based on the image acquisition situation. The synthetic image is converted according to the style of the latest acquired image to generate a change sample set suitable for the change detection of this latest image, thereby increasing and improving the change detection sample library and supporting the research on land cover mapping and dynamic change monitoring methods. Attached Figure Description

[0022] Figure 1 This is a network architecture diagram based on a cross-temporal generative adversarial variation sample simulation method; Figure 2 This is a flowchart of the steps of a cross-temporal generation adversarial variation sample simulation method; Figure 3 The results are simulations of changing samples generated based on cross-temporal adversarial processes. Detailed Implementation

[0023] To facilitate understanding and implementation of this invention by those skilled in the art, Figure 3 The following is a flowchart of the method. The present invention will be described in further detail below. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0024] This invention provides a variation sample simulation method based on cross-temporal generative adversarial methods, comprising three stages. In the dataset construction stage, a single temporal image set is used. and its semantic annotation set Randomly sample m labels and images from the set, and base them on random sequence values. Generate non-repeating ordered permutations and For each ordered arrangement, a batch parallel strategy is used to randomly synthesize the foreground of one image and the background of another image in the image space, thereby generating a new set of synthetic images and its annotation set. The composite image set and another set of geometrically consistent real-time images are cropped into image patches of the same size to complete the variation sample simulation dataset.

[0025] During the network training phase, a generator for generating realistic image style models across temporal phases of synthetic images was constructed. Discriminator Synthetic Image Model Generator that Generates Synthetic Images Across Time and Space Discriminator Using an unsupervised learning model and a cycle consistency loss function for loss constraint, a generative adversarial network model (S-to-T-cycleGAN) is obtained after training, which transforms the style of synthetic images into the style of real images.

[0026] During the network testing phase, synthetic images and real images are input, and style transfer from synthetic images to real images is performed based on the trained S-to-T-cycleGAN model, outputting style-transferred simulated images. That is, by using single-temporal images and tags, a set of simulated change samples is obtained through a cross-temporal generative adversarial change sample simulation method. , , , .

[0027] Please see Figure 1 and Figure 2 The present invention provides a method for simulating changing samples based on cross-temporal generative adversarial methods, comprising the following steps: Step 1: Perform random sampling using a single temporal image set and a labeled data set. Each sampling set contains m labels and images. Step 2: For each sampling set, generate a non-repeating ordered set of images and a set of labels based on the random sequence value k; Step 3: For each ordered arrangement, based on the information provided by the label, and combined with a batch parallel strategy, the foreground of one image and the background of another image are randomly synthesized in the image space to generate a new set of composite images. Step 4: Crop the synthetic image set and another set of real-time images with geometric consistency into image patches of the same size to obtain a simulated dataset of variations that can be used for network training.

[0028] Step 5: Use the composite image and the real image as a generator to generate a style model of the composite image across time and space. The input, through the generator Generate realistic image style The generator's encoder consists of convolution, regularization, and ReLU activation function modules, while the decoder consists of deconvolution, regularization, and ReLU activation function modules. The two structures are connected by nine repeating residual blocks. Step Six: Using the discriminator Identify the style of the generated target image The similarity between the style and the original real image is determined by a discriminator using a patchGAN structure. Step 7: Style the target image As a style model generator for cross-temporal generation of real images and composite images The input, through the generator Generate reconstructed synthetic images ; Step 8: Using the discriminator Identify the generated reconstructed synthetic image Similarity to the original synthetic image; Step 9: Use real and composite images as a style model generator to generate composite images from real images across time periods. The input, through the generator Generate target synthetic image The generator's encoder consists of convolution, regularization, and ReLU activation function modules, while the decoder consists of deconvolution, regularization, and ReLU activation function modules. The two structures are connected by nine repeating residual blocks. Step 10: Using the discriminator Identify the generated synthetic target image The discriminant, based on the similarity to the style of the original synthetic image, is a patchGAN structure. Step 11: Composite image of the target As a generator for generating realistic image style models across time and space in synthetic images The input, through the generator Generate reconstructed realistic images ; Step 12: Using the discriminator Identify the generated reconstructed real image Similarity to the original, authentic image; Step 13: Implement a transtemporal generative adversarial network architecture by generating real image styles from synthetic images across time and generating synthetic image styles from real images across time.

[0029] Step Fourteen: Synthesizing Images to Generate Realistic Image Styles Across Time. The loss function of the generative adversarial network consists of three parts: adversarial loss, cycle consistency loss, and reconstruction loss. The mathematical description of the loss function is as follows: The adversarial loss consists of two parts: the generation loss from the synthetic image style to the real image style, and the generation loss from the inverse transformation of the real image style to the synthetic image style. Both losses are calculated using the L1 norm.

[0030]

[0031] Cyclic consistency loss includes the process loss of converting the synthetic image style to the real image style and then reconstructing it back to the synthetic image style, and the process loss of converting the real image style to the synthetic image style and then reconstructing it back to the real image style. Both losses are calculated using the L1 norm.

[0032] The reconstruction loss includes the reconstruction loss between the style of the generated synthetic image and the style of the original synthetic image, as well as the reconstruction loss between the style of the generated real image and the style of the original real image. Both losses are calculated using the L1 norm.

[0033] Step 15: Synthesizing images across time to generate realistic image styles. The loss function of the generative adversarial network consists of the three loss components mentioned above, mathematically described as follows:

[0034] in, As a hyperparameter, the weighting of the three loss functions is adjusted to balance them.

[0035] Step 16: Input the synthetic image data to be tested and the real image data into the trained generative adversarial network model that generates real image styles across time phases of the synthetic image. Step 17: The synthesized image is output as a simulated image with a realistic image style through a network model; Step 18: Output the simulation change sample set { , , , .

[0036] Simulation results of variation samples based on cross-temporal generative adversarial mechanisms are as follows: Figure 3 As shown, the time phase 1 image and the pseudo time phase 2 image constitute the simulated change sample pair.

[0037] This invention also provides a simulation system for changing samples based on cross-temporal generative adversarial methods, comprising the following modules: The single-temporal image synthesis module is used to synthesize a set of single-temporal images; The dataset building module is used to build simulation datasets for changing samples; The model building module is used to build cross-temporal generative adversarial network models. The model training module is used to train a cross-temporal generative adversarial network model using an unsupervised learning model. The simulation generation module is used to perform style transfer from synthetic images to real images using a trained cross-temporal generative adversarial network model, and output simulated images with the style of real images.

[0038] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for simulating changing samples based on cross-temporal generative adversarial methods, characterized in that, The steps include the following: Step S1: Synthesize a single-temporal image set; In step S1, a single-temporal image set is obtained by random sampling using a single-temporal image set and a semantically labeled data set; Step S2: Construct a simulation dataset of changing samples; In step S2, for a single-temporal image set, a non-repeating ordered image set and a label set are generated based on the random sequence value k; For each ordered arrangement, based on the information provided by the label, and combined with a batch parallel strategy, the foreground of one image and the background of another image are randomly synthesized in the image space to generate a new set of synthetic images. The synthetic image set and another set of real temporal images with geometric consistency are cropped into image patches of the same size to obtain a variation sample simulation dataset. Step S3: Construct a cross-temporal generative adversarial network model; In step S3, the synthetic image and the real image are used as a generator to generate a style model of the synthetic image across time and space for the real image. The input, through the generator Generate realistic image style ; Through the discriminator Identify the style of the generated target image Similarity to the style of the original authentic image; target image style As a style model generator for cross-temporal generation of real images and composite images The input, through the generator Generate reconstructed synthetic images ; Through the discriminator Identify the generated reconstructed synthetic image Similarity to the original synthetic image; Using real and composite images as a cross-temporal generation technique for real images to create composite image style. The input, through the generator Generate target synthetic image ; Through the discriminator Identify the generated synthetic target image Similarity to the style of the original composite image; Composite image of the target As a generator for generating realistic image style models across time and space in synthetic images The input, through the generator Generate reconstructed realistic images ; Through the discriminator Identify the generated reconstructed real image Similarity to the original, authentic image; A cross-temporal generative adversarial network model is constructed by a bidirectional loop of generating realistic image styles from synthetic images across time and generating synthetic image styles from real images across time. Step S4: Train a cross-temporal generative adversarial network model using an unsupervised learning mode; Step S5: Use the trained cross-temporal generative adversarial network model to perform style transfer from synthetic images to real images, and output simulated images with the style of real images. Finally, the simulated images and the original single-temporal images form a set of variation samples for simulation.

2. The method for simulating changing samples based on cross-temporal generative adversarial methods as described in claim 1, characterized in that: A generator for creating real-image style models across temporal phases of synthetic images. The encoder and decoder structure consists of an encoder composed of convolution, regularization, and ReLU activation function modules, and a decoder composed of deconvolution, regularization, and ReLU activation function modules. The encoder and decoder are connected by several repeated residual blocks. Realistic image transtemporal generation composite image style generator It also uses an encoder and decoder structure, where the encoder consists of convolution, regularization and ReLU activation function modules, and the decoder consists of deconvolution, regularization and ReLU activation function modules. The encoder and decoder are connected by several repeated residual blocks. Discriminator and discriminator All are patchGAN structures.

3. The method for simulating changing samples based on cross-temporal generative adversarial methods as described in claim 1, characterized in that: Adopting adversarial losses Cyclic consistency loss and reconstruction loss Training a cross-temporal generative adversarial network; in, For the total loss function, As hyperparameters, the weighting of the three loss functions is adjusted to balance their proportions. These represent the phases at different times.

4. The method for simulating changing samples based on cross-temporal generative adversarial methods as described in claim 3, characterized in that: Adversarial loss It consists of two parts: one is the generation loss of converting the style of the synthetic image to the style of the real image, and the other is the generation loss of inversely transforming the style of the real image to the style of the synthetic image. 。 5. The method for simulating changing samples based on cross-temporal generative adversarial methods as described in claim 3, characterized in that: Cyclic consistency loss It includes the process loss of converting the style of a synthetic image to the style of a real image and then reconstructing it back into the style of a synthetic image, and the process loss of converting the style of a real image to the style of a synthetic image and then reconstructing it back into the style of a real image. Both of these losses are calculated using the L1 norm. 。 6. The method for simulating changing samples based on cross-temporal generative adversarial methods as described in claim 3, characterized in that: Reconstruction loss The reconstruction loss includes the style of the generated synthetic image and the style of the original synthetic image, as well as the style of the generated real image and the style of the original real image. The L1 norm is used to calculate the loss. 。 7. A variation sample simulation system based on cross-temporal generative adversarial methods, used to implement the method as described in any one of claims 1 to 6, characterized in that, Includes the following modules: The single-temporal image synthesis module is used to synthesize a set of single-temporal images; The dataset building module is used to build simulation datasets for changing samples; The model building module is used to build cross-temporal generative adversarial network models. The model training module is used to train a cross-temporal generative adversarial network model using an unsupervised learning model. The simulation generation module is used to perform style transfer from synthetic images to real images using a trained cross-temporal generative adversarial network model, and output simulated images with the style of real images. Finally, the simulated images and the original single-temporal images form a set of variation samples for simulation.

Images