Remote sensing image data augmentation method and device based on diffusion generation model

Abstract

The invention relates to a remote sensing image data augmentation method and device based on a diffusion generation model, and the method comprises the steps: data preprocessing, data noise addition, diffusion model training, data augmentation, sample quality evaluation and data mixing. The diffusion model training process comprises the steps that noise images in a training data set and time steps of the noise images are input into a model, the model predicts noise increased in each time step through a reverse process, then the noise image of the last time step is reversely deduced in combination with the noise image corresponding to the time step, and after multiple iterations, the noise image of the last time step is obtained. Gradually restoring the noise image into an original image; the data quality evaluation process comprises the step of measuring the authenticity of the generated data by adopting SSIM and LPIPS. The method has the advantages that the accuracy, flexibility, quality control and compatibility of remote sensing image data augmentation are improved, and wider and more efficient support is provided for training and testing of a remote sensing intelligent interpretation model.

Description

Technical Field

[0001] The present invention relates to the technical field of image data augmentation, and in particular to a remote sensing image data augmentation method and device based on a diffusion generation model. Background Art

[0002] In recent years, the research on intelligent interpretation of power satellite remote sensing in power inspection has made significant progress and is widely used in the monitoring and maintenance of transmission lines and substations. This technology combines remote sensing data, artificial intelligence, and big data analysis, aiming to improve the efficiency and accuracy of inspection work and reduce the cost and risk of manual inspection. However, the construction of sample libraries is a key link in power remote sensing interpretation, because the data sample library is the basis for learning and verification of artificial intelligence models, and its quality and quantity directly affect the performance of the model. At present, there is still a lack of sufficient labeled samples in the field of power remote sensing, and most of the existing sample libraries are general visual samples, which are difficult to effectively meet the specific needs of the field of power remote sensing. This deficiency not only limits the training and testing effects of the model, but also affects the performance of the model in practical applications. In order to solve these problems, we need to develop sample generation technology.

[0003] Existing research has gradually explored a variety of generative models for dataset augmentation. These generative models are mainly based on deep learning technologies such as generative adversarial networks (GANs) and variational autoencoders (VAEs), which enhance the diversity and quantity of samples by generating new high-quality images. However, they still have some shortcomings, which limit the quality and applicability of generated samples. The GAN training process is prone to mode collapse, which affects the quality and consistency of generated images. In the VAE model, since the fidelity of its generated samples depends on the distribution control of the latent space, the generated samples often have problems of blur or lack of details. Therefore, it is particularly necessary to explore new data augmentation technologies. Summary of the invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a remote sensing image data augmentation method based on a diffusion generation model, which can further improve the diversity and authenticity of samples, thereby better supporting the training of intelligent interpretation models and meeting the requirements of high precision and high reliability in power inspections.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] Technical Solution 1

[0007] A remote sensing image data augmentation method based on a diffusion generation model comprises the following steps: Data preprocessing: normalizing the pixel values ​​of power remote sensing image data to a preset range. Data denoising: taking the image data as the original image, gradually adding random noise to the original image according to the time step, obtaining the noise image at different time steps, and adding the noise image to the training data set. Diffusion model training: inputting the noise image and the time step of the noise image in the training data set into the model, the model predicts the noise added at each time step through a reverse process, and then infers the noise image of the previous time step by combining the noise image corresponding to the time step. After multiple iterations, the noise image is gradually restored to the original image. Data augmentation: inputting the noise image into the trained diffusion model, performing a step-by-step denoising process to restore the image, and the diffusion model outputs the final generated image data.

[0008] Preferably, the image data generated by the augmentation step is processed as follows: Sample quality assessment: The image data finally generated is screened using a similarity measurement function to obtain high-quality image data. Data mixing: An augmented training data set is constructed by mixing the high-quality image data with the original image.

[0009] Preferably, the data noise adding process is represented as follows: gradually adding Gaussian noise to the original image x0 until the image becomes a pure noise image x T , the noise image x output at each time step t t All are from the previous step noise image x t-1 Sampling from a Gaussian distribution centered on the state, from x t-1 to x t It can be expressed as:

[0010]

[0011] where α t is a hyperparameter, ∈ t-1 ~N(0,1) is the Gaussian noise randomly sampled at the t-1th time step, t is the time step, and the value range of t is [1,T]. From this formula, we can get x0 to x t The formula is expressed as:

[0012]

[0013] In the formula, ∈~N(0,1) is a randomly sampled Gaussian noise;

[0014] After gradually adding noise to the original image x0, we obtain the training data set (x t ,t,∈ t-1 ).

[0015] More preferably, the reverse process is expressed as: Among them, α t are model hyperparameters, x t represents the noise image after adding noise at the tth time step, ∈ θ (x t ,t) is the noise estimation function, σ t z represents the error between the predicted noise and the actual noise, z~N(0,1);

[0016] The loss function adopted by model learning is as follows: where ∈-∈ θ (x t ,t) measures the noise of the prediction ∈ θ 9x t , the error between t0 and the actual noise. This constraint enables the model to accurately predict the given noise image x t The diffusion model architecture adopts the UNet network, from (x t ,t) prediction∈ θ , the stochastic gradient descent algorithm is used to optimize the parameters of the UNet network.

[0018] Preferably, during the sample quality assessment process, for the finally generated image data I y , randomly select n remote sensing image samples, each sample is represented by I x , calculate I y And each I x The image structure similarity is calculated and the average is taken to decide whether to use the image data I y Specifically, SSIM and LPIPS are used to measure the authenticity of generated data. The formula for calculating SSIM is as follows: Where μ represents the mean value of image pixels, σ 2 represents the image variance, σ xy Represents the covariance of the two images, C is a constant used to avoid the situation where the denominator is 0; when calculating LPIPS, I y And each I x Input the pre-trained VGG model and obtain the distance S of its intermediate features lpips And find the mean μ and variance σ of the data set, and use the Sigmoid function to normalize it to the (0,1) interval. The overall augmented data screening function is expressed as:

[0019] Based on the same inventive concept, the present invention also provides a remote sensing image data augmentation device based on a diffusion generation model.

[0020] Technical Solution 2

[0021] A remote sensing image data augmentation device based on a diffusion generation model comprises a memory storing an executable program and a processor, wherein the processor runs the program to execute the method steps described in technical solution one.

[0022] The present invention has the following beneficial effects:

[0023] 1. The present invention first constructs a diffusion process model to gradually denoise the noisy image to generate remote sensing image samples with high fidelity, thereby significantly enriching the diversity and detail accuracy of the data samples. This method uses a highly versatile prediction network, is compatible with different remote sensing data formats, and can adapt to a variety of application scenarios. In addition, the present invention also designs an intelligent quality assessment module, which uses a variety of image quality assessment indicators to screen and optimize the generated samples in real time to ensure the reliability and authenticity of the generated samples.

[0024] 2. The present invention improves the accuracy, flexibility, quality control and compatibility of remote sensing image data augmentation, and provides more extensive and efficient support for the training and testing of remote sensing intelligent interpretation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the process of the present invention;

[0026] Figure 2 This is the diffusion model training process of the present invention. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0028] See also Figure 1 , a remote sensing image data augmentation method based on a diffusion generation model, comprising the following steps:

[0029] Step 10, data preprocessing: To ensure that the data has a consistent numerical range during training and avoid instability in training due to excessively large or small values, first normalize the pixel values ​​of the power remote sensing image to the range of [-1,1] as follows:

[0030]

[0031] Among them, x max is the maximum pixel value in the data set, x min is the minimum pixel value in the dataset.

[0032] Step 20, data noise addition: take the image data as the original image, gradually add random noise to the original image according to the time step, obtain the noise image at different time steps, and add the noise image to the training data set. The specific steps are as follows:

[0033] The image data image from the data set is taken as the original image, and Gaussian noise is gradually added to the original image according to the time step to convert the original image into a noisy image. Subsequently, the noisy image is restored to the original image through the reverse denoising process learned by the model. Specifically, a forward diffusion process is set, that is, Gaussian noise is gradually added to an original image x0 until the original image x0 is completely converted into a pure noise image x T This process can be viewed as a Markov chain, where the noise image x output at each time step t is t All are from the previous step noise image x t-1 Sampling from a Gaussian distribution centered on the state, from x t-1 to x t It can be expressed as:

[0034]

[0035] where α t is a hyperparameter, ∈ t-1 ~N(0,1) is the Gaussian noise randomly sampled at the t-1th time step, t is the time step, and the value range of t is [1,T]. From this formula, we can get x0 to x t The formula is expressed as:

[0036]

[0037] in, is a randomly sampled Gaussian noise. After gradually adding noise to the original image x0, we can obtain the training data set (x t ,t,∈ t-1 ).

[0038] Step 30, please refer to Figure 2 , Diffusion model training: The noise image and the time step of the noise image in the training data set are input into the model. The model predicts the noise added at each time step through the reverse process, and then combines the noise image corresponding to the time step to infer the noise image of the previous time step. After multiple iterations, the noise image is gradually restored to the original image. Through training, the relevant features of the existing remote sensing image data are learned to be compatible with the generation of new data. The reverse process can be expressed as:

[0039]

[0040] Among them, α t are model hyperparameters, x t represents the noise image after adding noise at the tth time step, ∈ θ (x t,t) is the noise estimation function, σ t z represents the error between the predicted noise and the actual noise, z~N(0,1);

[0041] The loss function adopted by model learning is as follows:

[0042]

[0043] where ∈-∈ θ (x t ,t) measures the noise of the prediction ∈ θ (x t ,t) and the true noise. This constraint enables the model to accurately predict the error between the given noise image x t The original noise under

[0044] The diffusion model architecture adopts the UNet network, from (x t ,t) prediction∈ θ , the stochastic gradient descent algorithm is used to optimize the parameters of the UNet network.

[0045] Step 40, data augmentation: the noisy image is input into the trained diffusion model, a step-by-step denoising process is performed to restore the image, and the diffusion model outputs the final generated image data.

[0046] Step 50, sample quality assessment: use the similarity measurement function to screen the image data finally generated to obtain high-quality image data; in the sample quality assessment process, the image data finally generated I y , randomly select n remote sensing image samples, each sample is represented by I x , calculate I y And each I x The image structure similarity is calculated and the average is taken to decide whether to use the image data I y Specifically, SSIM and LPIPS are used to measure the authenticity of generated data. The formula for calculating SSIM is as follows:

[0047]

[0048] Where μ represents the mean value of image pixels, σ 2 represents the image variance, σ xy Represents the covariance of two images, C is a constant used to avoid the situation where the denominator is 0;

[0049] When calculating LPIPS, I y And each I x Input the pre-trained VGG model and obtain the distance S of its intermediate features lpipsAnd find the mean μ and variance σ of the data set, and use the Sigmoid function to normalize it to the (0,1) interval. The overall augmented data screening function is expressed as:

[0050] Step 60, data mixing: By mixing high-quality image data with the original image, an augmented training data set is constructed to expand the diversity of remote sensing image data sets and provide richer and more diverse training samples for models in the field of power remote sensing. Through this data mixing strategy, the generalization ability of the model in different scenarios can be increased, thereby improving the model performance and robustness in power remote sensing related tasks.

[0051] The present invention realizes the augmentation of electric power remote sensing image data by introducing a diffusion model and an efficient quality assessment mechanism. The diffusion model has the ability to generate images in a layer-by-layer refined manner, and can gradually generate high-quality remote sensing images from noisy images, retaining rich spatial details, so that the generated samples are authentic and reliable; at the same time, the quality assessment mechanism adopts a similarity verification and screening mechanism, which screens the generated image data according to the image quality and feature distribution, retains high-quality image data, and effectively avoids the impact of low-quality data on model training. Improve the authenticity of augmented samples. This method ensures the high precision and high quality of the generated samples, provides rich and reliable training data for the intelligent interpretation model of electric power inspection, and greatly improves the accuracy and generalization ability of the augmented data.

[0052] Embodiment 2

[0053] A remote sensing image data augmentation device based on a diffusion generation model comprises a memory storing an executable program and a processor, wherein the processor runs the program to execute the method steps described in the first embodiment.

[0054] Since the device introduced in the second embodiment of the present invention is a hardware device for implementing the method of the first embodiment of the present invention, those skilled in the art can understand the specific implementation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All methods adopted in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0055] The above description is only a specific implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structural transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A remote sensing image data augmentation method based on a diffusion generation model, characterized in that: The steps include: Data preprocessing: normalize the pixel values ​​of power remote sensing image data to a preset range; Data noisy: Take the image data as the original image, add random noise to the original image step by step according to the time step, obtain the noisy image at different time steps, and add the noisy image to the training data set; Diffusion model training: The noise image and the time step of the noise image in the training data set are input into the model. The model predicts the noise added at each time step through the reverse process, and then combines the noise image corresponding to the time step to infer the noise image of the previous time step. After multiple iterations, the noise image is gradually restored to the original image. Data augmentation: The noisy image is input into the trained diffusion model, a step-by-step denoising process is performed to restore the image, and the diffusion model outputs the final generated image data.

2. The remote sensing image data augmentation method based on the diffusion generation model according to claim 1, characterized in that: The image data generated by the augmentation step is processed as follows: Sample quality assessment: Use the similarity measurement function to screen the final generated image data to obtain high-quality image data; Data Mixing: Construct an augmented training dataset by mixing high-quality image data with the original images.

3. The remote sensing image data augmentation method based on the diffusion generation model according to claim 1, characterized in that: The data noise addition process is represented as follows: gradually add Gaussian noise to the original image x0 until the image becomes a pure noise image x T , the noise image x output at each time step t t All are from the previous step noise image x t-1 Sampling from a Gaussian distribution centered on the state, from x t-1 to x t It can be expressed as: where α t is a hyperparameter, ∈ t-1 ~N(0,1) is the Gaussian noise randomly sampled at the t-1th time step, t is the time step, and the value range of t is [1,T]. From this formula, we can get x0 to x t The formula is expressed as: In the formula, ∈~N(0,1) is a randomly sampled Gaussian noise; After gradually adding noise to the original image x0, we obtain the training data set (x t ,t,∈ t-1 ).

4. The remote sensing image data augmentation method based on the diffusion generation model according to claim 1, characterized in that: The reverse process is expressed as: Among them, α t are model hyperparameters, x t represents the noise image after adding noise at the tth time step, ∈ θ (x t ,t) is the noise estimation function, σ t z represents the error between the predicted noise and the actual noise. z~N(0,1); The loss function adopted by model learning is as follows: where ∈-∈ θ (x t ,t) measures the noise of the prediction ∈ θ (x t ,t) and the actual noise. This constraint enables the model to accurately predict the error between the given noise image x t The original noise under The diffusion model architecture adopts the UNet network, from (x t ,t) prediction∈ θ , the stochastic gradient descent algorithm is used to optimize the parameters of the UNet network.

5. The remote sensing image data augmentation method based on the diffusion generation model according to claim 2, characterized in that: In the sample quality assessment process, the final generated image data I y , randomly select n remote sensing image samples, each sample is represented by I x , calculate I y And each I x The image structure similarity is calculated and the average is taken to decide whether to use the image data I y ; Specifically, SSIM and LPIPS are used to measure the authenticity of generated data, and the formula for calculating SSIM is as follows: Where μ represents the mean value of image pixels, σ 2 represents the image variance, σ xy Represents the covariance of two images, C is a constant used to avoid the situation where the denominator is 0; When calculating LPIPS, I y And each I x Input the pre-trained VGG model and obtain the distance S of its intermediate features lpips And find the mean μ and variance σ of the data set, and use the Sigmoid function to normalize it to the (0,1) interval. The overall augmented data screening function is expressed as:

6. A remote sensing image data augmentation device based on a diffusion generation model, characterized in that: It comprises a memory storing an executable program and a processor, wherein the processor runs the program to execute the method steps described in any one of claims 1 to 5.

Images