Sample generation method based on improved VAE and Poisson fusion

Abstract

The invention discloses a sample generation method based on improved VAE and Poisson fusion, and belongs to the technical field of image data processing, and the method comprises the steps: constructing a target image set D1, a background image set D2 and a sub-image set D3; constructing a YOLOv8s network, and training the YOLOv8s network by using D3 to obtain a classification model; identifying sub-images containing targets in the D3 by using a classification model to form a target sub-image set D4; generating a corresponding target area graph for each sub-graph in the D4 based on a CAM technology to form a first sample set R1, pre-training a variational auto-encoder by using the R1 to obtain a generative model, generating a plurality of generated images to form a second sample set R2, and combining the R1 and the R2 to obtain a target sample set R; and any image is selected from R and D2, and a Poisson fusion method is used for fusion. According to the method, the dependence on annotation data can be greatly reduced, the generalization ability and the background adaptability of the model are improved, and particularly, relatively high image generation quality can be kept under a complex background.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and particularly to a sample generation method based on improved VAE and Poisson fusion. Background Art

[0002] With the rapid development of technology, sample generation has shown great potential in fields such as image processing, planning and design, and post-disaster assessment. Sample generation refers to the process of determining a certain target, integrating it into a large number of background images, and obtaining a large number of generated samples containing the target. In image processing, ensuring the authenticity and high quality of generated samples is crucial for performing object detection, change detection, and automated analysis tasks. Specific application requirements often require high-quality sample generation and good coupling between the background and samples to meet the needs of sample recognition and extraction tasks. However, traditional sample generation methods often face the problems of lack of generalization and background adaptability of generated samples. Especially when dealing with complex backgrounds, the fusion of generated images and backgrounds is often unnatural, the boundary transition is not smooth, and it is difficult to achieve high-quality sample generation. In addition, traditional methods usually rely on a large amount of labeled data and lack the ability of automatic feature extraction, resulting in low efficiency.

[0003] YOLOv8 is a version in the YOLO (You Only Look Once) series of object detection algorithms. The "s" in YOLOv8s usually refers to small, that is, the small version of YOLOv8. While maintaining high detection accuracy, it optimizes the computational efficiency and inference speed of the model and is suitable for running on resource-constrained devices. YOLOv8 includes a backbone network, a neck network, and a head network. In the backbone network, a Feature Pyramid Network (FPN) structure is used to extract multi-scale features, generating 5 layers of feature maps P1~P5 from shallow to deep, which are respectively called the first feature map to the fifth feature map. P5 is the last layer output in the feature pyramid, usually having a smaller spatial resolution but containing richer semantic information, and can better capture the high-level features of the target. If the target categories in the classification task have obvious semantic features, P5 may be sufficient to support the classification task. The main role of the classification head is to convert the feature information in the feature map into specific detection results, including the position, confidence, and category information of the bounding box. This information can be used in subsequent steps such as Non-Maximum Suppression (NMS) to generate the final detection results.

[0004] VAE (Variational Autoencoder) is a deep generative model that combines the ideas of autoencoders and Bayesian inference. It generates new data samples by learning the latent representation of the data and can perform probabilistic modeling on the data. VAE consists of an encoder and a decoder. The encoder is used to map the input data x to a low-dimensional latent space to obtain the latent variable z, and the decoder is used to map the latent variable z back to the original data space to generate the reconstructed data. The Evidence Lower Bound (ELBO) is the core objective in VAE and is a lower bound on the log-likelihood of the data logp(x). By maximizing the ELBO, the log-likelihood of the data can be approximately maximized. The evidence lower bound is generally defined by the equation The first term on the right side of the equation is the reconstruction error, which is used to measure the similarity between the input data x and the reconstructed data. The second term on the right side of the equation is the KL divergence, which measures the difference between the distribution q(z∣x) output by the encoder and the prior distribution p(z). During training, the loss function loss needs to be minimized. Therefore, the Loss of VAE can be expressed as: .

[0005] Poisson Blending is an advanced image fusion technique mainly used to seamlessly fuse two images together so that the transition at their junction is natural without obvious boundary traces. It optimizes the pixel values in the fusion area by solving the Poisson equation, thereby achieving continuity in the gradient domain. Its steps can be specifically divided into: (1) Defining the fusion area: Select the part of the source image that needs to be fused and determine its position in the target image; (2) Calculating the gradient field: Calculate the gradient fields of the source image and the target image within the fusion area respectively; (3) Solving the Poisson equation: According to the gradient field of the source image and the boundary conditions of the target image, solve the Poisson equation to generate the fused image. Summary of the Invention

[0006] The object of the present invention is to provide a sample generation method based on improved VAE and Poisson blending to solve the problems of lack of generalization and background adaptability in image generation in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows: A sample generation method based on improved VAE and Poisson blending includes the following steps: S1, constructing a target image set D1, a background image set D2, and a sub-image set D3; Determining a single target, obtaining N images containing the target to form D1, N images not containing the target to form D2, cutting each image in D1 into several sub-images, and forming D3 with all the sub-images; S2. Construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first to fifth feature maps of different scales from the subgraph, and the classification head outputs the class probability of the subgraph based on the fifth feature map. S3. Classify the subgraphs in D3 using the classification model, and form a target subgraph set D4 consisting of the subgraphs containing the target. S4. Generate a corresponding target region map for each subgraph in D4 to form a first sample set R1. Among them, the method for generating the target region map r s of subgraph I s includes steps S41 to S43. S41. Input subgraph I s into the classification model to obtain its fifth feature map A and class probability p c , and generate a heat map L according to the following formula c ; , , In the formula, A k is the feature map corresponding to the fifth feature map A in channel k, is the global average of the gradients of A k , is the ReLU function, Z is the total number of pixels in A k , p c is the class probability of I s , is the feature value of the pixel at the i-th row and j-th column in A k ; S42. Generate a mask for each pixel in the heat map to obtain a mask map M c , and the mask M c of the pixel at the i-th row and j-th column in M c (i, j) is obtained according to the following formula; , In the formula, L c (i, j) is the feature value of the pixel at the i-th row and j-th column in L c , and θ1 is the mask threshold; S43. According to the formula , generate the target region map r s of subgraph I s , where is the Hadamard product; S5. Pre-train a variational autoencoder with R1 to obtain a generation model, and use the generation model to generate multiple generated images to form a second sample set R2. Combine R1 and R2 to obtain a target sample set R. S6. Randomly select an image from the target sample set R and label it as I A Randomly select an image from D2 and label it as I B , and generate a fused image using the Poisson fusion method.

[0008] Preferably, the S3 is specifically: preset a classification threshold, send D3 into a classification model to obtain the class probability of each sub-image, mark the sub-images with a probability class less than the classification threshold as not containing the target, and the rest as containing the target, and form a target sub-image set D4 from the sub-images containing the target.

[0009] Preferably, the variational autoencoder in S5 is an improved VAE, and training the improved VAE to obtain a generation model includes steps S51 - S53; S51. Obtain a variational autoencoder, including an encoder and a decoder; The parameters of the encoder are ϕ, which is used to map the input image r to the probability distribution in the latent space , and sample to obtain M1 latent variables, where the m-th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z); The parameters of the decoder are θ, which is used to map the latent variable back to the original data space to generate a reconstructed image; S52. Modify the evidence lower bound ELBO of the variational autoencoder as follows to obtain the improved VAE; , , In the formula, is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ (z|r)||M) is the KL divergence between q ϕ (z|r) and M, and KL(p θ (z)||M) is the KL divergence between p θ (z) and M; S53. Use R1 to train the improved VAE to maximize ELBO to obtain a generation model.

[0010] Preferably, 、KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated according to the following formulas respectively; , , , wherein, p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are respectively the mean and variance of q ϕ (z|r), μ2 and Σ2 are respectively the mean and variance of p θ (z), μ M , Σ M are respectively the mean and variance of the average distribution M, tr(·) represents the trace of a matrix, det(·) represents the determinant of a matrix, k is the distribution dimension, α is a constant, and I is the identity matrix.

[0011] Preferably, the JS divergence is smoothed.

[0012] Preferably, the Poisson fusion method in S6 is an improved Poisson fusion method, and its objective function F is: , , wherein, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω = I A | ∂Ω , f is the fusion image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter for controlling the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') in the fusion region Ω, I A (x', y') and I B (x', y') are respectively the pixel values of I A and I B at (x', y'), and ||·|| is the Euclidean norm.

[0013] The idea of the present invention is as follows: First, train a classification model to quickly classify sub - graphs and find the sub - graphs containing the target; then use the CAM technology to generate a heat map to quickly locate the target in the sub - graphs and generate a target region map; then use the improved VAE to train to obtain a generation model, which is used to quickly generate a large number of generated images containing the target according to the probability distribution of the target region map, and merge the sub - graphs containing the target and the generated images into a target sample set R; finally, use the improved improved Poisson fusion method to select an image from R as the foreground target and select an image from D2 as the background image for image fusion.

[0014] Compared with the prior art, the advantages of the present invention are: (1) Combine the classification model with the CAM technology. First, train a classification model based on the simplified YOLOv8s network. This model uses only the fifth feature map of the backbone network and the classification head for classification prediction, and can quickly obtain the classification results. Then, use the CAM technology to combine the class probabilities of the classification model, the fifth feature map, etc., and quickly perform accurate positioning of the target area and automatic feature extraction on the sub-images. This method can quickly locate the target boundary, ensure that the area of concern of the model is the key to its recognition and classification, and is more objective and accurate than visual judgment of the boundary. Moreover, it greatly reduces the dependence on labeled data, improves the generalization ability and background adaptability of the model, and can maintain a high image generation quality especially in complex backgrounds.

[0015] (2) Improve the variational autoencoder technology. On the basis of the original variational autoencoder, introduce the mean distribution M and the JS divergence. The KL divergence is asymmetric and the JS divergence is symmetric, which makes the metric more balanced and does not bias towards one of the distributions. And the JS divergence is more robust when dealing with the case where two probability distributions do not overlap at all or only partially overlap. The generative model obtained by improving the variational autoencoder enables the generated images to better capture the complex relationship between the samples and the background, significantly enhancing the sample diversity and the adaptability to complex backgrounds. Through this method, not only the morphology of the samples is concerned, but also the background details can be accurately processed, significantly improving the effect of image analysis.

[0016] (3) Improve the objective function of Poisson fusion. Based on the optimized Poisson fusion technology, it enhances the natural fusion effect between the generated samples and the background, ensuring the visual consistency and quality of the generated samples. By dynamically adjusting the image features, it optimizes the matching between the samples and the background, improving the overall quality and adaptability of the samples, especially performing excellently in complex backgrounds.

[0017] In summary, through the introduction of the CAM technology, the improvement of the variational autoencoder, and the improvement of the Poisson fusion method, this solution can achieve accurate positioning of the target area and automatic feature extraction, reduce the dependence on labeled data, improve the quality and application value of sample generation, and effectively solve the problems of lack of generalization and background adaptability in the existing technology. Brief Description of the Drawings

[0018] Figure 1 It is the flowchart of the present invention; Figure 2 It is the flowchart of generating the heat map of the present invention; Figure 3 It is the flowchart of improving the VAE. Detailed Embodiment

[0019] The present invention will be further described below in conjunction with the embodiments and the drawings.

[0020] Example 1: Refer to Figure 1 and Figure 2 , a sample generation method based on the fusion of improved VAE and Poisson, comprising the following steps: S1. Construct a target image set D1, a background image set D2, and a sub-image set D3; Determine a single target, obtain N images containing the target to form D1, N images not containing the target to form D2, cut each image in D1 into several sub-images, and form D3 with all the sub-images; S2. Construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first to fifth feature maps of different scales from the sub-images, and the classification head outputs the class probability of the sub-images based on the fifth feature map; S3. Classify the sub-images in D3 with the classification model, and form a target sub-image set D4 with the sub-images containing the target; S4. Generate a corresponding target region map for each sub-image in D4 to form a first sample set R1. Among them, the method for generating the target region map r s of sub-image I s includes steps S41 to S43; S41. Input sub-image I s into the classification model to obtain its fifth feature map A and class probability p c , and generate a heat map L according to the following formula c ; , , where A k is the feature map corresponding to the fifth feature map A in channel k, is the global average of the gradients of A k , is the ReLU function, Z is the total number of pixels in A k , p c is the class probability of I s , is the feature value of the pixel at the i-th row and j-th column in A k ; S42. Generate a mask for each pixel in the heat map to obtain a mask map M c , and the mask M c of the pixel at the i-th row and j-th column in M c is obtained according to the following formula; , where L c (i,j) is L cThe eigenvalue of the pixel in the i-th row and j-th column, where θ1 is the mask threshold; S43. Generate a sub-graph I according to the formula , to generate the target region graph r of s where s is the Hadamard product; S5. Use R1 to pre-train a variational autoencoder to obtain a generation model, and use the generation model to generate multiple generated images to form a second sample set R2. Combine R1 and R2 to obtain a target sample set R; S6. Select an image from the target sample set R and label it as I A , select an image from D2 and label it as I B , and use the Poisson fusion method to generate a fused image.

[0021] Figure 2 In, taking the derivative of the prediction target with respect to the feature map refers to the operation. Global average pooling to obtain weights refers to the calculation formula of, weighted summation of the feature map, corresponding to the heat map L c the formula of.

[0022] Example 2: Refer to Figure 1 to Figure 2 , specifically, in S3, a classification threshold is preset, D3 is sent into a classification model to obtain the class probability of each sub-graph. Sub-graphs with a probability class less than the classification threshold are marked as not containing the target, and the rest are marked as containing the target. The sub-graphs containing the target form a target sub-graph set D4. The rest is the same as in Example 1.

[0023] Example 3: Refer to Figure 1 to Figure 3 , on the basis of Example 1, a generation model can be directly trained using the existing technology variational autoencoder, or a generation model can be trained using an improved VAE. The training process of the improved VAE is described in detail in this example. The improved VAE is also improved based on the variational autoencoder in the existing technology. Training the improved VAE to obtain a generation model includes steps S51~S53; S51. Obtain a variational autoencoder, including an encoder and a decoder; The parameters of the encoder are ϕ, which is used to map the input image r to the probability distribution in the latent space , and sample to obtain M1 latent variables, where the m-th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z); The parameters of the decoder are θ, which is used to map the latent variable back to the original data space to generate a reconstructed image; S52. Modify the evidence lower bound ELBO of the variational autoencoder as follows to obtain the improved VAE; , , wherein, is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ (z|r)||M) is the KL divergence between q ϕ (z|r) and M, KL(p θ (z)||M) is the KL divergence between p θ (z) and M; S53. The improved VAE is trained with R1 to maximize the ELBO to obtain a generative model.

[0024] , KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated according to the following formulas respectively; , , , wherein, p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are the mean and variance of q ϕ (z|r) respectively, μ2 and Σ2 are the mean and variance of p θ (z) respectively, μ M , Σ M are the mean and variance of the average distribution M respectively, tr(·) represents the trace of a matrix, det(·) represents the determinant of a matrix, k is the dimension of the distribution, α is a constant, and I is the identity matrix.

[0025] The JS divergence is smoothed. In the VAE, to make the metric more balanced and not biased towards one of the distributions, the present invention introduces the JS divergence and the average distribution M, and the JS divergence is more robust in dealing with the case where two probability distributions do not overlap at all or only partially overlap. There may also be a small probability of division by zero problem in the calculation of the JS divergence, and the division by zero error in the calculation is effectively avoided by adding smoothing processing.

[0026] Example 4: Refer to Figure 1 to Figure 3 , in S6, the Poisson fusion method is an improved Poisson fusion method, and its objective function F is: , , wherein, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω = I A | ∂Ω , f is the fusion image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter for controlling the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') within the fusion region Ω, I A (x', y') and I B (x', y') are the pixel values of I A and I B at the position (x', y') respectively, and ||·|| is the Euclidean norm.

[0027] The rest is the same as that of Embodiment 1, Embodiment 2 or Embodiment 3.

[0028] Based on the existing Poisson fusion method, the present invention improves the objective function and introduces w(x', y'). When α is 0, the equation degenerates into the original Poisson fusion. When the gradient difference between I A and I B at the position (x', y') is large, w(x', y') is small, and substituting it into the objective function F means reducing the fusion intensity at the position (x', y'). In this way, the improved Poisson fusion method can automatically adapt to the feature differences of the images, making the fusion result more natural and harmonious visually.

[0029] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A sample generation method based on the fusion of improved VAE and Poisson, characterized in that, Including the following steps: S1. Construct a target image set D1, a background image set D2, and a sub-image set D3; Determine a single target, obtain N images containing the target to form D1, N images not containing the target to form D2, cut each image in D1 into several sub-images, and form D3 with all the sub-images; S2. Construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first feature map to the fifth feature map of different scales from the sub-images, and the classification head outputs the class probability of the sub-images based on the fifth feature map; S3. Classify the sub-images in D3 with the classification model, and form a target sub-image set D4 with the sub-images containing the target; S4. Generate a corresponding target region map for each sub - figure in D4 to form the first sample set R1. Among them, the method of generating the target region map r of sub - figure I s includes steps S41 - S43; s ​ S41, input the sub-graph I s into the classification model to obtain its fifth feature map A and class probability p c , and generate a heat map L according to the following formula c ; ， ， Where A k is the feature map corresponding to the fifth feature map A in channel k, is the global average value of the gradient of A k , is the ReLU function, Z is the total number of pixels in A k , p c is the class probability of I s , is the eigenvalue of the pixel at the i-th row and j-th column in A k ; S42, generate a mask for each pixel in the heat map to obtain a mask map M c , M c The mask M of the pixel in the i-th row and j-th column c (i, j) is obtained according to the following formula; ， Where L c (i,j) is the eigenvalue of the pixel at the i-th row and j-th column in L c , and θ1 is the mask threshold; S43, generate the target region graph r of sub-graph I according to the formula , where s is the Hadamard product s ; ​ S5. Use R1 to pre-train a variational autoencoder to obtain a generation model, and use the generation model to generate multiple generated images to form a second sample set R2, and merge R1 and R2 to obtain a target sample set R; S6. Select an image from the target sample set R and label it as I A . Select an image from D2 and label it as I B . Use the Poisson fusion method to generate a fused image 2. The sample generation method based on the fusion of improved VAE and Poisson according to claim 1, wherein, The specific content of S3 is: preset a classification threshold, send D3 into the classification model to obtain the class probability of each sub-image, mark the sub-images with a probability class less than the classification threshold as not containing the target, and the rest as containing the target, and form a target sub-image set D4 with the sub-images containing the target.

3. A sample generation method based on the fusion of improved VAE and Poisson according to claim 1, characterized in that The variational autoencoder in S5 is an improved VAE, and the steps for training the improved VAE to obtain the generation model include S51 to S53; S51. Obtain a variational autoencoder, including an encoder and a decoder; The encoder parameter is ϕ, which is used to map the input image r to a probability distribution in the latent space , and M1 latent variables are sampled, where the m-th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z); The parameters of the decoder are θ, which are used to map the latent variable back to the original data space to generate a reconstructed image; S52. Modify the evidence lower bound ELBO of the variational autoencoder as follows to obtain the improved VAE; ， ， In the formula, is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ (z|r)||M) is the KL divergence between q ϕ (z|r) and M, KL(p θ (z)||M) is the KL divergence between p θ (z) and M; S53. Use R1 to train the improved VAE to maximize ELBO to obtain the generation model.

4. A sample generation method based on the fusion of improved VAE and Poisson according to claim 3, characterized in that, , KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated respectively according to the following formula; ， ， ， where p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are the mean and variance of q ϕ (z|r) respectively, μ2 and Σ2 are the mean and variance of p θ (z) respectively, μ M and Σ M are the mean and variance of the average distribution M respectively, tr(·) represents the trace of a matrix, det(·) represents the determinant of a matrix, k is the distribution dimension, α is a constant, and I is the identity matrix.

5. The sample generation method based on the improved VAE and Poisson fusion according to claim 3, wherein, The JS divergence is smoothed.

6. A sample generation method based on the fusion of improved VAE and Poisson according to claim 1, characterized in that The Poisson fusion method in S6 is an improved Poisson fusion method, and its objective function F is: ， ， In the formula, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω =I A | ∂Ω , f is the fusion image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter for controlling the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') within the fusion region Ω, I A (x', y'), I B (x', y') are the pixel values of I A and I B at (x', y') respectively, and ||·|| is the Euclidean norm.

Images