A sample generation method based on the fusion of improved VAE and Poisson

Abstract

The present invention discloses a sample generation method based on the fusion of improved VAE and Poisson, belonging to the technical field of image data processing, including the steps of: constructing a target image set D1, a background image set D2, and a sub-image set D3; constructing a YOLOv8s network and training it with D3 to obtain a classification model; using the classification model to identify the sub-images containing the target in D3 to form a target sub-image set D4; generating corresponding target region maps for each sub-image in D4 based on the CAM technology to form a first sample set R1, pre-training a variational autoencoder with R1 to obtain a generation model, and generating multiple generated images to form a second sample set R2, merging R1 and R2 to obtain a target sample set R; randomly selecting one image from R and D2 respectively, and fusing them using the Poisson fusion method. The present invention can greatly reduce the dependence on labeled data, improve the generalization ability and background adaptability of the model, and especially maintain a high image generation quality under complex backgrounds.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and particularly to a sample generation method based on the fusion of improved VAE and Poisson. Background Art

[0002] With the rapid development of technology, sample generation has shown great potential in the fields of image processing, planning and design, and post-disaster assessment. Sample generation refers to the process of determining a certain target and integrating it into a large number of background images to obtain a large number of generated samples containing the target. In image processing, ensuring the authenticity and high quality of generated samples is crucial for performing object detection, change detection, and automated analysis tasks. Specific application requirements often require high-quality sample generation and good coupling between the background and samples to meet the needs of sample recognition and extraction tasks. However, traditional sample generation methods often face the problems of lack of generalization and background adaptability of generated samples. Especially when dealing with complex backgrounds, the fusion of generated images and backgrounds is often unnatural, the boundary transition is not smooth, and it is difficult to achieve high-quality sample generation. In addition, traditional methods usually rely on a large amount of labeled data and lack the ability of automatic feature extraction, resulting in low efficiency.

[0003] YOLOv8 is a version in the YOLO (You Only Look Once) series of object detection algorithms. The "s" in YOLOv8s usually refers to "small", that is, the small version of YOLOv8. While maintaining high detection accuracy, it optimizes the computational efficiency and inference speed of the model and is suitable for running on resource-constrained devices. YOLOv8 includes a backbone network, a neck network, and a head network. The feature pyramid network (FPN) structure is used in the backbone network to extract multi-scale features, generating 5 layers of feature maps P1~P5 from shallow to deep, which are respectively called the first feature map to the fifth feature map. P5 is the last layer output in the feature pyramid, usually having a smaller spatial resolution but containing richer semantic information, and can better capture the high-level features of the target. If the target categories of the classification task have obvious semantic features, P5 may be sufficient to support the classification task. The main role of the classification head is to convert the feature information in the feature map into specific detection results, including the position, confidence, and category information of the bounding box. These information can be used in subsequent steps such as non-maximum suppression (NMS) to generate the final detection results.

[0004] VAE (Variational Autoencoder) is a deep generative model that combines the ideas of autoencoders and Bayesian inference. It learns the latent representation of data, generates new data samples, and can perform probabilistic modeling on data. VAE consists of an encoder and a decoder. The encoder is used to map the input data x to a low-dimensional latent space to obtain the latent variable z, and the decoder is used to map the latent variable z back to the original data space to generate the reconstructed data. The Evidence Lower Bound (ELBO) is the core objective in VAE and is a lower bound on the log-likelihood of the data logp(x). By maximizing the ELBO, we can approximately maximize the log-likelihood of the data. The evidence lower bound is generally defined by the equation The first term on the right side of the equation is the reconstruction error, which is used to measure the similarity between the input data x and the reconstructed data. The second term on the right side of the equation is the KL divergence, which measures the difference between the distribution q(z∣x) output by the encoder and the prior distribution p(z). During training, the loss function loss needs to be minimized. Therefore, the Loss of VAE can be expressed as: .

[0005] Poisson Blending is an advanced image fusion technology mainly used to seamlessly fuse two images together, making the transition at their junction natural without obvious boundary traces. It optimizes the pixel values in the fusion area by solving the Poisson equation, thereby achieving continuity in the gradient domain. Its steps can be specifically divided into: (1) Defining the fusion area: Select the part of the source image that needs to be fused and determine its position in the target image; (2) Calculating the gradient field: Calculate the gradient fields of the source image and the target image within the fusion area respectively; (3) Solving the Poisson equation: According to the gradient field of the source image and the boundary conditions of the target image, solve the Poisson equation to generate the fused image. Summary of the Invention

[0006] The purpose of the present invention is to provide a sample generation method based on improved VAE and Poisson Blending, which solves the problems of lack of generalization and background adaptability in image generation in the prior art.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A sample generation method based on improved VAE and Poisson Blending includes the following steps:

[0008] S1, constructing a target image set D1, a background image set D2, and a sub-image set D3;

[0009] Determining a single target, obtaining N images containing the target to form D1, N images not containing the target to form D2, cutting each image in D1 into several sub-images, and forming D3 with all the sub-images;

[0010] S2: Construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first to fifth feature maps of different scales from the sub-image, and the classification head outputs the category probability of the sub-image based on the fifth feature map.

[0011] S3, use the classification model to classify the subgraphs in D3, and form the subgraphs containing the target into the target subgraph set D4;

[0012] S4, generate a corresponding target area map for each sub-map in D4 to form a first sample set R1, wherein the generated sub-map I s Target area map r s The method includes steps S41 to S43;

[0013] S41, subgraph I s Input the classification model and obtain its fifth feature map A and category probability p c , generate the heat map L according to the following formula c ;

[0014] ,

[0015] ,

[0016] Where A k The fifth feature map A corresponds to the feature map of channel k, A k The global average of the gradient, is the ReLU function, Z is A k The total number of pixels in the c For I s The category probability, A k The characteristic value of the pixel in the i-th row and j-th column;

[0017] S42, generate a mask for each pixel in the heat map to obtain a mask map M c , M c The mask M of the pixel in row i and column j c (i,j) is obtained according to the following formula;

[0018] ,

[0019] Where, L c (i,j) is L c The feature value of the pixel in the i-th row and j-th column in , θ1 is the mask threshold;

[0020] S43, according to the formula , generate subgraph Is Target region graph r s , where is the Hadamard product;

[0021] S5. Use R1 to pre-train a variational autoencoder to obtain a generative model, and use the generative model to generate multiple generated images to form a second sample set R2. Combine R1 and R2 to obtain the target sample set R;

[0022] S6. Select an image from the target sample set R and label it as I A , select an image from D2 and label it as I B , and use the Poisson fusion method to generate a fused image.

[0023] Preferably, the S3 is specifically: preset a classification threshold, send D3 into the classification model to obtain the class probability of each sub-graph, label the sub-graphs with a probability class less than the classification threshold as not containing the target, and the rest as containing the target. The sub-graphs containing the target form the target sub-graph set D4.

[0024] Preferably, the variational autoencoder in S5 is an improved VAE, and training the improved VAE to obtain the generative model includes steps S51 to S53;

[0025] S51. Obtain a variational autoencoder, including an encoder and a decoder;

[0026] The parameters of the encoder are ϕ, which is used to map the input image r to the probability distribution in the latent space , and sample to obtain M1 latent variables, where the m-th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z);

[0027] The parameters of the decoder are θ, which is used to map the latent variable back to the original data space to generate a reconstructed image;

[0028] S52. Modify the evidence lower bound ELBO of the variational autoencoder as follows to obtain the improved VAE;

[0029] ,

[0030] ,

[0031] In the formula, is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ(z|r)||M) is q ϕ The KL divergence between (z|r) and M, KL(p θ (z)||M) is p θ The KL divergence between (z) and M;

[0032] S53. Train the improved VAE with R1 to maximize the ELBO to obtain the generative model.

[0033] Preferably, , KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated according to the following formulas respectively;

[0034] ,

[0035] ,

[0036] ,

[0037] In the formula, p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are the mean and variance of q ϕ (z|r) respectively, μ2 and Σ2 are the mean and variance of p θ (z) respectively, μ M , Σ M are the mean and variance of the average distribution M respectively, tr(·) represents the trace of the matrix, det(·) represents the determinant of the matrix, k is the distribution dimension, α is a constant, and I is the identity matrix.

[0038] Preferably, the JS divergence is smoothed.

[0039] Preferably, the Poisson fusion method in S6 is an improved Poisson fusion method, and its objective function F is:

[0040] ,

[0041] ,

[0042] In the formula, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω =I A | ∂Ω , f is the fusion image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter to control the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') in the fusion region Ω, I A (x', y'), IB (x', y') are respectively I A and I B the pixel values at (x', y'), and ||·|| is the Euclidean norm.

[0043] The idea of the present invention is as follows: First, train a classification model to quickly classify sub - graphs and find the sub - graphs containing the target; then use the CAM technology to generate a heat map to quickly locate the target in the sub - graph and generate a target region map; then train an improved VAE to obtain a generation model, which is used to quickly generate a large number of generated images containing the target according to the probability distribution of the target region map, and merge the sub - graphs containing the target and the generated images into a target sample set R; finally, use an improved Poisson fusion method to select images from R as foreground targets and select images from D2 as background images for image fusion.

[0044] Compared with the prior art, the advantages of the present invention are as follows:

[0045] (1) Combine the classification model with the CAM technology. First, train a classification model based on the simplified YOLOv8s network, which only uses the fifth feature map of the backbone network and the classification head for classification prediction, and can quickly obtain the classification result. Then, use the CAM technology to combine the class probabilities, the fifth feature map, etc. of the classification model to quickly and accurately locate the target area of the sub - graph and automatically extract features. This method can quickly locate the target boundary, ensure that the area concerned by the model is the key to its recognition and classification, and is more objective and accurate than visual judgment of the boundary; and greatly reduces the dependence on labeled data, improves the generalization ability and background adaptability of the model, especially in complex backgrounds, it can maintain a high image generation quality.

[0046] (2) Improve the variational auto - encoder technology. On the basis of the original variational auto - encoder, introduce the mean distribution M and the JS divergence. The KL divergence is asymmetric and the JS divergence is symmetric, which makes the metric more balanced and will not be biased towards one of the distributions. And the JS divergence is more robust when dealing with the situation where two probability distributions do not overlap at all or only partially overlap. The generation model obtained by improving the variational auto - encoder enables the generated images to better capture the complex relationship between the samples and the background, significantly improving the sample diversity and the adaptability to complex backgrounds. By this method, not only the morphology of the samples is concerned, but also the background details can be accurately processed, significantly improving the effect of image analysis.

[0047] (3) Improve the objective function of Poisson fusion,

[0048] Based on the optimized Poisson fusion technology, it enhances the natural fusion effect between the generated samples and the background, ensuring the visual consistency and quality of the generated samples. By dynamically adjusting the image features, it optimizes the matching between the samples and the background, improving the overall quality and adaptability of the samples, especially performing excellently in complex backgrounds.

[0049] In summary, through the introduction of the CAM technology, the improvement of the variational autoencoder, and the improvement of the Poisson fusion method, this solution can achieve the precise positioning and automatic feature extraction of the target area, reduce the dependence on labeled data, improve the quality and application value of sample generation, and effectively solve the problems of lack of generalization and background adaptability in the existing technology. Brief Description of the Drawings

[0050] Figure 1 It is the flowchart of the present invention;

[0051] Figure 2 It is the flowchart of generating the heat map of the present invention;

[0052] Figure 3 It is the flowchart of improving the VAE. Detailed Embodiments

[0053] The present invention will be further described below in conjunction with the embodiments and the drawings.

[0054] Embodiment 1: Refer to Figure 1 and Figure 2 , a method for generating samples based on improved VAE and Poisson fusion, comprising the following steps:

[0055] S1, construct the target image set D1, the background image set D2, and the sub-image set D3;

[0056] Determine a single target, obtain N images containing the target to form D1, N images without the target to form D2, cut each image in D1 into several sub-images, and form D3 with all the sub-images;

[0057] S2, construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first to fifth feature maps of different scales from the sub-images, and the classification head outputs the class probability of the sub-images based on the fifth feature map;

[0058] S3, classify the sub-images in D3 with the classification model, and form the target sub-image set D4 with the sub-images containing the target;

[0059] S4, generate the corresponding target area map for each sub-image in D4 to form the first sample set R1. Among them, generate the target area map r s of the sub-image I sThe method includes steps S41 to S43;

[0060] S41, input the sub-graph I s into the classification model to obtain its fifth feature map A and class probability p c , and generate a heat map L according to the following formula c ;

[0061] ,

[0062] ,

[0063] In the formula, A k is the feature map corresponding to the fifth feature map A in channel k, is the global average gradient of A k , is the ReLU function, Z is the total number of pixels in A k , p c is the class probability of I s , is the eigenvalue of the pixel at the i-th row and j-th column in A k ;

[0064] S42, generate a mask for each pixel in the heat map to obtain a mask map M c , M c The mask M of the pixel at the i-th row and j-th column c (i, j) is obtained according to the following formula;

[0065] ,

[0066] In the formula, L c (i, j) is the eigenvalue of the pixel at the i-th row and j-th column in L c , and θ1 is the mask threshold;

[0067] S43, according to the formula , generate the target region map r s of the sub-graph I s , where is the Hadamard product;

[0068] S5, pre-train a variational autoencoder with R1 to obtain a generation model, and use the generation model to generate multiple generated images to form a second sample set R2, and merge R1 and R2 to obtain a target sample set R;

[0069] S6, randomly select an image from the target sample set R and label it as I A and randomly select an image from D2 and label it as I B , and use the Poisson fusion method to generate a fused image.

[0070] Figure 2Among them, the derivative of the prediction target with respect to the feature map refers to The operation of obtaining weights through global average pooling refers to The calculation formula for weighted summation of the feature map, corresponding to the heat map L c The formula.

[0071] Example 2: Refer to Figures 1 to 2 , specifically in S3, a classification threshold is preset, D3 is sent into the classification model to obtain the class probability of each sub - map, the sub - maps with probability classes less than the classification threshold are marked as not containing the target, and the rest are marked as containing the target. The sub - maps containing the target form the target sub - map set D4. The rest is the same as in Example 1.

[0072] Example 3: Refer to Figures 1 to 3 , based on Example 1, a generative model can be directly trained using the existing variational auto - encoder in the prior art, or an improved VAE can be used to train the generative model. In this example, the training process of the improved VAE is described in detail. The improved VAE is also obtained by improving the variational auto - encoder in the prior art. Training the improved VAE to obtain the generative model includes steps S51 - S53;

[0073] S51, obtain a variational auto - encoder, including an encoder and a decoder;

[0074] The parameters of the encoder are ϕ, which is used to map the input image r to the probability distribution in the latent space , and sample to obtain M1 latent variables, where the m - th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z);

[0075] The parameters of the decoder are θ, which is used to map the latent variable back to the original data space to generate a reconstructed image;

[0076] S52, modify the evidence lower bound ELBO of the variational auto - encoder as follows to obtain the improved VAE;

[0077] ,

[0078] ,

[0079] In the formula, is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ (z|r)||M) is the KL divergence of q ϕThe KL divergence between (z|r) and M, KL(p θ (z)||M) is the KL divergence between p θ (z) and M;

[0080] In S53, use R1 to train and improve the VAE to maximize the ELBO to obtain a generative model.

[0081] and KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated according to the following formulas respectively;

[0082] ,

[0083] ,

[0084] ,

[0085] In the formula, p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are the mean and variance of q ϕ (z|r) respectively, μ2 and Σ2 are the mean and variance of p θ (z) respectively, μ M and Σ M are the mean and variance of the average distribution M respectively, tr(·) represents the trace of a matrix, det(·) represents the determinant of a matrix, k is the dimension of the distribution, α is a constant, and I is the identity matrix.

[0086] The JS divergence is smoothed. In the VAE, to make the metric more balanced and not biased towards one of the distributions, the present invention introduces the JS divergence and the average distribution M, and the JS divergence is more robust in dealing with the case where two probability distributions do not overlap at all or only partially overlap. There may also be a small probability of division by zero problem in the calculation of the JS divergence, and the division by zero error in the calculation is effectively avoided by adding smoothing processing.

[0087] Example 4: Refer to Figures 1 to 3 , in S6, the Poisson fusion method is an improved Poisson fusion method, and its objective function F is:

[0088] ,

[0089] ,

[0090] In the formula, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω =I A | ∂Ω, f is the fused image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter for controlling the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') within the fusion region Ω, I A (x', y'), I B (x', y') are the pixel values of I A and I B at (x', y') respectively, ||·|| is the Euclidean norm.

[0091] The rest is the same as in Embodiment 1, Embodiment 2 or Embodiment 3.

[0092] Based on the existing Poisson fusion method, the present invention improves the objective function and introduces w(x', y'). When α is 0, the equation degenerates into the original Poisson fusion. When I A and I B have a large gradient difference at (x', y'), w(x', y') is small, and substituting it into the objective function F reduces the fusion intensity at (x', y'). In this way, the improved Poisson fusion method can automatically adapt to the feature differences of the images, making the fusion result more natural and harmonious visually.

[0093] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sample generation method based on the fusion of improved VAE and Poisson, characterized in that, It includes the following steps: S1. Construct a target image set D1, a background image set D2, and a sub-image set D3; Determine a single target, obtain N images containing the target to form D1, N images not containing the target to form D2, cut each image in D1 into several sub-images, and form D3 with all the sub-images; S2. Construct a YOLOv8s network and train it with D3 to obtain a classification model. The YOLOv8s network includes a backbone network and a classification head. The backbone network sequentially extracts the first feature map to the fifth feature map of different scales from the sub-images, and the classification head outputs the class probability of the sub-images based on the fifth feature map; S3. Classify the sub-images in D3 with the classification model, and form a target sub-image set D4 with the sub-images containing the target; S4. Generate a corresponding target region map for each sub - figure in D4 to form the first sample set R1. Among them, the method for generating the target region map r of sub - figure I s includes steps S41 - S43; s ​ S41, input the sub-graph I s into the classification model to obtain its fifth feature map A and class probability p c , and generate a heat map L according to the following formula c ; ， ， Where A k is the feature map corresponding to the fifth feature map A in channel k, is the global average of the gradients of A k , is the ReLU function, Z is the total number of pixels in A k , p c is the class probability of I s , is the eigenvalue of the pixel at the i-th row and j-th column in A k ; S42. Generate a mask for each pixel in the heat map to obtain a mask map M c , M c The mask M of the pixel in the i-th row and j-th column c (i, j) is obtained according to the following formula; ， where L c (i, j) is the eigenvalue of the pixel in the i-th row and j-th column of L c , and θ1 is the mask threshold; S43, generate the target region graph r of sub-graph I according to the formula , where s is the Hadamard product s ; ​ S5. Pre-train a variational autoencoder with R1 to obtain a generation model, and use the generation model to generate multiple generated images to form a second sample set R2, and merge R1 and R2 to obtain a target sample set R; S6. Select an image from the target sample set R and label it as I A , select an image from D2 and label it as I B , and generate a fused image using the Poisson fusion method; The variational autoencoder in S5 is an improved VAE, and training the improved VAE to obtain the generation model includes steps S51 to S53; S51. Obtain a variational autoencoder, including an encoder and a decoder; The encoder parameter is ϕ, which is used to map the input image r to a probability distribution in the latent space , and M1 latent variables are sampled, where the m-th latent variable is z (m) , 1 ≤ m ≤ M1, and the prior distribution of the latent variable is p θ (z); The parameters of the decoder are θ, which are used to map the latent variable back to the original data space to generate a reconstructed image; S52. Modify the evidence lower bound ELBO of the variational autoencoder as follows to obtain an improved VAE; ， ， where is the reconstruction error, JS α (q ϕ (z|r)||p θ (z)) is the JS divergence, M is the average distribution of q ϕ (z|r) and p θ (z), KL(q ϕ (z|r)||M) is the KL divergence between q ϕ (z|r) and M, KL(p θ (z)||M) is the KL divergence between p θ (z) and M; S53. Train the improved VAE with R1 to maximize ELBO to obtain a generation model.

2. The sample generation method based on the improved VAE and Poisson fusion according to claim 1, wherein The specific content of S3 is: preset a classification threshold, send D3 into the classification model, obtain the class probability of each sub-image, mark the sub-images with a probability class less than the classification threshold as not containing the target, and the rest as containing the target, and form a target sub-image set D4 with the sub-images containing the target.

3. The sample generation method based on the improved VAE and Poisson fusion according to claim 1, characterized in that, , KL(q ϕ (z|r)||M), KL(p θ (z)||M) are calculated respectively according to the following formula; ， ， ， where p θ (r|z (m) ) is the reconstructed data distribution corresponding to z (m) , μ1 and Σ1 are the mean and variance of q ϕ (z|r) respectively, μ2 and Σ2 are the mean and variance of p θ (z) respectively, μ M , Σ M are the mean and variance of the average distribution M respectively, tr(·) represents the trace of a matrix, det(·) represents the determinant of a matrix, k is the distribution dimension, α is a constant, and I is the identity matrix.

4. A sample generation method based on the fusion of improved VAE and Poisson according to claim 1, characterized in that, The JS divergence is smoothed.

5. The sample generation method based on the improved VAE and Poisson fusion according to claim 1, characterized in that The Poisson fusion method in S6 is an improved Poisson fusion method, and its objective function F is: ， ， Where, Ω is the fusion region, and the boundary condition of Ω is I B | ∂Ω =I A | ∂Ω , f is the fusion image to be solved, min is to take the minimum value, is the gradient operator, α1 is the parameter for controlling the contrast sensitivity, w(x', y') is the fusion intensity at the position (x', y') within the fusion region Ω, I A (x', y'), I B (x', y') are the pixel values of I A and I B at (x', y') respectively, ||·|| is the Euclidean norm.

Images