A data synthesis method based on conditional GAN network adaptive feature fusion

Abstract

The present application aims to provide a data synthesis method based on adaptive feature fusion of conditional GAN network, which adds new class labels in existing training samples, that is, synthesizes new target classes into existing training samples, and then adaptively adjusts the original target detection model. Specifically, a large number of foreground images are generated by using a GAN network. Since the size, shape and projection angle of each segmentation region in the background image are different, in order to naturally synthesize the foreground target into the background image, the present application performs three rounds of screening on the regions in the background image. Through the region screening, the foreground will not pass through the strong image discontinuous points, and the perspective of the foreground and the synthesized region is ensured to be consistent. Finally, the synthesis of the foreground image and the background image is realized in an adaptive fusion manner, and a synthesized image close to the real acquisition can be generated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a data synthesis method based on adaptive feature fusion of conditional GAN network. BACKGROUND

[0002] In practical applications, there is often a need to dynamically add new target categories to a deep learning target detection model according to actual needs, that is, to synthesize new category targets into existing training images, and then adjust the deep learning target detection model according to the training samples. In the process of synthesizing new category targets into existing training images, factors such as target diversity, shape and size, geometric deformation, and light changes can have a crucial impact. Existing synthesis methods ignore the effects of occlusion and perspective, which not only leads to synthesized samples that cannot approximate the effects of real shooting samples, but also causes the synthesized images to deform due to the inability to maintain the stereoscopic invariance of the foreground. In addition, relying only on a small amount of existing labeled foreground samples cannot meet the diversity of synthesized samples. SUMMARY

[0003] The main purpose of the present application is to provide a data synthesis method based on adaptive feature fusion of conditional GAN network, which aims to add new category labels to existing training samples, that is, to synthesize new target categories into existing training samples, and then adaptively adjust the source domain model.

[0004] To achieve the above purpose, the present application adopts the following contents:

[0005] A data synthesis method based on adaptive feature fusion of conditional GAN network, comprising the following steps:

[0006] Collecting foreground objects and generating foreground images using a conditional GAN network, the conditional GAN network can input conditions and generate multiple types of foreground image samples that meet the conditions according to the foreground images;

[0007] Obtaining a background image and pre-processing the background image, segmenting the background image according to local color and texture cues to obtain regions with continuous pixels, and using a convolutional neural network to predict a depth map for each segmented region;

[0008] According to the segmentation information and the depth information, the background image is subjected to a first round of screening to screen out regions that meet the area requirements and the projection angle requirements, and the regions are recorded as first candidate regions;

[0009] According to the surface normal a of the mask region in the foreground image and the surface normal b of the first candidate region in the background image, a second round of screening is performed to screen out regions whose included angle between a and b meets the threshold value from the first candidate region, and the regions are recorded as second candidate regions;

[0010] A homography matrix A of the mask region in the foreground image is calculated, a homography matrix B of the second candidate region is calculated, a final synthesis region is screened from the second candidate region, the homography matrix B is approximately orthogonal to the homography matrix A, and a rotation transformation of the foreground image is performed to make the homography matrix A orthogonal to the homography matrix B;

[0011] The foreground image after the rotation transformation is fused into the final synthesis region in an adaptive manner to generate a synthesis image close to a real acquisition.

[0012] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0013] In order to make the sample of data synthesis more diverse, the present application uses a conditional GAN network to generate a large number of foreground images. Specifically, a conditional GAN model is trained, and the generator and the discriminator both increase an additional information C as a condition. The condition can be category information or other modal data. By feeding the additional information C to the discriminator model and the generator model as part of the input layer, a conditional GAN is realized. Here, the conditional GAN is a GAN with a conditional constraint, and a condition variable is introduced in the modeling of the generator (G) and the discriminator (D). The condition variable is a category, and the foreground image with multiple categories and balanced sample quantity of each category is generated.

[0014] The method proposed in the present application can randomly fuse different foreground images and background images to increase new synthesis sample data.

[0015] In a natural shooting image, a foreground target object is usually contained in a well-defined background region. For a common three-dimensional target, it is generally required to be contained in a region with uniform color and texture to approximate this constraint. Since the size, shape and projection angle of each segmentation region in the background image are different, in order to make the foreground target naturally synthesized into the background image, the present application performs three rounds of screening on the regions in the background image. Through region screening, not only the foreground will not pass through strong image discontinuous points, but also the perspective of the foreground and the synthesis region is consistent. BRIEF DESCRIPTION OF DRAWINGS

[0016] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0017] Figure 1 is a flowchart of a data synthesis method based on adaptive feature fusion of a conditional GAN network according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a CGAN network;

[0019] Figure 3is a schematic diagram of a foreground image generated by a CGAN network;

[0020] Figure 4 is a schematic diagram of a gPb-UCM segmentation result on a background image;

[0021] Figure 5 is a schematic diagram of an upsampling manner proposed in the FCRN;

[0022] Figure 6 represents a structure diagram of an up-convolution in the FRCN;

[0023] Figure 7 is a schematic diagram of depth prediction and plane fitting on a background image. DETAILED DESCRIPTION

[0024] In order to more clearly illustrate the present application, the present application will be further described below in conjunction with preferred embodiments. It should be understood by those skilled in the art that the specific description below is illustrative rather than limiting, and should not limit the protection scope of the present application.

[0025] An embodiment of the present application proposes a data synthesis method based on adaptive feature fusion of a conditional GAN network, referring to Figure 1 , mainly comprising steps S1-S6:

[0026] S1: collect foreground objects, and generate foreground images using a conditional GAN network, in detail:

[0027] A CGAN (Conditional Adversarial Nets) is used, and both the generator and the discriminator increase additional information C as a condition, which can be category information or other modal data. By feeding the additional information C to the discriminator model and the generation model as part of the input layer, a conditional GAN is realized.

[0028] In the generation model, the prior input noise p(z) and the condition information c are combined, and the objective function of the conditional GAN is a binary minimax game with conditional probability.

[0029] In the generator, input z is randomly taken from the preposed random distribution, and is combined with the condition input c to form a new hidden representation G(z). In the discriminator, the real data x or the generated data G(z) are both input together with the condition information c for discrimination.

[0030] Figure 2 The CGAN network structure is shown in the schematic diagram, Figure 3The schematic diagram of CGAN network generating foreground image is shown. In this way, by constructing CGAN to generate foreground images of multiple categories, a specified number of foreground image samples can be generated in different categories according to requirements.

[0031] There are two ways to collect foreground objects: one is to select from an existing data set, and the other is to re-shoot images as foreground image samples.

[0032] When re-shooting images as foreground image samples, in order to accurately obtain the mask of the foreground object, the GrabCut algorithm is selected here, which uses a Gaussian mixture model to model the foreground and background in the foreground object, then constructs a Markov random field on the pixel label, and finally provides an accurate foreground mask through friendly human-computer interaction after multiple iterations.

[0033] S2: Obtain the background image and pre-process the background image, segment the background image according to local color and texture cues to obtain regions with continuous pixels, and use a convolutional neural network to predict a depth map for each segmented region, i.e., use a gPb-UCM image segmentation algorithm based on color and texture to determine the region position where the foreground image is placed, and use an image depth prediction algorithm based on a fully convolutional neural network to obtain the depth information of the background image. In detail:

[0034] 2.1 gPb-UCM image segmentation algorithm based on color and texture

[0035] This algorithm is a region segmentation algorithm relying on image edge information. First, the probability gPb (Globalized Probability of Boundary) of each pixel point in the graph as an edge is calculated, then the above result is converted into multiple closed regions using an improved OWT (Oriented Watershed Transform), and finally the closed regions are converted into a region tree using the UCM (Ultrametric Contour Map) method by setting different thresholds.

[0036] gPb is a weighted linear combination of mPb (Multiscale Probability of Boundary) and sPb (Spectral Probability of Boundary). mPb is a concept of multi-scale introduced on the basis of calculating the edge probability of pixel points in the three channels and the texture channel of the CIE color space of the picture. For each scale, the weighted sum of the Pb values of the four channels is calculated, and the formula is as follows:

[0037]

[0038] where i, s are different scales and four channels respectively, and a represents the weight of the channel under different scales. G is the gradient direction function of the pixel point. A circular area with a size of σ is selected at the pixel point (x, y), and is divided into two parts according to the angle β, and the chi-square distance of their histograms is calculated. Finally, the maximum response under different β angles is used to measure the edge strength of different pixel points:

[0039]

[0040] In addition, the saliency segmentation curve of the image is obtained by clustering the spectral changes. First, a sparse symmetric similarity matrix is constructed according to the mPb value between two pixel points i and j:

[0041]

[0042] where ij represents the line segment between two pixel points i and j, and ρ is a constant coefficient, which is set to 0.1. The matrix represents the similarity between pixels, and then:

[0043]

[0044]

[0045] The n+1-dimensional feature vector is calculated by formula (5), and the Gaussian directional derivative filter is convolved in the θ direction of each feature vector, so as to obtain the value of sPb:

[0046]

[0047] where the parameter is the physical definition of the eigenvalue problem. mPb represents the edge information of the image, and sPb is the salient boundary line of the image. The weighted sum of mPb and sPb is obtained to obtain the value of gPb:

[0048]

[0049] where ζ i,s and γ are constants, and then the gPb value is subjected to sigmoid transformation, so that it is distributed between [0, 1], which represents the probability of the pixel point as an edge, also known as the weight of the pixel. However, the value is also affected by the direction β, and the maximum value of gPb corresponding to the β taken by the pixel point (x, y) is taken as the edge probability.

[0050] Next, the edge probability of the minimum value in the region is taken as the seed point position, and the modified OWT algorithm is used to obtain suitable region and watershed arc information. Finally, the UCM algorithm based on graph merging is used to merge the segmented graphs to obtain a more complete region. By setting different thresholds for comparison, it is determined that a threshold of 0.2 best expresses the details of the background image region segmentation.

[0051] Reference Figure 4 This is an example of gPb-UCM segmentation of a background image. From left to right in the image: the input RCB image; the color- and texture-based gPb-UCM segmentation map; random color filling on the segmentation map; region filtering: regions suitable for placing the foreground image are filled with random colors, while regions unsuitable for placement retain the pixel values ​​of the original image.

[0052] 2.2 Image Depth Prediction Algorithm Based on Fully Convolutional Neural Network

[0053] The Fully Convolutional Residual Networks (FCRN) algorithm uses ResNet-50 to extract features from images. To predict depth information, the extracted features need to be upsampled. Common upsampling methods include bilinear interpolation and deconvolution. Bilinear interpolation blurs the edges of the feature map and introduces significant noise; while deconvolution can lead to large differences between adjacent pixels in the feature map, potentially causing a checkerboard effect that severely interferes with depth prediction. Therefore, FCRN proposes a novel upsampling structure called Up-Convolutional Blocks, detailed in [link to documentation]. Figure 5 As shown. By Figure 5 As can be seen, the feature map is first enlarged using a 2×2 pooling layer, which is actually interpolated with 0. Then, it goes through a 5×5 convolutional layer so that the positions filled with 0 are no longer all 0. Finally, the ReLU activation function is used to obtain the upsampling result. This operation can effectively reduce the checkerboard effect caused by deconvolution.

[0054] exist Figure 5 Building upon a), FCRN further introduces the concept of residual blocks, which superimpose the parameters of the feature maps before and after convolution, thus obtaining... Figure 5b) the up projection blocks, which on the one hand deepen the network depth by introducing more parameters, and on the other hand better preserve the feature information of the shallow layers in the up-sampling process. Through experiments, it is found that this structure is more effective for extracting the depth information of the image. In addition, since there are many 0 values after the up pooling layer in the up projection structure, followed by a 5x5 convolution layer, which makes it difficult for the network to train. Therefore, the FCRN algorithm is further improved, replacing a 5x5 large size convolution kernel with 3x3, 2x3, 3x2, 2x2 multiple convolution kernels, and merging the convolution results according to the spatial relationship. Not only the results obtained are the same as the original 5x5 large size convolution results, but also the convolution times are reduced, and the up-sampling process is accelerated, as shown in Figure 6 Through four times of up-sampling, a large-scale feature map can be obtained, and the depth information of each pixel can be predicted by regression.

[0055] The depth information depth obtained by the above method can convert the two-dimensional coordinates (u, v) of the pixels on the image into three-dimensional camera coordinates (X, Y, Z), and the formula is as follows:

[0056]

[0057] Where W and H are the width and height of the depth map respectively, and f is the focal length of the camera. Then, the RANSAC algorithm is used to fit the regional plane according to the three-dimensional coordinates. This method is a learning technique that estimates the best plane by randomly sampling data. Generally, the outliers (noise) in the data do not affect the fitted plane. First, a sample subset consisting of the minimum number of samples that can determine the model parameters is randomly selected from the sample points. Then, the relationship with the fitted model is determined according to the threshold. Those within the threshold are called inliers, and those outside are called outliers (abnormal points). Repeat the above operation multiple times. The maximum number of inliers accounts for the proportion of total data points, which is the best fitted plane. Finally, the plane normal vector can be obtained according to the fitted plane. For details, see Figure 7 .

[0058] In summary, the gPb-UCM image segmentation algorithm based on color and texture can obtain the specific area position of the foreground image placement, the image depth prediction algorithm based on the full convolution neural network can obtain the depth information of the background image, and then the RANSAC plane fitting can obtain the plane normal vector of the conveyor belt surface. According to the normal vector, perspective transformation is performed to make the defect foreground have a depth of field effect, and the synthesized defect sample is more realistic.

[0059] S3: According to the segmentation information and the depth information, the background image is subjected to a first round of screening to screen out regions that meet the area requirement and the projection angle requirement, and is recorded as the first candidate region.

[0060] Specifically, each segmented region in the background image is traversed according to the segmentation information to filter out regions with too small width and height, too small area, and too small width-to-height ratio; and regions with too small projection angle are filtered out according to the depth information.

[0061] S4: According to the surface normal a of the mask region in the foreground image and the surface normal b of the first candidate region in the background image, a second round of screening is performed to screen out a region with an included angle between a and b meeting a threshold value from the first candidate region, and the region is recorded as a second candidate region.

[0062] Specifically, the surface normal a of the mask region in the foreground image and the surface normal b of the first candidate region in the background image are calculated by using the RANSAC algorithm. The RANSAC algorithm is an iterative algorithm that can correctly estimate the parameters of a mathematical model from a set of data containing outliers. Given a set P containing N data points, it is assumed that most points in set P can be generated by a model, and at least n points (n

[0063] (1) Randomly select n data points from set P, i.e., calculate four points of the homography matrix;

[0064] (2) Use the n data points to fit the model M;

[0065] (3) Substitute the remaining points in set P into model M to calculate the number of internal points m;

[0066] (4) Compare the number of internal points of the current model M with that of the previous best model, and record the model parameters and the corresponding number of internal points with the maximum number of internal points;

[0067] (5) Repeat steps (1) to (4) until all iterations are completed, find the model M with the maximum number of internal points as the fitting result, and finally obtain the expression of the fitting plane:

[0068] Ax+By+Cz+D=0 (9)

[0069] [A;B;C] T is the normal vector of the fitting plane, and D is a constant. Assuming that the surface normal vectors of a region in the foreground image and the background image are a and b respectively, the two normal vectors are formalized as a=(A1, B1, C1) and b=(A2, B2, C2),

[0070] The included angle between the two plane normal vectors a and b is calculated as:

[0071]

[0072] Filtering out the region with an included angle between a and b not greater than 15° from the first candidate region as a second candidate region.

[0073] S5: Calculate the homography matrix A of the mask region in the foreground image, calculate the homography matrix B of the second candidate region, filter out the final synthesis region from the second candidate region, which is approximately orthogonal to the homography matrix A, and perform image rotation transformation on the foreground image so that the homography matrix A and the homography matrix B are orthogonal.

[0074] Specifically, the derivation process of the homography matrix is as follows: the following constraints are obtained from the equations formed by the pixel coordinate system and the world coordinate system:

[0075]

[0076] In the non-uniform coordinates,

[0077] x'2=x2 / z2,y'2=y2 / z2 (12)

[0078]

[0079]

[0080] The default value z1=1 does not lose generality, and then rearrange equations (13), (14) to obtain the following formula:

[0081] x'2(H 31 x1+H 32 y1+H 33 )=H 11 x1+H 12 y1+H 13 (15)

[0082] y'2(H 31 x1+H 32 y1+H 33 )=H 21 x1+H 22 y1+H 23 (16)

[0083] In order to obtain the homography matrix H, equations (15), (16) are written in matrix form:

[0084]

[0085]

[0086] Where,

[0087] h=(H 11 ,H 12 ,H13 ,H 21 ,H 22 ,G 23 ,H 31 ,H 32 ,H 33 ) T (19)

[0088] a x =(−x1,−y1,−1,0,0,0,x'2x1,x'2y1,x'2) T (20)

[0089] a y =(0,0,0,−x1,−y1,−1,y'2x1,y'2y1,y'2) T (21)

[0090] After obtaining the homography estimation H matrix of the foreground image and the background image, the two-norm between the two H is calculated as follows:

[0091]

[0092] Wherein, H1 represents the homography matrix in the foreground image, H2 represents the homography matrix in the background image, if the value of d2(H1, H2) is less than the threshold value, the region in the background image is regarded as the final candidate region which can be synthesized.

[0093] S6: The foreground image after rotation transformation is fused into the final synthesis region in an adaptive manner to generate a synthesis image which is close to the real acquisition.

[0094] Here, the final synthesis region selected can be subjected to fast Fourier calculation with the foreground image mask region to obtain a safe region for placing the foreground image, so that it can be ensured that the foreground will not exceed the boundary of the background after synthesis.

[0095] The data synthesis method provided in the application generates a large number of foreground images by using a GAN network, specifically, a class conditional GAN model is trained, and on the basis of not using manually annotated class labels, the labels generated by clustering in the discriminator feature space are used as conditions to generate foreground images unsupervisedly; different foreground images and background images can be randomly fused to increase new synthesis sample data. In order to enable the foreground target to be naturally synthesized into the background image, the region in the background image is screened for three rounds, through the region screening, not only the foreground will not pass through the strong image discontinuous point, but also the perspective of the foreground and the synthesis region is ensured to be consistent.

[0096] Obviously, the above embodiments of the present application are merely exemplary for clearly illustrating the present application, and are not intended to limit the implementation manners of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art, and here, all the implementation manners cannot be exhausted, and any obvious changes or variations derived from the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A data synthesis method based on conditional GAN network adaptive feature fusion, characterized in that, The method comprises the following steps: Collecting foreground objects and generating foreground images using a conditional GAN network that can input conditions to generate multiple types of foreground image samples that meet the conditions according to the foreground images; Obtaining a background image and pre-processing the background image, segmenting the background image according to local color and texture cues to obtain regions with continuous pixels, and using a convolutional neural network to predict a depth map for each segmented region; Performing a first round of screening on the background image according to segmentation information and depth information to screen out regions that meet area requirements and projection angle requirements, denoted as first candidate regions; Performing a second round of screening according to the surface normal a of the mask region in the foreground image and the surface normal b of the first candidate region in the background image to screen out regions whose included angle between a and b meets a threshold value from the first candidate regions, denoted as second candidate regions; Calculating the homography matrix A of the mask region in the foreground image and the homography matrix B of the second candidate region, and screening out the second candidate region whose homography matrix B is approximately orthogonal to the homography matrix A as the final synthesis region, and then performing a rotation transformation on the foreground image to make the homography matrix A orthogonal to the homography matrix B; Fusing the foreground image after the rotation transformation into the final synthesis region in an adaptive manner to generate a synthesis image that approximates a real collection.

2. The data synthesis method of claim 1, wherein, Wherein, The collection of foreground objects includes two ways, the first way is to select foreground image samples from existing image data sets, and the second way is to re-shoot images as foreground image samples.

3. The data synthesis method of claim 2, wherein, When re-shooting images as foreground image samples, a GrabCut algorithm is used to obtain a mask of the foreground image samples.

4. The data synthesis method of claim 1, wherein, Wherein, The pre-processing method of the background image is to determine the region position of the foreground image using a gPb-UCM image segmentation algorithm based on color and texture, and to obtain the depth information of the background image using a full convolutional neural network-based image depth prediction algorithm.

5. The data synthesis method of claim 1, wherein, Wherein, The specific method of the first round of screening is to traverse each segmented region in the background image according to the segmentation information to filter out regions that are too small in width and height, too small in area, and too small in aspect ratio, and to filter out regions that are too small in projection angle according to the depth information.

6. The data synthesis method of claim 1, wherein, Wherein, The surface normal a of the mask region in the foreground image and the surface normal b of the first candidate region in the background image are calculated according to the RANSAC algorithm.

Images