A method and system for augmenting semantic segmentation training data based on generative adversarial network (GAN)

Abstract

The method and system for augmenting semantic segmentation training data based on the generative adversarial network (GAN) disclosed in the present invention provide the pixel-level true value of the target object in the annotation image based on unsupervised learning, and obtain the mean corresponding to the foreground and background through intra-class mean calculation. The mean constrains the annotation image to reversely learn the input image and train the relevant parameters of the generator network in the GAN. Given an arbitrary synthetic annotation image, providing an arbitrary intra-class mean, adding random noise, a simulated target image can be obtained through the learned generative network. This method can generate more synthetic target images, which can be used as input, increasing a large amount of training set data, thereby improving the efficiency and accuracy of deep convolutional semantic segmentation network training.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a method and system for augmenting semantic segmentation training data based on a generative adversarial network (GAN). Background Art

[0002] The field of computer vision has undergone tremendous changes in recent years. Deep Neural Networks (DNN) have achieved great success in the classification task of ImageNet. Normally, the task that DNN can achieve is to input a picture and output a judgment, such as whether it is the result you want or not. The purpose of the adversarial generative network GAN is to use this network in reverse, that is, to output the object by inputting the name of an object. GAN is based on a game-based training process. There are generators G (generator) and discriminators D (Discriminator) in the network, and there are two data domains, X and Y. Generator G is responsible for taking the data in the X domain and trying to imitate it into Y', and hiding them in Y. The discriminator D tries hard to separate the fake data from the real data, and repeats this process until the generative model and the discriminative model cannot improve themselves. At this time, the generative model will become a perfect model.

[0003] In machine learning, we need a lot of data to train deep learning models to avoid overfitting. Deep neural networks have a lot of parameters, so if they are not trained with enough data, they tend to remember the entire training set, which results in good training results but poor results on the test set. The traditional data augmentation method used to expand training data still transforms the image itself rather than generating new data, so many times we still cannot obtain a lot of training data. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method for augmenting semantic segmentation training data based on a generative adversarial network (GAN), comprising the following steps:

[0005] Step S1: Match the input original image with the corresponding correct annotation image, and optimize To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, is the identifier;

[0006] Step S2: The generator G network extracts the target area of ​​the correctly labeled image, calculates the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimizes the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, For the identifier to identify and G(t),p data Represents the probability distribution of given data;

[0007] Step S3, generating different synthetic target images by arbitrarily changing one or more of the correct labeled images, the intra-class mean and the noise input into the trained G network.

[0008] Preferably, the step S2 further comprises: optimizing a cycle loss To constrain, where p data Represents the probability distribution of given data. By training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph.

[0009] The present invention also discloses a semantic segmentation training data augmentation system based on a generative adversarial network (GAN), comprising: a first training module for matching and learning the input original image with the corresponding correctly labeled image, and optimizing the training data to obtain the semantic segmentation training data. To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, The second training module is used for the generator G network to extract the target area of ​​the correctly labeled image, calculate the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimize the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, For the identifier to identify With G(t), p dataRepresents the probability distribution of given data; a synthetic target graph generation module is used to generate different synthetic target graphs by arbitrarily changing one or more of the correct labeled graph, the intra-class mean and the noise input into the trained G network.

[0010] Preferably, the second training module includes: a constraint module for optimizing a cycle loss To constrain, where p data Represents the probability distribution of given data. By training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph.

[0011] The present invention also discloses a semantic segmentation training data augmentation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0012] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0013] The deep learning training set expansion technology based on the adversarial generative network GAN disclosed in the present invention is based on unsupervised learning, provides the pixel-level true value of the target object in the annotation map, and obtains the mean corresponding to the foreground and the background through the intra-class mean calculation. The mean-constrained annotation map reversely learns to obtain the input map, and trains the relevant parameters of the generator network in the GAN. Given an arbitrary synthetic annotation map, providing an arbitrary intra-class mean, adding random noise, and a simulated target image can be obtained through the learned generation network. More synthetic target maps can be generated by this method, and these synthetic target maps can be used as input, which increases a large amount of training set data, thereby improving the efficiency and accuracy of deep convolutional semantic segmentation network training. Through the above method, any input picture A, which can be a picture of various forms such as RGB pictures, grayscale pictures, depth pictures, inspection pictures, thermal imaging pictures, radar point cloud projection pictures, B-ultrasound / color ultrasound pictures, etc., can be matched with its annotation map B. After the network training is completed, any synthetic annotation map is given, and a large amount of synthetic input can be obtained, thereby expanding the deep learning training set for semantic segmentation, achieving the effect of improving accuracy and accelerating the training of the semantic segmentation network.

[0014] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0016] Attached Figure 1 The RGB image and the corresponding correct annotation image are presented taking the RGB image as an example.

[0017] Attached Figure 2 A schematic diagram of G network training disclosed in an embodiment of the present invention.

[0018] Attached Figure 3 A schematic diagram of a training set expansion framework disclosed in an embodiment of the present invention.

[0019] Attached Figure 4 The figure is a flow chart of steps for generating an input original image and a corresponding annotation image according to an embodiment of the present invention.

[0020] Attached Figure 5 This is a schematic diagram of a specific flow chart of step S105 disclosed in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0024] Unless otherwise defined, the technical or scientific terms used herein shall have the common meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "an" do not indicate quantity restrictions, but indicate the existence of at least one.

[0025] In machine learning, we need a lot of data to train deep learning models to avoid overfitting. Deep neural networks have a lot of parameters, so if they are not trained with enough data, they tend to remember the entire training set, which will lead to good training results but poor results on the test set. However, many times we cannot obtain a large amount of training data, so this embodiment discloses a training set augmentation method based on a GAN network to solve the problem of insufficient training data.

[0026] The GAN network contains two types of networks G and D. Among them, G is a generator, and its function is to generate images. That is to say, after inputting a random code z, it will output a fake image G(z) automatically generated by the neural network. And pix2pix made a small change to the traditional GAN. It no longer inputs random noise, but inputs the pictures given by the user to establish the corresponding relationship between input and output. Therefore, pix2pix must use paired data for training, but in many cases paired data is difficult to obtain. Existing data augmentation is one of the commonly used techniques in deep learning. It is mainly used to increase the training data set, make the data set as diverse as possible, and make the trained model have stronger generalization ability. At present, data augmentation mainly includes horizontal / vertical flipping, rotation, scaling, cropping, shearing, translation, contrast, color jitter, noise, etc. Data augmentation is used in deep learning to improve the robustness of the network. However, the traditional data augmentation method still transforms the image itself instead of generating new data. As shown in the attached figure, the image is transformed into a real-time image, which is a very powerful image processing tool. The image is a very powerful image processing tool that can be used to generate a lot of data. The image is a very powerful image processing tool that can be used to generate a lot of data. Figure 1As shown, taking RGB pictures as an example, after inputting an RGB picture, the semantic segmentation network will output a semantic segmentation result, that is, each object in the picture is classified according to the same color, and different objects are classified according to different colors. Therefore, training a semantic segmentation network requires both a large number of RGB pictures similar to the left picture and a large number of ground truth correctly labeled pictures similar to the right picture. The traditional data augmentation method can only transform the image itself, but cannot generate new data, and it is also impossible to generate a ground truth picture that meets the requirements. The semantic segmentation training data augmentation method based on the adversarial generative network GAN disclosed in the present invention can generate a large number of such image pairs that can be used for semantic segmentation training, thereby enhancing the results of semantic segmentation.

[0027] This embodiment discloses a method for augmenting semantic segmentation training data based on a generative adversarial network (GAN). Based on the CycleGAN level, a small number of input images are used for learning, and some fake images that are extremely similar to the original images are generated for reverse training, so that the deep learning network can obtain a large amount of data sets for training and learning, and can obtain output results with higher accuracy and more stability. The basic expression formula of GAN is:

[0028]

[0029] In the above formula, G represents the generator, which outputs a generated image; D represents the discriminator, which outputs the probability of discriminating a real image, x is the real image, and z is the noise. That is, we train D to have the maximum discrimination accuracy, that is, the first and second terms in the above formula are both maximized, and train G to generate the most easily fooling pictures of the discriminator, that is, the second term in the above formula is minimized. Figure 1 Learning together can make deep convolutional networks more accurate and more robust.

[0030] The semantic segmentation training data augmentation method based on the Generative Adversarial Network (GAN) of this embodiment specifically includes the following steps:

[0031] Step S1: Match the input original image with the corresponding correct annotation image, and optimize To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, is the identifier, p data Represents the probability distribution of the given data. In the above formula, t is the input annotation map, and F(s) is the generated output annotation map.

[0032] Step S2: The generator G network extracts the target area of ​​the correctly labeled image, calculates the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimizes the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, For the identifier to identify With G(t), p data Represents the probability distribution of given data.

[0033] The G process extracts the target area of ​​the input correctly labeled map ground truth, and calculates the pixel average value in the same area of ​​the input target map, that is, the intra-class mean calculation, to constrain the reverse learning process. In the process of learning to generate the input target map from the input correctly labeled map t, the G network parameters are continuously optimized to obtain a more accurate and more satisfactory G network.

[0034] In some specific embodiments, step S2 further includes: optimizing a cycle loss To constrain, where p data Represents the probability distribution of given data. By training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph. That is, the complete expression of CycleGAN is L(G, F, DX, DY) = L GAN (G,DY,X,Y)+L GAN (F,DY,X,Y)+λL cyc (G,F).

[0035] Step S3, generating different synthetic target images by arbitrarily changing one or more of the correct labeled images, the intra-class mean and the noise input into the trained G network.

[0036] As attached Figure 2 As shown in the figure, S1i to Sni are different forms of input target images, that is, they come from the same image but may have different attributes such as RGB image, grayscale image, depth image, inspection image, thermal imaging image, radar point cloud projection image or B-ultrasound / color ultrasound image, etc. i Perform the above matching learning. Control the reverse G by calculating the mean value within the class i process, optimize the parameters of the G1 network, Get the most suitable G and F through training.

[0037] After training the G network, we can set a correctly labeled ground truth map at will, give a class mean, that is, the average pixel value of the target area, and randomly provide a noise Z. After passing through the G network, we can generate an output map similar to the target but different in type, thus achieving our goal of expanding the training data. Each time we change one of the three randomly set values, we can obtain a completely new learned target map.

[0038] Attached Figure 3 is a schematic diagram of the training set expansion framework, ti' is an arbitrarily set annotation map ground truth, and the class mean and noise are arbitrarily set. Such a set of random data is input into the trained G1 network to generate an output map that is very similar to the target but different, namely G(ti+Zi). The specific form of the output map, namely RGB image, grayscale image, depth image, inspection image, thermal imaging image, radar point cloud projection image or B-ultrasound / color ultrasound image, is determined by the G1 network. Different G networks can output different image forms of the same target.

[0039] The deep learning training set expansion technology based on the generative adversarial network GAN disclosed in this embodiment is based on unsupervised learning, provides the pixel-level true value of the target object in the annotation image, and obtains the mean corresponding to the foreground and background through intra-class mean calculation. The mean-constrained annotation image (Figure B) is reversely learned to obtain the input image (Figure A), and the relevant parameters of the generator network in the GAN are trained. When we give an arbitrary synthetic annotation image, provide an arbitrary intra-class mean, add random noise, and a simulated target image can be obtained through the learned generation network. This method can generate more synthetic target images, which can be used as input, increasing a large amount of training set data, thereby improving the efficiency and accuracy of deep convolutional semantic segmentation network training. Through the above method, any input image A, which can be a variety of forms of images such as RGB images, grayscale images, depth images, inspection images, thermal imaging images, radar point cloud projection images, B-ultrasound / color ultrasound images, etc., can be matched with its annotation image B. After the network training is completed, given any synthetic annotation image, a large number of synthetic inputs can be obtained, thereby expanding the deep learning training set for semantic segmentation, achieving the effect of improving accuracy and accelerating the training of the semantic segmentation network.

[0040] The semantic segmentation training data augmentation method based on the generative adversarial network GAN disclosed in this embodiment uses an original RGB image s and its corresponding correct annotation image t as input to train the network. After training, we reversely give any randomly generated calibration image, and by calculating its intra-class mean, we can directly generate an original RGB image. The generated RGB and annotation images realize data augmentation, which can then be applied to semantic segmentation to improve the robustness of the network. When there are different inputs, such as RGB and depth, the semantic segmentation training data augmentation method based on the generative adversarial network GAN can also be used to generate multiple modes of images. However, it is first necessary to obtain an original RGB image s and its corresponding correct annotation image t, and use the trained network to generate the original input image.

[0041] Therefore, in some embodiments of training data augmentation for the disparity map of a stereo camera, the semantic segmentation training data augmentation method based on the generative adversarial network GAN also includes: obtaining the latest deep convolutional neural network for 3D reconstruction by training a deep convolutional neural network, which can simulate the annotation map ground truth to avoid a large amount of manual calibration, so as to accurately and quickly obtain the correct annotation map t corresponding to the RGB image s. That is, the original image for input in the aforementioned step S1 and the corresponding correct annotation map are obtained by using the trained deep convolutional neural network.

[0042] As attached Figure 4 As shown, this step specifically includes:

[0043] Step S101 , obtaining a set of stereoscopic images of the same size as the original size, wherein the stereoscopic images include a left image and a right image, and obtaining a corresponding first dense disparity map through local stereoscopic vision matching.

[0044] Specifically, first input a stereo image of the same size as the original size, including the left image I L and right figure I R , the corresponding first dense disparity map D0 is obtained through the local stereo vision matching equation, where the local stereo vision matching equation is specifically: Among them, (i, j) is the coordinate of the center point of the block selected in the left image, d is the disparity, and (s, t) is the coordinate of the adjacent point of the selected block.

[0045] In some other embodiments, another local stereo vision matching equation may also be used:

[0046] Among them, E(T) represents the average value of template T. Take an image block in the same horizontal direction of the corrected left and right images and translate it step by step for similarity matching. You can take the average value of the pixels in the block. By default, the disparity of the pixels in the block is the same. The point with the smallest difference or the highest similarity is regarded as the disparity value. The disparity value of point p in the disparity map is the difference between the horizontal coordinates of the point in the left and right views. Among them, (i, j) is the coordinate of the center point of the block selected in the left image, d is the disparity, and (s, t) is the coordinate of the adjacent point of the selected block.

[0047] By placing the disparity of each point at the corresponding position, i.e. (i, j), we can get the corresponding dense disparity map D0.

[0048] Step S102, scaling the left image and the right image multiple times at different ratios to obtain multiple sets of stereoscopic images.

[0049] Since the disparity result obtained by the stereo vision matching equation in step S101 above is not accurate enough, if it is directly used as the correct labeled data required for fully supervised deep learning, the training results are often unsatisfactory. Therefore, it is necessary to scale the input original stereo image to different scales. Specifically, the imresize formula can be used to perform the operation, for example, img2=imresize(img1,ratio), where img1 is the original image, img2 is the scaled image, and ratio is a decimal between 0 and 1. For example, the size of the original image img1 is 1000x500, and the ratio is 0.1, then the size of the scaled image img2 will become 100x50, and multiple groups of left images of different scales will be obtained. L1 ,...,I Lk and right figure I R1 ,…,I Rk .

[0050] Step S103 , performing binocular stereo matching on the left image and the right image in each group of stereo images obtained after scaling, to obtain a small-scale dense disparity map corresponding to the group of stereo images.

[0051] After each scaling, each set of corresponding scaled stereo images, such as I Li and I Ri Perform binocular stereo matching. The specific matching method can adopt the stereo vision matching equation in step S1 to obtain the corresponding small-scale dense disparity map D i. Since the left and right images of different scales obtained by scaling are in the shape of a tower, we call this multi-scale structure a pyramid structure. In the specific application of deep learning, this pyramid structure can be implemented through a series of pooling operations with different moving steps. The operator consists of a series of average pooling operators, and the step λ is between 1 and N, thereby generating a series of images of different scales, where the relationship between the step λ and the above ratio is ratio = 1 / λ, and then each pair of processed left and right images is generated by a local stereo vision matching algorithm to generate a small-scale dense disparity map, where the specific stereo vision matching algorithm is the same as that in step S101.

[0052] Step S104 , upsampling the dense disparity maps of different scales to their original scales to obtain a plurality of second dense disparity maps of the same size as the original scales.

[0053] Specifically, since it is insensitive to the image intensity difference, the first dense disparity map obtained by local stereo vision matching in step S1 is scaled multiple times at different ratios to obtain N original disparity images with different ratios, and the disparity images of the original ratios are multiplied by their corresponding upsampling ratios. Then, the disparity images generated by the stereo image groups of N different ratios are calculated by the standard deviation calculation operator as follows: P = [u, v] T is the image pixel, D i is the disparity image of the i-th scale, S is the output of the SDC operation, where S is an image of the same size as D, where the value of each point represents the credibility of the disparity of that point, the smaller the value, the more reliable it is, and vice versa.

[0054] Then, the dense disparity maps of different scales are upsampled and resized to the original ratio to obtain multiple second dense disparity maps of the same size as the original. Specifically, the imresize formula can be used to solve it, such as img3 = imresize (img2, λ) * λ, where img2 is the scaled image, img3 is the upsampled image, and λ is a decimal greater than 1. For example, the size of the scaled image img2 is 100x100, such as λ is 1.3, then the size of the upsampled image img3 will become 1300x1300, and each disparity after upsampling needs to be multiplied by λ, that is, 1.3, and finally a dense disparity map of the same size as the original can be obtained.

[0055] Step S105: Perform neural network training based on the plurality of second dense disparity maps to obtain a neural network for target detection. Figure 5 As shown, step S5 specifically includes:

[0056] Step S1051 , filtering each pixel in the second dense disparity map according to a preset filtering rule to obtain a third dense disparity map.

[0057] Since the smaller the scaled stereo image is, the less it is affected by noise, the fewer pixels are obtained in the dense disparity map, but the more accurate the disparity is, and the more blurred the part near the edge of the image is. However, the disparity values ​​of the most credible pixels corresponding to different scaling scales are very small, so in this embodiment, a pre-set screening rule, such as a credibility voting step, can be used to screen out the most reliable pixels. Among them, step S51 of using the preset screening rule specifically includes the following contents:

[0058] The variance of the disparity value of each pixel in each second dense disparity map is calculated.

[0059] If the variance is greater than a preset threshold, the pixel is deleted; otherwise, the pixel is retained to form a third dense disparity map.

[0060] Specifically, by calculating the variance of the second dense disparity maps of each pixel at different scales, if the variance is less than a certain threshold, we believe that the disparity value of the pixel is credible. Finally, filter out the unreliable pixels, and use the remaining disparity pixels as a semi-dense disparity map as the correctly labeled data for the training of the fully supervised deep neural network. That is, by processing S with the voting operator, a voting map V is obtained. When S(p)≤1, V(p)=1, and when S(p)>1, V(p) is discarded. That is, a multi-scale credibility voting module is set up. By calculating the variance of the disparity maps of each pixel at different scales, if the variance is less than a certain threshold, we believe that the disparity value of the pixel is credible. Finally, we filter out the unreliable pixels, and use the remaining disparity pixels (forming a semi-dense disparity map) as the correctly labeled data for the training of the fully supervised deep neural network. That is, by processing S with the voting operator, a voting map V is obtained. When When V(p)=1, when V(p) is discarded.

[0061] Step S1052: Using the third dense disparity map as correctly labeled data to perform neural network training, to obtain a neural network for target detection in binocular 3D reconstruction.

[0062] In this embodiment, any fully supervised deep neural network with good performance is selected in the training stage to learn binocular stereo vision matching. By optimizing a joint loss function L = L P +λ1·L R +λ2·L S , where λ1 is L R The weight of LS weights, as a rule of thumb they can be set to 0.1 here. L R Represents reconstruction loss, specifically:

[0063] Among them, SSIM is the structural similarity, a=0.8 is the weight, P is any point in the image, I l It is the original binocular left image. S represents the smoothing loss, where Where L P It represents the multi-scale credibility voting module guided loss, specifically: Where p is any point in an image, l(x) is a smooth function, x is the independent variable of the l function, is the semi-dense disparity image generated by our multi-scale confidence voting module, is the predicted disparity map, is the number of pixels observed in the semi-dense disparity image, generated by the multi-scale confidence voting module described above. P Considering only sparse pixels, we use L R To increase the constraints on dense prediction pixels. Given a set of 3D images, according to the predicted disparity value, we can get the pixel density from the right image I R Reconstruct the left image I L To make the reconstruction process differentiable, we perform it using a bilinear sampler. R Defined as a combination of structural similarity indicators at a single scale. Based on Utilize I R The reconstructed image, is the reconstructed image In a specific embodiment, a simplified 3x3 band-filtered structural similarity index is selected, in which α=0.85 is selected, and L is further added. S term to smooth the predicted disparity image. Since disparity discontinuities often occur where images change significantly, Representing the gradient of an image, fully supervised deep neural networks can show impressive performance even with only semi-dense disparity ground truth.

[0064] In some specific embodiments, the step further includes:

[0065] By judging The difference between the disparity map and the point (i, j) is used to determine whether the corresponding point (i, j) is an occlusion area. is the estimated left disparity map, The estimated right disparity map, if the difference is greater than a preset threshold, is considered to be an occluded area and is deleted.

[0066] Specifically, we can judge The difference between the corresponding point (i, j) and the occlusion area is determined and the occlusion area is removed. is the estimated left disparity map, is the estimated right disparity map. If the difference is greater than the set threshold, it is considered to be an occluded area. Because these areas are often only visible in one of the left or right images. Then the left and right disparity consistency is achieved, that is, any point in the left image has at most one corresponding point in the right image.

[0067] The neural network training method disclosed in steps S101-S105 of the above embodiment performs unsupervised calibration on the input left and right camera images, obtains multiple disparity maps by continuously reducing the left and right camera images, calculates the variances of the corresponding points of these disparity maps respectively, and deletes the pixels exceeding the preset threshold to obtain a disparity map to train the deep convolutional neural network, so that the latest deep convolutional neural network used for 3D reconstruction can easily adapt to various new situations in practice, avoiding a large amount of manual calibration, and can be deployed in any supervised 3D reconstruction system.

[0068] By proposing a pyramid module that continuously reduces the left and right images of the camera to obtain a series of left and right images of different scales to simulate a labeled image ground truth to avoid manual calibration, it can be used in any deep convolutional neural network DCNN. When matching the left and right images of the camera, a square area is taken on the same horizontal line and continuously translated for similarity matching to obtain the optimal disparity map. It can be determined that the larger the square area, the easier the similarity matching, the smaller the noise and the more blurred the disparity. The smaller the square area, the greater the noise and the more accurate the disparity. Taking into account the amount of calculation, this embodiment obtains multiple disparity maps by fixing the size of the square area for similarity matching and continuously reducing the left and right images of the camera. Since these disparity maps obtained by the same camera left and right images are very similar, this embodiment calculates the variance of the corresponding points of these disparity maps respectively, and it is considered to be an invalid value if it exceeds a threshold value, so as to obtain a rough disparity map to train the deep convolutional neural network DCNN, and finally the latest DCNN for three-dimensional reconstruction can be easily adapted to new situations in practice. The present invention avoids a large amount of manual calibration and can be deployed in any supervised three-dimensional reconstruction system to convert it into a self-supervised method.

[0069] In some other embodiments, a semantic segmentation training data augmentation system based on a generative adversarial network (GAN) is disclosed, including: a first training module for matching and learning the input original image with the corresponding correctly labeled image, and optimizing the image. To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, The second training module is used for the generator G network to extract the target area of ​​the correctly labeled image, calculate the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimize the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, For the identifier to identify With G(t), p data Represents the probability distribution of given data; a synthetic target graph generation module is used to generate different synthetic target graphs by arbitrarily changing one or more of the correct labeled graph, the intra-class mean and the noise input into the trained G network.

[0070] The second training module specifically includes: a constraint module for optimizing a cycle loss To constrain, where p data Represents the probability distribution of given data. By training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph.

[0071] It should be noted that the aforementioned embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the semantic segmentation training data augmentation system based on the adversarial generative network GAN disclosed in the embodiment, since it corresponds to the neural network training method disclosed in the aforementioned embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0072] The present invention also provides a semantic segmentation training data augmentation device based on a generative adversarial network (GAN), comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the various steps of the semantic segmentation training data augmentation method based on the generative adversarial network (GAN) as described in the above embodiments are implemented.

[0073] The apparatus for augmenting semantic segmentation training data based on the generative adversarial network GAN may include, but is not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is merely an example of an apparatus for augmenting semantic segmentation training data based on the generative adversarial network GAN, and does not constitute a limitation on the apparatus for augmenting semantic segmentation training data based on the generative adversarial network GAN, and may include more or fewer components than shown in the diagram, or a combination of certain components, or different components, for example, the apparatus for augmenting semantic segmentation training data based on the generative adversarial network GAN may also include input and output devices, network access devices, buses, and the like.

[0074] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the apparatus for augmenting the training data for semantic segmentation based on the generative adversarial network GAN, and uses various interfaces and lines to connect the various parts of the apparatus for augmenting the training data for semantic segmentation based on the generative adversarial network GAN.

[0075] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the semantic segmentation training data augmentation device based on the adversarial generation network GAN by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0076] If the data management method of the semantic segmentation training data augmentation device based on the adversarial generative network GAN is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned embodiments of the semantic segmentation training data augmentation method based on the adversarial generative network GAN. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0078] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0080] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

Claims

1. A method for augmenting semantic segmentation training data based on generative adversarial network (GAN), characterized in that: The steps include: S1, matching the input original image with the corresponding correct annotation image, through optimization To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, is the identifier; The trained deep convolutional neural network is used to obtain the original image for input and the corresponding correct annotation image, which specifically includes the following steps: Acquire a set of stereoscopic images of the same size as the original size, the stereoscopic images including a left image and a right image, obtain a corresponding first dense disparity map through local stereoscopic vision matching; scale the left image and the right image multiple times at different ratios to obtain multiple sets of stereoscopic images; Perform binocular stereo matching on the left image and the right image in each group of stereo images obtained after scaling to obtain a small-scale dense disparity map corresponding to the group of stereo images; Upsampling the dense disparity maps of different scales into the original scales to obtain multiple second dense disparity maps of the same size as the original scales; Filtering each pixel in the second dense disparity map according to a preset filtering rule to obtain a third dense disparity map; The third dense disparity map is used as the correctly labeled data for neural network training, by optimizing a joint loss function L = LP + λ1·LR + λ2·LS, where λ1 is L R The weight of L S Weight, L R represents the reconstruction loss, where SSIM is the structural similarity, a is the weight, P is any point in the image, Il is the original left binocular image, L S represents the smoothing loss, where Where L P represents the multi-scale credibility voting module guidance loss, Where p is any point in an image, l(x) is a smooth function, x is the independent variable of the l function, is a semi-dense disparity image generated by the multi-scale credibility voting module, is the predicted disparity map, is the number of pixels observed in the semi-dense disparity image, Based on Utilize I R The reconstructed image, is the reconstructed image The number of pixels observed in , and the neural network for object detection for binocular 3D reconstruction are obtained; S2, the generator G network extracts the target area of ​​the correctly labeled image, calculates the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimizes the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, and DS is the identifier used to identify With G(t), pdata represents the probability distribution of given data; S3, generates different synthetic target images by arbitrarily changing one or more of the correct labeled images, intra-class mean and noise input into the trained G network.

2. The method for augmenting semantic segmentation training data based on generative adversarial network (GAN) according to claim 1, characterized in that: The step S2 also includes: optimizing a cycle loss To perform constraints, where pdata represents the probability distribution of the given data, and by training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph.

3. A semantic segmentation training data augmentation system based on generative adversarial network GAN, characterized in that: include: The first training module is used to match the input original image with the corresponding correct annotation image. To achieve recognition and F(s), where is the input original image, i.e. the input target image of the training target, is the correctly labeled graph, F(s) is the generated labeled graph, The invention is a recognizer; wherein a trained deep convolutional neural network is used to obtain the original image for input and the corresponding correctly labeled image, specifically including: obtaining a group of stereo images of the same size as the original size, wherein the stereo images include a left image and a right image, and obtaining a corresponding first dense disparity map through local stereo vision matching; scaling the left image and the right image multiple times at different ratios to obtain multiple groups of stereo images; performing binocular stereo matching on the left image and the right image in each group of stereo images obtained after scaling to obtain a small-scale dense disparity map corresponding to the group of stereo images; upsampling each dense disparity map of different ratios to the original ratio to obtain multiple second dense disparity maps of the same size as the original; filtering each pixel in the second dense disparity map according to a preset filtering rule to obtain a third dense disparity map; using the third dense disparity map as the correctly labeled data for neural network training, and optimizing a joint loss function L=LP+λ1·LR+λ2·LS, wherein λ1 is L R The weight of L S Weight, L R represents the reconstruction loss, where SSIM is the structural similarity, a is the weight, P is any point in the image, Il is the original left binocular image, L S represents the smoothing loss, where Where L P represents the multi-scale credibility voting module guidance loss, Where p is any point in an image, l(x) is a smooth function, x is the independent variable of the l function, is a semi-dense disparity image generated by the multi-scale credibility voting module, is the predicted disparity map, is the number of pixels observed in the semi-dense disparity image, Based on Utilize I R The reconstructed image, is the reconstructed image The number of pixels observed in , and the neural network for object detection for binocular 3D reconstruction are obtained; The second training module is used for the generator G network to extract the target area of ​​the correctly labeled image, calculate the pixel average value in the same area of ​​the input target image to constrain the reverse learning process, and continuously optimize the G network parameters in the process of learning to generate the input target image from the correctly labeled image. To identify the correctly labeled graph and the generated original input graph, is the original input image, To correctly label the graph, G(t) is the generated original input graph, and DS is the identifier used to identify With G(t), pdata represents the probability distribution of given data; The synthetic target graph generation module is used to generate different synthetic target graphs by arbitrarily changing one or more of the correct labeled graph, the intra-class mean and the noise input into the trained G network.

4. The semantic segmentation training data augmentation system based on the generative adversarial network (GAN) according to claim 3, characterized in that: The second training module includes: Constraint module, used to optimize a cycle loss To perform constraints, where pdata represents the probability distribution of the given data, and by training the G network model and the F network model, F(G(t))≈t and G(F(s))≈s, that is, the generated annotation graph is close to the correct annotation graph, and the generated original input graph is close to the input target graph.

5. A semantic segmentation training data augmentation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Images