Learnable superpixel-guided parallax optimization method and device

Abstract

The present invention provides a method and device for disparity optimization based on learnable superpixel guidance, wherein the method comprises: obtaining a first image and a second image input by a user; inputting the first image and the second image into a disparity optimization model to obtain a disparity map output by the disparity optimization model; wherein the first image and the second image are obtained by shooting the same object with a binocular camera; the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic maps, spatial position maps and real disparity maps are predetermined according to the picture samples, and correspond one to one with the picture samples. The method and device for disparity optimization based on learnable superpixel guidance provided by the embodiments of the present invention realize effective interaction between a superpixel network and a disparity network.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for parallax optimization guided by learnable superpixels. Background Art

[0002] Stereo matching, also known as disparity estimation, calculates the disparity map representing the reference image based on two images taken by a calibrated binocular camera. The disparity map can be converted into a scene depth map by the camera focal length and the baseline of the two cameras. It has a wide range of applications in autonomous driving, living body detection, 3D reconstruction and other fields. Superpixels are irregular pixel blocks with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics. Superpixels are widely used in disparity estimation methods.

[0003] However, the superpixel segmentation algorithm used in the prior art is not differentiable and cannot effectively interact with the disparity estimation task, so it cannot fully explore the rich local information in the superpixel to effectively improve the stereo matching performance. In addition, the existing superpixel algorithm has poor adaptability to the disparity estimation tasks in different application scenarios. Summary of the invention

[0004] The present invention provides a method and device for disparity optimization based on learnable superpixel guidance, which are used to solve the defect that the superpixel segmentation algorithm in the prior art cannot effectively interact with the disparity estimation task, and realize disparity optimization based on learnable superpixel guidance.

[0005] In a first aspect, the present invention provides a disparity optimization method based on learnable superpixel guidance, comprising:

[0006] Obtain the first image and the second image input by the user;

[0007] Inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model;

[0008] Among them, the first image and the second image are obtained by shooting the same object with a binocular camera; the disparity optimization model is obtained after training based on the data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic map, the spatial position map and the real disparity map are predetermined based on the picture samples and correspond one-to-one to the picture samples.

[0009] Optionally, the disparity optimization model includes a superpixel branch network, a disparity branch network, and a superpixel guided disparity optimization network;

[0010] The step of inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model includes:

[0011] Inputting the first map into the superpixel branch network to obtain a pixel-superpixel association map output by the superpixel branch network;

[0012] Inputting the first image and the second image into the disparity branch network to obtain an initial disparity image output by the disparity branch network;

[0013] The pixel-superpixel association map and the initial disparity map are input into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network.

[0014] Optionally, the superpixel guided disparity optimization network comprises: a superpixel reconstruction module, an error map generation module and a disparity optimization module;

[0015] The step of inputting the pixel-superpixel association map and the initial disparity map into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network comprises:

[0016] Inputting the pixel superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain a reconstructed disparity map output by the superpixel reconstruction module;

[0017] Inputting the reconstructed disparity map and the initial disparity map into the error map generation module to obtain an error map output by the error map generation module;

[0018] The reconstructed disparity map, the initial disparity map, the error map and the first map are input into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module.

[0019] Optionally, inputting the pixel-superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain the reconstructed disparity map output by the superpixel reconstruction module comprises:

[0020] Downsampling the initial disparity map based on a downsampling formula and the pixel superpixel association map to obtain a process disparity map;

[0021] The process disparity map is upsampled based on an upsampling formula and the pixel superpixel association map to obtain a reconstructed disparity map.

[0022] Optionally, the downsampling formula is:

[0023]

[0024] The upsampling formula is:

[0025]

[0026] Where s represents a superpixel, s = 1, 2...n, n is the number of superpixels, n is a positive integer, D1 represents the process sampling map, p represents the pth pixel in the superpixel s, p is a positive integer, D init represents the initial disparity map, D2 represents the reconstructed disparity map, N p represents the neighborhood superpixel of the p-th pixel, Q s (p) represents the probability that pixel p belongs to superpixel s, Q p (s) represents the probability that superpixel s is a superpixel in the neighborhood of pixel p.

[0027] Optionally, the disparity optimization module includes a lightweight convolutional neural network unit;

[0028] The step of inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module comprises:

[0029] Inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the lightweight convolutional neural network unit to obtain a residual map output by the lightweight convolutional neural network unit;

[0030] The residual map and the initial disparity map are added to obtain the optimized disparity map.

[0031] Optionally, the disparity optimization model is trained according to the following method:

[0032] Training the disparity branch network and the disparity optimization module based on the data set and the disparity supervision loss function;

[0033] The superpixel guided disparity optimization network is trained based on the data set, the disparity supervision loss function and the superpixel supervision function.

[0034] Optionally, the disparity supervision loss function is:

[0035] L D =m1×L D_init +m2×L D_r ;

[0036] L D_init =S(D init -D gt );

[0037] L D_r =S(D r -D gt );

[0038]

[0039] Where, L D represents the parallax supervision loss function, m1 is the weighting coefficient, L D_init represents the loss function of the disparity branch network, m2 is the weighting coefficient, L D_r represents the loss function of the parallax optimization model, S(x) represents the smooth L1 loss function, and D init represents the initial disparity map, D gt represents the real disparity map, D r represents an optimized disparity map, the initial disparity map and the optimized disparity map are obtained based on the picture sample, and the real disparity map is pre-stored in the data set and corresponds to the picture sample.

[0040] Optionally, the training of the superpixel guided disparity optimization network based on the data set, the disparity supervision loss function and the superpixel supervision function includes:

[0041] Inputting the real semantic map, the spatial position map and the real disparity map corresponding to the image sample in the data set into the superpixel reconstruction module respectively, and obtaining the reconstructed semantic map, the reconstructed spatial position map and the reconstructed disparity map output by the superpixel reconstruction module;

[0042] Obtaining a cross entropy loss between the true semantic map and the reconstructed semantic map, a first L2 loss between the spatial position map and the reconstructed spatial position map, and a second L2 loss between the true disparity map and the reconstructed disparity map;

[0043] The superpixel guided disparity optimization network is trained based on the cross entropy loss, the first L2 loss, the second L2 loss, the disparity supervision loss function and the superpixel supervision function.

[0044] Optionally, the superpixel supervision function is:

[0045] L=L D +λ1L S ;

[0046] L S =λ1L sem +λ2L pos +λ3L d ;

[0047] Where L represents the superpixel supervision function, L D represents the parallax supervision loss function, λ1, λ2, λ3 and λ4 are weighting coefficients, L semis the cross entropy loss between the true semantic graph and the reconstructed semantic graph, L pos is the first L2 loss between the spatial position map and the reconstructed spatial position map, L d is the second L2 loss between the real disparity map and the reconstructed disparity map.

[0048] In a second aspect, the present invention further provides a disparity optimization device based on learnable superpixel guidance, comprising:

[0049] An acquisition unit, used to acquire a first image and a second image input by a user;

[0050] an output unit, configured to input the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model;

[0051] Among them, the first image and the second image are obtained by photographing the same object with a binocular camera, and the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic map, the spatial position map and the real disparity map are predetermined based on the picture samples and correspond one-to-one to the picture samples.

[0052] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the disparity optimization method based on learnable superpixel guidance as described in the first aspect are implemented.

[0053] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the disparity optimization method based on learnable superpixel guidance as described in the first aspect.

[0054] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the disparity optimization method based on learnable superpixel guidance as described in the first aspect.

[0055] The disparity optimization method and device based on learnable superpixel guidance provided by the present invention construct a superpixel network based on a neural network, so that the superpixel algorithm can be combined with the disparity network and trained together, thereby optimizing the disparity under the guidance of superpixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 This is one of the flow charts of the parallax optimization method based on learnable superpixel guidance provided by an embodiment of the present invention;

[0058] Figure 2 This is the second flow chart of the parallax optimization method based on learnable superpixel guidance provided by an embodiment of the present invention;

[0059] Figure 3 is a schematic diagram of superpixel initialization provided by an embodiment of the present invention;

[0060] Figure 4 Schematic diagram of the process of generating an initial disparity map provided by an embodiment of the present invention

[0061] Figure 5 is a schematic diagram of a process of superpixel reconstruction provided by an embodiment of the present invention;

[0062] Figure 6 is a schematic diagram of a process of parallax optimization provided by an embodiment of the present invention;

[0063] Figure 7 is a schematic diagram of the process of disparity optimization model training provided by an embodiment of the present invention;

[0064] Figure 8 is a structural schematic diagram of a disparity optimization device based on learnable superpixel guidance provided by an embodiment of the present invention;

[0065] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] The technical terms involved in the present invention are introduced as follows:

[0068] Convolutional Neural Network (CNN): It is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0069] Superpixel: An irregular pixel block with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics.

[0070] Pixel-superpixel association map: represents the probability that each pixel belongs to the corresponding superpixel.

[0071] Stereo matching: The process of calculating the disparity value corresponding to each pixel of the reference image (usually the left image) based on two binocular images taken by a calibrated binocular camera.

[0072] Parallax: Parallax is the pixel-level difference between the positions of corresponding points in the left and right images at a certain point in a 3D scene.

[0073] In the prior art, superpixels are widely used in disparity estimation methods, which assume the smoothness of disparity within superpixels and use tilted plane equations to represent the disparity distribution within each superpixel. The general steps are: first, the reference image is divided into multiple regions by the existing superpixel algorithm, and then the plane fitting of the regional disparity and subsequent optimization are performed. The existing superpixel algorithm is not differentiable, but the construction of a neural network requires a differentiable function, and it is necessary to be able to calculate the gradient, so that the neural network can be trained based on back propagation. If a network is not differentiable and the gradient cannot be calculated, it cannot be trained into a neural network. Therefore, the existing non-differentiable superpixel algorithm cannot be trained together with the disparity estimation model based on the neural network, resulting in the superpixel algorithm being unable to effectively interact with the disparity estimation task, and thus being unable to fully explore the rich local information in the superpixel to effectively improve the stereo matching performance, and the existing superpixel algorithm may not be adaptable to the disparity estimation tasks of different application scenarios.

[0074] The disparity optimization method guided by learnable superpixels provided in the embodiments of the present invention realizes the combination of a superpixel algorithm and a disparity network, and can optimize the disparity under the guidance of superpixels.

[0075] Combine the following Figure 1-Figure 7 A disparity optimization method based on learnable superpixel guidance provided by an embodiment of the present invention is described.

[0076] Figure 1 is one of the flow charts of the parallax optimization method based on learnable superpixel guidance provided by an embodiment of the present invention, such as Figure 1 As shown, the disparity optimization method based on learnable superpixel guidance provided by the embodiment of the present invention includes:

[0077] Step 110, obtaining a first image and a second image input by a user;

[0078] Specifically, the first image and the second image are obtained by shooting the same object with a binocular camera. The binocular camera can shoot left and right viewpoint images of the same scene. Exemplarily, the first image is selected as a reference image, and the disparity optimization model outputs a disparity map of the reference image. The first image can be a left image taken by the binocular camera, and the second image can be a right image taken by the binocular camera. It can be understood that in the embodiment of the present invention, the correspondence between the first image and the second image and the left and right images of the binocular camera, and the selection of the reference image are exemplary explanations made for a better understanding of the present invention. The embodiment of the present invention does not specifically limit the correspondence between the left and right images and the selection of the reference image.

[0079] Step 120, inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model;

[0080] Among them, the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic maps, the spatial position maps and the real disparity maps are predetermined based on the picture samples and correspond one-to-one to the picture samples.

[0081] The image samples in this step refer to the first image sample and the second image sample taken by the binocular camera in advance. The semantic map is obtained by superpixel segmentation of the image based on the semantic segmentation rule. The semantic segmentation rule refers to assigning a category label (such as car, building or sky, etc.) to each pixel in the image, and performing superpixel segmentation of the image according to the label. The real semantic map is a semantic map sample obtained by pre-annotating the reference image in the image sample. The spatial position map is obtained by pre-segmenting the reference image based on the pixel position proximity rule. The pixel position proximity rule refers to dividing pixels with similar positions into one superpixel. The real disparity map is a real and effective disparity map obtained by pre-annotating the reference image.

[0082] The disparity optimization model provided by the embodiment of the present invention includes a superpixel network and a disparity network constructed based on a neural network. The superpixel network can be combined with the disparity network so that the superpixel and the disparity network can be trained together to optimize the disparity under the guidance of the superpixel. Since the disparity optimization model can optimize the disparity under the guidance of the learnable superpixel, the disparity optimization model can combine the advantages of the neural network, extract information from the global and multiple local areas, explore the rich global information and local information in the first image and the second image, thereby improving the disparity estimation performance, and using the learnability of the neural network to enhance the adaptability of the disparity optimization model to different application scenarios.

[0083] Optionally, Figure 2 FIG. 2 is a flow chart of a parallax optimization method based on learnable superpixel guidance provided by an embodiment of the present invention. Figure 2 As shown, the disparity optimization model includes a superpixel branch network, a disparity branch network, and a superpixel guided disparity optimization network;

[0084] The step of inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model includes:

[0085] Step 210, inputting the first map into the superpixel branch network to obtain a pixel-superpixel association map output by the superpixel branch network;

[0086] Specifically, the superpixel branch extracts features from the first image through a convolutional neural network and directly outputs a pixel-superpixel association map Q of the first image. The pixel-superpixel association map represents the probability that each pixel belongs to each superpixel. The superpixel size can be set according to the image size or user needs. The superpixel size refers to the number of pixels contained in each superpixel.

[0087] Figure 3 is a schematic diagram of superpixel initialization provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, firstly, a regular grid is used to initialize the superpixel segmentation, and each small box represents an initialized superpixel. Then, based on the initialized superpixel segmentation, a convolutional neural network of the autoencoder type is used to directly output the pixel-superpixel association map Q. Optionally, in order to reduce the computational burden, the network only predicts the probability that each pixel belongs to the 9 superpixels in the neighborhood (9-channel representation). The superpixel initialization step defines the neighborhood superpixels of each pixel. As shown in the figure, Figure 3 As shown, for all pixels in the initialized superpixel 4, the neighborhood 9 superpixels are shown in boxes 0 to 8. Optionally, by setting a plurality of different superpixel sizes, a multi-scale pixel-superpixel association map can be obtained.

[0088] Step 220, inputting the first image and the second image into the disparity branch network to obtain an initial disparity image output by the disparity branch network;

[0089] In one embodiment, the first image is a left image L, and the second image is a right image R. The left image L and the right image R are input into a disparity branch network, and an initial disparity map is output through the disparity branch network based on a convolutional neural network.

[0090] Figure 4 is a schematic diagram of a process for generating an initial disparity map provided by an embodiment of the present invention, such as Figure 4As shown, the disparity branch network can adopt the steps of the 3D stereo matching network, including feature extraction, cost volume construction, cost aggregation, and disparity regression.

[0091] Feature extraction step: extract the high-level features F of the left image L through CNNs l And the high-level features F of R on the right r ,The feature extraction networks share parameters for the left and right images.

[0092] Cost volume construction step: by horizontally translating F at different disparity levels r , and with F l Connected together, a 4D cost volume is constructed to explicitly represent the matching relationship at different disparity levels.

[0093] Cost aggregation step: The 4D cost volume is filtered through 3D CNN to obtain the aggregated cost volume.

[0094] Disparity regression step: Get the initial disparity map through disparity regression.

[0095] It is understandable that the above is a disparity branch network, and a binocular disparity network based on a convolutional neural network such as PSMnet or Ganet can also be used. In the case where the disparity branch network is a multi-layer deep neural network, the disparity branch network can output multiple initial disparity maps. For example, the disparity branch network can output an initial disparity map D init1 , in the initial disparity map D in Based on the initial disparity map D init2 , in the initial disparity map D init Based on the initial disparity map D init3 , the final output initial disparity map D init3 The initial disparity map D as the output of the disparity branch network init .

[0096] Optionally, to better achieve multi-task learning, the disparity branch network and the superpixel branch network can share a shallow feature extraction network.

[0097] Step 230: input the pixel-superpixel association map and the initial disparity map into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network.

[0098] Specifically, the pixel-superpixel association map Q (“pixel-superpixel association map” may also be written as “pixel-superpixel association map” in the following text) obtained in step 210 is used to guide the initial disparity map D in step 220. initThe optimized disparity map is obtained by optimizing the pixel superpixel association map Q obtained in step 210 can be a multi-scale pixel superpixel association map. Based on the disparity consistency assumption within the same superpixel, if the disparity estimate at a certain pixel deviates greatly from the disparity estimates of other pixels in the same superpixel or the disparity average, the disparity estimate at the pixel may be wrong. Based on this assumption, an optimization network is constructed to guide the disparity optimization.

[0099] Optionally, the superpixel guided disparity optimization network comprises: a superpixel reconstruction module, an error map generation module and a disparity optimization module;

[0100] The step of inputting the pixel-superpixel association map and the initial disparity map into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network comprises:

[0101] Step 310, inputting the pixel-superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain a reconstructed disparity map output by the superpixel reconstruction module;

[0102] Optionally, Figure 5 is a schematic diagram of a process of superpixel reconstruction provided by an embodiment of the present invention, such as Figure 5 As shown, the step of inputting the pixel superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain the reconstructed disparity map output by the superpixel reconstruction module includes:

[0103] Step 311, downsampling the initial disparity map based on a downsampling formula and the pixel superpixel association map to obtain a process disparity map;

[0104] The downsampling formula is:

[0105]

[0106] Where s represents a superpixel, s = 1, 2...n, n is the number of superpixels, n is a positive integer, D1 represents the process sampling map, p represents the pth pixel in the superpixel s, p is a positive integer, D init is the initial disparity map, Q s (p) represents the probability that pixel p belongs to superpixel s. It can be understood that Q s (p) can be obtained through the pixel-superpixel correlation map.

[0107] The pixel superpixel association map Q obtained by the superpixel branch network is used to compare the initial disparity map D initThe superpixel downsampling is performed to obtain the process disparity map D1. As shown in the sampling formula, the superpixel downsampling process takes the weighted average of the disparity values ​​of all pixels p belonging to the same superpixel s (the weight is the probability that the superpixel s represented in the pixel-superpixel association map Q is the neighborhood superpixel of pixel p) as the disparity value of the pixel s after downsampling.

[0108] Step 321 , upsampling the process disparity map based on an upsampling formula and the pixel superpixel association map to obtain a reconstructed disparity map.

[0109] The upsampling formula is:

[0110]

[0111] Where s represents a superpixel, s = 1, 2...n, n is the number of superpixels, n is a positive integer, D1 represents the process sampling map, p represents the pth pixel in the superpixel s, p is a positive integer, D2 represents the reconstructed disparity map, N p represents the neighborhood superpixel of the p-th pixel, Q p (s) represents the probability that superpixel s is a superpixel in the neighborhood of pixel p. It can be understood that Q p (s) can be obtained through the pixel-superpixel correlation map.

[0112] The same pixel superpixel association map Q process disparity map is used to super-pixel upsample D1 to obtain the reconstructed disparity map D2, as shown in the upsampling formula. The superpixel upsampling process assigns the weighted average of the disparity within each superpixel s to each pixel belonging to the superpixel to obtain the reconstructed disparity map D2. The reconstructed disparity map D2 is consistent with the initial disparity map D init The spatial dimensions are consistent.

[0113] Step 320, inputting the reconstructed disparity map and the initial disparity map into the error map generation module to obtain an error map output by the error map generation module;

[0114] The error map is used to represent the difference between the reconstructed disparity map and the initial disparity map. Exemplarily, the difference in pixel values ​​of pixels at corresponding positions between the reconstructed disparity map and the initial disparity map can be used as the error value to generate the error map, or the error matching points in the initial disparity map can be found by a correction algorithm, and the error map is generated based on the error matching points. The embodiment of the present invention does not limit how the error map generation module generates the error map.

[0115] Step 330 : input the reconstructed disparity map, the initial disparity map, the error map and the first map into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module.

[0116] Optionally, Figure 6is a schematic diagram of a process of parallax optimization provided by an embodiment of the present invention, such as Figure 6 As shown, the disparity optimization module includes a lightweight convolutional neural network unit;

[0117] The step of inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module comprises:

[0118] Step 331, inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the lightweight convolutional neural network unit to obtain a residual map output by the lightweight convolutional neural network unit;

[0119] The first image, the reconstructed disparity map, the initial disparity map and the error map are sent as mixed features to a lightweight convolutional neural network for further optimization to obtain a residual map, which can also be called a disparity residual map.

[0120] Step 332: Add the residual map and the initial disparity map to obtain the optimized disparity map.

[0121] The disparity residual map D re Add initial D init The optimized disparity map D is obtained, which can be expressed as: D = D re +D init .

[0122] In the embodiment of the present application, by guiding the initial disparity optimization through learnable superpixels, the prediction errors of occluded areas, textureless areas and detail areas can be effectively reduced, thereby improving the disparity prediction quality.

[0123] Optionally, the superpixel branch network, the disparity branch network and the learnable superpixel-guided disparity optimization module can be jointly trained in end-to-end supervision. The supervision is divided into disparity supervision and superpixel supervision. Figure 7 is a flow chart of the parallax optimization model training provided by an embodiment of the present invention, such as Figure 7 As shown, the parallax optimization model is trained according to the following method:

[0124] Step 410, training the disparity branch network and the disparity optimization module based on the data set and the disparity supervision loss function;

[0125] The disparity branch network and the disparity optimization module are subjected to disparity supervision. Optionally, the disparity supervision loss function in step 410 may be existing loss functions such as a pixel-wise loss function, a perceptual loss function or a topology-aware loss function, and improvements thereof. The embodiment of the present invention does not limit the type of the disparity supervision loss function.

[0126] Optionally, the Smooth L1 loss function is used to supervise the initial disparity map D init And optimize the disparity map D r , the disparity supervision loss function is:

[0127] L D =m1×L D_init +m2×L D_r ;

[0128] L D_init =S(D init -D gt );

[0129] L D_r =S(D r -D gt );

[0130]

[0131] Where, L D represents the parallax supervision loss function, m1 is the weighting coefficient, L D_init represents the loss function of the disparity branch network, m2 is the weighting coefficient, L D_r represents the loss function of the parallax optimization model, S(x) represents the smooth L1 loss function, and D init represents the initial disparity map, D gt represents the real disparity map, D r represents an optimized disparity map, the initial disparity map and the optimized disparity map are obtained based on the image sample, and the real disparity map is pre-stored in the data set and corresponds to the image sample. The smooth L1 loss function S(x) is an L2 loss function when |x|<1 and an L1 loss function when |x|≥1.

[0132] The final disparity supervision L D is the supervision of the initial disparity map L D_init and supervision L for optimizing the disparity map D_r It can be understood that, as described above, in the case where the disparity branch network is a multi-layer deep neural network, exemplarily, the disparity branch network outputs the initial disparity map D init , initial disparity map D init and the initial disparity map D init3 , at this time, m1×L in the disparity supervision loss function D_init It can be expressed as:

[0133] m1×L D_init =m 1_1 ×L D_init1 +m 1_1 ×LD_init2 +m 1_3 ×L D_init3 ;

[0134] In the formula, m 1_1 、m 1_2 and m 1_3 is the weighting coefficient; L D_init1 is the initial disparity map D init1 The loss function of L D_init2 is the initial disparity map D init2 The loss function of L D_init3 is the initial disparity map D init3 The loss function of .

[0135] Optionally, m 1_1 0.5, m 1_2 0.7, m 1_3 is 1.0, m2 is 1.0.

[0136] Step 420: training the superpixel guided disparity optimization network based on the data set, the disparity supervision loss function and the superpixel supervision function.

[0137] The superpixel guided disparity optimization network is subjected to superpixel supervision. Optionally, the selection of the loss function refers to the introduction in step 410 and will not be described in detail here.

[0138] Optionally, superpixel supervision includes semantic reconstruction supervision, spatial position reconstruction supervision and disparity reconstruction supervision. The training of the superpixel guided disparity optimization network based on the data set, the disparity supervision loss function and the superpixel supervision loss function includes:

[0139] Step 421, inputting the real semantic map, the spatial position map and the real disparity map corresponding to the image sample in the data set into the superpixel reconstruction module respectively, to obtain the reconstructed semantic map, the reconstructed spatial position map and the reconstructed disparity map output by the superpixel reconstruction module;

[0140] Specifically, the pixel superpixel association map Q predicted by the superpixel branch network is compared with the real semantic map Sem gt , spatial position map Pos, and real disparity map D gt Perform continuous superpixel downsampling and superpixel upsampling to obtain the reconstructed semantic map Sem rec , spatial position map Pos rec And the disparity map D rec .

[0141] Step 422, obtaining a cross entropy loss between the real semantic map and the reconstructed semantic map, a first L2 loss between the spatial position map and the reconstructed spatial position map, and a second L2 loss between the real disparity map and the reconstructed disparity map;

[0142] Specifically, calculate the true semantic graph Sem gt and reconstruct the semantic graph Sem rec The cross entropy loss between the spatial position map Pos and the reconstructed spatial position map Pos rec The L2 loss between them and the true disparity map D gt and reconstruct the disparity map D rec L2 loss.

[0143] Step 423: training the superpixel guided disparity optimization network based on the cross entropy loss, the first L2 loss, the second L2 loss, the disparity supervision loss function and the superpixel supervision loss function.

[0144] Optionally, the superpixel supervision loss function is:

[0145] L=L D +λ1L S ;

[0146] L S =λ1L sem +λ2L pos +λ3L d ;

[0147] Where L represents the superpixel supervision loss function, L D represents the parallax supervision loss function, λ1, λ2, λ3 and λ4 are weighting coefficients, L sem is the cross entropy loss between the true semantic graph and the reconstructed semantic graph, L pos is the first L2 loss between the spatial position map and the reconstructed spatial position map, L d is the second L2 loss between the real disparity map and the reconstructed disparity map. S The supervision loss is the real semantic map Sem gt and reconstruct the semantic graph Sem rec The cross entropy loss between the spatial position map Pos and the reconstructed spatial position map Pos rec The L2 loss between them and the true disparity map D gt and reconstruct the disparity map D rec The weighted sum of the L2 losses.

[0148] Optionally, λ1 is 1, λ2 is 0.1, λ3 is 0.125, and λ4 is 0.1.

[0149] The following is a description of a disparity optimization device based on learnable superpixel guidance provided in an embodiment of the present invention. The disparity optimization device based on learnable superpixel guidance described below and the disparity optimization method based on learnable superpixel guidance described above can refer to each other.

[0150] Figure 8 is a schematic diagram of the structure of a disparity optimization device based on learnable superpixel guidance provided by an embodiment of the present invention, such as Figure 8 As shown, the parallax optimization device provided by the embodiment of the present invention includes an acquisition unit 510 and an output unit 520:

[0151] An acquisition unit 510 is used to acquire a first image and a second image input by a user;

[0152] An output unit 520, configured to input the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model;

[0153] Among them, the first image and the second image are obtained by photographing the same object with a binocular camera, and the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic map, the spatial position map and the real disparity map are predetermined based on the picture samples and correspond one-to-one to the picture samples.

[0154] It should be noted here that the above-mentioned device provided in the embodiment of the present invention can implement all the method steps implemented in the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0155] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a disparity optimization method based on learnable superpixel guidance, the method comprising: obtaining a first image and a second image input by a user; inputting the first image and the second image into a disparity optimization model to obtain a disparity map output by the disparity optimization model; wherein the first image and the second image are obtained by shooting the same object with a binocular camera, the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes a picture sample and a corresponding real semantic map, a spatial position map and a real disparity map; the real semantic map, the spatial position map and the real disparity map are predetermined according to the picture sample and correspond to the picture sample one by one.

[0156] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0157] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the disparity optimization method based on learnable superpixel guidance provided by the above methods, and the method includes: obtaining a first image and a second image input by a user; inputting the first image and the second image into a disparity optimization model to obtain a disparity map output by the disparity optimization model; wherein the first image and the second image are obtained by photographing the same object with a binocular camera, and the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic map, the spatial position map and the real disparity map are predetermined based on the picture samples, and correspond one-to-one to the picture samples.

[0158] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the disparity optimization method based on learnable superpixel guidance provided by the above-mentioned methods, the method comprising: obtaining a first image and a second image input by a user; inputting the first image and the second image into a disparity optimization model to obtain a disparity map output by the disparity optimization model; wherein the first image and the second image are obtained by photographing the same object with a binocular camera, and the disparity optimization model is obtained after training based on a data set, and is used to optimize the disparity under the guidance of superpixels; the data set comprises picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic maps, the spatial position maps and the real disparity maps are predetermined based on the picture samples, and correspond one-to-one to the picture samples.

[0159] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A disparity optimization method based on learnable superpixel guidance, characterized in that: include: Obtain the first image and the second image input by the user; Inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model; The first image and the second image are obtained by shooting the same object with a binocular camera; the disparity optimization model is obtained after training based on the data set, and is used to optimize the disparity under the guidance of superpixels; the data set includes picture samples and corresponding real semantic maps, spatial position maps and real disparity maps; the real semantic maps, the spatial position maps and the real disparity maps are predetermined according to the picture samples and correspond to the picture samples one by one; The disparity optimization model includes a superpixel branch network, a disparity branch network and a superpixel guided disparity optimization network; The step of inputting the first image and the second image into a disparity optimization model to obtain a disparity image output by the disparity optimization model includes: Inputting the first map into the superpixel branch network to obtain a pixel-superpixel association map output by the superpixel branch network; Inputting the first image and the second image into the disparity branch network to obtain an initial disparity image output by the disparity branch network; Inputting the pixel-superpixel association map and the initial disparity map into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network; The superpixel guided disparity optimization network includes: a superpixel reconstruction module, an error map generation module and a disparity optimization module; The step of inputting the pixel-superpixel association map and the initial disparity map into the superpixel-guided disparity optimization network to obtain an optimized disparity map output by the superpixel-guided disparity optimization network comprises: Inputting the pixel superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain a reconstructed disparity map output by the superpixel reconstruction module; Inputting the reconstructed disparity map and the initial disparity map into the error map generation module to obtain an error map output by the error map generation module; The reconstructed disparity map, the initial disparity map, the error map and the first map are input into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module.

2. The method for disparity optimization based on learnable superpixel guidance according to claim 1, characterized in that: The step of inputting the pixel superpixel association map and the initial disparity map into the superpixel reconstruction module to obtain a reconstructed disparity map output by the superpixel reconstruction module comprises: Downsampling the initial disparity map based on a downsampling formula and the pixel superpixel association map to obtain a process disparity map; The process disparity map is upsampled based on an upsampling formula and the pixel superpixel association map to obtain a reconstructed disparity map.

3. The method for disparity optimization based on learnable superpixel guidance according to claim 2, characterized in that: The downsampling formula is: ； The upsampling formula is: ； Where s represents superpixel, , n is the number of superpixels, n is a positive integer, represents the process sampling graph, p represents the p-th pixel in the superpixel s, p is a positive integer, represents the initial disparity map, represents the reconstructed disparity map, represents the neighborhood superpixel of the p-th pixel, represents the probability that pixel p belongs to superpixel s, represents the probability that superpixel s is a superpixel in the neighborhood of pixel p.

4. The method for disparity optimization based on learnable superpixel guidance according to claim 1, characterized in that: The disparity optimization module includes a lightweight convolutional neural network unit; The step of inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the disparity optimization module to obtain the optimized disparity map output by the disparity optimization module comprises: Inputting the reconstructed disparity map, the initial disparity map, the error map and the first map into the lightweight convolutional neural network unit to obtain a residual map output by the lightweight convolutional neural network unit; The residual map and the initial disparity map are added to obtain the optimized disparity map.

5. The method for disparity optimization based on learnable superpixel guidance according to any one of claims 1 to 4, characterized in that: The parallax optimization model is trained as follows: Training the disparity branch network and the disparity optimization module based on the data set and the disparity supervision loss function; The superpixel guided disparity optimization network is trained based on the data set, the disparity supervision loss function and the superpixel supervision function.

6. The method for disparity optimization based on learnable superpixel guidance according to claim 5, characterized in that: The disparity supervision loss function is: ； ； ； ； In the formula, represents the disparity supervision loss function, is the weighting coefficient, represents the loss function of the disparity branch network, is the weighting coefficient, represents the loss function of the parallax optimization model, represents the smooth L1 loss function, represents the initial disparity map, represents the true disparity map, represents an optimized disparity map, the initial disparity map and the optimized disparity map are obtained based on the picture sample, and the real disparity map is pre-stored in the data set and corresponds to the picture sample.

7. The method for disparity optimization based on learnable superpixel guidance according to claim 5, characterized in that: The training of the superpixel guided disparity optimization network based on the data set, the disparity supervision loss function and the superpixel supervision function comprises: Inputting the real semantic map, the spatial position map and the real disparity map corresponding to the image sample in the data set into the superpixel reconstruction module respectively, and obtaining the reconstructed semantic map, the reconstructed spatial position map and the reconstructed disparity map output by the superpixel reconstruction module; Obtaining a cross entropy loss between the true semantic map and the reconstructed semantic map, a first L2 loss between the spatial position map and the reconstructed spatial position map, and a second L2 loss between the true disparity map and the reconstructed disparity map; The superpixel guided disparity optimization network is trained based on the cross entropy loss, the first L2 loss, the second L2 loss, the disparity supervision loss function and the superpixel supervision function.

8. The method for disparity optimization based on learnable superpixel guidance according to claim 7, characterized in that: The superpixel supervision function is: ； ； In the formula, represents the superpixel supervision function, represents the disparity supervision loss function, and is the weighting coefficient, is the cross entropy loss between the true semantic graph and the reconstructed semantic graph, is the first L2 loss between the spatial position map and the reconstructed spatial position map, is the second L2 loss between the real disparity map and the reconstructed disparity map.

Images