Parallax data set generation method and parallax estimation network formation method

Abstract

The invention provides a parallax data set generation method and a parallax estimation network formation method. The parallax data set generation method comprises the following steps: acquiring a focused RGB image and a depth map; the diffusion radius of a blurring kernel corresponding to each pixel in the depth map is calculated, and a disparity map is formed; obtaining a point spread function model; layering the focused RGB image according to the depth map; convolution is carried out on each layer by adopting a point spread function model and mixing is carried out to obtain a simulation defocused RGB image; noise distribution is added to the simulation defocused RGB image, and the simulation defocused RGB image with the noise distribution added serves as a parallax data set. According to the parallax data set generation method, noise distribution is added, so that the simulated defocused RGB image is closer to reality, and boundary information is more accurate. According to the parallax estimation network forming method, the QPD parallax image has obvious advantages in parallax accuracy and background noise.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for generating a disparity data set and a method for forming a disparity estimation network. Background Art

[0002] Quad-phase (QPD) image sensors are widely used in mobile phones due to their advantages such as accurate focus and high dynamic range imaging. In addition to being used for focusing, due to the unique physical structure of QPD image sensors, parallax can be estimated using a single QPD camera. However, the lack of relevant datasets on the market has led to very little related research. In addition, the narrow baseline and uneven brightness of QPD image sensors have made algorithm design difficult.

[0003] Existing method 1 before improvement: PSF-based RGBD data generation method, which obtains disparity through preset camera parameters and input depth results; generates Dual-PD (two-phase detection) images through disparity and calibrated PSF; calibrates the disparity distribution of simulated data and real data. Limitations of method 1 before improvement: There is a blurred boundary effect when generating simulated images; there is a domain generalization problem with real data; since the input of the simulated image is a computer-generated image, there is no noise and no brightness imbalance.

[0004] Existing method 2 before improvement: RGBD data generation method based on rectangular diffusion, which obtains disparity according to preset camera parameters and input depth results; obtains the rectangular diffusion range of each pixel by disparity, and evenly distributes each pixel point within the rectangular diffusion range; accelerates the rectangular diffusion calculation through integral image. Limitations of method 2 before improvement: The PSF of the real image sensor is not evenly distributed, which is different from the characteristics of the actual image sensor, resulting in rectangular spots in the bright spots of the simulated image. No modeling is performed on the noise distribution and brightness non-uniformity of the real image sensor. Summary of the invention

[0005] The object of the present invention is to provide a disparity data set generation method and a disparity estimation network formation method. The disparity data set generation method provided by the present invention adds noise distribution to make the simulated defocused RGB image closer to reality, and the boundary information obtained by the simulated defocused RGB generated by the simulation method is more accurate. The disparity estimation network formation method provided by the present invention has obvious advantages in the accuracy of disparity and background noise in the QPD disparity map.

[0006] The present invention provides a method for generating a disparity data set, comprising:

[0007] S1. Obtain the corresponding focused RGB image and depth map; calculate the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and the preset camera parameters to form a corresponding disparity map;

[0008] S2. Obtaining a point spread function model by calibration or direct calling, wherein the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model;

[0009] S3, layering the focused RGB image according to the depth map; convolving each layer using the point spread function model and mixing to obtain a simulated defocused RGB image corresponding to the focused RGB image;

[0010] S4. Add two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and use the simulated defocused RGB image after adding the noise distribution as a disparity data set.

[0011] Furthermore, step S2, a method for calibrating the two-phase detection point spread function model, specifically comprises:

[0012] S211, collecting two-phase detection focusing raw images: using a Dual-PD camera, aligning it with the center of the calibration pattern, adjusting the position of the camera focus motor to focus, and collecting the two-phase detection focusing raw images;

[0013] S212, collecting two-phase detection defocus raw images: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting the two-phase detection defocus raw images;

[0014] S213, data alignment: using the enhanced correlation coefficient image alignment method, taking the two phase detection defocused raw images as a reference, aligning the two phase detection focused raw images with the two phase detection defocused raw images to obtain an aligned focused image; and calculating the two phase aligned green channel focused images corresponding to the aligned focused image.

[0015] S214, point spread function estimation: based on the two-phase detection defocus raw images, respectively calculate and obtain the two-phase green channel defocused images of the two-phase defocused left image and the right image in two directions; respectively calculate the blur kernels of the two-phase green channel defocused images of the two directions relative to the green channel focused images after the two phases are aligned, and obtain the left and right point spread function models.

[0016] Further, according to the diffusion radius corresponding to each layer of the focused RGB image and the left point diffusion function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused left image;

[0017] By analogy, the focused RGB image is calculated through the right point spread function model to obtain a simulated defocused right image;

[0018] The simulated defocused left image and the simulated defocused right image constitute the simulated defocused RGB image.

[0019] Furthermore, step S2, a method for calibrating the four-phase detection point spread function model, specifically comprises:

[0020] S221, collecting the four-phase detection focus raw image: using the QPD camera, aligning it with the center of the calibration pattern, adjusting the position of the camera focus motor to focus, and collecting the four-phase detection focus raw image;

[0021] S222, collecting a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and collecting the four-phase detection defocus raw image;

[0022] S223, data alignment: using the enhanced correlation coefficient image alignment method, taking the four-phase detection defocus raw image as a reference, aligning the four-phase detection focus raw image with the four-phase detection defocus raw image to obtain an aligned focused image; and calculating an aligned green channel focused image corresponding to the aligned focused image;

[0023] S224, point spread function estimation: based on the four-phase detection defocus raw image, respectively calculate and obtain the green channel defocus images of the four directions of the four-phase defocus, namely, the upper image, the lower image, the left image and the right image; respectively calculate the blur kernel of the green channel defocus image of the four directions of the four-phase defocus relative to the aligned green channel focused image to obtain four point spread function models of upper, lower, left and right.

[0024] Further, according to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused upper image;

[0025] By analogy, the focused RGB image is calculated by the lower point spread function model to obtain a simulated defocused lower image; the focused RGB image is calculated by the left point spread function model to obtain a simulated defocused left image; the focused RGB image is calculated by the right point spread function model to obtain a simulated defocused right image;

[0026] The simulated defocused left image, the simulated defocused right image, the simulated defocused upper image and the simulated defocused lower image constitute the simulated defocused RGB image.

[0027] Furthermore, in step S2, the method for calibrating the point spread function model uses a calibration graph comprising a graph consisting of a disk and a grid.

[0028] Further, in step S3, the point spread function model is used to convolve each layer and mix to obtain the simulated defocused RGB image, using the following formula:

[0029]

[0030] Among them, B s is the pixel value after diffusion calculation, I is the pixel intensity value, K is the PSF kernel function, (x 0 ,y 0 ) is the pixel coordinate; the pixel that needs to be calculated is dispersed according to the range and diffusion form of the diffusion function, and the final simulated defocused RGB image is the accumulation of all pixels after dispersion.

[0031] Furthermore, the disparity data set generation method further includes: performing compensation calculation on the simulated defocused RGB image, and the compensation method specifically includes:

[0032] providing a gray image of the same color intensity as the focused RGB image;

[0033] According to the diffusion radius and the point spread function model, a diffusion calculation is performed on the gray image to obtain a gray fuzzy image;

[0034] Compensate the gray fuzzy image to the gray image to obtain a compensation weight; the compensation weight is a color intensity ratio of the gray image to the gray fuzzy image;

[0035] Multiplying the simulated defocus RGB image by the compensation weight to obtain a compensated simulated defocus RGB image; and

[0036] The two-phase detection or four-phase detection noise distribution is added to the compensated simulated defocused RGB image, and the simulated defocused RGB image after adding the noise distribution is used as a disparity data set.

[0037] The present invention also provides a method for generating a disparity data set, comprising:

[0038] Collecting a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and collecting the four-phase detection defocus raw image;

[0039] According to the four-phase detection defocus raw image, a green channel defocus left image, a green channel defocus right image, a green channel defocus upper image and a green channel defocus lower image of four-phase defocus are respectively calculated;

[0040] In the four-phase detection defocused raw image, four adjacent pixels of the same color form a group; four groups arranged in a Bayer format form a unit; the unit includes a first green group Gb and a second green group Gr that are diagonally distributed;

[0041] The average value of the upper left and lower left pixels of the first green group Gb and the upper left and lower left pixels of the second green group Gr in each unit is taken as the green left pixel value after the four-in-one of the unit; each unit is assigned its own green left pixel value to form the green channel defocused left image;

[0042] The average value of the upper right and lower right pixels of the first green group Gb and the upper right and lower right pixels of the second green group Gr in each unit is taken as the green right pixel value after the four-in-one of the unit; each unit is assigned its own green right pixel value to form the green channel defocused right image;

[0043] The average value of the upper left and upper right pixels of the first green group Gb and the upper left and upper right pixels of the second green group Gr in each unit is taken as the green upper pixel value after the four-in-one of the unit; each unit is assigned its own green upper pixel value to form the green channel defocused upper image;

[0044] The average value of the four pixels, namely, the lower left and lower right pixels of the first green group Gb and the lower left and lower right pixels of the second green group Gr in each unit is taken as the green lower pixel value after the four-in-one of the unit; each unit is assigned its own green lower pixel value to form the green channel defocused image;

[0045] Perform disparity phase detection on the green channel defocused left image and the green channel defocused right image, and

[0046] The green channel defocused upper image and the green channel defocused lower image are subjected to disparity phase detection to finally obtain a disparity map.

[0047] Furthermore, the disparity label of the real scene is obtained by using the disparity calibration method through the checkerboard image. The specific process is as follows:

[0048] S1. Use a single QPD camera with a variable lens position to capture a chessboard image. Due to its rich texture information, the chessboard scene can accurately calculate the disparity map and obtain its corresponding disparity map;

[0049] S2, maintaining the same lens position, replacing the checkerboard with stickers; aligning the image of the sticker with the checkerboard image, and the disparity thereof is the disparity map obtained in step S1.

[0050] Furthermore, it also includes:

[0051] In the testing stage, a four-phase detection defocus raw image is input, and according to the four-phase detection defocus raw image, a green channel defocus left image, a green channel defocus right image, a green channel defocus upper image and a green channel defocus lower image of four-phase defocus are respectively calculated;

[0052] The green channel defocused left image, the green channel defocused right image, the green channel defocused upper image and the green channel defocused lower image of the four-phase defocus are input into the four-phase disparity model to obtain a disparity map.

[0053] The present invention also provides a method for forming a disparity estimation network, comprising:

[0054] S01. Establish a QPD disparity dataset, wherein the QPD disparity dataset includes: an upper image, a lower image, a left image, and a right image;

[0055] S02, using an illumination invariant feature extraction module to extract features of the upper image and the lower image, and mapping the upper image and the lower image to a dense feature map; using a recursive full field transform algorithm to calculate the visual similarity between all pixel pairs in the upper image and the lower image to construct a correlation volume and generate relevant features; the relevant features are passed through a number of convolutional layers and the disparity update is predicted, and an upper and lower phase disparity map is generated through a sub-pixel disparity estimation module;

[0056] The illumination invariant feature extraction module is used to extract features of the left image and the right image, and a recursive full field transform algorithm is used to calculate the visual similarity between all pixel pairs in the left image and the right image to construct a correlation volume and generate relevant features; the relevant features are transmitted through a plurality of convolutional layers and the disparity update is predicted, and the left and right phase disparity maps are generated through the sub-pixel disparity estimation module;

[0057] S03: Using a confidence-based disparity fusion module to fuse the upper and lower phase disparity maps with the left and right phase disparity maps to obtain the final disparity image.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention provides a disparity data set generation method and a disparity estimation network formation method, the disparity data set generation method, including: S1, obtaining the corresponding focused RGB image and depth map; according to the imaging principle and the preset camera parameters, calculating the diffusion radius of the blur kernel corresponding to each pixel in the depth map to form the corresponding disparity map; S2, obtaining the point spread function model by calibration or direct calling, the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; using the point spread function model to convolve each layer and mix to obtain a simulated defocused RGB image corresponding to the focused RGB image; S4, adding two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and using the simulated defocused RGB image after adding the noise distribution as the disparity data set. The disparity data set generation method provided by the present invention adds noise distribution, so that the simulated defocused RGB image is closer to reality, and the boundary information obtained by the simulated defocused RGB generated by the simulation method is more accurate. The disparity estimation network formation method provided by the present invention has obvious advantages in the accuracy of disparity and background noise in the QPD disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The figure is a flow chart of a method for generating a disparity dataset according to an embodiment of the present invention.

[0061] Figure 2 Schematic diagram of the principle of a method for generating a disparity dataset according to an embodiment of the present invention.

[0062] Figure 3 It is a PSF calibration graph in the disparity data set generation method according to an embodiment of the present invention.

[0063] Figure 4 It is a PSF graph in the disparity data set generation method according to an embodiment of the present invention.

[0064] Figure 5 It is a compensation schematic diagram of a disparity data set generation method according to an embodiment of the present invention.

[0065] Figure 6 A schematic diagram of adding noise in the method for generating a disparity dataset according to an embodiment of the present invention.

[0066] Figure 7 These are the corresponding focused RGB image and depth map obtained in the disparity dataset generation method according to the embodiment of the present invention.

[0067] Figure 8 This is a comparison diagram of the simulated defocused RGB image generated by the present invention and the improved method 1.

[0068] Figure 9a A schematic diagram of generating a disparity map from a real defocused RGB image according to the present invention.

[0069] Figure 9b Schematic diagram of a checkerboard image, a corresponding poster image, and a corresponding disparity image in a single depth of field scene.

[0070] Fig.9c Schematic diagram of a checkerboard image, a corresponding poster image, and a corresponding disparity image in a progressive depth of field scene.

[0071] Figure 9d Schematic diagram of a chessboard image, a corresponding poster image, and a corresponding disparity image in a multi-depth scene.

[0072] Fig.10 A disparity map is generated for testing and verifying a real defocused RGB image of the present invention.

[0073] Fig.11 Comparison of the QPD disparity datasets generated by the present invention and the first method before improvement.

[0074] Fig.12 Comparison of the disparity maps generated by the present invention and the first method before improvement.

[0075] Fig.13 The DPD disparity map and QPD disparity map generated by the present invention.

[0076] Fig.14 This is a comparison table of the DPD parallax generated by the present invention and the traditional PDAF parallax.

[0077] Fig.15 Comparison of the QPD disparity maps generated by the present invention and the third method before improvement.

[0078] Fig.16 Comparison of the QPD disparity maps generated by the present invention and the improved method 4.

[0079] Fig.17 Comparison of the QPD disparity maps generated by the present invention and the method 5 before improvement. DETAILED DESCRIPTION

[0080] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0081] For ease of description, some embodiments of the present application may use spatially relative terms such as "above", "below", "top", "below", etc. to describe the relationship between one element or component and another (or other) elements or components as shown in the various figures of the embodiments. It should be understood that in addition to the orientations described in the drawings, the spatially relative terms are also intended to include different orientations of the device in use or operation. For example, if the device in the drawings is turned over, the elements or components described as being "below" or "below" other elements or components will subsequently be positioned as being "above" or "above" other elements or components. The terms "first", "second", etc. below are used to distinguish between similar elements and are not necessarily used to describe a specific order or time sequence. It is to be understood that these terms used in this way are interchangeable where appropriate.

[0082] The embodiment of the present invention provides a method for generating a disparity dataset. Figure 1 As shown, including:

[0083] S1. Obtain the corresponding focused RGB image and depth map; calculate the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and the preset camera parameters to form the corresponding disparity map;

[0084] S2, by calibrating or directly calling the point spread function model, the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model;

[0085] S3, layering the focused RGB image according to the depth map; convolving each layer using a point spread function model and mixing to obtain a simulated defocused RGB image corresponding to the focused RGB image;

[0086] S4. Add two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and use the simulated defocused RGB image after adding the noise distribution as a disparity data set.

[0087] The following describes in detail the steps of the method for generating a disparity dataset according to an embodiment of the present invention.

[0088] Step S1: Figure 2 As shown, the corresponding focused RGB image and depth map are obtained; the corresponding focused RGB image and depth map are obtained by retrieving a known database or shooting after the camera is focused. The focused RGB image can be obtained by shooting with a conventional single phase detection camera. According to the imaging principle and the preset camera parameters, the diffusion radius of the blur kernel corresponding to each pixel in the depth map is calculated to form a corresponding disparity map. The depth in this article is the object distance, and the depth map records the geometric information of the scene, which indicates the distance of the object in the scene from the optical center of the camera. The calculation formula for the diffusion radius r is as follows:

[0089]

[0090] Among them, d is the diffusion diameter, α is a constant, L is the aperture size, f is the focal length, z is the object distance when focusing, and D is the depth. The preset camera parameters include: focal length, aperture, phase plane distance and other parameters.

[0091] It should be understood that the phase parallax of the left and right images of the Dual-PD (dual phase detection) focused RGB image is 0; the phase parallax of the left and right images of the QPD (quad phase detection) focused RGB image is 0, and the phase parallax of the upper and lower images of the QPD (quad phase detection) focused RGB image is 0; that is, the focused RGB image has basically no parallax, and what needs to be generated by the disparity data set of the present invention is a defocused image, and both the defocused image of the Dual-PD (dual phase detection) and the defocused image of the QPD (quad phase detection) have parallax.

[0092] Step S2: Calibrate or directly call the point spread function model, such as Figure 3 and Figure 4 As shown, a method for calibrating a point spread function model uses a calibration figure including a figure composed of a disk and a grid. The point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model. The point spread function model can be obtained by a calibration method, or by filter simulation, or by directly giving it. The point spread function model can be estimated or calibrated using a Butterworth filter function or a Gaussian filter function, or an existing point spread function model can be directly called, or the shape of a blur function can be defined. A clean and clear image can be rendered from a blurred input image, and the point spread function model can also be obtained by rendering. The point spread function model of the present invention can be a non-uniform diffusion model or a uniform diffusion model.

[0093] Step S2, a method for calibrating a two-phase detection point spread function model, specifically comprising:

[0094] S211, collecting two-phase detection focusing raw images: using a Dual-PD camera, aligning it with the center of the calibration pattern, adjusting the position of the camera focus motor to focus, and collecting two-phase detection focusing raw images;

[0095] S212, collecting two-phase detection defocus raw images: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting two-phase detection defocus raw images;

[0096] S213, data alignment: using the enhanced correlation coefficient image alignment method, taking the two-phase detection defocused raw images as the reference, aligning the two-phase detection focused raw images with the two-phase detection defocused raw images to obtain an aligned focused image; and calculating the two-phase aligned green channel focused images corresponding to the aligned focused image.

[0097] S214, point spread function estimation: based on the two-phase detection defocus raw images, respectively calculate and obtain the two-phase green channel defocused images of the two-phase defocused left image and the right image in two directions; respectively calculate the blur kernel of the two-phase green channel defocused images of the two directions relative to the green channel focused image after the two phases are aligned, and obtain the left and right point spread function models.

[0098] According to the diffusion radius corresponding to each layer of the focused RGB image and the left point diffusion function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of distance to obtain the simulated defocused left image;

[0099] By analogy, the focused RGB image is calculated through the right point spread function model to obtain the simulated defocused right image;

[0100] The simulated defocused left image and the simulated defocused right image constitute a simulated defocused RGB image.

[0101] Step S2, a method for calibrating a four-phase detection point spread function model, specifically comprising:

[0102] S221, collecting four-phase detection focus raw images: using a QPD camera, aligning it with the center of the calibration pattern, adjusting the position of the camera focus motor to focus, and collecting four-phase detection focus raw images;

[0103] S222, collecting four-phase detection defocus raw images: adjusting the focus motor position of the QPD camera to blur the image, and collecting four-phase detection defocus raw images;

[0104] S223, data alignment: using the enhanced correlation coefficient image alignment method, taking the four-phase detection defocused raw image as a reference, aligning the four-phase detection focused raw image with the four-phase detection defocused raw image to obtain an aligned focused image; and calculating the aligned green channel focused image corresponding to the aligned focused image;

[0105] S224, point spread function estimation: based on the four-phase detection defocus raw image, respectively calculate and obtain the green channel defocused images of the four directions of the four-phase defocus, namely, the upper image, the lower image, the left image and the right image; respectively calculate the blur kernel of the green channel focused image after the relative alignment of the green channel defocused images of the four directions of the four-phase defocus, and obtain the four point spread function models of upper, lower, left and right.

[0106] According to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of distance to obtain the simulated defocused upper image;

[0107] By analogy, the focused RGB image is calculated through the lower point spread function model to obtain the simulated defocused lower image; the focused RGB image is calculated through the left point spread function model to obtain the simulated defocused left image; the focused RGB image is calculated through the right point spread function model to obtain the simulated defocused right image;

[0108] The simulated defocused left image, the simulated defocused right image, the simulated defocused upper image and the simulated defocused lower image constitute a simulated defocused RGB image.

[0109] In step S2, Figure 4 As shown, the four-phase detection point spread function model includes: an upper point spread function model (upper PSF), a lower point spread function model (lower PSF), a left point spread function model (left PSF) and a right point spread function model (right PSF); Figure 4 It can be seen that the point spread function models (blur kernels) in the four directions of up, down, left, and right are different, and the shapes of the blur kernels are different in different directions, so that the defocused RGB image can be simulated more specifically and accurately.

[0110] Step S3, layering the focused RGB image according to the depth map; convolving each layer using a point spread function model and mixing to obtain a simulated defocused RGB image corresponding to the focused RGB image.

[0111] In step S3, the point spread function model is used to convolve each layer to blur the image and mask of the depth layer to obtain a simulated defocused RGB image. The diffusion calculation formula is as follows:

[0112]

[0113] Among them, B s is the pixel value after diffusion calculation, I is the pixel intensity value, K is the PSF kernel function, (x 0 ,y 0 ) is the pixel coordinate; the pixel to be calculated is dispersed according to the range and diffusion form of the diffusion function, and the final simulated defocused RGB image is the accumulation of all pixels after dispersion. The layers after diffusion calculation are mixed in the order of distance to obtain the simulated defocused (blurred) image B corresponding to the focused RGB image s .

[0114] Different depths correspond to different point spread functions, and different orientation maps correspond to different point spread functions. The point spread function is a mathematical description of the shape and intensity of the imaging spot. According to the depth map, the layering is performed, and the depth corresponds to the blur kernel radius; the depth and direction are combined to obtain the conversion from a clear image to a blurred image.

[0115] The disparity data set generation method further includes: performing compensation calculation on the simulated defocused RGB image, and the compensation method specifically includes: Figure 5As shown,

[0116] Provide a gray image B_gray with the same color intensity as the focused RGB image; the color intensity of the gray image B_gray is, for example, 128, and the gray image B_gray is used as a calibration image;

[0117] According to the diffusion radius and point spread function model, the gray image B_gray is diffused and calculated to obtain the gray blur image B_gray_blur;

[0118] The gray blur image B_gray_blur is compensated to the gray image B_gray to obtain a compensation weight W; the compensation weight W is a color intensity ratio of the gray image B_gray to the gray blur image B_gray_blur;

[0119] The simulated defocused RGB image B s Multiply by the compensation weight W to get the compensated simulated defocused RGB image B out ;as well as

[0120] The simulated defocused RGB image B after compensation out Add two-phase detection or four-phase detection noise distribution, and use the simulated defocused RGB image after adding the noise distribution as the disparity dataset.

[0121] Figure 5 In the figure, the simulated defocused RGB image B before compensation s The transition zone across the depth of field is unnatural, for example, there are stripes, which are more obvious on the left side of the car in the picture. The gray blur image B_gray_blur also has stripes. The gray blur image B_gray_blur is compensated to the gray image B_gray, and the gray blur image B_gray_blur is de-striped and compensated to the gray image B_gray to obtain the compensation weight W. s Multiply by the compensation weight W to get the compensated simulated defocus RGB image B out . Figure 5 As can be seen in the upper right corner, the simulated defocused RGB image B after compensation out There are no more stripes in the transition zone area across the depth of field, and the transition is natural.

[0122] Step S4: adding two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and using the simulated defocused RGB image after adding the noise distribution as a disparity data set.

[0123] Since the simulated defocused RGB image is generated by simulating the focused RGB image or a computer-generated image (also called a CG image), the noise distribution of the simulated defocused RGB image is inconsistent with that of the QPD image sensor. The QPD image sensor is calibrated to obtain the noise distribution, and noise is added to the simulated defocused RGB image to alleviate the domain generalization problem. Figure 6 As shown in the figure, QPD noise calibration shows that the ideal noise model is as follows:

[0124]

[0125] Among them, σ y 2 is the noise variance, σ d are all noise sources associated with the sensor’s readout and amplification circuits, σ q is the quantization noise. According to the noise variance σ y 2 and the average grayscale value of the photoelectric conversion output (μ y -μ y , dark), the global system gain K can be obtained from the slope, and σ can be obtained from the intercept. d 2 By collecting two frames of color card images of the QPD image sensor at different gains, the noise distribution is obtained by fitting the noise curve. The influence of FPN noise can be eliminated by two frame inputs.

[0126] Figure 7 These are the corresponding focused RGB image and depth map obtained in the disparity dataset generation method according to the embodiment of the present invention. Figure 8 This is a comparison of the simulated defocused RGB images generated by the present invention and the improved method 1. Figure 7 and Figure 8 As shown, compared with the simulated defocus RGB generated by the first method before the improvement, the boundary information (especially in the red circle) obtained by the simulated defocus RGB generated by the disparity data set generation method of the present invention, i.e. the simulation method, is more accurate.

[0127] The above introduces the formation of a disparity dataset by simulation. The following introduces the formation of a disparity dataset by using real data.

[0128] Figure 9a Schematic diagram of generating a disparity map from a real defocused RGB image according to the present invention. Figure 9a As shown,

[0129] An embodiment of the present invention provides a method for generating a disparity dataset, comprising:

[0130] Collecting the defocused raw image of quad phase detection (QPD): Adjust the focus motor position of the QPD camera to blur the image and collect the defocused raw image of quad phase detection;

[0131] According to the four-phase detection defocus raw image, the four-phase defocused green channel defocused left image G is calculated respectively. L , Green channel defocused right image G R , green channel defocused image G U And the green channel defocused image G D ;

[0132] In the four-phase detection defocused raw image, four adjacent pixels of the same color form a group; four groups arranged in the Bayer format form a unit; the unit includes a first green group Gb and a second green group Gr distributed diagonally;

[0133] The average value of the upper left and lower left pixels of the first green group Gb and the upper left and lower left pixels of the second green group Gr in each unit is taken as the green left pixel value after the unit is four-in-one; each unit is assigned its own green left pixel value to form a green channel defocused left image Left-G;

[0134] The average value of the upper right and lower right pixels of the first green group Gb and the upper right and lower right pixels of the second green group Gr in each unit is taken as the green right pixel value after the unit is four-in-one; each unit is assigned its own green right pixel value to form a green channel defocused right image Right-G;

[0135] The average value of the upper left and upper right pixels of the first green group Gb and the upper left and upper right pixels of the second green group Gr in each unit is taken as the green upper pixel value after the unit is four-in-one; each unit is assigned its own green upper pixel value to form a green channel defocused upper image Up-G;

[0136] The average value of the four pixels, namely, the lower left and lower right pixels of the first green group Gb and the lower left and lower right pixels of the second green group Gr in each unit, is taken as the green lower pixel value after the unit is four-in-one; each unit is assigned its own green lower pixel value to form a green channel defocused lower image Down-G;

[0137] Perform disparity phase detection on the green channel defocused left image Left-G and the green channel defocused right image Right-G, and

[0138] The disparity phase detection is performed on the green channel defocused upper image Up-G and the green channel defocused lower image Down-G to finally obtain a disparity map.

[0139] The real data acquisition process proposes a method to obtain the disparity Ground Truth (real label) of the real scene by using a disparity calibration method using a checkerboard image. Figure 9b Schematic diagram of a checkerboard image (left), the corresponding poster image (middle), and the corresponding disparity image (right) in a single depth of field scene. Fig.9cSchematic diagram of a checkerboard image (left), the corresponding poster image (middle), and the corresponding disparity image (right) in a progressive depth of field scene. Figure 9d Schematic diagram of a checkerboard image (left), the corresponding poster image (middle), and the corresponding disparity image (right) in a multi-depth scene.

[0140] The disparity label of the real scene is obtained by using the disparity calibration method through the checkerboard image. The specific process is as follows:

[0141] S1. Use a single QPD camera with a variable lens position to capture a checkerboard image. Due to its rich texture information, the checkerboard scene can accurately calculate the disparity map and obtain its corresponding disparity map;

[0142] S2. Keep the same lens position and replace the chessboard with a sticker (such as the corresponding poster image); align the sticker image with the chessboard image, and the disparity thereof is the disparity map obtained in step S1.

[0143] Disparity dataset generation methods, such as Fig.10 As shown, it also includes:

[0144] In the test phase, the four-phase detection (QPD) defocus raw image is input, and the four-phase defocus green channel defocus left image, green channel defocus right image, green channel defocus upper image and green channel defocus lower image are calculated according to the four-phase detection defocus raw image.

[0145] The four-phase defocused green channel defocused left image, green channel defocused right image, green channel defocused upper image and green channel defocused lower image are input into the four-phase detection disparity estimation network model to obtain the disparity map.

[0146] Fig.11 This is a comparison of the QPD disparity data set generated by the present invention and the improved method 1. Fig.11 As shown in Figure 2, the validity of the data, due to the challenges of obtaining disparity in natural scenes, the quality of the disparity map is evaluated by segmentation. The threshold t is used to distinguish the foreground from the background. By changing the threshold, the maximum intersection-over-union (IoU) value can be generated to measure the maximum overlap between the foreground and its corresponding disparity. Where IoU is the value obtained by dividing the intersection of A and B by the union of A and B. A represents the foreground and B represents the background.

[0147] Fig.11 Quantitative test results are provided, which show that the combination of the simulated QPD data of the present invention with the real QPD data during the training process (see Fig.11The last row of data) will produce the best results. In addition, the method of the present invention shows better performance on the disparity boundary (minHD), but the performance is slightly reduced in the correctness of the disparity map (maxIoU). Combining real data and simulated data for iterative training can improve the generalization ability of the disparity estimation network. Compared with the simulation method of the improved method 1, the method proposed in the present invention can improve the accuracy of the disparity edge.

[0148] Fig.12 The disparity map generated by the present invention is compared with the disparity map generated by the first method before improvement. Fig.12 The boundary of the red circle is not clear. Fig.12 This shows that the data simulation method of the present invention can obtain more accurate boundary results.

[0149] Fig.13 The DPD disparity map and QPD disparity map generated by the present invention. Fig.14 This is a comparison table of the DPD parallax generated by the present invention and the traditional PDAF parallax.

[0150] To verify the correctness of the algorithm, the calculation results of the present invention are compared with the traditional PDAF results, where the DPD disparity map input of the present invention is a Dual-PD (two-phase detection) focused RGB image, and the QPD disparity map input is a QPD (four-phase detection) focused RGB image. Dual-PD and four-phase (QPD) technologies have been applied to cameras because they enable camera devices (such as mobile phone cameras) to achieve higher-speed autofocus. For two-phase raw data, two left and right images with phase difference can be obtained in the same frame. For four-phase raw data, four images with phase difference, upper left, upper right, lower left, and lower right, can be obtained in the same frame.

[0151] Fig.14 It shows that the error between the DPD disparity calibration result of the present invention and the result of the traditional PDAF algorithm is very small, indicating that the disparity estimation result is correct. PDAF (Phase Detection Auto Focus) technology is widely used in the auto-focus of smartphone cameras.

[0152] The embodiment of the present invention further provides a disparity estimation network, including:

[0153] S01. Establish a QPD disparity dataset, where the QPD disparity dataset includes: an upper image, a lower image, a left image, and a right image;

[0154] Four adjacent pixels of the same color in the QPD image form a group, and there is parallax between any two of the four pixels in the same group; based on the QPD image, an upper image, a lower image, a left image, and a right image are obtained by calculation.

[0155] The upper image takes the average value of the upper left and upper right pixels in each color group as the pixel value of the group after four-in-one; the lower image takes the average value of the lower left and lower right pixels in each color group as the pixel value of the group after four-in-one; the left image takes the average value of the upper left and lower left pixels in each color group as the pixel value of the group after four-in-one; the right image takes the average value of the upper right and lower right pixels in each color group as the pixel value of the group after four-in-one.

[0156] S02, using an illumination invariant feature extraction module to extract features of the upper image and the lower image, and mapping the upper image and the lower image to a dense feature map; using a recursive full field transform algorithm to calculate the visual similarity between all pixel pairs in the upper image and the lower image to construct a correlation volume and generate relevant features; the relevant features are passed through several convolutional layers and the disparity update is predicted, and the upper and lower phase disparity maps are generated by a sub-pixel disparity estimation module;

[0157] The illumination invariant feature extraction module is used to extract the features of the left and right images. The recursive full field transform algorithm is used to calculate the visual similarity between all pixel pairs in the left and right images to construct the correlation volume and generate the correlation features. The correlation features are passed through several convolutional layers and the disparity update is predicted. The sub-pixel disparity estimation module generates the left and right phase disparity maps.

[0158] S03, using a confidence-based disparity fusion module to fuse the upper and lower phase disparity maps with the left and right phase disparity maps to obtain a final disparity image.

[0159] A disparity estimation network that effectively utilizes the characteristics of QPD data. The network consists of three main modules: illumination invariant feature extraction, sub-pixel disparity estimation, and confidence-based disparity fusion.

[0160] Fig.15 The QPD disparity map generated by the present invention is compared with that generated by the third method before improvement. The third method before improvement is a self-supervised monocular depth estimation network, which uses a combination of depth estimation and posture estimation networks to predict the depth in a single frame image. When training with monocular data, three adjacent frames of images need to be input. The third method before improvement is designed for general scenarios and does not take into account the characteristics of QPD / DPD image sensors. Fig.15 It can be seen that the calculation results of the present invention are compared with the monocular depth results of the three methods before the improvement. Whether it is the accuracy of disparity or background noise, the QPD disparity map generated by the present invention has obvious advantages.

[0161] Fig.16The QPD disparity map generated by the present invention is compared with that generated by the fourth method before improvement. The fourth method before improvement is based on the disparity estimation network of the DPD image sensor, and estimates the disparity by using the symmetry of the DPD PSF, which is an unsupervised method. The fourth method before improvement includes: obtaining the PSF according to the left and right images of the DP input and the blur kernel and other parameters; the disparity is proportional to the blur kernel, and a preliminary disparity is obtained; the input image and the preliminary disparity result are subjected to bilateral filtering to obtain an optimized disparity map; the optimized disparity map and the input image are subjected to guided filtering to obtain the final optimized disparity result. The fourth method before improvement is based on the Dual-PD (two-phase detection) pixel blur kernel symmetry design algorithm, and different depths will appear in the same depth of field; the blur kernel of the actual image sensor PSF is asymmetric, and the principle assumption is too strong. By Fig.16 It can be seen that compared with the fourth method before improvement, the QPD disparity map generated by the present invention has obvious advantages in boundary accuracy, which shows that the unsupervised method of the fourth method before improvement is not general enough.

[0162] Fig.17 The QPD disparity map generated by the present invention is compared with that generated by the fifth method before improvement. The fifth method before improvement is a disparity estimation network based on stereo matching, which uses CNN for feature extraction and matching to estimate disparity, and is a supervised method. The fifth method before improvement includes: image pairs are passed to the feature extractor for long-distance pixel modeling; features are processed by the subsequent multi-scale decoupled LSTM network, which can carry more semantic information in iterations; the refinement module upsamples the disparity map and performs disparity, and normalizes it before processing through the feedforward structure to alleviate the limitation of domain generalization problem. Limitations of the fifth method before improvement: The feature extraction method is based on the feature extractor, which leads to a lack of local information and blurred boundaries. Fig.17 It can be seen that, compared with the improved method 5, the QPD disparity map generated by the present invention has obvious advantages in boundary accuracy and can restore sharp boundary information.

[0163] In summary, the present invention provides a disparity data set generation method and a disparity estimation network formation method, the disparity data set generation method, including: S1, obtaining the corresponding focused RGB image and depth map; according to the imaging principle and the preset camera parameters, calculating the diffusion radius of the blur kernel corresponding to each pixel in the depth map to form a corresponding disparity map; S2, obtaining the point spread function model by calibration or direct call, the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; using the point spread function model to convolve each layer and mix to obtain a simulated defocused RGB image corresponding to the focused RGB image; S4, adding two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and using the simulated defocused RGB image after adding the noise distribution as the disparity data set. The disparity data set generation method provided by the present invention adds noise distribution, so that the simulated defocused RGB image is closer to reality, and the boundary information obtained by the simulated defocused RGB generated by the simulation method is more accurate. The disparity estimation network formation method provided by the present invention has obvious advantages in the accuracy of disparity and background noise in the QPD disparity map.

[0164] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0165] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for generating a disparity dataset, characterized in that: include: S1. Obtain the corresponding focused RGB image and depth map; Calculating the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and preset camera parameters to form a corresponding disparity map; S2. Obtaining a point spread function model by calibration or direct calling, wherein the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; Convolving each layer with the point spread function model and mixing to obtain a simulated defocused RGB image corresponding to the focused RGB image; S4. Add two-phase detection or four-phase detection noise distribution to the simulated defocused RGB image, and use the simulated defocused RGB image after adding the noise distribution as a disparity dataset.

2. The method for generating a disparity dataset according to claim 1, wherein: Step S2, a method for calibrating the two-phase detection point spread function model, specifically comprising: S211, collecting the two-phase detection focus raw image: using the Dual-PD camera, aligning it with the center of the calibration pattern, adjusting the position of the camera focus motor to focus, and collecting the two-phase detection focus raw image; S212, collecting a dual-phase detection defocus raw image: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting the dual-phase detection defocus raw image; S213. Data alignment: Using the enhanced correlation coefficient image alignment method, with the two phase detection defocused raw images as a reference, align the two phase detection focused raw images with the two phase detection defocused raw images to obtain an aligned focused image; and calculate the two phase-aligned green channel focused images corresponding to the aligned focused image. S214, point spread function estimation: based on the two-phase detection defocus raw images, respectively calculate and obtain the two-phase green channel defocus images of the two-phase defocused left image and the right image; respectively calculate the blur kernel of the two-phase green channel defocus images of the two directional images relative to the green channel focused image after the two phases are aligned, and obtain the left and right point spread function models.

3. The method for generating a disparity dataset according to claim 2, wherein: Performing diffusion calculation on pixels corresponding to each layer according to the diffusion radius corresponding to each layer of the focused RGB image and the left point spread function model; mixing the layers after the diffusion calculation in order of distance to obtain a simulated defocused left image; Similarly, the focused RGB image is calculated by the right point spread function model to obtain a simulated defocused right image; The simulated defocused left image and the simulated defocused right image constitute the simulated defocused RGB image.

4. The method for generating a disparity dataset according to claim 1, wherein: Step S2, a method for calibrating the four-phase detection point spread function model, specifically comprising: S221. Acquire a four-phase detection focus raw image: Use a QPD camera to align with the center of the calibration pattern, adjust the camera focus motor position to focus, and acquire the four-phase detection focus raw image; S222, collecting a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and collecting the four-phase detection defocus raw image; S223, data alignment: using the enhanced correlation coefficient image alignment method, with the four-phase detection defocus raw image as a reference, aligning the four-phase detection focus raw image with the four-phase detection defocus raw image to obtain an aligned focused image; and calculating an aligned green channel focused image corresponding to the aligned focused image; S224, point spread function estimation: based on the four-phase detection defocus raw image, respectively calculate and obtain the green channel defocus images of the four directions of the four-phase defocus, namely the upper image, the lower image, the left image and the right image; respectively calculate the blur kernel of the green channel defocus image of the four directions of the four-phase defocus relative to the aligned green channel focused image to obtain the four point spread function models of upper, lower, left and right.

5. The method for generating a disparity dataset according to claim 4, wherein: Performing diffusion calculation on pixels corresponding to each layer according to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model; mixing the layers after the diffusion calculation in order of distance to obtain a simulated defocused upper image; Similarly, the focused RGB image is calculated by the lower point spread function model to obtain a simulated defocused image; The focused RGB image is calculated by the left point spread function model to obtain a simulated defocused left image; The focused RGB image is calculated by the right point spread function model to obtain a simulated defocused right image; The simulated defocused left image, the simulated defocused right image, the simulated defocused upper image, and the simulated defocused lower image constitute the simulated defocused RGB image.

6. The method for generating a disparity dataset according to claim 2 or 4, wherein: In step S2, the method for calibrating the point spread function model uses a calibration graph comprising a graph consisting of a disk and a grid.

7. The method for generating a disparity dataset according to claim 1, wherein: In step S3, the point spread function model is used to convolve each layer and mix to obtain the simulated defocused RGB image, using the following formula: Among them, B s is the pixel value after diffusion calculation, I is the pixel intensity value, K is the PSF kernel function, and (x0, y0) is the pixel coordinate; the pixel currently to be calculated is dispersed according to the range and diffusion shape of the diffusion function, and the final simulated defocused RGB image is the accumulation of all pixels after dispersion.

8. The method for generating a disparity dataset according to claim 1, wherein: The disparity dataset generation method further includes: performing compensation calculation on the simulated defocused RGB image, wherein the compensation method specifically includes: providing a grayscale image having the same color intensity as the focused RGB image; Performing diffusion calculation on the gray image according to the diffusion radius and the point spread function model to obtain a gray fuzzy image; Compensating the gray fuzzy image to the gray image to obtain a compensation weight; the compensation weight is a color intensity ratio of the gray image to the gray fuzzy image; Multiplying the simulated defocus RGB image by the compensation weight to obtain a compensated simulated defocus RGB image; and The two-phase detection or four-phase detection noise distribution is added to the compensated simulated defocused RGB image, and the simulated defocused RGB image after adding the noise distribution is used as a disparity dataset.

9. A method for generating a disparity dataset, characterized in that: include: Acquiring a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and acquiring the four-phase detection defocus raw image; According to the four-phase detection defocus raw image, a green channel defocused left image, a green channel defocused right image, a green channel defocused upper image and a green channel defocused lower image of the four-phase defocus are respectively calculated; In the four-phase detection defocus raw image, four adjacent pixels of the same color form a group; four of the groups arranged in a Bayer format form a unit; the unit includes a first green group Gb and a second green group Gr that are diagonally distributed; The average value of the four pixels (the upper left and lower left pixels of the first green group Gb and the upper left and lower left pixels of the second green group Gr) in each unit is taken as the green left pixel value of the unit after four-in-one; each unit is assigned its own green left pixel value to form the green channel defocused left image; The average value of the four pixels (the upper right and lower right pixels of the first green group Gb and the upper right and lower right pixels of the second green group Gr) in each unit is taken as the green right pixel value of the unit after four-in-one combination; each unit is assigned its own green right pixel value to form the green channel defocused right image; The average value of the four pixels, namely, the upper left and upper right pixels of the first green group Gb and the upper left and upper right pixels of the second green group Gr in each unit, is taken as the green upper pixel value after the four-in-one process of the unit; each unit is assigned its own green upper pixel value to form the green channel defocused upper image; The average value of the four pixels, namely, the lower left and lower right pixels of the first green group Gb and the lower left and lower right pixels of the second green group Gr in each unit, is taken as the green lower pixel value of the unit after four-in-one processing; each unit is assigned its own green lower pixel value to form the green channel defocused image; Performing parallax phase detection on the green channel defocused left image and the green channel defocused right image, and Disparity phase detection is performed on the green channel defocused upper image and the green channel defocused lower image to finally obtain a disparity map.

10. The method for generating a disparity dataset according to claim 9, wherein: The disparity calibration method is used to obtain the disparity label of the real scene through the checkerboard image. The specific process is as follows: S1. Use a single QPD camera with a variable lens position to capture a checkerboard image. Due to its rich texture information, the checkerboard scene can accurately calculate the disparity map and obtain its corresponding disparity map. S2. Maintaining the same lens position, replace the checkerboard with a sticker; align the sticker image with the checkerboard image, and the disparity thereof is the disparity map obtained in step S1.

11. The method for generating a disparity dataset according to claim 1, wherein: Also includes: In the testing phase, a four-phase detection defocus raw image is input, and according to the four-phase detection defocus raw image, a four-phase defocus green channel defocus left image, a green channel defocus right image, a green channel defocus upper image, and a green channel defocus lower image are calculated respectively; The green channel defocused left image, the green channel defocused right image, the green channel defocused upper image, and the green channel defocused lower image of the four-phase defocus are input into the four-phase disparity model to obtain a disparity map.

12. A method for forming a disparity estimation network, characterized in that: include: S01. Establish a QPD disparity dataset, where the QPD disparity dataset includes an upper image, a lower image, a left image, and a right image. S02, extracting features of the upper image and the lower image using an illumination invariant feature extraction module, and mapping the upper image and the lower image to dense feature maps; using a recursive full field transform algorithm to calculate the visual similarity between all pixel pairs in the upper image and the lower image to construct a correlation volume and generate correlation features; passing the correlation features through several convolutional layers and predicting disparity updates, and generating upper and lower phase disparity maps through a sub-pixel disparity estimation module; The illumination-invariant feature extraction module is used to extract features of the left image and the right image, and a recursive full-field transform algorithm is used to calculate the visual similarity between all pixel pairs in the left image and the right image to construct a correlation volume and generate correlation features; the correlation features are passed through several convolutional layers and disparity updates are predicted, and left and right phase disparity maps are generated by the sub-pixel disparity estimation module; S03 , using a confidence-based disparity fusion module to fuse the upper and lower phase disparity maps with the left and right phase disparity maps to obtain the final disparity image.

Images