Joint phase estimation and defocusing fuzzy data set generation and network formation method

Abstract

According to the combined phase estimation and defocusing fuzzy data set generation method provided by the invention, matching is carried out after a full-bright image is projected, so that the shadow defect can be avoided, and the depth estimation of the shadow part of the projected image is correct; structured light projection fringes are improved, Gray codes and phase shift codes are combined, the moire phenomenon is avoided, and accurate edge parallax is obtained. The invention also provides a DPD-based joint phase estimation and defocusing fuzzy network forming method, and the method comprises the steps: setting an independent parallax estimation module, and obtaining an accurate parallax result; according to the parallax image, an image alignment module is added, the deblurring performance is improved, and parallax estimation is more accurate. The invention further provides a QPD-based combined phase estimation and defocusing fuzzy network forming method, parallax fusion and feature fusion are carried out by fully utilizing parallax information and image feature information according to the imaging principle of a QPD image sensor, and a clearer deblurring effect and a more accurate depth estimation effect are obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a joint phase estimation and defocus blur dataset generation and network formation method. Background Art

[0002] The quad-phase (QPD) image sensor is a design in which four pixels share a microlens, which can measure all phase differences of the object from top to bottom, left to right, and all four directions, to achieve faster and more accurate autofocus. In addition to focusing, it can also be used for applications such as image depth and deblurring. Compared with single-purpose input, it can achieve better results by utilizing additional phase information. Joint phase estimation and deblurring can achieve better results, but due to the lack of relevant datasets, there are few related studies.

[0003] Existing method before improvement: DPD (two-phase detection) data simulation method based on rectangular kernel and joint disparity and defocus blur network design. The method includes: S1, constructing a real data set: collecting defocused-in-focus image pairs in real scenes and using a depth estimation method based on defocused image sequences to obtain the corresponding real depth labels; S2, constructing a simulation data set: modeling the imaging principle of DPD images, proposing a diffusion model based on rectangular kernels to simulate and obtain DPD images, whose input is based on open source RGBD images (NYU dataset) and output is DPD defocused images; S3 first obtains the initial depth map through DepthNet, and inputs the initial depth map and DPD image pairs into the Deblur network to obtain in-focus images and inverse depth maps. Limitations of method 1: (1) The DPD data simulation method based on rectangular kernels is prone to rectangular spots; (2) The initial depth map is obtained through DepthNet, and the initial depth map is used to optimize the deblurring effect, but in actual tests, the depth estimation effect is not good, resulting in unclear deblurring effect.

[0004] Existing improved method 2: DPD depth and deblurring data set based on binocular structured light. This method projects a Gray code image through a projector and decodes the Gray code image to obtain the absolute depth of the object being measured. Figure 1 This is a schematic diagram showing the comparison between the regular stripes in the input model and the moiré patterns that appear in the projector model in the previous method 2. Figure 2 This is a schematic diagram showing the comparison between the composite image of the additional stripes of the object under test and the shadow part of the parallax image in the improved method 2. Figure 1 As shown, the ideal input model projected by the projector onto the object to be measured (see Figure 1 Left) is a regular and densely distributed Gray code stripe pattern, while the pattern projected by an actual low-end projector is the projector model (see Figure 1 Right) Moiré patterns appear. Figure 2As shown in the figure, after the measured object projects the Gray code stripes, the composite image of the additional stripes of the measured object (see Figure 1 Left) The red circle is the shaded area; the parallax image obtained by method 2 (see Figure 1 Right) The red circle shows that the shadow part cannot obtain depth information. Limitations of Method 2: (1) Gray code is prone to moiré in low-end projectors and mobile device cameras, resulting in matching failure. (2) The shadow area of ​​the projected stripes has no phase information, which makes it impossible to provide correct depth information.

[0005] Existing improved method 3: joint disparity and defocus blur network design based on iteration. This method includes: S1, extracting features of the input DPD image pair through the feature extraction module to obtain F{L,R}; S2, extracting features of the input DPD image pair through the feature extraction module, matching the features to obtain a disparity map, and designing image blur kernels of different scales based on the disparity map; S3, restoring a clear in-focus image through the feature reconstruction module. Limitations of method 3: (1) The deep network is an unsupervised algorithm, and the depth estimation effect is poor; (2) Since the depth result is required to estimate the blur kernel, the deblurring effect is poor and pseudo texture will appear. Summary of the invention

[0006] The purpose of the present invention is to provide a joint phase estimation and defocus blur dataset generation method, which overcomes the limitations of method 2; the present invention uses the full-brightness image after projection and then performs matching, and the depth estimation of the shadow part of the projected image is correct. The present invention improves the structured light projection stripes, combines the Gray code with the phase shift code, and obtains the absolute phase; avoids the occurrence of moiré phenomenon, and obtains accurate edge parallax.

[0007] Another object of the present invention is to provide a joint phase estimation and defocus blur network formation method based on DPD. The present invention sets up an independent disparity estimation module, and adds an image alignment module according to the result of disparity estimation, thereby improving the deblurring performance and making the disparity estimation more accurate.

[0008] Another object of the present invention is to provide a QPD-based joint phase estimation and defocus blur network formation method, which makes full use of disparity information and image information, performs disparity fusion and feature fusion, and obtains clearer deblurring effect and more accurate depth estimation effect.

[0009] The present invention provides a method for generating a joint phase estimation and defocus blur dataset, comprising:

[0010] S1. Provide a binocular acquisition system, which includes a projector and two cameras distributed on the left and right. Calibrate the two cameras to obtain distortion parameters of the two cameras and perform distortion correction. Obtain the external parameters of the two cameras through the calibration images, and perform epipolar correction on the binocular acquisition system.

[0011] S2, placing the object to be measured in the middle of the field of view, using the projector to project a full-brightness image to the object to be measured, using the two cameras to respectively photograph the object to be measured, obtaining a defocused left image and a right image, and matching the left image and the right image to obtain a first parallax image;

[0012] S3, using the projector to project a phase shift code pattern and a Gray code pattern onto the object to be measured, forming a stripe pattern on the object to be measured, using the two cameras to respectively photograph the integrated pattern of the additional stripes of the object to be measured, obtaining a defocused left integrated image and a right integrated image, decoding the left integrated image and the right integrated image to obtain an absolute phase, and using the absolute phase for matching to obtain a second parallax image;

[0013] S4, fusing the first parallax image with the second parallax image to obtain a fused parallax image;

[0014] S5. Adjust the motor position of any one of the two cameras to capture an in-focus image of the same scene, and align the in-focus image with the out-of-focus image.

[0015] Furthermore, calibrating the two cameras specifically includes:

[0016] The projector is used to project the full-brightness image onto the calibration plate, and the two cameras are used to respectively photograph a plurality of calibration images of the calibration plate at different positions and postures within the field of view.

[0017] Furthermore, the fully bright image refers to a projection image that is entirely white, and the pixel value of the entire area of ​​the fully bright image is 255.

[0018] Furthermore, step S2 specifically includes: taking the left image and the right image as the first input of RAFT-stereo, and matching to obtain the first disparity image; wherein the left image and the right image are both Raw images.

[0019] Furthermore, step S3 specifically includes:

[0020] The projector first projects N sinusoidal phase-shift code patterns onto the surface of the object to be measured, and then projects M pairs of Gray code patterns onto the surface of the object to be measured; the two cameras photograph the object to be measured, obtain the defocused left integrated image and the right integrated image, calculate the phase difference at each position to obtain the wrapped phase; use the mean value of each pixel of the N sinusoidal phase-shift code patterns as a threshold, perform threshold segmentation on the M pairs of Gray code images to obtain a threshold image, and obtain the absolute phase information of the object to be measured through the threshold image and the wrapped phase;

[0021] An absolute phase map is calculated from the left integrated image and the right integrated image, and the absolute phase map is used as a second input of RAFT-stereo to obtain the second parallax image.

[0022] The present invention also provides a joint phase estimation and defocus blur removal network formation method based on DPD, comprising:

[0023] S1. Provide a DPD blurred image, and split the DPD blurred image into a left image and a right image;

[0024] S2, extracting features from the left image and the right image through an independent disparity estimation module and matching them, and finally performing disparity estimation to obtain a disparity image;

[0025] S3, extracting features from the left image and the right image through a feature extraction module to obtain left image features and right image features;

[0026] S4, an image alignment module uses the disparity image to align the left image features and the right image features, and superimposes the aligned features to obtain a fusion feature;

[0027] S5. Up-sampling the fused features to obtain a deblurred output image.

[0028] Furthermore, it also includes:

[0029] The disparity estimation module includes: a feature extractor, a correlation pyramid and a GRU-based update operator.

[0030] Furthermore, it also includes:

[0031] During the testing phase, the DPD blurred image is input into the DPD joint phase estimation and defocus blur removal network model, and the disparity image and the deblurred output image are output.

[0032] The present invention also provides a QPD-based joint phase estimation and defocus blurring network formation method, comprising:

[0033] S1. Provide a QPD blurred image, and split the DPD blurred image into a left image, a right image, an upper image, and a lower image;

[0034] Inputting the left image and the right image into a first joint deblurring and disparity estimation module to obtain a first disparity image and a first deblurring feature;

[0035] Inputting the upper image and the lower image into a second joint deblurring and disparity estimation module to obtain a second disparity image and a second deblurring feature;

[0036] S2, inputting the first disparity image and the second disparity image into a disparity fusion module for fusion based on image content to obtain a final fused disparity image;

[0037] S3. Input the first deblurring feature and the second deblurring feature into a feature fusion module for feature fusion and then upsample to obtain a final deblurring output image.

[0038] Furthermore, it also includes:

[0039] During the testing phase, the QPD blurred image is input into the QPD joint phase estimation and defocus blur removal network model, and the disparity image and the deblurred output image are output.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention provides a joint phase estimation and defocus blur dataset generation method, which overcomes the limitations of method 2; the present invention uses the full-brightness image after projection and then matches, which can avoid the defect of shadow, so that the shadow part of the projected image matches correctly, thereby making the depth estimation of the shadow part of the projected image correct. The present invention improves the structured light projection stripes, combines the Gray code with the phase shift code, obtains the absolute phase, avoids the moiré phenomenon, and obtains accurate edge parallax.

[0042] The present invention also provides a joint phase estimation and defocus blurring network formation method based on DPD. The present invention sets an independent disparity estimation module to obtain accurate disparity results; according to the disparity estimation result, i.e., the disparity image, an image alignment module is added to improve the deblurring performance. Compared with method one and method three, the present invention has accurate disparity estimation and good deblurring effect.

[0043] The present invention also provides a method for forming a joint phase estimation and defocus blurring network based on QPD. According to the imaging principle of the QPD image sensor, the disparity information and image information are fully utilized to obtain a first disparity image and a first deblurring feature from the left image and the right image; the second disparity image and the second deblurring feature are obtained from the upper image and the lower image; the first disparity image and the second disparity image are fused based on the image content to obtain a final fused disparity image; the first deblurring feature and the second deblurring feature are feature-fused and then upsampled to obtain a final deblurring output image. The information of the four phases of the QPD image sensor is fully utilized to perform disparity fusion and feature fusion to obtain a clearer deblurring effect and a more accurate depth estimation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a schematic diagram showing the comparison between the regular stripes in the input model and the moiré patterns that appear in the projector model in the previous method 2.

[0045] Figure 2 This is a schematic diagram for comparing the composite image of the additional stripes of the object being measured in the improved method 2 and the shadow part of the parallax image.

[0046] Figure 3a The present invention is a flowchart of a method for generating a joint phase estimation and defocus blur dataset according to an embodiment of the present invention.

[0047] Figure 3b Schematic diagram of a full-brightness image, phase-shift code, and Gray code according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram for comparing the image of the object being measured and the disparity image in the improved method 2.

[0049] Figure 5 It is a schematic diagram of comparing the image of the object under test and the parallax image of the present invention.

[0050] Figure 6 It is a schematic diagram of the joint phase estimation and defocus blur removal network formation method based on DPD of the present invention.

[0051] Figure 7 It is a schematic diagram comparing an input blurred image, a deblurred output image of method 2, and a deblurred output image of the present invention.

[0052] Figure 8 It is a schematic diagram comparing another example of an input blurred image, a deblurred output image of method 2, and a deblurred output image of the present invention.

[0053] Fig. 9 It is a schematic diagram of the joint phase estimation and defocus blur removal network formation method based on QPD of the present invention.

[0054] Fig.10 Schematic diagram for comparing a blurred input image, a deblurred output image of method one, and a deblurred output image of the present invention.

[0055] Fig.11 Schematic diagram of comparison between an input blurred image, a disparity image of method one, and a disparity image of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0057] For ease of description, some embodiments of the present application may use spatially relative terms such as "above", "below", "top", "below", etc. to describe the relationship between one element or component and another (or other) elements or components as shown in the various figures of the embodiments. It should be understood that in addition to the orientations described in the drawings, the spatially relative terms are also intended to include different orientations of the device in use or operation. For example, if the device in the drawings is turned over, the elements or components described as being "below" or "below" other elements or components will subsequently be positioned as being "above" or "above" other elements or components. The terms "first", "second", etc. below are used to distinguish between similar elements and are not necessarily used to describe a specific order or time sequence. It is to be understood that these terms used in this way are interchangeable where appropriate.

[0058] The embodiment of the present invention provides a method for generating a joint phase estimation and defocus blur dataset. Figure 3a As shown, including:

[0059] Step S1, providing a binocular acquisition system, the binocular acquisition system includes a projector and two cameras distributed on the left and right, calibrating the two cameras, obtaining distortion parameters of the two cameras and performing distortion correction; obtaining the external parameters of the two cameras through the calibration images, and performing epipolar correction on the binocular acquisition system;

[0060] Step S2, placing the object to be measured in the middle of the field of view, using a projector to project a full-brightness image onto the object to be measured, using two cameras to respectively photograph the object to be measured, obtaining a defocused left image and a right image, and matching the left image and the right image to obtain a first parallax image;

[0061] Step S3, using a projector to project a phase shift code pattern and a Gray code pattern onto the object to be measured, forming a stripe pattern on the object to be measured, using two cameras to respectively shoot the integrated pattern of the additional stripes of the object to be measured, obtaining a defocused left integrated image and a right integrated image, decoding the left integrated image and the right integrated image to obtain an absolute phase, and using the absolute phase for matching to obtain a second parallax image;

[0062] Step S4, fusing the first parallax image with the second parallax image to obtain a fused parallax image;

[0063] Step S5: adjust the motor position of any one of the two cameras to capture an in-focus image of the same scene, and align the in-focus image with the out-of-focus image.

[0064] The following describes in detail the steps of the method for generating a joint phase estimation and defocus blur dataset according to an embodiment of the present invention.

[0065] In step S1, a binocular acquisition system is provided, and the binocular acquisition system is a binocular structured light system based on a QPD image sensor. The QPD data set contains two parts, a defocused image, an in-focus image corresponding to the defocused image, and a parallax image corresponding to the defocused image. The two cameras are also called binocular cameras. In one example, the two cameras can be calibrated without the projector working and without projecting any image. The two cameras directly capture multiple calibration images of the calibration plate at different positions and postures within the field of view. Figure 3b As shown, in another example, the calibration of the two cameras can be performed by projecting a full-bright image onto the calibration plate through a projector to increase the illumination intensity of the measured object when photographing; specifically, it includes: using a projector to project a full-bright image onto the calibration plate, the full-bright image means that the projected image is all white, and the pixel values ​​of the entire area of ​​the full-bright image are all 255, and the full-bright image is, for example, 8 bits. The two cameras respectively shoot multiple calibration images of the calibration plate at different positions and different postures within the field of view. The distortion parameters of the two cameras are obtained through calibration and distortion correction is performed; the external parameters of the two cameras are obtained through the calibration image, and the epipolar correction is performed on the binocular acquisition system. Through the epipolar correction, the left and right cameras only have a position difference in the horizontal direction (left and right direction), and there is no position difference in the camera height direction (located at the same height).

[0066] Step S2: Place the object to be measured in the middle of the field of view and use a projector to project a full-brightness image onto the object to be measured. Figure 3bAs shown, the full-bright image means that the projected image is all white, and the pixel value of the entire area of ​​the full-bright image is 255, and the full-bright image is, for example, 8 bits. The object to be measured is photographed by two cameras respectively, and the left image and the right image of the binocular camera out of focus are obtained. The left image and the right image are used as the first input of RAFT-stereo, and the first disparity image is obtained by matching; wherein the left image and the right image are both Raw images. The full-bright image can be, for example, a full-white image, and the pixel value of the entire area of ​​the full-bright image is the same. The full-bright image makes the illumination intensity of the object to be measured better when photographed. Step S2 uses auxiliary image information, that is, the projected full-bright image, to improve the illumination intensity of the object to be measured when photographed. When matching the left image and the right image, the defect of the shadow part of the projected image can be avoided, and the problem that the shadow area cannot obtain depth information in the existing method 2 before the improvement is solved. Step S2 The shadow part of the projected image is matched correctly, the depth estimation of the shadow part of the projected image is correct, and the background depth noise level is low. The first disparity image obtained in step S2 has a good overall effect, but the edge of the first disparity image will be inaccurate.

[0067] Step S3: Figure 3b As shown, a projector is used to project a phase shift code pattern and a Gray code pattern onto the object to be measured; the specific projector first projects N sinusoidal phase shift code patterns onto the surface of the object to be measured, and then projects M Gray code patterns onto the surface of the object to be measured. A stripe pattern is formed on the object to be measured, and two cameras are used to respectively shoot the integrated pattern of the additional stripes of the object to be measured, and the left integrated image and the right integrated image of the binocular camera are obtained, and the phase difference of each position is calculated to obtain the wrapped phase; the mean value of each pixel of the N sinusoidal phase shift code patterns is used as the threshold, and the M Gray code images are threshold-segmented to obtain the threshold image, and the absolute phase information of the object to be measured is obtained through the threshold image and the wrapped phase; the absolute phase map is calculated from the left integrated image and the right integrated image, and the absolute phase map is used as the second input of RAFT-stereo to obtain the second parallax image. Step S3 obtains the absolute phase by projecting the phase shift code pattern and the Gray code pattern onto the object to be measured, and the absolute phase is matched to obtain the second parallax image. Disadvantages of step S3: matching is performed only based on phase information (such as absolute phase). Since there is no phase information in the shadow part, the matching of the shadow part will fail, while the matching of the area outside the shadow is correct, which will result in incorrect disparity estimation. Advantages of step S3: the obtained second disparity image can provide accurate disparity edges.

[0068] RAFT-Stereo is a new stereo vision deep structure based on the optical flow network RAFT, which can be applied to stereo matching. Through multi-level convolutional GRUs, it can more effectively propagate information across images and achieve accurate real-time inference. On the dataset of the present invention, RAFT-Stereo shows excellent generalization and accuracy. RAFT-Stereo utilizes RAFT's unique 3D volume and iterative refinement method. RAFT-Stereo only uses 2D convolutions and a lightweight cost volume. Compared with previous stereo networks, RAFT-Stereo has better generalization ability and does not require complex loss terms. The method of RAFT-Stereo consists of three main components: a feature extractor, a correlation pyramid, and a GRU-based update operator. The update operator iteratively retrieves features from the correlation pyramid and updates the disparity field.

[0069] S4. Fusing the first disparity image with the second disparity image to obtain a fused disparity image; and converting the disparity into a depth map according to the calibration result.

[0070] S5. Adjust the motor position of any one of the two cameras to capture an in-focus image of the same scene; align the in-focus image with the out-of-focus image using the out-of-focus image as a reference standard.

[0071] The joint phase estimation and defocusing blur dataset includes: corresponding defocused images and in-focus images, and fused disparity images; wherein, step S2 obtains the defocused images, specifically the defocused left image and right image of the object under test without stripes; step S3 obtains the defocused images, specifically the defocused left composite image and right composite image of the object under test with stripes; the fused disparity image is obtained through steps S1 to S4; the in-focus image taken by any camera is obtained through step S5; the joint phase estimation and defocusing blur dataset is obtained through steps S1 to S5.

[0072] In step S2, auxiliary image information is projected onto the object to be measured, that is, a full-bright image is projected, and the left image and the right image are matched to obtain the first parallax image. Matching after projecting the full-bright image will avoid the defects of the shadow, so that the shadow part of the projected image is matched correctly, and the depth of the shadow part of the projected image is estimated correctly. The first parallax image obtained in step S2 has a good overall effect, but the edge of the first parallax image will be inaccurate. In step S3, the absolute phase is obtained by projecting a phase shift code pattern and a Gray code pattern onto the object to be measured, and the second parallax image is obtained by matching using the absolute phase. Disadvantages of step S3: Matching only relies on phase information (such as absolute phase). Since there will be no phase information in the shadow part, the matching of the shadow part will fail, while the matching of the area outside the shadow is correct, which will result in an error in the parallax estimation. Advantages of step S3: The second parallax image obtained can provide accurate parallax edges. The phase information of step S3 is obtained by projecting a stripe image, and there is no phase information in the shadow part. Even if it is the shadow part, the full-bright image of step S2 has information and can be used to match and obtain parallax. Step S4 combines step S2 and step S3 to complement each other, and can obtain both accurate edge disparity and accurate disparity results (especially the correct depth estimation of the shadow part of the projected image). Specifically, the advantages of correct matching of the shadow part of the projected image and correct depth estimation of the shadow part of the projected image in step S2 are taken; the advantages of accurate disparity edge provided by step S3 are taken.

[0073] Figure 4 This is a schematic diagram for comparing the image of the object being measured and the disparity image in the improved method 2. Figure 5 FIG. 1 is a schematic diagram showing the comparison between the image of the object under test and the parallax image of the present invention. Figure 4 As shown in Figure 2, the depth of the shadow projected by the object under test cannot be estimated in Method 2, and the moiré pattern in the parallax image is serious, which causes serious noise in the depth calculation. Figure 5 As shown, the depth estimation of the projected shadow part (such as the shadow part of the pear image) in the parallax image of the present invention is correct, and the background depth noise level is low. The disadvantage of the second method before the improvement that it cannot be applied to low-end projectors, that is, Gray code is prone to moiré phenomenon in low-end projectors and mobile phone device cameras, resulting in matching failure. When the stripes are dense, moiré phenomenon is prone to occur; the present invention combines the phase shift code with the Gray code to reduce the stripe density and make the stripes sparse, avoiding the interference of moiré, and uses RAFT-stereo to match the image information itself and the phase information to obtain the parallax, thereby alleviating the defect that the stripe shadow part cannot obtain the phase. The depth estimation of the projected shadow part in the parallax image of the present invention is correct, and the background depth noise level is low.

[0074] The above introduces a phase estimation and defocus blur dataset based on DPD / QPD proposed by the present invention. Based on this dataset, the present invention proposes a joint phase and defocus blur network, which performs significantly better than other algorithms. The present invention makes improvements to the problems of inaccurate disparity estimation and poor deblur recovery effect of the previous methods one and three.

[0075] Figure 6 FIG. 1 is a schematic diagram of a joint phase estimation and defocus blur removal network formation method based on DPD of the present invention. Figure 6 As shown, the present invention makes two improvements: (1) the present invention sets an independent disparity estimation module to obtain accurate disparity results; (2) according to the disparity estimation results, an image alignment module is added to improve the deblurring performance. Since the split left image L and right image R will have disparity when the DPD image is blurred, simple feature fusion will lead to poor image restoration performance. The present invention uses disparity to align and fuse features.

[0076] like Figure 6 As shown, the joint phase estimation and defocus blur removal network formation method based on DPD of the present invention includes:

[0077] S1. Provide a DPD blurred image, and split the DPD blurred image to obtain a left image L and a right image R. Specifically, the DPD blurred image can sample G channel information, and split the G channel to obtain a left image L and a right image R.

[0078] S2, extract features from the left image L and the right image R through an independent disparity estimation module and match them, and finally perform disparity estimation to obtain the disparity image d; the disparity estimation module includes: a feature extractor, a correlation pyramid (C1 to C4) and a GRU-based update operator. The feature extractor extracts features based on the feature encoder and the context encoder. The feature encoder is applied to the left and right images and generates a dense feature map for constructing the correlation volume. The context encoder has the same structure as the feature encoder, but batch normalization is used instead of instance normalization. The context features are used to initialize the hidden state of the update operator and are injected into the GRU in each iteration. The correlation pyramid is used to improve the accuracy and robustness of the corrected stereo vision. The correlation volume is based on the dot product between the feature vectors and is used to calculate the similarity between images. The update operator of the GRU iteratively retrieves features from the correlation pyramid and updates the disparity field.

[0079] S3, extract features from the left image L and the right image R through the feature extraction module to obtain the left image feature F L and right image feature F R ;

[0080] S4, the image alignment module uses the disparity image d to align the left image feature F Land right image feature F R Perform alignment and superimpose the aligned features to obtain fused features;

[0081] S5. Upsample the fused features to obtain a deblurred output image out.

[0082] The DPD-based joint phase estimation and defocus blur network formation method also includes: in the testing phase, the DPD blurred image is input into the DPD joint phase estimation and defocus blur network model, and the disparity image and the deblurred output image are output.

[0083] In order to verify the effectiveness of the joint phase estimation and defocus blur network based on DPD, the deblurring effect of the present invention is compared with that of method 2 (the deblurring effect of method 2 is better than that of method 1). For a fair comparison, the comparison is performed on the public dataset Canon dataset. Since the Canon dataset is a deblurring dataset, only the deblurring effects are compared. Figure 7 It is a schematic diagram comparing an input blurred image, a deblurred output image of method 2, and a deblurred output image of the present invention. Figure 8 FIG. 1 is a schematic diagram comparing an input blurred image, a deblurred output image of method 2, and a deblurred output image of the present invention. Figure 7 and Figure 8 As shown, the deblurring effect of the joint phase estimation and defocus blur network based on DPD of the present invention is obviously better than that of method two. Since the deblurring effect of method two is better than that of method one, the deblurring effect of the joint phase estimation and defocus blur network based on DPD of the present invention is also better than that of method one.

[0084] Based on the DPD joint model, the present invention designs a QPD joint phase estimation and deblurring network to remove the reconstruction module in image restoration. According to the imaging principle of the QPD image sensor, the disparity information and image information are fully utilized.

[0085] The present invention is based on the joint phase estimation and defocus blurring network formation method of QPD, such as Fig. 9 As shown, including:

[0086] S1, providing a QPD blurred image, splitting the DPD blurred image to obtain a left image L, a right image R, an upper image S, and a lower image X;

[0087] Input the left image L and the right image R into the first joint deblurring and disparity estimation module M1 to obtain a first disparity image D1 and a first deblurring feature F1;

[0088] Input the upper image S and the lower image X into the second joint deblurring and disparity estimation module M2 to obtain a second disparity image D2 and a second deblurring feature F2;

[0089] S2, inputting the first disparity image D1 and the second disparity image D2 into the disparity fusion module Df for fusion based on the image content to obtain a final fused disparity image D;

[0090] S3, inputting the first deblurred feature F1 and the second deblurred feature F2 into the feature fusion module Ff for feature fusion and then upsampling to obtain the final deblurred output image out.

[0091] The method for forming a joint phase estimation and defocus blur network based on QPD also includes: in the testing phase, in order to verify the effectiveness of the QPD joint phase estimation and defocus blur network, the input is kept consistent and only the input model is changed. Specific process: input the QPD blurred image into the QPD joint phase estimation and defocus blur network model, and output the disparity image and the deblurred output image.

[0092] Fig.10 Schematic diagram for comparing a blurred input image, a deblurred output image of method one, and a deblurred output image of the present invention. Fig.11 Schematic diagram of the comparison between the input blurred image, the parallax image of method 1, and the parallax image of the present invention. Fig.10 and Fig.11 As shown in FIG. 1 , the deblurring effect and disparity image of the joint phase estimation and defocus blur removal network based on QPD of the present invention are significantly better than those of method 1. Fig.10 The deblurring effect of the present invention is better than that of method 1, and the image after deblurring of the present invention is clearer (especially the red circle on the right side of the figure). Fig.11 The middle image is the disparity image of method 1, which has severe noise and poor depth estimation effect; Fig.11 The right picture is a parallax image of the present invention, and the depth estimation is accurate. In particular, the thumb in the red circle in the picture has an obvious color difference with the background behind it, indicating that the depth estimation is accurate.

[0093] In summary, the present invention provides a joint phase estimation and defocus blur dataset generation method, which overcomes the limitations of method 2; the present invention uses the full-brightness image after projection and then matches, which can avoid the defect of shadow, so that the shadow part of the projected image matches correctly, thereby making the depth estimation of the shadow part of the projected image correct. The present invention improves the structured light projection stripes, combines the Gray code with the phase shift code, and obtains the absolute phase; avoids the occurrence of moiré phenomenon, and obtains accurate edge parallax.

[0094] The present invention also provides a joint phase estimation and defocus blurring network formation method based on DPD. The present invention sets an independent disparity estimation module to obtain accurate disparity results; according to the disparity estimation result, i.e., the disparity image, an image alignment module is added to improve the deblurring performance. Compared with method one and method three, the present invention has accurate disparity estimation and good deblurring effect.

[0095] The present invention also provides a method for forming a joint phase estimation and defocus blurring network based on QPD. According to the imaging principle of the QPD image sensor, the disparity information and image information are fully utilized to obtain a first disparity image and a first deblurring feature from the left image and the right image; the second disparity image and the second deblurring feature are obtained from the upper image and the lower image; the first disparity image and the second disparity image are fused based on the image content to obtain a final fused disparity image; the first deblurring feature and the second deblurring feature are feature-fused and then upsampled to obtain a final deblurring output image. The information of the four phases of the QPD image sensor is fully utilized to perform disparity fusion and feature fusion to obtain a clearer deblurring effect and a more accurate depth estimation effect.

[0096] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0097] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A joint phase estimation and defocus blur dataset generation method, It is characterized in that include: S1. Provide a binocular acquisition system, which includes a projector and two cameras distributed on the left and right. Calibrate the two cameras to obtain distortion parameters of the two cameras and perform distortion correction. Obtain the external parameters of the two cameras through the calibration images, and perform epipolar correction on the binocular acquisition system. S2, placing the object to be measured in the middle of the field of view, using the projector to project a full-brightness image to the object to be measured, using the two cameras to respectively photograph the object to be measured, obtaining a defocused left image and a right image, and matching the left image and the right image to obtain a first parallax image; S3, using the projector to project a phase shift code pattern and a Gray code pattern onto the object to be measured, forming a stripe pattern on the object to be measured, using the two cameras to respectively photograph the integrated pattern of the additional stripes of the object to be measured, obtaining a defocused left integrated image and a right integrated image, decoding the left integrated image and the right integrated image to obtain an absolute phase, and using the absolute phase for matching to obtain a second parallax image; S4, fusing the first parallax image with the second parallax image to obtain a fused parallax image; S5. Adjust the motor position of any one of the two cameras to capture an in-focus image of the same scene, and align the in-focus image with the out-of-focus image.

2. The method for generating a joint phase estimation and defocus blur dataset according to claim 1, It is characterized in that In step S1, calibrating the two cameras specifically includes: The projector is used to project the full-brightness image onto the calibration plate, and the two cameras are used to respectively photograph a plurality of calibration images of the calibration plate at different positions and postures within the field of view.

3. The method for generating a joint phase estimation and defocus blur dataset according to claim 1, It is characterized in that The fully bright image means that the projection image is all white, and the pixel value of the entire area of ​​the fully bright image is 255.

4. The method for generating a joint phase estimation and defocus blur dataset according to claim 1, It is characterized in that Step S2 specifically includes: taking the left image and the right image as the first input of RAFT-stereo, and matching to obtain the first disparity image; wherein the left image and the right image are both Raw images.

5. The method for generating a joint phase estimation and defocus blur dataset according to claim 4, It is characterized in that Step S3 specifically includes: The projector first projects N sinusoidal phase-shift code patterns onto the surface of the object to be measured, and then projects M pairs of Gray code patterns onto the surface of the object to be measured; the two cameras photograph the object to be measured, obtain the defocused left integrated image and the right integrated image, calculate the phase difference at each position to obtain the wrapped phase; use the mean value of each pixel of the N sinusoidal phase-shift code patterns as a threshold, perform threshold segmentation on the M pairs of Gray code images to obtain a threshold image, and obtain the absolute phase information of the object to be measured through the threshold image and the wrapped phase; An absolute phase map is calculated from the left integrated image and the right integrated image, and the absolute phase map is used as a second input of the RAFT-stereo to obtain the second parallax image.

6. A joint phase estimation and defocus blurring network formation method based on DPD, It is characterized in that include: S1. Provide a DPD blurred image, and split the DPD blurred image into a left image and a right image; S2, extracting features from the left image and the right image through an independent disparity estimation module and matching them, and finally performing disparity estimation to obtain a disparity image; S3, extracting features from the left image and the right image through a feature extraction module to obtain left image features and right image features; S4, an image alignment module uses the disparity image to align the left image features and the right image features, and superimposes the aligned features to obtain a fusion feature; S5. Up-sampling the fused features to obtain a deblurred output image.

7. The method for forming a joint phase estimation and defocus blur removal network based on DPD as claimed in claim 6, It is characterized in that Also includes: The disparity estimation module includes: a feature extractor, a correlation pyramid and a GRU-based update operator.

8. The method for forming a joint phase estimation and defocus blur removal network based on DPD as claimed in claim 6, It is characterized in that Also includes: During the testing phase, the DPD blurred image is input into the DPD joint phase estimation and defocus blur removal network model, and the disparity image and the deblurred output image are output.

9. A QPD-based joint phase estimation and defocus blurring network formation method, It is characterized in that include: S1. Provide a QPD blurred image, and split the DPD blurred image into a left image, a right image, an upper image, and a lower image; Inputting the left image and the right image into a first joint deblurring and disparity estimation module to obtain a first disparity image and a first deblurring feature; Inputting the upper image and the lower image into a second joint deblurring and disparity estimation module to obtain a second disparity image and a second deblurring feature; S2, inputting the first disparity image and the second disparity image into a disparity fusion module for fusion based on image content to obtain a final fused disparity image; S3. Input the first deblurring feature and the second deblurring feature into a feature fusion module for feature fusion and then upsample to obtain a final deblurring output image.

10. The QPD-based joint phase estimation and defocus blurring network formation method according to claim 9, It is characterized in that Also includes: During the testing phase, the QPD blurred image is input into the QPD joint phase estimation and defocus blur removal network model, and the disparity image and the deblurred output image are output.

Images