Data simulation method

Through the simulation method based on diffusion calculation of RGB-depth data, the problems of cross-field boundary artifacts and rectangular spots in existing PD data simulation are solved, and a natural transition simulated defocused image is generated, which improves the data quality and defuzzing effect. .

CN118941467APending Publication Date: 2024-11-12HOWAY TECH (WUHAN) CO LTD

Patent Information

Application Number
CN202410858556.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When simulating defocus blur, existing PD data simulation methods have problems with artifacts across depth of field boundaries and rectangular spots, making it difficult to effectively simulate real optical principles.

Method used

The simulation method based on diffusion calculation of RGB-depth data is adopted. By acquiring the focused RGB image and depth map, the fuzzy nucleus diffusion radius of each pixel is calculated, and the diffusion calculation is performed using the point diffusion function model to generate a simulated defocused image of natural transition.

Benefits of technology

Severe artifacts and rectangular spot problems across the depth of field boundaries are avoided, a more natural depth of field transition is achieved, and the quality of simulation data and defuzzing effect is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941467A_ABST
    Figure CN118941467A_ABST
Patent Text Reader

Abstract

The invention provides a data simulation method. The data simulation method comprises the following steps: acquiring a corresponding focusing RGB image and a depth map; according to an imaging principle and preset camera parameters, the diffusion radius of a blurring kernel corresponding to each pixel in the depth map is calculated; estimating or directly calling a point spread function model, wherein the point spread function model comprises a two-phase detection point spread function model and a four-phase detection point spread function model; layering the focused RGB image according to the depth map; according to the diffusion radius corresponding to each layer and the point spread function model, performing diffusion calculation on the pixels corresponding to each layer; and mixing the layers after diffusion calculation according to a far-near sequence to obtain a simulated defocused RGB image. According to the method, a simulation method of RGB-depth data based on diffusion calculation is adopted, diffusion calculation is performed on a point spread function, cross-depth-of-field boundary transition is more natural, and cross-depth-of-field boundary serious artifacts are avoided; and no rectangular light spot problem exists.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a data simulation method. Background Art

[0002] PDAF (Phase Detection Auto Focus) technology is widely used in smartphone camera autofocus. PD cameras are also used in deblurring, depth estimation, dereflection, depth of field simulation and other fields. Applications such as deblurring and depth estimation based on PD data are mainly based on deep learning and require a large amount of training data. The lack of PD data has also become a difficulty in this direction. PD data simulation can quickly obtain a large number of data sets. The current simulation methods are all for Dual-PD (two-phase detection) data. There are mainly two simulation methods:

[0003] The first method: a simulation method based on collection calculation using RGB-depth data. The first advantage: the point spread function is closer to the real PD camera device and has diversified morphology. The first disadvantage: (1) It does not conform to the optical principle of defocus blur. The optical principle of defocus blur can be understood as a scattering operation, that is, each pixel of the input disperses its influence to its influence range. (2) The convolution implementation will produce more serious visual artifacts. The convolution algorithm is based on the collection operation, that is, each pixel of the output will collect the influence of nearby pixels in the input, and produce more serious artifacts across the depth of field boundary.

[0004] The second method: a simulation method based on rectangular uniform scattering calculation using RGB-depth data. The second advantage: it is consistent with the optical scattering principle of defocus blur. The second disadvantage: for most CMOS sensors or cameras, the point spread function is not uniformly distributed, and is not half of the left and right sides of the rectangle. There are a large number of rectangular spots in the blurred image simulated by the second method. Summary of the invention

[0005] The purpose of the present invention is to provide a data simulation method, which adopts a simulation method based on diffusion calculation of RGB-depth data. The present invention is a diffusion calculation of a point spread function. Compared with the first method before improvement, the transition across the depth of field boundary is more natural, avoiding the generation of serious artifacts across the depth of field boundary; compared with the second method before improvement, there is no rectangular spot problem.

[0006] The present invention provides a data simulation method, comprising:

[0007] S1. Obtain the corresponding focused RGB image and depth map; calculate the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and preset camera parameters;

[0008] S2, estimating or directly calling a point spread function model, wherein the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model;

[0009] S3, layering the focused RGB image according to the depth map;

[0010] S4, performing diffusion calculation on pixels corresponding to each layer according to the diffusion radius corresponding to each layer and the point spread function model;

[0011] S5. Mix the layers after diffusion calculation in order of distance to obtain a simulated defocused RGB image corresponding to the focused RGB image.

[0012] Furthermore, step S2, the method for estimating the four-phase detection point spread function model, specifically includes:

[0013] S211, collecting the four-phase detection focusing raw image: using the QPD camera, aligning it with the center of the plane image, adjusting the position of the camera focus motor to focus, and collecting the four-phase detection focusing raw image;

[0014] S212, collecting a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and collecting the four-phase detection defocus raw image;

[0015] S213, data alignment: using the enhanced correlation coefficient image alignment method, taking the four-phase detection defocus raw image as a reference, aligning the four-phase detection focus raw image with the four-phase detection defocus raw image to obtain an aligned focused image; and calculating the aligned G channel focused image corresponding to the aligned focused image.

[0016] Further, in step S211, in the four-phase detection focus raw image, four adjacent pixels of the same color are grouped; in each color group, the average value of the four pixels is taken as the pixel value of the group after four-in-one; each group is assigned a respective average value to form a synthesized focused image; in the synthesized focused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a first unit, and the green average value of Gb and Gr in each of the first units is taken as the green pixel value of the first unit after four-in-one; each first unit is assigned a respective green average value to form a green channel focused image;

[0017] In step S212, in the four-phase detection defocused raw image, four adjacent pixels of the same color are grouped; in each color group, the average value of the four pixels is taken as the pixel value of the group after four-in-one; each group is assigned its own average value to form a synthesized defocused image; in the synthesized defocused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a second unit, and the green average value of Gb and Gr in each of the second units is taken as the green pixel value of the second unit after four-in-one; each second unit is assigned its own green average value to form a green channel defocused image;

[0018] In step S213, the enhanced correlation coefficient image alignment method is used to align the green channel focused image with the green channel defocused image with the green channel defocused image as a reference, and an affine transformation matrix is ​​calculated; the B, Gb, Gr and R channels of the synthesized focused image are respectively transformed and synthesized by the affine transformation matrix to obtain an aligned focused image; in the aligned focused image, each four adjacent groups of pixels arranged in a Bayer format are taken as a third unit, and the green average value of Gb and Gr in each of the third units is taken as the green pixel value after the third unit is four-in-one; each of the third units is assigned its own green average value to form the aligned G channel focused image.

[0019] Furthermore, step S2, the method for estimating the four-phase detection point spread function model, specifically includes:

[0020] S214a, point spread function estimation: according to the four-phase detection defocus raw image, respectively calculate and obtain the G channel defocus images of the four direction images of the four-phase defocus, namely, the upper image, the lower image, the left image and the right image; respectively calculate the blur kernel of the G channel defocus image of each of the four direction images of the four-phase defocus relative to the aligned G channel focused image, and obtain four point spread function models of upper, lower, left and right;

[0021] The upper image is obtained by taking the average value of the upper left and upper right pixels in each color group of the four-phase detection defocus raw image, as the pixel value of the group after four-in-one; the lower image is obtained by taking the average value of the lower left and lower right pixels in each color group, as the pixel value of the group after four-in-one; the left image is obtained by taking the average value of the upper left and lower left pixels in each color group, as the pixel value of the group after four-in-one; the right image is obtained by taking the average value of the upper right and lower right pixels in each color group, as the pixel value of the group after four-in-one.

[0022] Furthermore, in the upper image, lower image, left image and right image of the four-phase defocus, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels of B, Gb, Gr and R, and the green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

[0023] Furthermore, step S2, the method for estimating the four-phase detection point spread function model, specifically includes:

[0024] S214b, point spread function estimation: according to the four-phase detection defocus raw image, respectively calculate and obtain the G channel defocus images of the four direction images of the upper left image, the lower left image, the upper right image and the lower right image of the four-phase defocus; respectively calculate the blur kernel of the G channel defocus image of each of the four direction images relative to the aligned G channel focused image, and obtain four point spread function models of the upper left, lower left, upper right and lower right;

[0025] The upper left image is obtained by taking the pixel value of the upper left pixel in each color group of the four-phase detection defocus raw image as the pixel value of the group after four-in-one; the lower left image is obtained by taking the pixel value of the lower left pixel in each color group as the pixel value of the group after four-in-one; the upper right image is obtained by taking the pixel value of the upper right pixel in each color group as the pixel value of the group after four-in-one; the lower right image is obtained by taking the pixel value of the lower right pixel in each color group as the pixel value of the group after four-in-one.

[0026] Furthermore, in the four-phase defocused upper left image, lower left image, upper right image and lower right image, each adjacent four groups of pixels arranged in Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels B, Gb, Gr and R. The green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

[0027] Furthermore, step S2, the method for estimating the two-phase detection point spread function model, specifically includes:

[0028] S221, collecting two-phase detection focusing raw images: using a Dual-PD camera, aligning it with the center of the plane image, adjusting the position of the camera focus motor to focus, and collecting the two-phase detection focusing raw images;

[0029] S222, collecting two-phase detection defocus raw images: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting the two-phase detection defocus raw images;

[0030] S223, data alignment: using the enhanced correlation coefficient image alignment method, taking the two phase detection defocused raw images as a reference, aligning the two phase detection focused raw images with the two phase detection defocused raw images to obtain an aligned focused image; and calculating the two phase aligned G channel focused images corresponding to the aligned focused image.

[0031] Further, in step S221, in the two-phase detection focus raw image, two adjacent pixels of the same color are grouped; in each color group, the average value of the two pixels is taken as the pixel value of the group after two-in-one; each group is assigned its own average value to form a two-phase synthesized focus image; in the two-phase synthesized focus image, each four adjacent groups of pixels arranged in the Bayer format are taken as a fourth unit, and the green average value of Gb and Gr in each of the fourth units is taken as the green pixel value of the fourth unit after four-in-one; each of the fourth units is assigned its own green average value to form a two-phase green channel focus image;

[0032] In step S222, in the two-phase detection defocused raw image, two adjacent pixels of the same color are grouped; in each color group, the average value of the two pixels is taken as the pixel value of the group after two-in-one; each group is assigned its own average value to form a two-phase synthesized defocused image; in the two-phase synthesized defocused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a fifth unit, and the green average value of Gb and Gr in each of the fifth units is taken as the green pixel value of the fifth unit after four-in-one; each of the fifth units is assigned its own green average value to form a two-phase green channel defocused image;

[0033] In step S223, the enhanced correlation coefficient image alignment method is used to align the two-phase green channel focused image with the two-phase green channel defocused image with the two-phase green channel defocused image, and an affine transformation matrix is ​​calculated; the B, Gb, Gr and R channels of the two-phase synthesized focused image are respectively transformed and synthesized by the affine transformation matrix to obtain the two-phase aligned focused image; in the two-phase aligned focused images, each four adjacent groups of pixels arranged in Bayer format are taken as a sixth unit, and the green average value of Gb and Gr in each of the sixth units is taken as the green pixel value after the four-in-one sixth unit; each of the sixth units is assigned its own green average value to form the two-phase aligned G channel focused image.

[0034] Furthermore, step S2, the method for estimating the two-phase detection point spread function model, specifically includes:

[0035] S224, point spread function estimation: according to the two-phase detection defocus raw images, respectively calculate and obtain the two-phase G channel defocus images of the two-phase defocused left image and the right image in two directions; respectively calculate the blur kernel of the two-phase G channel defocused images of the two directions relative to the G channel focused image after the two phases are aligned, and obtain the left and right point spread function models;

[0036] The left image is obtained by taking the left pixel in each color group of the two phase detection defocus raw images as the pixel value of the group after the two are combined into one; the right image is obtained by taking the right pixel in each color group as the pixel value of the group after the two are combined into one.

[0037] Furthermore, in the two-phase defocused left image and right image, each adjacent four groups of pixels arranged in Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels B, Gb, Gr and R. The green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

[0038] Furthermore, step S2 further includes:

[0039] S25. Adjust the camera motor to obtain the four-phase detection defocus raw image with different blur levels, and repeat steps S23 and S24 to obtain the point spread function with different blur levels.

[0040] Further, when the focused RGB image is in a two-phase detection format, the two-phase detection point spread function model includes: a left point spread function model and a right point spread function model;

[0041] According to the diffusion radius corresponding to each layer of the focused RGB image and the left point diffusion function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused left image;

[0042] By analogy, the focused RGB image is calculated through the right point spread function model to obtain a simulated defocused right image;

[0043] The simulated defocused left image and the simulated defocused right image constitute the simulated defocused RGB image.

[0044] Further, when the focused RGB image is in a four-phase detection format, the four-phase detection point spread function model includes: an upper point spread function model, a lower point spread function model, a left point spread function model and a right point spread function model;

[0045] According to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused upper image;

[0046] By analogy, the focused RGB image is calculated by the lower point spread function model to obtain a simulated defocused lower image; the focused RGB image is calculated by the left point spread function model to obtain a simulated defocused left image; the focused RGB image is calculated by the right point spread function model to obtain a simulated defocused right image;

[0047] The simulated defocused left image, the simulated defocused right image, the simulated defocused upper image and the simulated defocused lower image constitute the simulated defocused RGB image.

[0048] Further, when the focused RGB image is in a four-phase detection format, the four-phase detection point spread function model includes: an upper left point spread function model, a lower left point spread function model, an upper right point spread function model and a lower right point spread function model;

[0049] According to the diffusion radius corresponding to each layer of the focused RGB image and the upper left point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of distance to obtain a simulated defocused upper left image;

[0050] By analogy, the focused RGB image is calculated by the lower left point spread function model to obtain a simulated defocused lower left image; the focused RGB image is calculated by the upper right point spread function model to obtain a simulated defocused upper right image; the focused RGB image is calculated by the lower right point spread function model to obtain a simulated defocused lower right image;

[0051] The simulated defocused upper left image, the simulated defocused lower left image, the simulated defocused upper right image, and the simulated defocused lower right image constitute the simulated defocused RGB image.

[0052] Furthermore, the corresponding focused RGB image and the depth map are obtained by retrieving a known database or shooting with a camera focused.

[0053] Furthermore, the data simulation method further includes: performing compensation calculation on the simulated defocused RGB image, and the compensation method specifically includes:

[0054] providing a gray image of the same color intensity as the focused RGB image;

[0055] Calculate the diffusion radius and the point spread function model according to the camera parameters and the depth map, perform diffusion calculation on the focused RGB image and the gray image respectively, and obtain the simulated defocused RGB image and the gray blur image;

[0056] Compensate the gray fuzzy image to the gray image to obtain a compensation weight; the compensation weight is a color intensity ratio of the gray image to the gray fuzzy image;

[0057] The simulated defocus RGB image is multiplied by the compensation weight to obtain a compensated simulated defocus RGB image.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention provides a data simulation method, including: S1, obtaining a corresponding focused RGB image and a depth map; calculating the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and the preset camera parameters; S2, estimating or directly calling a point spread function model, the point spread function model including a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; S4, performing diffusion calculation on the pixels corresponding to each layer according to the diffusion radius and the point spread function model corresponding to each layer; S5, mixing the layers after the diffusion calculation in the order of near and far to obtain a simulated defocused RGB image corresponding to the focused RGB image. The present invention adopts a simulation method based on diffusion calculation of RGB-depth data. The present invention is a diffusion calculation of a point spread function, and compared with the first method before the improvement, the transition across the depth of field boundary is more natural, avoiding the generation of serious artifacts across the depth of field boundary; compared with the second method before the improvement, there is no rectangular spot problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flowchart of a data simulation method according to an embodiment of the present invention.

[0061] Figure 2 Schematic diagram of a method for forming a G channel focused image after alignment in a data simulation method according to an embodiment of the present invention.

[0062] Figure 3 It is a schematic diagram of a method for forming G channel defocused images of four directions of images, namely, upper image, lower image, left image and right image, corresponding to the QPD defocused raw image in the data simulation method of an embodiment of the present invention.

[0063] Figure 4 Schematic diagram of four point spread function models of upper, lower, left and right in the data simulation method of an embodiment of the present invention.

[0064] Figure 5Schematic diagram of compensation calculation in the data simulation method according to an embodiment of the present invention.

[0065] Figure 6 This is a comparison diagram of a simulated defocused (blurred) image generated by the data simulation method of the present invention and a blurred image simulated and generated by the first method before improvement.

[0066] Figure 7 This is a comparison diagram of a simulated defocused (blurred) image generated by the data simulation method of the present invention and a blurred image simulated and generated by the second method before improvement.

[0067] Figure 8 A comparison chart of deblurring effects is shown for using the simulation data of the present invention and the first method before improvement for image deblurring training. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in a very simplified form and use an inaccurate scale, which is only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0069] For ease of description, some embodiments of the present application may use spatially relative terms such as "above", "below", "top", "below", etc. to describe the relationship between one element or component and another (or other) elements or components as shown in the various figures of the embodiments. It should be understood that in addition to the orientations described in the drawings, the spatially relative terms are also intended to include different orientations of the device in use or operation. For example, if the device in the drawings is turned over, the elements or components described as being "below" or "below" other elements or components will subsequently be positioned as being "above" or "above" other elements or components. The terms "first", "second", etc. below are used to distinguish between similar elements and are not necessarily used to describe a specific order or time sequence. It is to be understood that these terms used in this way are interchangeable where appropriate.

[0070] The embodiment of the present invention provides a data simulation method, such as Figure 1 As shown, including:

[0071] S1. Obtain the corresponding focused RGB image and depth (object distance) map; calculate the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and the preset camera parameters;

[0072] S2, estimating or directly calling a point spread function model, where the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model;

[0073] S3, layering the focused RGB image according to the depth map;

[0074] S4, performing diffusion calculation on the pixels corresponding to each layer according to the diffusion radius and point spread function model corresponding to each layer;

[0075] S5. Mix the layers after diffusion calculation in order of distance to obtain a simulated defocused (blurred) image corresponding to the focused RGB image.

[0076] The steps of the data simulation method according to the embodiment of the present invention are described in detail below with reference to the accompanying drawings.

[0077] Step S1, obtain the corresponding focused RGB image and depth (object distance) map; the depth in this article is the object distance. The depth map records the geometric information of the scene, which represents the distance of the object in the scene from the optical center of the camera. According to the imaging principle and the preset camera parameters, the diffusion radius r of the blur kernel corresponding to each pixel in the depth map is calculated. The calculation formula of the diffusion radius r is as follows:

[0078]

[0079] Among them, d is the diffusion diameter, α is a constant, L is the aperture size, f is the focal length, z is the object distance when focusing, and D is the object distance.

[0080] The corresponding focused RGB image and depth map are obtained by retrieving the known database or shooting after the camera is focused. The preset camera parameters include: focal length, aperture, phase plane distance and other parameters. The camera parameters and object distance can determine the blur degree of the image. The phase parallax of the left and right images of Dual-PD (two-phase detection) is proportional to the blur degree of the image. The phase parallax of the left and right images of QPD (four-phase detection) is proportional to the blur degree of the image, and the phase parallax of the upper and lower images of QPD is also proportional to the blur degree of the image.

[0081] Step S2, estimate or directly call the point spread function (PSF) model, the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model. The point spread function model can be estimated using a Butterworth filter function or a Gaussian filter function, or an existing point spread function model can be directly called, or the shape of a blur function can be defined. A clean and clear image is rendered from a blurred input image, and the point spread function model can also be obtained by rendering. The point spread function model of the present invention can be a non-uniform diffusion model or a uniform diffusion model. In one example, the point spread function model can be, for example, a uniform rectangular diffusion model.

[0082] In the process of imaging an object, when the image distance is equal to the focal length, the image formed by an object point on the imaging plane through the lens should also be a point, but due to factors such as diffraction, scattering and aberration, this point will spread into a light spot. The point spread function is a mathematical description of the shape and intensity of this light spot. For example, if the distance between the imaging plane and the lens changes, the light will be projected onto the imaging plane before it converges to a point. At this time, the image formed is no longer a point, but a light spot. In other words, when an object is imaged, if it is not an object point-image point mapping, but an object point-image spot mapping, a defocused image will appear, and the image will become blurred and difficult to view. The point spread function describes the response of an imaging optical system to a point light source (object).

[0083] In the Dual-PD image obtained by using the Dual-PD camera, two adjacent pixels share a micro lens, and the two adjacent pixels can use the same color filter (such as red R, green G, blue B filter). In the Dual-PD image, two adjacent pixels of the same color are grouped together, and there is parallax between the two pixels in the same group.

[0084] In a QPD image obtained by using a QPD camera, four adjacent pixels share a micro lens, and four adjacent pixels can use the same color filter (e.g., red R, green G, blue B filter). In the QPD image, four adjacent pixels of the same color are grouped together, and there is parallax between each of the four pixels in the same group.

[0085] In a first example, a method for estimating a four-phase detection point spread function model specifically includes:

[0086] S211, such as Figure 2 As shown, a four-phase detection (QPD) focus (infocus) raw image is collected: a QPD camera is used to align with the center of the plane image, the position of the camera focus motor is adjusted to focus, and a QPD focus raw image is collected. In the four-phase detection focus raw image, four adjacent pixels of the same color are grouped together; in each color group, the average value of each of the four pixels is taken as the pixel value of the group after four-in-one; each group is assigned its own average value to form a synthesized focused image C-infocus. In the synthesized focused image C-infocus, each of the four adjacent groups of pixels arranged in the Bayer format is taken as a first unit, and the first unit includes four groups of pixels B, Gb, Gr and R. The green average value of Gb and Gr in each first unit is taken as the green pixel value after four-in-one of the first unit; each first unit is assigned its own green average value to form a green channel focused image G-infocus.

[0087] S212, collect QPD defocus raw image: adjust the focus motor position of the QPD camera to blur the image, and collect the QPD defocus raw image; in the four-phase detection defocus raw image, four adjacent pixels of the same color are grouped; in each color group, take the average value of each of the four pixels as the pixel value of the group after four-in-one; each group is assigned its own average value to form a synthesized defocus image C-defocus. In the synthesized defocus image C-defocus, each adjacent four groups of pixels arranged in the Bayer format are used as a second unit, and the second unit includes four groups of pixels B, Gb, Gr and R. The green average value of Gb and Gr in each second unit is taken as the green pixel value after four-in-one of the second unit; each second unit is assigned its own green average value to form a green channel defocus image G-defocus.

[0088] S213, data alignment: using the enhanced correlation coefficient (ECC) image alignment method, taking the QPD defocus raw image as a reference, aligning the QPD focus raw image with the QPD defocus raw image to obtain the aligned focused image Aligned C-infocus; and calculating the aligned G channel focused image Aligned G-infocus corresponding to the aligned focused image Aligned C-infocus. Specifically, using the enhanced correlation coefficient (ECC) image alignment method, taking the green channel defocus image G-defocus as a reference, aligning the green channel focused image G-infocus with the green channel defocus image G-defocus, and calculating the affine transformation matrix W, which can be a 3x3 matrix. The B, Gb, Gr, and R channels of the synthesized focused image C-infocus are transformed and synthesized by the affine transformation matrix W respectively to obtain the aligned focused image Aligned C-infocus. The aligned focused image serves as the deblurring target of the four-phase detection defocus image.

[0089] In the aligned focused image Aligned C-infocus, each four adjacent groups of pixels arranged in the Bayer format are used as a third unit, and the third unit includes four groups of pixels, namely B, Gb, Gr and R. The green average value of Gb and Gr in each third unit is taken as the green pixel value after the third unit is four-in-one; each third unit is assigned its own green average value to form the aligned G channel focused image Aligned G-infocus.

[0090] S214a, such as Figure 2 and Figure 3As shown, point spread function estimation: According to the QPD defocus raw image, the G channel defocus images of the four direction images of the four phase defocus, namely, the upper image up, the lower image down, the left image left and the right image right, are calculated respectively; the blur kernel of the G channel focused image AlignedG-infocus after the G channel defocus images of the four direction images are relatively aligned is calculated respectively, and the result is as follows Figure 4 The four point spread function models (PSF) shown are upper, lower, left and right. Specifically, the blur kernel of the G channel defocused image up G-defocus of the upper image is calculated to obtain the upper point spread function model (upper PSF); the blur kernel of the G channel focused image Aligned G-infocus of the lower image is calculated to obtain the lower point spread function model (lower PSF); the blur kernel of the G channel defocused image left G-defocus of the left image is calculated to obtain the left point spread function model (left PSF); the blur kernel of the G channel focused image Aligned G-infocus of the right image is calculated to obtain the right point spread function model (right PSF). Figure 4 It can be seen that the point spread function models (blur kernels) in the four directions of up, down, left, and right are different, and the shapes of the blur kernels are different in different directions. In this way, the defocused RGB image can be simulated more specifically and accurately.

[0091] Specifically, the upper image up is obtained by taking the average value of the upper left and upper right pixels in each color group of the QPD defocused raw image, as the pixel value of the group after four-in-one; the lower image down is obtained by taking the average value of the lower left and lower right pixels in each color group, as the pixel value of the group after four-in-one; the left image left is obtained by taking the average value of the upper left and lower left pixels in each color group, as the pixel value of the group after four-in-one; the right image right is obtained by taking the average value of the upper right and lower right pixels in each color group, as the pixel value of the group after four-in-one.

[0092] In the upper image up, lower image down, left image left and right image right of the four-phase defocus, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels of B, Gb, Gr and R. The green average value of Gb and Gr in each unit is taken as the green pixel value after the four-in-one of the unit; each unit is assigned its own green average value to form its corresponding G channel defocused image.

[0093] In the second example, the method for estimating the four-phase detection point spread function model specifically also includes: S214b, point spread function estimation: according to the four-phase detection defocus raw image, respectively calculate the G channel defocus images of the four direction images of the four-phase defocus, namely the upper left image, the lower left image, the upper right image and the lower right image; respectively calculate the blur kernel of the G channel focused image after the relative alignment of the G channel defocus images of the four direction images, and obtain four point spread function models of the upper left, lower left, upper right and lower right.

[0094] The upper left image is obtained by taking the pixel value of the upper left pixel in each color group of the four-phase detection defocus raw image as the pixel value of the group after the four-in-one; the lower left image is obtained by taking the pixel value of the lower left pixel in each color group as the pixel value of the group after the four-in-one; the upper right image is obtained by taking the pixel value of the upper right pixel in each color group as the pixel value of the group after the four-in-one; the lower right image is obtained by taking the pixel value of the lower right pixel in each color group as the pixel value of the group after the four-in-one.

[0095] In the upper left image, lower left image, upper right image and lower right image of the four-phase defocus, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels of B, Gb, Gr and R. The green average value of Gb and Gr in each unit is taken as the green pixel value after the unit is four-in-one; each unit is assigned its own green average value to form its corresponding G channel defocused image.

[0096] In the third example, step S2, a method for estimating a two-phase detection point spread function model, specifically includes:

[0097] S221, collecting two-phase detection focus raw images: using a Dual-PD camera, aligning it with the center of the plane image, adjusting the position of the camera focus motor to focus, and collecting two-phase detection focus raw images; in the two-phase detection focus raw images, two adjacent pixels of the same color are grouped; in each color group, the average value of the two pixels is taken as the pixel value of the group after two-in-one; each group is assigned its own average value to form a two-phase synthesized focus image; in the two-phase synthesized focus image, each adjacent four groups of pixels arranged in the Bayer format are taken as a fourth unit, and the green average value of Gb and Gr in each fourth unit is taken as the green pixel value of the fourth unit after four-in-one; each fourth unit is assigned its own green average value to form a two-phase green channel focus image;

[0098] S222, collecting two-phase detection defocus raw images: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting two-phase detection defocus raw images; in the two-phase detection defocus raw images, two adjacent pixels of the same color are a group; in each color group, the average value of each two pixels is taken as the pixel value of the group after two-in-one; each group is assigned its own average value to form a two-phase synthesized defocused image; in the two-phase synthesized defocused image, each adjacent four groups of pixels arranged in Bayer format are taken as a fifth unit, and the green average value of Gb and Gr in each fifth unit is taken as the green pixel value after four-in-one of the fifth unit; each fifth unit is assigned its own green average value to form a two-phase green channel defocused image.

[0099] S223, data alignment: using the enhanced correlation coefficient image alignment method, taking the two-phase detection defocused raw image as the reference, aligning the two-phase detection focus raw image with the two-phase detection defocused raw image to obtain the aligned focused image; and calculating the two-phase aligned G channel focused image corresponding to the aligned focused image. Specifically, using the enhanced correlation coefficient image alignment method, taking the two-phase green channel defocused image as the reference, aligning the two-phase green channel focused image with the two-phase green channel defocused image, and calculating the affine transformation matrix; transforming the B, Gb, Gr and R channels of the two-phase synthesized focused image by the affine transformation matrix respectively and synthesizing them to obtain the two-phase aligned focused image; in the two-phase aligned focused image, each adjacent four groups of pixels arranged in the Bayer format are taken as a sixth unit, and the green average value of Gb and Gr in each sixth unit is taken as the green pixel value after the six units are four-in-one; each sixth unit is assigned its own green average value to form the two-phase aligned G channel focused image.

[0100] S224, point spread function estimation: according to the two-phase detection defocus raw images, the two-phase G channel defocus images of the two-phase defocused left image and the right image are calculated respectively; the blur kernel of the two-phase G channel defocused images of the two-phase defocused images relative to the G channel focused image after the two phases are aligned is calculated respectively, and the left and right point spread function models are obtained; the left image is to take the left pixel in each color group of the two-phase detection defocus raw images as the pixel value of the group after two-in-one; the right image is to take the right pixel in each color group as the pixel value of the group after two-in-one. In the two-phase defocused left and right images, each image is arranged in the Bayer format. Each adjacent four groups of pixels are regarded as a unit, and the unit includes four groups of pixels B, Gb, Gr and R. The green average value of Gb and Gr in each unit is taken as the green pixel value after the unit is four-in-one; each unit is assigned its own green average value to form its corresponding G channel defocused image.

[0101] Step S2 also includes: S25, adjusting the camera motor to obtain four-phase detection defocus raw images with different blur levels, repeating steps S23 and S24 to obtain point spread functions with different blur levels.

[0102] S3, layering the focused RGB image according to the depth map;

[0103] S4. According to the diffusion radius and point spread function model corresponding to each layer, the pixels corresponding to each layer are diffused. The diffusion calculation formula is as follows:

[0104]

[0105] Among them, B scatter is the pixel value after diffusion calculation, I is the pixel intensity value, K is the PSF kernel function, and x is the pixel coordinate. The pixel that needs to be calculated is dispersed according to the range of the diffusion function and the diffusion form (i.e., the diffusion weight), and the final image is the accumulation of all pixels after dispersion.

[0106] S5. Mix the layers after diffusion calculation in order of distance to obtain a simulated defocused (blurred) image corresponding to the focused RGB image.

[0107] Different depths correspond to different point spread functions, and different orientation maps correspond to different point spread functions. The point spread function is a mathematical description of the shape and intensity of the imaging spot. According to the depth map, the layering is performed, and the depth corresponds to the blur kernel radius; the depth and direction are combined to obtain the conversion from a clear image to a blurred image.

[0108] When the focused RGB image is in a two-phase detection format, the two-phase detection point spread function model includes: a left point spread function model and a right point spread function model; according to the diffusion radius corresponding to each layer of the focused RGB image and the left point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in order of distance to obtain a simulated defocused left image; and so on, the focused RGB image is calculated through the right point spread function model to obtain a simulated defocused right image; the simulated defocused left image and the simulated defocused right image constitute a simulated defocused RGB image.

[0109] When the focused RGB image is in a four-phase detection format, in one example, the four-phase detection point spread function model includes: an upper point spread function model, a lower point spread function model, a left point spread function model and a right point spread function model; according to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in order of distance to obtain a simulated defocused upper image; and so on, the focused RGB image is calculated through the lower point spread function model to obtain a simulated defocused lower image; the focused RGB image is calculated through the left point spread function model to obtain a simulated defocused left image; the focused RGB image is calculated through the right point spread function model to obtain a simulated defocused right image; the simulated defocused left image, the simulated defocused right image, the simulated defocused upper image and the simulated defocused lower image constitute a simulated defocused RGB image.

[0110] When the focused RGB image is in a four-phase detection format, in another example, the four-phase detection point spread function model includes: an upper left point spread function model, a lower left point spread function model, an upper right point spread function model and a lower right point spread function model; according to the diffusion radius corresponding to each layer of the focused RGB image and the upper left point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in order of distance to obtain a simulated defocused upper left image; and so on, the focused RGB image is calculated by the lower left point spread function model to obtain a simulated defocused lower left image; the focused RGB image is calculated by the upper right point spread function model to obtain a simulated defocused upper right image; the focused RGB image is calculated by the lower right point spread function model to obtain a simulated defocused lower right image; the simulated defocused upper left image, the simulated defocused lower left image, the simulated defocused upper right image and the simulated defocused lower right image constitute a simulated defocused RGB image.

[0111] The data simulation method further includes: performing compensation calculation on the simulated defocused RGB image to make the edge transition across the depth of field more natural. The compensation method specifically includes: Figure 5 As shown,

[0112] Provide a gray image B_gray of the same color intensity as the focused RGB image;

[0113] Calculate the diffusion radius and point spread function model according to the camera parameters and the depth map depth, perform diffusion calculations on the focused RGB image and the gray image B_gray respectively, and obtain the simulated defocused RGB image RGB_blur and the gray blur image B_gray_blur;

[0114] The gray blur image B_gray_blur is compensated to the gray image B_gray to obtain a compensation weight W; the compensation weight W is a color intensity ratio of the gray image B_gray to the gray blur image B_gray_blur;

[0115] Multiply the simulated defocus RGB image RGB_blur by the compensation weight W to obtain the compensated simulated defocus RGB image RGB_blur_out.

[0116] Figure 5 In the figure, the simulated defocused RGB image RGB_blur before compensation has unnatural transition zones across the depth of field, such as stripes, which are more obvious on the left side of the car. The gray blur image B_gray_blur also still has stripes. The gray blur image B_gray_blur is compensated to the gray image B_gray, and the gray blur image B_gray_blur is de-striped and compensated to the gray image B_gray to obtain the compensation weight W. The simulated defocused RGB image RGB_blur is multiplied by the compensation weight W to obtain the simulated defocused RGB image RGB_blur_out after compensation. Figure 3 It can be seen in the upper right corner that after compensation, there are no more stripes in the transition zone across the depth of field of the simulated defocus RGB image RGB_blur_out, and the transition is natural.

[0117] like Figure 6 As shown, a focused RGB image is input, and a blurred image needs to be simulated and generated. The blurred image simulated and generated by the first method before improvement in the background technology produces serious artifacts across the depth of field boundary, as shown in the red circle in the figure. The blurred image simulated and generated by the diffusion calculation method based on the RGB-depth data point spread function of the present invention has a more natural transition across the depth of field boundary without artifacts.

[0118] like Figure 7 As shown, a focused RGB image is input, and a blurred image needs to be simulated and generated. The blurred image simulated and generated by the second method before improvement in the background technology has a large number of rectangular spots, as shown in the red circle in the figure. The blurred image simulated and generated by the diffusion calculation method based on the RGB-depth data point spread function of the present invention has no rectangular spot problem, and the transition across the depth of field boundary is natural.

[0119] Deblurring effect comparison, such as Figure 8 As shown in FIG. 1 , the simulated data is used for image deblurring training, and the dataset simulated by the first method before the improvement and the dataset simulated by the present invention are used for training using the same deblurring model. The deblurring effect of the model trained by the simulated dataset of the present invention is better, as shown in FIG. Figure 8 In the green circle, the model trained by the simulation data set of the present invention is clearer after deblurring.

[0120] In one example, deblurred image training specifically includes: inputting two phase detection defocused RGB images including a left image and a right image into a deblurring model, and using the focused RGB image as a supervision for training, so that the model training reaches a converged state.

[0121] The test of the deblurred image specifically includes inputting two phase detection defocused RGB images including a left image and a right image into a deblurring model, and outputting a clear RGB image, that is, a corresponding focused RGB image, after deblurring processing.

[0122] In another example, deblurred image training specifically includes: inputting the left image, right image, upper image and lower image corresponding to the four-phase detection defocused RGB image into the deblurring model, and using the focused RGB image as supervision for training to make the model training reach a convergence state.

[0123] The test of the deblurred image specifically includes inputting the left image, right image, upper image and lower image corresponding to the four-phase detection defocused RGB image into the deblurring model, and outputting a clear RGB image, that is, the corresponding focused RGB image, after deblurring processing.

[0124] To determine whether the data simulation method of the present invention is used, a white dot and black background image can be used, where the white dot represents the diffusion radius and the black background represents the spot range. Assuming that the white dot and black background image is of different depths, the diffusion radius and the morphology of the blur kernel corresponding to different depths can be inferred. By finding the diffusion radius and the morphology of the blur kernel corresponding to each depth of a set of simple multi-depth focused RGB images and depth images, simulation calculations are performed. If the same or similar results are obtained as the other party, it can be suspected that the data simulation method of the present invention is used.

[0125] Data-driven defocus deblurring using the dual image views available on Dual-PD (dual phase detection) sensors has shown promising results. A significant challenge in this line of research is acquiring dual-PD data. Although many cameras have dual-PD sensors, only a limited number of cameras have access to low-level dual-PD sensor images. In addition, capturing training data for defocus deblurring involves a time-consuming and cumbersome setup that requires adjusting the camera's aperture. Some cameras with dual-PD sensors, such as smartphones, do not have an adjustable aperture, further limiting the ability to generate the necessary training data.

[0126] The data simulation method provided by the present invention proposes a QPD data simulation method based on the Dual-PD data simulation. Both Dual-PD data and QPD data can be simulated to solve the data capture bottleneck. The data simulation method of the present invention simulates the optical image formation on two-phase detection and four-phase detection sensors, and can be applied to virtual scenes rendered using standard computer software. Using these realistic simulated images, the deblurring effect is improved. The data simulation images of the present invention can be used to train models for video deblurring applications.

[0127] In summary, the present invention provides a data simulation method, including: S1, obtaining the corresponding focused RGB image and depth map; according to the imaging principle and the preset camera parameters, calculating the diffusion radius of the blur kernel corresponding to each pixel in the depth map; S2, estimating the point spread function model, the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; S4, performing diffusion calculation on the pixels corresponding to each layer according to the diffusion radius and point spread function model corresponding to each layer; S5, mixing the layers after the diffusion calculation in the order of near and far to obtain a simulated defocused RGB image corresponding to the focused RGB image. The present invention adopts a simulation method based on diffusion calculation of RGB-depth data. The present invention is a diffusion calculation of the point spread function. Compared with the first method before the improvement, the transition across the depth of field boundary is more natural, avoiding the generation of serious artifacts across the depth of field boundary; compared with the second method before the improvement, there is no rectangular spot problem.

[0128] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the method disclosed in the embodiment, since it corresponds to the device disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0129] The above description is only a description of the preferred embodiment of the present invention, and is not any limitation on the scope of rights of the present invention. Any technical personnel in this field can make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A data simulation method, It is characterized in that include: S1, obtain the corresponding focused RGB image and depth map; Calculate the diffusion radius of the blur kernel corresponding to each pixel in the depth map according to the imaging principle and preset camera parameters; S2, estimating or directly calling a point spread function model, wherein the point spread function model includes a two-phase detection point spread function model and a four-phase detection point spread function model; S3, layering the focused RGB image according to the depth map; S4, performing diffusion calculation on pixels corresponding to each layer according to the diffusion radius corresponding to each layer and the point spread function model; S5. Mix the layers after diffusion calculation in order of distance to obtain a simulated defocused RGB image corresponding to the focused RGB image.

2. The data simulation method according to claim 1, It is characterized in that Step S2, a method for estimating the four-phase detection point spread function model, specifically comprising: S211, collecting the four-phase detection focusing raw image: using the QPD camera, aligning it with the center of the plane image, adjusting the position of the camera focus motor to focus, and collecting the four-phase detection focusing raw image; S212, collecting a four-phase detection defocus raw image: adjusting the focus motor position of the QPD camera to blur the image, and collecting the four-phase detection defocus raw image; S213, data alignment: using the enhanced correlation coefficient image alignment method, taking the four-phase detection defocus raw image as a reference, aligning the four-phase detection focus raw image with the four-phase detection defocus raw image to obtain an aligned focused image; and calculating the aligned G channel focused image corresponding to the aligned focused image.

3. The data simulation method according to claim 2, It is characterized in that In step S211, in the four-phase detection focus raw image, four adjacent pixels of the same color are grouped; in each color group, the average value of the four pixels is taken as the pixel value of the group after four-in-one; each group is assigned a respective average value to form a synthesized focused image; in the synthesized focused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a first unit, and the green average value of Gb and Gr in each of the first units is taken as the green pixel value of the first unit after four-in-one; each first unit is assigned a respective green average value to form a green channel focused image; In step S212, in the four-phase detection defocused raw image, four adjacent pixels of the same color are grouped; in each color group, the average value of the four pixels is taken as the pixel value of the group after four-in-one; each group is assigned its own average value to form a synthesized defocused image; in the synthesized defocused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a second unit, and the green average value of Gb and Gr in each of the second units is taken as the green pixel value of the second unit after four-in-one; each second unit is assigned its own green average value to form a green channel defocused image; In step S213, the enhanced correlation coefficient image alignment method is used to align the green channel focused image with the green channel defocused image with the green channel defocused image as a reference, and an affine transformation matrix is ​​calculated; the B, Gb, Gr and R channels of the synthesized focused image are respectively transformed and synthesized by the affine transformation matrix to obtain an aligned focused image; in the aligned focused image, each four adjacent groups of pixels arranged in a Bayer format are taken as a third unit, and the green average value of Gb and Gr in each of the third units is taken as the green pixel value after the third unit is four-in-one; each of the third units is assigned its own green average value to form the aligned G channel focused image.

4. The data simulation method according to claim 2, It is characterized in that Step S2, the method for estimating the four-phase detection point spread function model specifically includes: S214a, point spread function estimation: according to the four-phase detection defocus raw image, respectively calculate and obtain the G channel defocus images of the four direction images of the four-phase defocus, namely, the upper image, the lower image, the left image and the right image; respectively calculate the blur kernel of the G channel defocus image of each of the four direction images of the four-phase defocus relative to the aligned G channel focused image, and obtain four point spread function models of upper, lower, left and right; The upper image is obtained by taking the average value of the upper left and upper right pixels in each color group of the four-phase detection defocus raw image, as the pixel value of the group after four-in-one; the lower image is obtained by taking the average value of the lower left and lower right pixels in each color group, as the pixel value of the group after four-in-one; the left image is obtained by taking the average value of the upper left and lower left pixels in each color group, as the pixel value of the group after four-in-one; the right image is obtained by taking the average value of the upper right and lower right pixels in each color group, as the pixel value of the group after four-in-one.

5. The data simulation method according to claim 4, It is characterized in that In the upper image, lower image, left image and right image of the four-phase defocus, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels of B, Gb, Gr and R. The green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

6. The data simulation method according to claim 2, It is characterized in that Step S2, the method for estimating the four-phase detection point spread function model specifically includes: S214b, point spread function estimation: according to the four-phase detection defocus raw image, respectively calculate and obtain the G channel defocus images of the four direction images of the upper left image, the lower left image, the upper right image and the lower right image of the four-phase defocus; respectively calculate the blur kernel of the G channel defocus image of each of the four direction images relative to the aligned G channel focused image, and obtain four point spread function models of the upper left, lower left, upper right and lower right; The upper left image is obtained by taking the pixel value of the upper left pixel in each color group of the four-phase detection defocus raw image as the pixel value of the group after four-in-one; the lower left image is obtained by taking the pixel value of the lower left pixel in each color group as the pixel value of the group after four-in-one; the upper right image is obtained by taking the pixel value of the upper right pixel in each color group as the pixel value of the group after four-in-one; the lower right image is obtained by taking the pixel value of the lower right pixel in each color group as the pixel value of the group after four-in-one.

7. The data simulation method according to claim 6, It is characterized in that In the upper left image, lower left image, upper right image and lower right image of the four-phase defocused image, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels of B, Gb, Gr and R. The green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

8. The data simulation method according to claim 1, It is characterized in that Step S2, a method for estimating the two-phase detection point spread function model, specifically comprising: S221, collecting two-phase detection focusing raw images: using a Dual-PD camera, aligning it with the center of the plane image, adjusting the position of the camera focus motor to focus, and collecting the two-phase detection focusing raw images; S222, collecting two-phase detection defocus raw images: adjusting the focus motor position of the Dual-PD camera to blur the image, and collecting the two-phase detection defocus raw images; S223, data alignment: using the enhanced correlation coefficient image alignment method, taking the two phase detection defocused raw images as a reference, aligning the two phase detection focused raw images with the two phase detection defocused raw images to obtain an aligned focused image; and calculating the two phase aligned G channel focused images corresponding to the aligned focused image.

9. The data simulation method according to claim 8, It is characterized in that In step S221, in the two-phase detection focus raw image, two adjacent pixels of the same color are grouped; in each color group, the average value of the two pixels is taken as the pixel value of the group after two-in-one; each group is assigned a respective average value to form a two-phase synthesized focus image; in the two-phase synthesized focus image, each four adjacent groups of pixels arranged in the Bayer format are taken as a fourth unit, and the green average value of Gb and Gr in each of the fourth units is taken as the green pixel value of the fourth unit after four-in-one; each of the fourth units is assigned a respective green average value to form a two-phase green channel focus image; In step S222, in the two-phase detection defocused raw image, two adjacent pixels of the same color are grouped; in each color group, the average value of the two pixels is taken as the pixel value of the group after two-in-one; each group is assigned its own average value to form a two-phase synthesized defocused image; in the two-phase synthesized defocused image, each four adjacent groups of pixels arranged in the Bayer format are taken as a fifth unit, and the green average value of Gb and Gr in each of the fifth units is taken as the green pixel value of the fifth unit after four-in-one; each of the fifth units is assigned its own green average value to form a two-phase green channel defocused image; In step S223, the enhanced correlation coefficient image alignment method is used to align the two-phase green channel focused image with the two-phase green channel defocused image with the two-phase green channel defocused image, and an affine transformation matrix is ​​calculated; the B, Gb, Gr and R channels of the two-phase synthesized focused image are respectively transformed and synthesized by the affine transformation matrix to obtain the two-phase aligned focused image; in the two-phase aligned focused images, each four adjacent groups of pixels arranged in Bayer format are taken as a sixth unit, and the green average value of Gb and Gr in each of the sixth units is taken as the green pixel value after the four-in-one sixth unit; each of the sixth units is assigned its own green average value to form the two-phase aligned G channel focused image.

10. The data simulation method according to claim 8, It is characterized in that Step S2, the method for estimating the two-phase detection point spread function model specifically includes: S224, point spread function estimation: according to the two-phase detection defocus raw images, respectively calculate and obtain the two-phase G channel defocus images of the two-phase defocused left image and the right image in two directions; respectively calculate the blur kernel of the two-phase G channel defocused images of the two directions relative to the G channel focused image after the two phases are aligned, and obtain the left and right point spread function models; The left image is obtained by taking the left pixel in each color group of the two phase detection defocus raw images as the pixel value of the group after the two are combined into one; the right image is obtained by taking the right pixel in each color group as the pixel value of the group after the two are combined into one.

11. The data simulation method according to claim 10, It is characterized in that In the two-phase defocused left image and right image, each adjacent four groups of pixels arranged in the Bayer format in each image are regarded as a unit, and the unit includes four groups of pixels, namely B, Gb, Gr and R. The green average value of Gb and Gr in each of the units is taken as the green pixel value after the four-in-one of the unit; each of the units is assigned its own green average value to form the corresponding G channel defocused image.

12. The data simulation method according to claim 2, It is characterized in that Step S2 also includes: S25. Adjust the camera motor to obtain the four-phase detection defocus raw image with different blur levels, and repeat steps S23 and S24 to obtain the point spread function with different blur levels.

13. The data simulation method according to claim 1, It is characterized in that When the focused RGB image is in a two-phase detection format, the two-phase detection point spread function model includes: a left point spread function model and a right point spread function model; According to the diffusion radius corresponding to each layer of the focused RGB image and the left point diffusion function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused left image; By analogy, the focused RGB image is calculated through the right point spread function model to obtain a simulated defocused right image; The simulated defocused left image and the simulated defocused right image constitute the simulated defocused RGB image.

14. The data simulation method according to claim 1, It is characterized in that When the focused RGB image is in a four-phase detection format, the four-phase detection point spread function model includes: an upper point spread function model, a lower point spread function model, a left point spread function model and a right point spread function model; According to the diffusion radius corresponding to each layer of the focused RGB image and the upper point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of near and far to obtain a simulated defocused upper image; By analogy, the focused RGB image is calculated by the lower point spread function model to obtain a simulated defocused lower image; the focused RGB image is calculated by the left point spread function model to obtain a simulated defocused left image; the focused RGB image is calculated by the right point spread function model to obtain a simulated defocused right image; The simulated defocused left image, the simulated defocused right image, the simulated defocused upper image and the simulated defocused lower image constitute the simulated defocused RGB image.

15. The data simulation method according to claim 1, It is characterized in that When the focused RGB image is in a four-phase detection format, the four-phase detection point spread function model includes: an upper left point spread function model, a lower left point spread function model, an upper right point spread function model and a lower right point spread function model; According to the diffusion radius corresponding to each layer of the focused RGB image and the upper left point spread function model, the pixels corresponding to each layer are diffused; the layers after the diffusion calculation are mixed in the order of distance to obtain a simulated defocused upper left image; By analogy, the focused RGB image is calculated by the lower left point spread function model to obtain a simulated defocused lower left image; the focused RGB image is calculated by the upper right point spread function model to obtain a simulated defocused upper right image; the focused RGB image is calculated by the lower right point spread function model to obtain a simulated defocused lower right image; The simulated defocused upper left image, the simulated defocused lower left image, the simulated defocused upper right image, and the simulated defocused lower right image constitute the simulated defocused RGB image.

16. The data simulation method according to claim 1, It is characterized in that The corresponding focused RGB image and the depth map are obtained by retrieving a known database or shooting with a camera focused.

17. The data simulation method according to claim 1, It is characterized in that The data simulation method further includes: performing compensation calculation on the simulated defocused RGB image, and the compensation method specifically includes: providing a gray image of the same color intensity as the focused RGB image; Calculate the diffusion radius and the point spread function model according to the camera parameters and the depth map, perform diffusion calculation on the focused RGB image and the gray image respectively, and obtain the simulated defocused RGB image and the gray blur image; Compensate the gray fuzzy image to the gray image to obtain a compensation weight; the compensation weight is a color intensity ratio of the gray image to the gray fuzzy image; The simulated defocus RGB image is multiplied by the compensation weight to obtain a compensated simulated defocus RGB image.

Citation Information

Patent Citations

  • Image field depth drawing method and device

    CN106558089A

  • Under-screen ToF depth camera imaging noise model construction method

    CN118052728A

  • Image processing apparatus, image forming apparatus, image processing method, and storage medium

    US20150254537A1

  • Learnable point spread functions for image rendering

    WO2024099564A1

Cited By

  • Quantum unitary matrix quasi-probability decomposition method and device based on diffusion model

    CN122264151A

  • Quantum unitary matrix quasi-probability decomposition method and device based on diffusion model

    CN122264151B