Dual-pixel image simulation method and application thereof

Abstract

The invention belongs to the related technical field of image processing, and discloses a dual-pixel image simulation method and application thereof, and the method comprises the steps: obtaining a clear image in an effective imaging region of a camera and the space coordinates of each pixel; for each space object point, calculating a point spread function of the space object point falling on the front surface of the DP sensor through ray tracing refraction; respectively calculating light intensity distribution of left and right sub-pixels according to the characteristics of the DP sensor, and carrying out light weight distribution on the point spread function to generate a dual-pixel point spread function; further integrating to obtain a dual-pixel point spread function map in a scene corresponding to the clear image; and carrying out convolution on the clear image and the double-pixel point spread function map pixel by pixel to generate a simulation double-pixel image. According to the method, the simulation image highly consistent with the real DP image can be quickly generated, downstream image processing application is effectively supported, the model generalization ability is improved, meanwhile, the calculation cost is reduced, and the real-time or batch processing requirement is met.

Description

Technical Field

[0001] The present invention belongs to the technical field related to image processing, and more specifically, relates to a dual-pixel image simulation method and application thereof. Background Art

[0002] Dual-Pixel (DP) sensors have broad application prospects in the field of image processing due to their unique phase characteristics. Figure 1 The figure shows a schematic diagram of the DP sensor obtaining left and right disparity. Unlike traditional single-pixel sensors, the DP sensor divides each pixel into left and right sub-pixels, and extracts all left and right sub-pixels to form images, so that two images of left and right perspectives can be obtained in one shot. In the figure, red is the left sub-pixel and blue is the right sub-pixel, which is the dual-pixel point spread function (DP PSF). By capturing the left and right sub-pixel images of each pixel, the DP sensor is able to provide rich phase information in a single shot. This information is crucial for a variety of image processing downstream applications such as depth estimation, deblurring, and image enhancement. However, since the DP image features depend entirely on the camera hardware, datasets for DP sensors customized for specific cameras and paired with downstream applications have become very scarce.

[0003] To overcome this challenge, researchers have attempted to simulate DP images using ideal optical system models. However, these simulation methods often violate the true laws of optical propagation, resulting in poor generalization capabilities on real DP data.

[0004] Therefore, it is necessary to provide an efficient and accurate DP image simulation method to meet the demand for high-quality DP data in diverse downstream applications and promote the further development of image processing technology. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the prior art, the present invention provides a dual-pixel image simulation method and its application, the purpose of which is to provide an efficient and accurate DP image simulation method to generate a simulated DP image that is close to the real DP image in phase and aberration characteristics, so as to support a variety of downstream image processing applications.

[0006] To achieve the above object, according to a first aspect of the present invention, a dual-pixel image simulation method is provided, which includes:

[0007] S1. Obtain a clear image and the spatial coordinates of each pixel within the effective imaging area of the camera. The spatial coordinates of different pixels represent different spatial object points.

[0008] S2. For each spatial object point, select n sampling points on the aperture stop image plane of the camera lens system, construct n rays pointing from the spatial object point to the n sampling points respectively, and form a ray set for the spatial object point;

[0009] S3, obtaining the landing point of each light ray on the front surface of the composite pixel after being refracted by the camera lens system and the direction of the light ray at the landing point;

[0010] S4. For each light ray that falls on the front surface of the composite pixel, expand it into a beam of parallel light, obtain the left sub-pixel light intensity of the parallel light ray when it reaches the left sub-pixel of the corresponding composite pixel after passing through the DP sensor microlens, and the right sub-pixel light intensity when it reaches the right sub-pixel, calculate the ratio of the left sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light ray in the left sub-pixel, and calculate the ratio of the right sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light ray in the right sub-pixel;

[0011] S5. Accumulate and integrate the ray weights of all rays in the ray set of the same spatial object point on each sub-pixel to obtain a dual-pixel point spread function of the spatial object point, and integrate the dual-pixel point spread functions of all spatial object points to obtain a dual-pixel point spread function map for the scene corresponding to the clear image;

[0012] S6. Convolve the clear image with the dual-pixel point spread function map pixel by pixel to generate a simulated dual-pixel image of the scene corresponding to the clear image.

[0013] Optionally, the portion of parallel light passing through the DP sensor microlens undergoes phase delay, and the delay formula is:

[0014]

[0015] Where r s is the distance between the landing point of the sub-beam and the center of the DP sensor microlens, λ is the wavelength of light, f is the focal length of the DP sensor microlens, and △φ is the phase delay.

[0016] Optionally, the method further includes:

[0017] After obtaining the dual-pixel point spread functions of multiple spatial object points, a neural network is trained with the spatial object points as input and the dual-pixel point spread functions as output, and the trained neural network is used to predict the dual-pixel point spread functions of other spatial object points.

[0018] Optionally, the neural network is an MLP network.

[0019] Optionally, S6 includes:

[0020] Restore the clear image from pixel distribution to energy distribution;

[0021] The two-pixel point spread function map is spatially aligned with the clean image and then convolved pixel by pixel;

[0022] The convolution result is converted from energy distribution back to pixel distribution to generate a simulated dual-pixel image.

[0023] Optionally, an inverse gamma mapping is used to restore the clear image from pixel distribution to energy distribution, and a gamma mapping is used to convert the convolution result from energy distribution back to pixel distribution.

[0024] According to a second aspect of the present invention, a method for constructing an end-to-end dual-pixel image generation system is provided, comprising:

[0025] Acquire a simulated dual-pixel image of the selected clear image using the dual-pixel image simulation method described in any one of the above items;

[0026] A deep learning network is trained with clear images as input and the corresponding simulated dual-pixel images as output, resulting in an end-to-end dual-pixel image generation system.

[0027] According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0028] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the above methods when executed by a processor.

[0029] According to a fourth aspect of the present invention, there is provided a computer program product, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of any of the above methods.

[0030] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0031] The present invention provides an efficient and accurate DP image simulation method. For each spatial object point, a light set is constructed. By tracing the rays in the light set, the point spread function (PSF) of the rays on the front surface of the composite pixel is obtained. The rays are then expanded into parallel light. The light intensities of the parallel light reaching different sub-pixels after diffraction by the DP sensor microlens are calculated. The light weight of each sub-pixel is calculated based on the light intensity of each sub-pixel. By integrating the light weights, the dual-pixel PSF of the spatial object point can be obtained. By analyzing the refraction and diffraction of light, the present invention can quickly generate simulated images that are highly consistent with the real DP image, effectively supporting downstream image processing applications, improving the model's generalization capabilities, while reducing computing costs and meeting real-time or batch processing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the DP sensor obtaining left and right parallax;

[0033] Figure 2 is a flowchart of the steps of a dual-pixel image simulation method in one embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the effective imaging area of a camera in one embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of calculating a dual-pixel point spread function in one embodiment of the present invention;

[0036] Figure 5 Schematic diagram of diffracted light intensity distribution of a DP sensor in one embodiment of the present invention;

[0037] Figure 6 is a schematic diagram of a DP PSF prediction network in one embodiment of the present invention;

[0038] Figure 7 is a flow chart of obtaining a simulated DP image in one embodiment of the present invention;

[0039] Figure 8 is a schematic structural diagram of an end-to-end dual-pixel image generation system in one embodiment of the present invention;

[0040] Figure 9 This is a qualitative analysis diagram of a simulated DP image in one embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0042] Example 1

[0043] The present invention proposes a dual-pixel image simulation method, such as Figure 2 FIG2 is a flowchart of a dual-pixel image simulation method according to an embodiment of the present invention, wherein the steps are described in detail below.

[0044] S1. Obtain a clear image and the spatial coordinates of each pixel within the effective imaging area of the camera. The spatial coordinates of different pixels represent different spatial object points.

[0045] Specifically, we use the color and depth (RGB-Depth) paired dataset in the existing database as the data source to obtain a clear image and the spatial coordinates corresponding to each pixel (i.e., each spatial object point). The RGB image is used as the clear image, and the depth image can provide the spatial coordinates of each pixel.

[0046] It is understandable that the coordinates of the spatial object point are not unlimited, they are clamped within the effective imaging area of the camera, such as Figure 3 The figure shows a schematic diagram of the effective imaging area of a camera in one embodiment of the present invention. This area is defined by the field of view of the camera lens, the sensor size, and the minimum and maximum depth settings, and has a truncated cone shape. During the shooting process, spatial object points can only be located within this area and cannot be placed arbitrarily.

[0047] S2. For each spatial object point, select n sampling points on the aperture stop image plane of the camera lens system, construct n rays pointing from the spatial object point to the n sampling points respectively, and form a ray set for the spatial object point.

[0048] Taking any spatial object point p as an example, n sampling points are selected on the aperture stop image plane of the camera lens system, and n rays A1~A1 are constructed from the same spatial object point p to the n sampling points respectively. n , forming the set of rays of the object point p in this space.

[0049] Specifically, the aperture stop image plane of the camera lens system in object space is used as the entrance pupil. N points are densely sampled on this entrance pupil plane. Starting from a spatial object point p, a ray set A consisting of n rays is constructed. These rays all originate from the spatial object point p. The initial direction vector of each ray in ray set A points from the spatial object point p to the corresponding sampling point in the entrance pupil. If a ray emitted by spatial object point p is blocked by surrounding objects, it cannot propagate to the entrance pupil.

[0050] S3. Obtain the landing point of each light ray on the front surface of the composite pixel after being refracted by the camera lens system and the direction of the light ray at the landing point.

[0051] like Figure 4 Figure 1 shows a schematic diagram of calculating a dual-pixel point spread function (PSF) in accordance with one embodiment of the present invention. Light originates from an object point p in space, passes through various surfaces of the lens system, undergoes precise refraction, and ultimately lands on the DP sensor surface (i.e., the front surface of the composite pixel). There, it is diffracted by the microlenses of the DP sensor's composite pixel and absorbed by the sub-pixels, ultimately forming the DP PSF.

[0052] Specifically, this step is to obtain each ray A k The point O on the front surface of the composite pixel after refraction by the lens system k And at point O k The direction of light D k , k=1,2,……,n.

[0053] Each ray A in the ray set A k As it passes through each surface of the camera lens system, its position and direction strictly adhere to the unique optical properties of the mirror surface and are precisely refracted according to Snell's law. After a series of refractions through the lens, the light ray set A ultimately lands on the front surface of the composite pixel. The point where A lands on the front surface of the composite pixel is represented by O, and the direction from which the light enters O is represented by D. The cumulative integral of the landing point O of the light ray set A at each composite pixel yields the simulated PSF. The core of this step is to accurately calculate the propagation path and landing point of each ray in the light ray set through the lens using optical principles.

[0054] S4. For each light ray that falls on the front surface of the composite pixel, expand it into a beam of parallel light, obtain the left sub-pixel light intensity of the parallel light that reaches the left sub-pixel of the corresponding composite pixel after passing through the DP sensor microlens, and the right sub-pixel light intensity of the parallel light that reaches the right sub-pixel, calculate the ratio of the left sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light in the left sub-pixel, calculate the ratio of the right sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light in the right sub-pixel.

[0055] This step is to use the characteristics of the DP sensor and diffraction analysis to distribute light weights to the simulated PSF.

[0056] Specifically, through lens ray tracing, we can know that the kth ray A k The landing point O on the front surface of the composite pixel k and direction D k Based on the landing point O k , we can calculate A k The serial number (i, j) of the DP composite pixel where it is located, and the surface center coordinates of the composite pixel can be calculated as (x i ,y i ). This step is to analyze the kth ray A in the ray set A according to the landing point and direction of the ray. k Integral weights are assigned to the left sub-pixel and the right sub-pixel of the composite pixel, so that the DP PSF of the spatial object point can be obtained by subsequently calculating the cumulative integral.

[0057] like Figure 5 The figure shows the distribution diagram of the diffracted light intensity of the DP sensor in one embodiment of the present invention. According to the density of light, the kth light falling on the front surface of the composite pixel is expanded into a beam of parallel light, and the complex amplitude of the parallel light is recorded as U s , the direction of parallel light remains D k After passing through the front surface of the composite pixel, the phase of the part passing through the microlens is delayed. The delay formula is as follows:

[0058]

[0059] Where r s is the distance between the landing point of the sub-beam and the center of the microlens, λ is the wavelength of light, and f is the focal length of the microlens. The part that does not pass through the microlens does not experience phase delay. After passing through the front surface, the parallel light U s The microlens is modulated into U s ′ , the propagation process from the back surface of the composite pixel to the front surface of the sub-pixel can be described by the Fresnel diffraction formula:

[0060]

[0061] Where h is the distance between the sub-pixel and the microlens, k' is the wave number, (x f ,y f ) is the coordinate of the light beam propagating to the front surface of the sub-pixel when the front surface of the sub-pixel is used as the observation screen, U(x f ,y f ) is the complex amplitude of the light beam propagating to the front surface of the sub-pixel. The modulus of the complex amplitude is the light intensity. The light intensity integral of the left sub-pixel front surface is added to the original parallel beam U sThe integral ratio of the light intensity of the kth light A k The light weight assigned to the left sub-pixel within this composite pixel Correspondingly, the light intensity integral ratio of the front surface of the right sub-pixel is used as the light weight allocated to the right sub-pixel

[0062] S5. Accumulate and integrate the light weights of all light rays in the light set of the same spatial object point on each sub-pixel to obtain the dual-pixel point spread function of the spatial object point, integrate the dual-pixel point spread functions of all spatial object points, and obtain a dual-pixel point spread function map under the scene corresponding to the clear image.

[0063] To calculate the DP PSF, the ray weight distribution of all rays on each sub-pixel is cumulatively integrated. Taking the PSF calculation of the left sub-pixel as an example, the calculation process can be based on the accumulation of the contribution of each ray in the ray set A, that is:

[0064]

[0065] in, Represents light A k The weight of the light in the left sub-pixel of the DP composite pixel (i, j). k The final landing point is not on the DP composite pixel (i, j), and its value is zero. L The same steps can also be used to calculate PSF R .

[0066] Furthermore, in order to improve the accuracy of the calculation, different optical models can be introduced, including wave optics method, geometric optics method and Monte Carlo ray tracing simulation, to more comprehensively describe the energy distribution characteristics of light.

[0067] Furthermore, the aberration characteristics of the lens system, such as spherical aberration and coma, can also be considered to obtain a more accurate DP PSF.

[0068] Furthermore, to quickly calculate the DP PSF, after obtaining the DP PSFs of multiple spatial points, a neural network can be trained using the spatial points as input and the simulated DP PSFs as the true values. After training, the neural network can be used to directly predict the DP PSFs of the spatial points, thus avoiding the high computational cost of the simulated DP PSFs.

[0069] like Figure 6The figure shows a schematic diagram of the DP PSF prediction network in one embodiment of the present invention. An MLP network is designed and trained to quickly predict the DPPSF of spatial objects at different locations and under different lenses, reducing the time cost of ray tracing calculations. The network inputs the coordinates of spatial object points within the camera's effective imaging area, and outputs the DP PSF corresponding to the object point. Through training, the MLP network can accurately predict the DP PSF in various scenarios. Once training is complete, the DP PSF of all spatial object points can be directly output through the MLP, thus bypassing the huge time cost of ray tracing and avoiding the need to perform complex ray tracing operations each time.

[0070] S6. Convolve the clear image with the dual-pixel point spread function map pixel by pixel to generate a simulated dual-pixel image of the scene corresponding to the clear image.

[0071] like Figure 7 The figure shows a schematic flow chart for dual-pixel image simulation in one embodiment of the present invention. Specifically, a rendering pipeline is constructed, taking RGB-Depth paired data as input. After considering the occlusion relationship between adjacent points in the depth map, each point in the depth map is treated as a spatial object point and its corresponding DPPSF is calculated. Finally, each point in the clear image is convolved pixel by pixel with the DPPSF of the corresponding point calculated by simulation, thereby generating a simulated DPPSF image.

[0072] During the rendering process, inverse gamma mapping is used to convert the clear RGB image from pixel distribution values to energy distribution values, ensuring that the image and point spread function have consistent energy distributions before convolution. Each pixel's DP PSF is spatially aligned with the clear image, and convolution is performed pixel by pixel to ensure that the phase and aberration characteristics of the simulated image are consistent with those of the real DP image. After convolution, gamma mapping is used to convert the result from energy distribution values back to pixel distribution values, resulting in the final simulated DP image. This rendering network design optimizes computational speed and can meet the needs of real-time or batch processing.

[0073] Example 2

[0074] The present invention also proposes a method for constructing an end-to-end dual-pixel image generation system, comprising:

[0075] The dual-pixel image simulation method proposed in Example 1 is used to obtain a simulated dual-pixel image of the selected clear image; a deep learning network is trained with the clear image as input and the corresponding simulated dual-pixel image as output to obtain an end-to-end dual-pixel image generation system.

[0076] The above end-to-end image generation system efficiently predicts and simulates dual-pixel images, significantly reducing computational costs while ensuring the fidelity of image structure and details.

[0077] like Figure 8 The figure shows a schematic diagram of the structure of an end-to-end dual-pixel image generation system in one embodiment of the present invention. This embodiment designs and trains a dual-pixel image generation system for generating simulated dual-pixel images directly based on RGB-Depth paired data. Specifically, the system uses RGB-Depth paired data as input and uses the simulated DP image obtained by DP PSF rendering as a supervisory signal for training. After training, the network can generate DP images end-to-end, thereby effectively bypassing the calculation and image rendering process based on DP PSF, and significantly reducing the time overhead caused by the repeated execution of complex ray tracing, diffraction and convolution operations during the generation process.

[0078] refer to Figure 8 As shown in the figure, the end-to-end dual-pixel image generation system includes an input module, an encoding module, a decoding module, and a training supervision module. The input module receives RGB-Depth paired data, providing the system with raw information. The encoding module introduces sinusoidal position encoding to address the spatial non-uniformity problem of dual-pixel PSFs, enhancing the network's ability to perceive absolute position information. At the same time, by equally dividing the original feature maps in each layer of the encoding module along the column direction, the structural modeling capability of spatial variation features is improved. The encoded data is processed by the decoding module. The training supervision module uses the simulated DP image rendered by the DP PSF as the supervision signal. By calculating the L2 loss between the predicted DP image and the simulated DP image, the system parameters are optimized, ultimately achieving end-to-end generation of DP images and avoiding complex computational processes.

[0079] In this embodiment, the spatial non-uniformity of the dual-pixel PSF makes it difficult for traditional convolutional neural networks to accurately model its spatial variation. To address this, the encoding module uses sinusoidal position encoding to enhance absolute position information perception and further improves the modeling efficiency of spatially varying features by designing a structure that equally divides the feature map columns.

[0080] like Figure 9The figure shows a qualitative analysis diagram of a simulated dual-pixel image in an embodiment of the present invention. The figure shows the comparison effect between the real image and the simulated image of the left-eye dual-pixel image under the condition of F / 4 aperture. The figure includes images taken under the real F / 4 aperture, images taken under the real F / 20 aperture, and F / 4 aperture images simulated by different methods based on the F / 20 aperture image. The simulation methods include the dual-pixel image simulation method proposed in the present invention and the traditional Circle of Confusion (COC) simulation method. From the comparison results, it can be seen that the image simulated by the COC method has obvious differences from the real image in terms of pattern size, shape integrity and texture position, which manifests as problems such as abnormal pattern scaling, edge deformation and texture drift. In contrast, the simulation method proposed in the present invention is more consistent with the real image in terms of structure restoration and detail fidelity, can more accurately reproduce the real imaging effect, and has a higher simulation fidelity, verifying the superior performance of the method of the present invention in dual-pixel image simulation.

[0081] like Figure 9 As shown in the rightmost column, by inputting the real RGB-Depth image at F / 20 aperture into the image generation system constructed in Example 2, the predicted simulated image at F / 4 aperture can be directly output. From the image comparison results, it can be seen that the end-to-end image generation system proposed in the present invention is highly consistent with the simulated image obtained by the simulation method proposed in Example 1 in terms of structure restoration and detail fidelity, and can effectively reproduce the real imaging characteristics. However, when generating a pair of dual-pixel images, the simulation method of Example 1 takes an average of 890 seconds in a hardware environment equipped with an Intel i7-12700K processor and an NVIDIA RTX 3090 GPU; while the end-to-end image generation system based on Example 2 can complete the same task in only 0.3 seconds, significantly reducing the time cost and having extremely high generation efficiency.

[0082] In summary, the dual-pixel image simulation method and dual-pixel image generation system proposed in this invention can be widely applied in downstream applications, including but not limited to depth estimation, image deblurring, image refocusing, image coding and high-precision multimodal reconstruction, disparity estimation, rain and reflection removal, and more. By accurately simulating DP images, this invention aims to narrow the domain gap between simulated and real DP images, thereby improving the generalization and performance of downstream application models.

[0083] Example 3

[0084] The present invention also relates to an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0085] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory.

[0086] Example 4

[0087] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0088] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0089] Example 5

[0090] An embodiment of the present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the method of the above embodiment of the present invention.

[0091] The technical features of the above embodiments can be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be noted that the phrases "in one embodiment", "for example", "and another example", etc. of the present invention are intended to illustrate the present invention and are not intended to limit the present invention.

[0092] The above-described embodiments merely illustrate several implementations of the present invention. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, and all such variations and improvements fall within the scope of protection of the present invention.

Claims

1. A dual-pixel image simulation method, characterized in that: include: S1. Obtain a clear image and the spatial coordinates of each pixel within the effective imaging area of the camera. The spatial coordinates of different pixels represent different spatial object points. S2. For each spatial object point, select n sampling points on the aperture stop image plane of the camera lens system, construct n rays pointing from the spatial object point to the n sampling points respectively, and form a ray set for the spatial object point, where n>2; S3, obtaining the landing point of each light ray on the front surface of the composite pixel after being refracted by the camera lens system and the direction of the light ray at the landing point; S4. For each light ray that falls on the front surface of the composite pixel, expand it into a beam of parallel light, obtain the left sub-pixel light intensity of the parallel light ray when it reaches the left sub-pixel of the corresponding composite pixel after passing through the DP sensor microlens, and the right sub-pixel light intensity when it reaches the right sub-pixel, calculate the ratio of the left sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light ray in the left sub-pixel, and calculate the ratio of the right sub-pixel light intensity to the original parallel light intensity and use it as the light weight of the corresponding light ray in the right sub-pixel; S5. Accumulate and integrate the ray weights of all rays in the ray set of the same spatial object point on each sub-pixel to obtain a dual-pixel point spread function of the spatial object point, and integrate the dual-pixel point spread functions of all spatial object points to obtain a dual-pixel point spread function map for the scene corresponding to the clear image; S6. Convolve the clear image with the dual-pixel point spread function map pixel by pixel to generate a simulated dual-pixel image of the scene corresponding to the clear image.

2. The dual-pixel image simulation method according to claim 1, wherein: The part of parallel light passing through the DP sensor microlens is phase delayed. The delay formula is: Where r s is the distance between the landing point of the sub-beam and the center of the DP sensor microlens, λ is the wavelength of light, f is the focal length of the DP sensor microlens, and △φ is the phase delay.

3. The dual-pixel image simulation method according to claim 1, wherein: The method further includes: after obtaining the dual-pixel point spread functions of multiple spatial object points, training a neural network with the spatial object points as input and the dual-pixel point spread functions as output, and using the trained neural network to predict the dual-pixel point spread functions of other spatial object points.

4. The dual-pixel image simulation method according to claim 3, wherein: The neural network is an MLP network.

5. The dual-pixel image simulation method according to claim 1, wherein S6 include: Restore the clear image from pixel distribution to energy distribution; The two-pixel point spread function map is spatially aligned with the clean image and then convolved pixel by pixel; The convolution result is converted from energy distribution back to pixel distribution to generate a simulated dual-pixel image.

6. The dual-pixel image simulation method according to claim 5, wherein: Inverse gamma mapping is used to restore the clear image from pixel distribution to energy distribution, and gamma mapping is used to convert the convolution result from energy distribution back to pixel distribution.

7. A method for constructing an end-to-end dual-pixel image generation system, characterized in that: include: Acquire a simulated dual-pixel image of the selected clear image using the dual-pixel image simulation method according to any one of claims 1 to 6; A deep learning network is trained with clear images as input and the corresponding simulated dual-pixel images as output, resulting in an end-to-end dual-pixel image generation system.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images