Super-depth-of-field image generation method, three-dimensional reconstruction method and 3D camera

Abstract

The invention discloses a super field depth image generation method, a three-dimensional reconstruction method and a 3D camera, and belongs to the technical field of image processing. The super depth-of-field image generation method comprises the following steps: acquiring a long exposure image; inputting the long exposure image into a super depth-of-field image generation model to obtain a super depth-of-field image; the super-depth-of-field image generation model is obtained by training based on the long-exposure image sample and a to-be-processed image set corresponding to the long-exposure image sample. According to the method, the super-depth-of-field image is generated based on the deep learning technology, the efficiency and accuracy of generating the super-depth-of-field image are improved, the method can be applied to the 3D camera, and the imaging performance and efficiency of the 3D camera are improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a method for generating a super-depth-of-field image, a three-dimensional reconstruction method and a 3D camera. Background Art

[0002] As a device for obtaining the three-dimensional morphological information of an object, a 3D camera uses technologies such as structured light, time of flight (ToF), and binocular stereo vision to quickly and accurately capture three-dimensional data, and is widely used in industrial automation, robot navigation and other fields. However, traditional 3D cameras are limited by the depth of field when collecting depth information, especially when the distance of the object varies greatly or has complex geometric features, the clarity of the image may decrease. This depth of field limitation affects the accuracy and integrity of 3D data in complex scenes. The depth of field extension method in the related art usually designs a lens with multiple focal points to capture target images at different distances, and then uses image fusion technology to fuse multiple images taken at different focal lengths to create an image with a greater depth of field. Although this method can obtain high-quality ultra-depth of field images, the time-consuming image fusion will significantly affect the efficiency and also have a negative impact on the miniaturization and ease of use of the 3D camera. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a method for generating a super depth of field image, a three-dimensional reconstruction method and a 3D camera, which generates a super depth of field image based on deep learning technology, improves the efficiency and accuracy of generating super depth of field images, and can be applied to 3D cameras to improve the imaging performance and efficiency of 3D cameras.

[0004] In a first aspect, the present application provides a method for generating a super depth of field image, the method comprising:

[0005] Acquire long exposure images;

[0006] Inputting the long exposure image into a super depth of field image generation model to obtain a super depth of field image;

[0007] The super depth of field image generation model is trained based on long exposure image samples and a set of images to be processed corresponding to the long exposure image samples.

[0008] In the above technical scheme, an image with a super depth of field effect is generated by acquiring a long exposure image of the target object and inputting it into a super depth of field image generation model. The super depth of field image generation model is trained based on long exposure image samples and a set of images to be processed corresponding to the long exposure image samples, thereby improving the ability of the model to generate super depth of field images, thereby improving the quality of the generated super depth of field images. The super depth of field images are generated based on deep learning technology, which improves the efficiency and accuracy of generating super depth of field images. The model can be applied to 3D cameras to improve the imaging performance and efficiency of 3D cameras.

[0009] According to one embodiment of the present application, the super depth of field image generation model is trained by the following steps:

[0010] Adjusting the focal length of the target lens to obtain the set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, each of the images to be processed includes the same target object;

[0011] Controlling the target lens to perform long-range zooming to obtain the long-exposure image sample, wherein the long-exposure image sample fuses the imaging of the target object at all focal lengths within the zoom range corresponding to the image set to be processed;

[0012] Based on the set of images to be processed, generating a fused image;

[0013] The long exposure image sample is used as input, and the fused image is used as a corresponding label to train the super depth of field image generation model.

[0014] In the above technical scheme, by adjusting the focal length of the target lens, a set of images to be processed is obtained, the set of images to be processed includes multiple images to be processed that contain the same target object and have different depths of field, the target lens is controlled to perform long-range zoom, and long-exposure image samples that fuse the target object at all focal lengths within the zoom range corresponding to the set of images to be processed are obtained, a fused image is generated based on the set of images to be processed, and a super-depth-of-field image generation model is trained with the long-exposure image samples as input and the fused image as a label. The trained super-depth-of-field image generation model can generate super-depth-of-field images based on long-exposure images, thereby improving the efficiency and accuracy of generating super-depth-of-field images.

[0015] According to an embodiment of the present application, generating a fused image based on the set of images to be processed includes:

[0016] Calculating the clarity of a clear area of ​​each of the to-be-processed images in the to-be-processed image set;

[0017] Determining a clarity weight of each of the images to be processed based on the clarity of a clear area of ​​each of the images to be processed;

[0018] Based on the clarity weight of each of the images to be processed, the images to be processed are fused to obtain the fused image.

[0019] In the above technical solution, the clarity of the clear area of ​​each image to be processed in the set of images to be processed is calculated, a corresponding weight is assigned to each image according to the clarity, and based on the clarity weight, the images to be processed are fused to obtain a fused image. The fused image is used as a label to train a super depth of field image generation model. The trained super depth of field image generation model can generate super depth of field images based on long exposure images, thereby improving the efficiency and accuracy of generating super depth of field images.

[0020] According to an embodiment of the present application, before generating a fused image based on the set of images to be processed, the method further includes:

[0021] The set of images to be processed is unified in the spatial domain to align the images to be processed in the set of images to be processed.

[0022] In the above technical scheme, before generating a fused image based on the set of images to be processed, the set of images to be processed is unified in the spatial domain to achieve alignment of the spatial positions of the images to be processed. By aligning the images to be processed, adverse effects such as blur or misalignment that may occur in the image fusion process can be reduced, thereby improving the quality of the fused image. The fused image is used as a label to train a super depth of field image generation model. The trained super depth of field image generation model can generate super depth of field images based on long exposure images, thereby improving the efficiency and accuracy of generating super depth of field images.

[0023] In a second aspect, the present application provides a three-dimensional reconstruction method, the method comprising:

[0024] Projecting a preset grating pattern onto a surface of a target object and adjusting the focal length of a target lens to obtain a long exposure image of the target object;

[0025] Inputting the long exposure image into a super depth of field image generation model to obtain a super depth of field two-dimensional image;

[0026] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0027] In the above technical scheme, a preset grating pattern is projected onto the surface of the target object, the focal length of the target lens is adjusted, and the target lens is controlled to capture a long exposure image of the target object, thereby generating a super depth-of-field two-dimensional image. The depth information is calculated based on the deformation of the grating pattern, and the three-dimensional reconstruction model of the target object is generated by fusion, thereby realizing three-dimensional model reconstruction combined with the super depth-of-field two-dimensional image.

[0028] According to one embodiment of the present application, projecting a preset grating pattern onto the surface of a target object and adjusting the focal length of a target lens to obtain a long exposure image includes:

[0029] Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process;

[0030] The target lens is controlled to collect a long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

[0031] In the above technical scheme, a preset periodic grating pattern is projected onto the surface of the target object, the grating pattern is phase-shifted in four steps, the target lens is controlled to continuously and dynamically zoom within the working range, and long exposure images of the target object in each phase shift and frequency state are collected. A set of multi-frequency phase-shifted long exposure images are obtained, which provides a data basis for subsequently generating ultra-depth-of-field two-dimensional images based on the combination of multi-frequency phase-shifted long exposure images and then generating a three-dimensional reconstruction model.

[0032] According to one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0033] The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises:

[0034] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0035] Unwrapping the main phase information to obtain an unwrapped phase;

[0036] The unwrapped phase is converted into a depth value to obtain the depth information.

[0037] In the above technical scheme, multiple super-depth of field phase-shift images are processed by the four-step phase shift method, the main phase information of the pixel points is extracted, and the multi-frequency heterodyne method is used to unwrap the phase step by step, and the unwrapped phase is converted into a depth value, thereby achieving high-precision calculation of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0038] In a third aspect, the present application provides a 3D camera, comprising: an imaging module and a control module, the imaging module comprising a target lens and a super depth of field fusion module, the super depth of field fusion module being used to deploy a super depth of field image generation model, the control module being communicatively connected to the imaging module, and the control module being configured to: execute the super depth of field image generation method as described in the first aspect above.

[0039] In the above technical scheme, the 3D camera includes an imaging module and a control module. The imaging module includes a target lens and a super depth of field fusion module. The control module executes a super depth of field image generation method to generate a super depth of field image. The super depth of field image has higher clarity and realizes a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded. The super depth of field fusion module that deploys the super depth of field image generation model can realize the miniaturization of the 3D camera for mobile devices, and improves the flexibility and efficiency of the super depth of field image generation method.

[0040] According to an embodiment of the present application, the 3D camera further includes: a projection module, the projection module is communicatively connected to the control module, and the control module is further configured as follows:

[0041] Controlling the projection module to project a preset grating pattern onto the surface of the target object, and controlling the target lens of the imaging module to adjust the focal length, so as to obtain a long exposure image of the target object;

[0042] Inputting the long exposure image into the super depth of field image generation model in the super depth of field fusion module to obtain a super depth of field two-dimensional image;

[0043] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0044] In the above technical solution, the 3D camera also includes a projection module. The control module controls the projection module to project a preset grating pattern onto the surface of the target object, dynamically adjusts the focal length of the target lens to obtain a long exposure image of the target object, and inputs the long exposure image into the super depth of field image generation model in the super depth of field fusion module to generate a super depth of field two-dimensional image, and calculates the depth information according to the deformation of the grating pattern, and fuses to generate a three-dimensional reconstruction model of the target object, thereby realizing three-dimensional model reconstruction combined with the super depth of field two-dimensional image.

[0045] According to one embodiment of the present application, the projection module includes a surface structured light projection system, and the control module controls the projection module to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens to obtain a long exposure image of the target object, including:

[0046] Controlling the surface structured light projection system to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process;

[0047] The target lens is controlled to collect a long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

[0048] In the above technical solution, the control module of the 3D camera controls the surface structured light projection system in the projection module, projects a preset periodic grating pattern onto the surface of the target object, performs a four-step phase shift on the grating pattern, controls the target lens to continuously and dynamically adjust the focal length, and obtains long exposure images of the target object under different phase shift and frequency states, which provides a data basis for the subsequent generation of a super-depth of field two-dimensional image based on the reflected light intensity image and then generating a three-dimensional reconstruction model.

[0049] According to an embodiment of the present application, the target lens is a liquid lens, and adjusting the focal length of the target lens includes: adjusting a driving voltage or a driving current of the liquid lens to change the diopter of the liquid lens.

[0050] In the above technical solution, the 3D camera uses a liquid lens as the target lens, and changes the refractive power of the liquid lens by precisely adjusting the driving voltage or current, thereby achieving flexible adjustment of the focal length to adapt to different shooting requirements or environmental conditions.

[0051] According to one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0052] The control module calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including:

[0053] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0054] Unwrapping the main phase information to obtain an unwrapped phase;

[0055] The unwrapped phase is converted into a depth value to obtain the depth information.

[0056] In the above technical solution, the 3D camera processes multiple ultra-depth phase-shifted images through a four-step phase shift method, extracts the main phase information of the pixel points, and uses a multi-frequency heterodyne method to gradually unwrap the phase, and converts the unwrapped phase into a depth value, thereby achieving high-precision calculation of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0057] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0058] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0059] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the super-depth of field image generation method as described in the first aspect above, or to implement the three-dimensional reconstruction method as described in the second aspect above.

[0060] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0061] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0063] Figure 1 is a flowchart of a method for generating a super depth of field image provided by some embodiments of the present application;

[0064] Figure 2 is a schematic diagram of a super depth of field image generation model provided in some embodiments of the present application;

[0065] Figure 3 is a schematic diagram of generating a fused image provided by some embodiments of the present application;

[0066] Figure 4 is a flowchart of a three-dimensional reconstruction method provided in some embodiments of the present application;

[0067] Figure 5 is a schematic diagram of the structure of a 3D camera provided in some embodiments of the present application;

[0068] Figure 6 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application.

[0069] Description of reference numerals:

[0070] 500: 3D camera; 501: imaging module; 502: control module; 503: target lens;

[0071] 504: super depth of field fusion module; 505: projection module; 506: surface structured light projection system;

[0072] 600: electronic device; 601: processor; 602: memory. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0074] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0075] The super depth of field image generation method provided in the embodiment of the present application can be executed by a 3D camera or a functional module or functional entity in a 3D camera that can implement the super depth of field image generation method. The super depth of field image generation method provided in the embodiment of the present application is described below using a 3D camera as an example of the execution subject.

[0076] Figure 1 FIG. 1 is a flow chart of a method for generating a super depth of field image provided by some embodiments of the present application. Figure 1 As shown, the super depth of field image generation method includes: step 110 and step 120.

[0077] Step 110: Acquire a long exposure image.

[0078] It should be noted that the long exposure image is an image obtained by adjusting the focal length of the target lens during the exposure process and continuously maintaining the exposure state, thereby fusing the imaging information of the target object at different focal lengths.

[0079] Step 120: input the long exposure image into a super depth of field image generation model to obtain a super depth of field image; the super depth of field image generation model is trained based on long exposure image samples and a set of images to be processed corresponding to the long exposure image samples.

[0080] Figure 2 Schematic diagram of a super depth of field image generation model provided by some embodiments of the present application. Figure 2 As shown, in some embodiments, the super depth of field image generation model is an end-to-end convolutional neural network model, which is designed to process blurred images with long exposure and panoramic depth information. The input of the model is a blurred image during the shooting process, which contains blurred information in different regions due to insufficient depth of field, and the output is a clear image after super depth of field processing. In order to make the input image consistent with the output image in size, so that each pixel can be accurately mapped and restored in detail, optionally, the U-Net framework is used as the network structure of the super depth of field image generation model. The U-Net framework consists of an encoder and a decoder, and has skip connections to maintain image details. The encoder gradually reduces the spatial size of the input image by stacking multiple convolutional layers (Conv) and pooling layers (Pooling), while continuously increasing the number of feature maps. The convolutional layer extracts local features of the image through convolution operations, while the pooling layer reduces the size of the image and extracts more abstract features. The decoder is used to gradually restore the spatial size of the image, convert the feature map into an output image, and gradually upsample the low-resolution feature map to the original image size through deconvolution or upsampling operations.

[0081] In the process of training the super depth of field image generation model, for each long exposure image sample {F} T , there are corresponding image sets to be processed {G1,G2,…,G n-1 ,G n} T, where n represents the number of images to be processed in the set of images to be processed, and T represents the sample size. The set of images to be processed is composed of a series of images with different depths of field (i.e., images to be processed), which are obtained by adjusting the focal length of the target lens and shooting the same target object multiple times. Each image to be processed is shot at a specific focal length, so at this focal length, a part of the target object may present a clear imaging effect. In the case where the clear area of ​​the image to be processed in the set of images to be processed can cover all the regions of interest of the target object, the fused image generated based on the set of images to be processed can achieve clear imaging of the target object in the entire range, thereby improving the training effect of the super-depth of field image generation model. It can be understood that the region of interest refers to the area where the target object or a specific part of the target object exists.

[0082] In the above technical scheme, an image with a super depth of field effect is generated by acquiring a long exposure image of the target object and inputting it into a super depth of field image generation model. The super depth of field image generation model is trained based on long exposure image samples and a set of images to be processed corresponding to the long exposure image samples, thereby improving the ability of the model to generate super depth of field images, thereby improving the quality of the generated super depth of field images. The super depth of field images are generated based on deep learning technology, which improves the efficiency and accuracy of generating super depth of field images. The model can be applied to 3D cameras to improve the imaging performance and efficiency of 3D cameras.

[0083] In one embodiment of the present application, the super depth of field image generation model is trained by the following steps:

[0084] Adjusting the focal length of the target lens to obtain the set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, each of the images to be processed includes the same target object;

[0085] Controlling the target lens to perform long-range zooming to obtain the long-exposure image sample, wherein the long-exposure image sample fuses the imaging of the target object at all focal lengths within the zoom range corresponding to the image set to be processed;

[0086] Based on the set of images to be processed, generating a fused image;

[0087] The long exposure image sample is used as input, and the fused image is used as a corresponding label to train the super depth of field image generation model.

[0088] It can be understood that, in the process of adjusting the focal length of the target lens, the target lens is controlled to collect the images to be processed of the target object, and the images to be processed are combined to generate a set of images to be processed {G1, G2, ..., G n-1 ,G n} T, where n represents the number of images to be processed and T represents the number of sets of images to be processed. When acquiring each image to be processed, the focal length of the target lens should remain unchanged. It can be understood that the set of images to be processed should contain at least two images to be processed acquired at different focal lengths, i.e., G n Represents the images to be processed at different focal lengths, and these images to be processed all capture the same target object. It should be noted that each image in the set of images to be processed has a clear imaging effect at the corresponding focal length. When the images in the set of images to be processed can cover all the regions of interest of the target object, it is helpful for the subsequently generated super depth of field image to achieve clear imaging in the entire range of the target object.

[0089] In the process of acquiring long-exposure images, the target lens is controlled to zoom over a long distance. The zoom range should include the focal length used when each image to be processed in the image set to be processed was taken. That is, the focal length range of the long-exposure image should be consistent with the range of the image set to be processed, thereby ensuring that the same area can be covered during imaging. The acquired long-exposure image is denoted as {F} T , where T represents the number of different long exposure images, which corresponds to the image set to be processed. It should be noted that when taking long exposure images, the exposure time must be controlled to avoid overexposure of the image.

[0090] It can be understood that the long exposure image sample integrates the imaging of the target object at all focal lengths within the zoom range corresponding to the image set to be processed. In the process of acquiring the long exposure image sample, the focal length variation range of the target lens for long-range zoom covers the focal length used when each image to be processed in the image set to be processed was shot. Optionally, during the exposure process, the target lens is gradually adjusted from the minimum focal length used when each image to be processed was shot to the maximum focal length used when each image to be processed was shot (or gradually adjusted from the maximum focal length to the minimum focal length). During this process, the target lens will pass through the shooting focal length of each image in the image set to be processed. The exposure time should be controlled to avoid overexposure of the image, and the target lens should be controlled to complete the full zoom within the zoom range within the exposure time.

[0091] A fused image is generated based on a set of images to be processed. It can be understood that the fused image combines clear areas in different images to be processed and is an image with higher quality and wider depth of field, and can be considered as a super depth of field image.

[0092] The fused image is used as the label corresponding to the long exposure image sample, that is, the expected output of the super depth of field image generation model, which is compared with the actual output of the super depth of field image generation model (the image output by receiving the long exposure image sample as input) and the loss value is calculated.

[0093] Optionally, the loss value is calculated based on the mean squared error (MSE) and the structure similarity index measure (SSIM). The calculation formula for the mean squared error is as follows:

[0094]

[0095] Among them, L MSE is the mean square error, M represents the total number of pixels in the actual output image of the model, and I output (i) is the i-th pixel value of the model’s actual output image, I fusion (i) is the i-th pixel value of the fused image. It can be understood that the actual output image of the model should be consistent with the size of the fused image, so M can also represent the total number of pixels in the fused image;

[0096] The calculation formula of structural similarity is as follows:

[0097]

[0098] Among them, L SSIM is the structural similarity, μ x and μ y Respectively represent the mean of the pixel values ​​of the two images x and y, and Respectively represent the variance of the pixel values ​​of the two images x and y, σ xy They represent the covariance of the pixel values ​​of the two images x and y respectively, and C1 and C2 are constants set to avoid the denominator being zero.

[0099] Based on the mean square error and structural similarity, the parameters are adjusted according to the back propagation algorithm during the training of the super depth of field image generation model, and the difference between the actual output image of the model and the fused image (expected output) is gradually optimized. The gradient is calculated by the chain rule, and the weights and bias of the super depth of field image generation model are updated according to the loss value. It can be considered that the fully trained super depth of field image generation model has the ability to output super depth of field images, thereby realizing the generation of super depth of field images.

[0100] In the above technical scheme, by adjusting the focal length of the target lens, a set of images to be processed is obtained, the set of images to be processed includes multiple images to be processed that contain the same target object and have different depths of field, the target lens is controlled to perform long-range zoom, and long-exposure image samples that fuse the target object at all focal lengths within the zoom range corresponding to the set of images to be processed are obtained, a fused image is generated based on the set of images to be processed, and a super-depth-of-field image generation model is trained with the long-exposure image samples as input and the fused image as a label. The trained super-depth-of-field image generation model can generate super-depth-of-field images based on long-exposure images, thereby improving the efficiency and accuracy of generating super-depth-of-field images.

[0101] In one embodiment of the present application, the super depth of field image generation model is optimized, including:

[0102] Pruning the super depth of field image generation model;

[0103] Based on the quantization technology, the pruned super depth of field image generation model is optimized.

[0104] In some embodiments, unnecessary or redundant weights and neurons in the super depth of field image generation model are removed by model pruning to reduce the scale and computational complexity of the model, so that the model can run efficiently on resource-constrained devices such as mobile terminals, including: evaluating the importance of each neuron or weight in the super depth of field image generation model, for example, measuring it according to the absolute value of the weight (weights with smaller absolute values ​​contribute less to the network); setting weights, neurons or convolution filters with low importance to zero or completely deleting them; retraining the pruned model to restore accuracy; repeating the pruning process for multiple iterations, pruning at multiple stages, and gradually reducing the size of the super depth of field image generation model. Optionally, weight pruning can be expressed by the following formula:

[0105] W pruned ={w i ||w i >ε}

[0106] Among them, w i is the i-th weight in the super depth of field image generation model, ε is a threshold used to measure the importance of the weight, W pruned The pruned weight set in the super depth of field image generation model.

[0107] In some embodiments, the super depth of field image generation model is further optimized using quantization technology. Model quantization can not only reduce the size of the model, but also accelerate the reasoning process, so that the model can run efficiently on resource-constrained devices such as mobile terminals. Quantization refers to converting high-order floating-point weights into floating-point weights or integer weights with lower digits. The quantization algorithm calculates the range of weight values ​​and maps the range to a smaller integer space. Optionally, weight quantization is expressed by the following formula:

[0108]

[0109] Among them, w orig is the original floating point weight, w min and w max is the minimum and maximum value of the weight, n is the number of digits of the quantized weight, round indicates the rounding operation, and w quant is the quantized floating point weight.

[0110] After optimizing the pruned super-depth image generation model, the super-depth image generation model can be deployed on a mobile device or mobile platform. Based on the hardware characteristics of the mobile device or mobile platform, the dedicated hardware acceleration functions in the mobile device, such as GPU, NPU, etc., can significantly improve the reasoning speed of the model. GPU acceleration improves the efficiency of image processing through parallel computing, while NPU is optimized for neural network reasoning, which can further improve the computing speed and reduce power consumption.

[0111] In the above technical solution, the super depth of field image generation model is optimized, and unnecessary or redundant weights and neurons are removed through model pruning. The high-order floating-point weights are converted into low-order representations using quantization technology to further reduce the model size and accelerate the reasoning process. This allows the super depth of field image generation model to run efficiently on resource-constrained devices such as mobile terminals, while maintaining high accuracy and performance.

[0112] In one embodiment of the present application, generating a fused image based on the set of images to be processed includes:

[0113] Calculating the clarity of a clear area of ​​each of the to-be-processed images in the to-be-processed image set;

[0114] Determining a clarity weight of each of the images to be processed based on the clarity of a clear area of ​​each of the images to be processed;

[0115] Based on the clarity weight of each of the images to be processed, the images to be processed are fused to obtain the fused image.

[0116] Figure 3 is a schematic diagram of generating a fused image provided by some embodiments of the present application. Figure 3 As shown, optionally, calculating the clarity of the clear area of ​​each image to be processed includes:

[0117] Perform a mean filter operation on the image to be processed to generate a corresponding mean filter image. The difference image can be generated by calculating the feature difference between the image to be processed and its mean filter image. The feature difference refers to the difference between the original pixel value and its mean filter pixel value at the same coordinate position, which reflects the degree of change of the pixel in the image relative to its local environment. The difference image can be obtained by the following formula:

[0118] D(x,y)=|I(x,y)-I mean (x,y)|

[0119] Wherein, D(x,y) represents the pixel value of the difference image at the pixel position (x,y), and I(x,y) is the pixel value of the original image to be processed.

[0120] Guided filtering is performed on the difference image to obtain an image area map containing high-frequency information. The clarity of the image to be processed is calculated based on the feature difference in the image area map. The larger the feature difference, the more drastic the pixel change and the richer the image details, so the higher the image clarity.

[0121] Optionally, based on the clarity of the clear area of ​​each image to be processed, the process of determining the clarity weight of each image to be processed is expressed as:

[0122]

[0123] Among them, W i represents the clarity weight of the i-th image to be processed, C i represents the clarity score of the clear area in the i-th image to be processed, and n represents the total number of images to be processed.

[0124] Optionally, based on the clarity weight of each image to be processed, the process of fusing the images to be processed to obtain a fused image is expressed as:

[0125]

[0126] Among them, I fusion Represents the pixel value of the fused image at position (x, y), I i (x, y) represents the pixel value of the i-th image to be processed at position (x, y), W i (x,y) represents the clarity weight of the i-th image to be processed at position (x,y).

[0127] In the above technical solution, the clarity of the clear area of ​​each image to be processed in the set of images to be processed is calculated, a corresponding weight is assigned to each image according to the clarity, and based on the clarity weight, the images to be processed are fused to obtain a fused image. The fused image is used as a label to train a super depth of field image generation model. The trained super depth of field image generation model can generate super depth of field images based on long exposure images, thereby improving the efficiency and accuracy of generating super depth of field images.

[0128] In one embodiment of the present application, before generating a fused image based on the set of images to be processed, the method further includes:

[0129] The set of images to be processed is unified in the spatial domain to align the images to be processed in the set of images to be processed.

[0130] It can be understood that through spatial domain transformation, multiple images to be processed are unified in the spatial domain and aligned. This unified processing in the spatial domain can effectively reduce the misalignment problem caused by differences in viewing angles or changes in shooting conditions during image stitching, and improve the accuracy of geometric docking of multiple images to be processed in the process of generating a fused image.

[0131] Optionally, performing spatial domain unification on the set of images to be processed includes:

[0132] Based on the key point detection algorithm, key point detection is performed on each image to be processed in the image set to be processed to obtain a key point coordinate set corresponding to each image to be processed, and the key point coordinate set includes multiple key point coordinates; the feature descriptor of each key point coordinate is calculated to obtain the key point feature; based on the key point feature, the key points of each image to be processed in the image set to be processed are matched to obtain the matched key points; based on the matched key points, the transformation matrix between the images to be processed is calculated, and the spatial domain transformation is performed on each image to be processed in the image set to be processed based on the transformation matrix to obtain multiple aligned images, thereby completing the spatial domain unification of the image set to be processed.

[0133] In the above technical scheme, before generating a fused image based on the set of images to be processed, the set of images to be processed is unified in the spatial domain to achieve alignment of the spatial positions of the images to be processed. By aligning the images to be processed, adverse effects such as blur or misalignment that may occur in the image fusion process can be reduced, thereby improving the quality of the fused image. The fused image is used as a label to train a super depth of field image generation model. The trained super depth of field image generation model can generate super depth of field images based on long exposure images, thereby improving the efficiency and accuracy of generating super depth of field images.

[0134] Figure 4 is a flow chart of a 3D reconstruction method provided by some embodiments of the present application. Figure 4 As shown, the three-dimensional reconstruction method includes: step 410, step 420 and step 430.

[0135] Step 410: Project a preset grating pattern onto the surface of a target object, and adjust the focal length of a target lens to obtain a long exposure image of the target object.

[0136] In some embodiments, projecting a preset grating pattern onto a surface of a target object and adjusting the focal length of a target lens to obtain a long exposure image includes:

[0137] Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process;

[0138] The target lens is controlled to collect a long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

[0139] Optionally, a set of periodic grating patterns are projected onto the surface of the target object. These grating patterns are usually composed of stripes or bands and are designed with a specific frequency and periodic distribution to form a clearly visible projection pattern on the surface of the target object. The projection light intensity can be expressed by the following formula:

[0140] I(x)=I A +I B cos(wx+δ+Δφ)

[0141] Among them, I(x) represents the projected light intensity of the current row x column, I A represents the mean component, I B represents the amplitude component, w represents the frequency, δ represents the initial phase value, and Δφ represents the phase shift of each projection.

[0142] During the projection process, the grating pattern is phase-shifted in four steps based on the four-step phase shift method, that is, the initial phase of the grating pattern is gradually changed (such as 0 degrees, 90 degrees, 180 degrees and 270 degrees). The orderly change of the phase causes the reflected light intensity of the grating pattern on the surface of the target object to change periodically, thereby forming a series of reflected light intensity images with specific phase differences. Optionally, a multi-frequency heterodyne method is used on the basis of the phase-shifted grating pattern to adjust the frequency of the grating pattern, that is, as the phase gradually changes, the frequency of the grating pattern also changes accordingly. The joint adjustment of phase and frequency makes the characteristics of the light intensity of the grating pattern changing with position more obvious, which can effectively reflect the geometric shape and depth information of the surface of the target object.

[0143] It can be understood that the working range is a focal length range that can cover all regions of interest of the target object.

[0144] Under each phase shift and frequency state, the target lens is controlled to continuously dynamically zoom within the working range to collect long exposure images of the target object. In this way, a set of multi-frequency phase shift long exposure images are obtained. The multi-frequency phase shift long exposure images cover the imaging information of the target object at the current phase, providing a data basis for subsequent image processing and three-dimensional reconstruction. Optionally, the target lens is a liquid lens, and its refractive power is changed by adjusting the driving voltage or driving current of the liquid lens to achieve imaging at different focal lengths.

[0145] In the above technical scheme, a preset periodic grating pattern is projected onto the surface of the target object, the grating pattern is phase-shifted in four steps, the target lens is controlled to continuously and dynamically zoom within the working range, and long exposure images of the target object in each phase shift and frequency state are collected. A set of multi-frequency phase-shifted long exposure images are obtained, which provides a data basis for subsequently generating ultra-depth-of-field two-dimensional images based on the combination of multi-frequency phase-shifted long exposure images and then generating a three-dimensional reconstruction model.

[0146] Step 420: input the long exposure image into a super depth of field image generation model to obtain a super depth of field two-dimensional image;

[0147] Step 430: Calculate the depth information of the surface of the target object according to the deformation of the preset grating pattern, and generate a three-dimensional reconstruction model of the target object based on the depth information and the super depth of field two-dimensional image.

[0148] According to the deformation of the preset grating pattern after being projected on the surface of the target object, the depth information of the object surface can be calculated using a related algorithm. This depth information reflects the distance or height difference of each point on the object surface relative to the reference plane. Combined with the super-depth of field two-dimensional image generated in step 420, which provides detailed information such as the texture and reflection intensity of the object surface, a three-dimensional reconstruction model of the target object is generated. The three-dimensional reconstruction model is a digital representation that accurately reproduces the geometric shape, surface texture and depth information of the target object.

[0149] In the above technical scheme, a preset grating pattern is projected onto the surface of the target object, the focal length of the target lens is adjusted, and the target lens is controlled to capture a long exposure image of the target object, thereby generating a super depth-of-field two-dimensional image. The depth information is calculated based on the deformation of the grating pattern, and the three-dimensional reconstruction model of the target object is generated by fusion, thereby realizing three-dimensional model reconstruction combined with the super depth-of-field two-dimensional image.

[0150] In one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0151] The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises:

[0152] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0153] Unwrapping the main phase information to obtain an unwrapped phase;

[0154] The unwrapped phase is converted into a depth value to obtain the depth information.

[0155] It can be understood that the multiple super-depth phase-shift images refer to a series of images obtained by projecting a preset grating pattern onto the surface of the target object and combining the four-step phase-shift method and the super-depth image generation method during the three-dimensional reconstruction process.

[0156] Optionally, the depth information of the target object surface is calculated, and the main phase information of each pixel point of the multiple super-depth phase-shift images is extracted by calculating the multiple super-depth phase-shift images using a four-step phase-shift formula. The four-step phase-shift formula can be expressed as:

[0157]

[0158] The main phase can be solved as:

[0159]

[0160] Among them, φ(x,y) represents the main phase, also called the main phase value, I1(x,y), I2(x,y), I3(x,y), and I4(x,y) represent the reflected light intensity images of the pixel point (x,y) under different phase shift states, respectively.

[0161] It can be understood that the range of the phase main value calculated by the inverse tangent function is [-π,π]. Although the main phase value is unique within a grating period, the phase value may be repeated in different periods. Therefore, in order to ensure that each point has a unique code when matching the phase value to avoid mismatching, the calculated phase main value needs to be unwrapped.

[0162] Based on the main phase value, unwrapping is performed to determine the period number n of the main phase value, and 2nπ is added to the main phase value to obtain a continuous absolute phase value, which is expressed as:

[0163] Φ(x,y)=φ(x,y)+2πn

[0164] Here, Φ represents the unwrapping phase.

[0165] Optionally, in order to determine the period number n where the main phase value is located, a multi-frequency heterodyne formula is used to calculate multiple super-depth frequency-converted images. The core of the multi-frequency heterodyne method is to use the differences between different frequencies to gradually eliminate the ambiguity of the phase by calculating the phase difference between the frequencies. Specifically, a base frequency is selected as a reference, and the main phase values ​​of other frequencies are converted to the phase space of the base frequency. By comparing the phase change patterns of different frequencies, the true period range of each main phase value can be determined, thereby solving the continuous phase, and then unwrapping step by step from low frequency to high frequency, using lower frequency phase information to unwrap the approximate absolute phase range, and then using higher frequency phase information to further refine the unwrapping results.

[0166] Optionally, the unwrapped phase data is converted into depth values ​​based on a triangulation method.

[0167] In the above technical scheme, multiple super-depth of field phase-shift images are processed by the four-step phase shift method, the main phase information of the pixel points is extracted, and the multi-frequency heterodyne method is used to unwrap the phase step by step, and the unwrapped phase is converted into a depth value, thereby achieving high-precision calculation of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0168] The super depth of field image generation method provided in the embodiment of the present application can be executed by a 3D camera. In the embodiment of the present application, the super depth of field image generation method performed by a 3D camera is taken as an example to illustrate the 3D camera provided in the embodiment of the present application.

[0169] Figure 5 3D camera provided by some embodiments of the present application. Figure 5 As shown, the 3D camera 500 includes: an imaging module 501 and a control module 502, the imaging module 501 includes a target lens 503 and a super depth of field fusion module 504, the super depth of field fusion module 504 is used to deploy a super depth of field image generation model, the control module 502 is communicatively connected to the imaging module 501, and the control module 502 is configured to: execute the above-mentioned super depth of field image generation method.

[0170] Optionally, the super depth of field fusion module 504 may be a mobile device, or a functional module or component on the mobile device.

[0171] In the above technical scheme, the 3D camera includes an imaging module and a control module. The imaging module includes a target lens and a super depth of field fusion module. The control module executes a super depth of field image generation method to generate a super depth of field image. The super depth of field image has higher clarity and realizes a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded. The super depth of field fusion module deployed with the super depth of field image generation model can realize the miniaturization of the 3D camera for mobile devices and improve the flexibility and efficiency of the super depth of field image generation method.

[0172] In one embodiment of the present application, the 3D camera 500 further includes: a projection module 505, wherein the projection module 505 is in communication connection with the control module 502, and the control module 502 is further configured as follows:

[0173] Control the projection module 505 to project a preset grating pattern onto the surface of the target object, and control the target lens 503 of the imaging module 501 to adjust the focal length, so as to obtain a long exposure image of the target object;

[0174] Inputting the long exposure image into the super depth of field image generation model in the super depth of field fusion module 504 to obtain a super depth of field two-dimensional image;

[0175] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0176] In the above technical solution, the 3D camera also includes a projection module. The control module controls the projection module to project a preset grating pattern onto the surface of the target object, dynamically adjusts the focal length of the target lens to obtain a long exposure image of the target object, and inputs the long exposure image into the super depth of field image generation model in the super depth of field fusion module to generate a super depth of field two-dimensional image, and calculates the depth information according to the deformation of the grating pattern, and fuses to generate a three-dimensional reconstruction model of the target object, thereby realizing three-dimensional model reconstruction combined with the super depth of field two-dimensional image.

[0177] In one embodiment of the present application, the projection module 505 includes a surface structured light projection system 506, and the control module 502 controls the projection module 505 to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens to obtain a long exposure image of the target object, including:

[0178] Controlling the surface structured light projection system 506 to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process;

[0179] The target lens 503 is controlled to collect the long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

[0180] In the above technical solution, the control module of the 3D camera controls the surface structured light projection system in the projection module, projects a preset periodic grating pattern onto the surface of the target object, performs a four-step phase shift on the grating pattern, controls the target lens to continuously and dynamically adjust the focal length, and obtains long exposure images of the target object under different phase shift and frequency states, which provides a data basis for the subsequent generation of a super-depth of field two-dimensional image based on the reflected light intensity image and then generating a three-dimensional reconstruction model.

[0181] In one embodiment of the present application, the target lens 503 is a liquid lens, and adjusting the focal length of the target lens 503 includes: adjusting a driving voltage or a driving current of the liquid lens to change the diopter of the liquid lens.

[0182] Adjusting the focal length of a liquid lens is achieved by adjusting the drive voltage or drive current applied to the lens. This adjustment method directly affects the liquid inside the liquid lens, changing its shape or refractive index, and thus changing the diopter of the lens. Diopter is a measure of the focusing ability of a lens and is directly related to the focal length: the shorter the focal length, the higher the diopter; the longer the focal length, the lower the diopter. By precisely controlling the drive voltage or current, the focal length of the liquid lens can be flexibly adjusted to adapt it to different shooting needs or environmental conditions.

[0183] In the above technical solution, the 3D camera uses a liquid lens as the target lens, and changes the refractive power of the liquid lens by precisely adjusting the driving voltage or current, thereby achieving flexible adjustment of the focal length to adapt to different shooting requirements or environmental conditions.

[0184] In one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0185] The control module 502 calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including:

[0186] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0187] Unwrapping the main phase information to obtain an unwrapped phase;

[0188] The unwrapped phase is converted into a depth value to obtain the depth information.

[0189] In the above technical solution, the 3D camera processes multiple ultra-depth phase-shifted images through a four-step phase shift method, extracts the main phase information of the pixel points, and uses a multi-frequency heterodyne method to gradually unwrap the phase, and converts the unwrapped phase into a depth value, thereby achieving high-precision calculation of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0190] The 3D camera in the embodiment of the present application may be an electronic device or a component in an electronic device, and the embodiment of the present application does not make any specific limitation.

[0191] The 3D camera in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0192] The 3D camera provided in the embodiment of the present application can implement each process implemented in the embodiment of the super depth of field image generation method or the embodiment of the three-dimensional reconstruction method. To avoid repetition, they will not be described here.

[0193] In some embodiments, Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, each process of the above-mentioned super-depth-of-field image generation method embodiment or the three-dimensional reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0194] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0195] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the super depth of field image generation method embodiment or the three-dimensional reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0196] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0197] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned super-depth-of-field image generation method or three-dimensional reconstruction method.

[0198] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0199] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned super-depth of field image generation method embodiment or the three-dimensional reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0200] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0201] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0203] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0204] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0205] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for generating a super depth of field image, characterized in that: include: Acquire long exposure images; Inputting the long exposure image into a super depth of field image generation model to obtain a super depth of field image; The super depth of field image generation model is trained based on long exposure image samples and a set of images to be processed corresponding to the long exposure image samples.

2. The method for generating a super depth of field image according to claim 1, characterized in that: The super depth of field image generation model is trained by the following steps: Adjusting the focal length of the target lens to obtain the set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, each of the images to be processed includes the same target object; Controlling the target lens to perform long-range zooming to obtain the long-exposure image sample, wherein the long-exposure image sample fuses the imaging of the target object at all focal lengths within the zoom range corresponding to the image set to be processed; Based on the set of images to be processed, generating a fused image; The long exposure image sample is used as input, and the fused image is used as a corresponding label to train the super depth of field image generation model.

3. The method for generating a super depth of field image according to claim 2, characterized in that: The step of generating a fused image based on the set of images to be processed includes: Calculating the clarity of a clear area of ​​each of the to-be-processed images in the to-be-processed image set; Determining a clarity weight of each of the images to be processed based on the clarity of a clear area of ​​each of the images to be processed; Based on the clarity weight of each of the images to be processed, the images to be processed are fused to obtain the fused image.

4. The method for generating a super depth of field image according to claim 3, characterized in that: Before generating a fused image based on the set of images to be processed, the method further includes: The set of images to be processed is unified in the spatial domain to align the images to be processed in the set of images to be processed.

5. A three-dimensional reconstruction method, characterized in that: include: Projecting a preset grating pattern onto a surface of a target object and adjusting the focal length of a target lens to obtain a long exposure image of the target object; Inputting the long exposure image into a super depth of field image generation model to obtain a super depth of field two-dimensional image; According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

6. The three-dimensional reconstruction method according to claim 5, characterized in that: The method of projecting a preset grating pattern onto the surface of a target object and adjusting the focal length of a target lens to obtain a long exposure image includes: Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process; The target lens is controlled to collect a long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

7. The three-dimensional reconstruction method according to claim 6, characterized in that: The super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images; The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises: According to the intensity changes of the grating images of different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel; Unwrapping the main phase information to obtain an unwrapped phase; The unwrapped phase is converted into a depth value to obtain the depth information.

8. A 3D camera, characterized in that: include: An imaging module and a control module, wherein the imaging module includes a target lens and a super-depth-of-field fusion module, the super-depth-of-field fusion module is used to deploy a super-depth-of-field image generation model, the control module is communicatively connected to the imaging module, and the control module is configured to: execute the super-depth-of-field image generation method as described in any one of claims 1-4.

9. The 3D camera according to claim 8, characterized in that: Also includes: A projection module, wherein the projection module is communicatively connected to the control module, and the control module is further configured as follows: Controlling the projection module to project a preset grating pattern onto the surface of the target object, and controlling the target lens of the imaging module to adjust the focal length, so as to obtain a long exposure image of the target object; Inputting the long exposure image into the super depth of field image generation model in the super depth of field fusion module to obtain a super depth of field two-dimensional image; According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

10. The 3D camera according to claim 9, characterized in that: The projection module includes a surface structured light projection system, and the control module controls the projection module to project a preset grating pattern onto the surface of a target object, and adjusts the focal length of a target lens to obtain a long exposure image of the target object, including: Controlling the surface structured light projection system to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process; The target lens is controlled to collect a long exposure image of the target object in each phase shift and frequency state, and a group of multi-frequency phase shift long exposure images is obtained.

11. The 3D camera according to any one of claims 8 to 10, characterized in that: The target lens is a liquid lens, and adjusting the focal length of the target lens includes: adjusting a driving voltage or a driving current of the liquid lens to change the diopter of the liquid lens.

12. The 3D camera according to claim 10, characterized in that: The super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images; The control module calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including: According to the intensity changes of the grating images of different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel; Unwrapping the main phase information to obtain an unwrapped phase; The unwrapped phase is converted into a depth value to obtain the depth information.

Images