Super-depth-of-field image generation method, three-dimensional reconstruction method and 3D camera

Abstract

The invention discloses a super field depth image generation method, a three-dimensional reconstruction method and a 3D camera, and belongs to the technical field of image processing. The super-depth-of-field image generation method comprises the steps that the focal length of a target lens is adjusted, a to-be-processed image set is acquired, the to-be-processed image set comprises a plurality of to-be-processed images with different depths of field, and each to-be-processed image comprises the same target object; determining key point features of the to-be-processed image set based on a key point detection algorithm; based on the key point features, key points of the to-be-processed image set are matched, and matched key points are obtained; calculating a transformation matrix based on the matched key points, and performing spatial domain transformation on the to-be-processed image set based on the transformation matrix to obtain a plurality of aligned images; and performing depth-of-field fusion on the plurality of aligned images to generate a super-depth-of-field image. The target object is comprehensively and clearly presented, the field depth range of the target lens is expanded, the method can be applied to the 3D camera, and the imaging performance of the 3D camera in various distances and scenes is improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to a method for generating a super-depth-of-field image, a three-dimensional reconstruction method and a 3D camera. Background Art

[0002] As a device for obtaining the three-dimensional shape information of an object, 3D cameras use technologies such as structured light and binocular stereo vision to quickly and accurately capture three-dimensional data, and are widely used in industrial automation, robot navigation and other fields. However, traditional 3D cameras are often limited by the depth of field, and it is difficult to maintain image clarity when the distance of the object changes greatly or the geometric features are complex, which affects the accuracy and integrity of the 3D data. Therefore, breaking through the depth of field limitation and improving the imaging performance of 3D cameras in various distances and scenes has become an urgent problem to be solved. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a super depth of field image generation method, a three-dimensional reconstruction method and a 3D camera to generate a super depth of field image, which has higher clarity, realizes a comprehensive and clear presentation of the target object, expands the depth of field range of the target lens, and can be applied to a 3D camera to improve the imaging performance of the 3D camera in various distances and scenes.

[0004] In a first aspect, the present application provides a method for generating a super depth of field image, the method comprising:

[0005] Adjusting the focal length of the target lens to obtain a set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, each of the images to be processed includes the same target object;

[0006] Determining key point features of the image set to be processed based on a key point detection algorithm;

[0007] Based on the key point features, matching the key points of the image set to be processed to obtain matched key points;

[0008] Based on the matched key points, a transformation matrix is ​​calculated, and based on the transformation matrix, a spatial domain transformation is performed on the set of images to be processed to obtain a plurality of aligned images;

[0009] Perform depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image.

[0010] In the above technical solution, multiple images of different depths of field containing the same target object are obtained by adjusting the focal length of the target lens, the key point features of these images are extracted and matched using a key point detection algorithm, the transformation matrix is ​​calculated to transform the images in the spatial domain, image alignment is achieved, and the aligned images are subjected to depth of field fusion to generate a super depth of field image. The super depth of field image has higher clarity and achieves a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded, and it can be applied to 3D cameras to improve the imaging performance of 3D cameras in various distances and scenes.

[0011] According to an embodiment of the present application, determining key point features of the image set to be processed based on a key point detection algorithm includes:

[0012] Preprocessing each of the to-be-processed images in the to-be-processed image set to obtain each of the to-be-processed images that have been preprocessed;

[0013] Based on a key point detection algorithm, key point detection is performed on each of the pre-processed images to be processed to obtain a key point coordinate set corresponding to each of the images to be processed, wherein the key point coordinate set includes a plurality of key point coordinates;

[0014] Calculate the feature descriptor of each key point coordinate to obtain the key point feature;

[0015] The key point features are optimized to obtain the key point features of the image set to be processed.

[0016] In the above technical scheme, the images to be processed are preprocessed, and the key points in each preprocessed image to be processed are identified and located based on the key point detection algorithm to form a corresponding key point coordinate set, and the feature descriptor of each key point is calculated to obtain the key point feature, and the key point feature is optimized to obtain the key point feature of the image set to be processed. The key point feature of the image set to be processed can be used to match the key points of the image set to be processed, thereby realizing image alignment and depth of field fusion to generate a super depth of field image.

[0017] According to an embodiment of the present application, the key point detection is performed on each of the pre-processed images to be processed based on the key point detection algorithm to obtain a key point coordinate set corresponding to each of the images to be processed, including:

[0018] Down-sampling the pre-processed images to be processed to obtain first resolution images corresponding to the images to be processed;

[0019] Based on a key point detection algorithm, key point detection is performed on the first resolution image corresponding to each of the images to be processed to obtain a key point coordinate set corresponding to each of the first resolution images;

[0020] The key point coordinate sets corresponding to each of the first resolution images are up-sampled to obtain the key point coordinate sets corresponding to each of the to-be-processed images.

[0021] In the above technical scheme, each pre-processed image to be processed is down-sampled to generate each first-resolution image with a lower resolution, which helps to reduce the calculation amount of key point detection and speed up the processing speed. Key point detection is performed on the first-resolution image based on the key point detection algorithm to obtain the key point coordinate set corresponding to the first-resolution image. The key point coordinate set corresponding to each first-resolution image is up-sampled to obtain the key point coordinate set corresponding to each image to be processed, thereby improving the efficiency of key point detection in the process of generating super depth of field images.

[0022] According to an embodiment of the present application, matching the key points of the image set to be processed based on the key point features to obtain matched key points includes:

[0023] Based on the nearest neighbor matching algorithm, the similarity between the key point features of the image set to be processed is calculated to determine the best matching candidate for each key point, and a plurality of preliminary matching key points are obtained;

[0024] Based on a random sampling consistency algorithm, key points that do not meet the spatial geometric consistency among the multiple preliminary matched key points are eliminated to obtain the matched key points.

[0025] In the above technical scheme, by combining the nearest neighbor matching algorithm and the random sampling consistency algorithm, the matching of key points in the processed image set is realized. The similarity between the key point features is calculated based on the nearest neighbor matching algorithm to obtain the preliminary matched key points, and then the random sampling consistency algorithm is used for iterative screening to eliminate the erroneous matching points that do not conform to the spatial geometric consistency to obtain the optimized matched key points, which effectively improves the matching accuracy of the key points. The matched key points are used to calculate the transformation matrix, which can realize image alignment and depth of field fusion to generate super depth of field images.

[0026] According to an embodiment of the present application, the step of calculating a transformation matrix based on the matched key points and performing a spatial domain transformation on the set of images to be processed based on the transformation matrix to obtain a plurality of aligned images includes:

[0027] Sorting the set of images to be processed according to focal length, and setting the image to be processed with the longest or shortest focal length as the first reference image;

[0028] Calculating, based on the matched key points, a transformation matrix of the first reference image and an adjacent image to be processed of the first reference image;

[0029] Based on the transformation matrix, transforming the adjacent image to be processed of the first reference image to obtain a second reference image;

[0030] Repeat the steps of calculating the transformation matrix based on the matched key points and transforming adjacent images to be processed based on the transformation matrix until all images to be processed in the set of images to be processed are transformed into the same coordinate system to obtain multiple aligned images.

[0031] In the above technical scheme, by selecting the reference image by sorting by focal length, gradually calculating the transformation matrix based on the matched key points and performing spatial domain transformation on the adjacent images to be processed, the alignment of multiple images in the image set to be processed is achieved, which effectively addresses the deformation problem caused by the change of lens focal length during the imaging process, improves the stability and reliability of the graphic alignment in the image set to be processed, and provides a basis for the subsequent depth of field fusion of the aligned images.

[0032] According to an embodiment of the present application, performing depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image includes:

[0033] Extracting features from each of the aligned images based on a guided filtering algorithm to generate an image region map containing high-frequency information;

[0034] Calculating a fusion weight of each of the aligned images based on the high-frequency information;

[0035] A fusion operation is performed on all the aligned images based on the fusion weight to generate a super depth of field image.

[0036] In the above technical scheme, the high-frequency information of each aligned image is extracted through a guided filtering algorithm to generate an image area map containing high-frequency information, and then the fusion weight is calculated based on the high-frequency information, and multiple aligned images are fused based on the fusion weight to generate a super depth of field image. The image clarity is evaluated through the high-frequency information, and higher weights are given to clear areas, which effectively supplements the blurred areas and improves the clarity of the super depth of field image in each depth of field area.

[0037] In a second aspect, the present application provides a three-dimensional reconstruction method, comprising:

[0038] Projecting a preset grating pattern onto the surface of a target object, and adjusting the focal length of the target lens, to obtain a set of reflected light intensity images;

[0039] Taking the reflected light intensity image set as the image set to be processed, executing the super depth of field image generation method as described in the first aspect above, to generate a super depth of field two-dimensional image;

[0040] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super depth of field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0041] In the above technical scheme, a preset grating pattern is projected onto the surface of the target object, and a set of reflected light intensity images is obtained by combining the four-step phase shift method and dynamic adjustment of the lens focal length to generate a super-depth-of-field two-dimensional image. The depth information is calculated based on the deformation of the grating pattern, and the three-dimensional reconstruction model of the target object is generated by fusion, thereby realizing the reconstruction of the three-dimensional model in combination with the super-depth-of-field two-dimensional image.

[0042] According to one embodiment of the present application, projecting a preset grating pattern onto the surface of a target object and adjusting the focal length of a target lens to obtain a set of reflected light intensity images includes:

[0043] Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process;

[0044] Controlling the target lens to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images;

[0045] Adjusting the focal length of the target lens to obtain a series of multiple phase-shifted images;

[0046] The series of multiple phase-shifted images constitute the reflected light intensity image set.

[0047] In the above technical scheme, a preset periodic grating pattern is projected onto the surface of the target object, and combined with the four-step phase shift method and dynamic adjustment of the lens focal length, a series of multiple phase shift images covering different phase shift states and depths of field are obtained to form a reflected light intensity image set. By combining the surface structured light projection system and the dynamic focusing capability of the liquid lens, the depth of field range of the camera is effectively expanded, providing a data basis for the subsequent generation of ultra-depth-of-field two-dimensional images based on the reflected light intensity image set and then generating a three-dimensional reconstruction model.

[0048] According to one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0049] The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises:

[0050] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0051] Unwrapping the main phase information to obtain an unwrapped phase;

[0052] The unwrapped phase is converted into a depth value to obtain the depth information.

[0053] In the above technical scheme, multiple super-depth of field phase-shift images are processed by the four-step phase shift method, the main phase information of the pixel points is extracted, and the phase is unwrapped step by step using the multi-frequency heterodyne method, and the unwrapped phase is converted into a depth value, thereby achieving high-resolution and high-precision acquisition of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0054] In a third aspect, the present application provides a 3D camera, comprising: an imaging module and a control module, wherein the imaging module comprises a target lens, the control module is communicatively connected to the imaging module, and the control module is configured to: execute the super-depth-of-field image generation method as described in the first aspect above.

[0055] In the above technical solution, the 3D camera includes an imaging module and a control module. The imaging module includes a target lens. The control module executes a super depth-of-field image generation method to generate a super depth-of-field image. The super depth-of-field image has higher clarity and achieves a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded, and the imaging performance of the 3D camera in various distances and scenes is improved.

[0056] According to an embodiment of the present application, the 3D camera further includes: a projection module, the projection module is communicatively connected to the control module, and the control module is further configured as follows:

[0057] Controlling the projection module to project a preset grating pattern onto the surface of the target object, adjusting the focal length of the target lens, and controlling the target lens to collect a set of reflected light intensity images;

[0058] Taking the reflected light intensity image set as the image set to be processed, executing the super-depth-of-field image generation method according to any one of claims 1 to 6 to generate a super-depth-of-field two-dimensional image;

[0059] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0060] In the above technical solution, the 3D camera also includes a projection module. The control module controls the projection module to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens to obtain a set of reflected light intensity images, thereby generating a super-depth-of-field two-dimensional image, and calculating the depth information based on the deformation of the grating pattern, and fusing it to generate a three-dimensional reconstruction model of the target object, thereby realizing three-dimensional model reconstruction combined with the super-depth-of-field two-dimensional image.

[0061] According to one embodiment of the present application, the projection module includes a surface structured light projection system, and the control module controls the projection module to project a preset grating pattern onto the surface of the target object, and at the same time adjusts the focal length of the target lens, and controls the target lens to collect a reflected light intensity image set, including:

[0062] Controlling the surface structured light projection system to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process;

[0063] Controlling the target lens to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images;

[0064] Adjusting the focal length of the target lens to obtain a series of multiple phase-shifted images;

[0065] The series of multiple phase-shifted images constitute the reflected light intensity image set.

[0066] In the above technical scheme, the 3D camera projection module includes a surface structured light projection system. The control module controls the surface structured light projection system to project a preset periodic grating pattern onto the surface of the target object, and performs a four-step phase shift on the grating pattern, dynamically adjusts the focal length of the target lens, and obtains a series of multiple phase-shifted images covering different phase shift states and depths of field, forming a set of reflected light intensity images. By combining the dynamic focusing capability of the surface structured light projection system and the liquid lens, the depth of field range of the camera is effectively expanded, providing a data basis for the subsequent generation of ultra-depth-of-field two-dimensional images based on the reflected light intensity images and then generating a three-dimensional reconstruction model.

[0067] According to an embodiment of the present application, the target lens is a liquid lens, and adjusting the focal length of the target lens includes: adjusting a driving voltage or a driving current of the liquid lens to change the diopter of the liquid lens.

[0068] In the above technical solution, the 3D camera uses a liquid lens as the target lens, and changes the refractive power of the liquid lens by precisely adjusting the driving voltage or current, thereby achieving flexible adjustment of the focal length to adapt to different shooting requirements or environmental conditions.

[0069] According to one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0070] The control module calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including:

[0071] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0072] Unwrapping the main phase information to obtain an unwrapped phase;

[0073] The unwrapped phase is converted into a depth value to obtain the depth information.

[0074] In the above technical solution, the 3D camera control module processes multiple ultra-depth of field phase-shift images through a four-step phase shift method, extracts the main phase information of the pixel points, and uses a multi-frequency heterodyne method to gradually unwrap the phase, and converts the unwrapped phase into a depth value, thereby achieving high-resolution and high-precision acquisition of the depth information on the surface of the target object, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and realizing high-precision three-dimensional reconstruction in complex scenes.

[0075] In a fourth aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0076] In a fifth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0077] In a sixth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the super-depth of field image generation method as described in the first aspect above, or to implement the three-dimensional reconstruction method as described in the second aspect above.

[0078] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the super-depth-of-field image generation method as described in the first aspect above, or implements the three-dimensional reconstruction method as described in the second aspect above.

[0079] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0081] Figure 1 is a flowchart of a method for generating a super depth of field image provided by some embodiments of the present application;

[0082] Figure 2 is a schematic diagram of key point detection provided by some embodiments of the present application;

[0083] Figure 3 is a schematic diagram of sorting a set of images to be processed provided by some embodiments of the present application;

[0084] Figure 4 A schematic diagram of the effect of super depth of field imaging results provided by some embodiments of the present application;

[0085] Figure 5 is a flowchart of a three-dimensional reconstruction method provided in some embodiments of the present application;

[0086] Figure 6 is a schematic diagram of the structure of a 3D camera provided in some embodiments of the present application;

[0087] Figure 7 It is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application.

[0088] Description of reference numerals:

[0089] 600: 3D camera; 601: imaging module; 602: control module;

[0090] 603: target lens; 604: projection module; 605: surface structured light projection system;

[0091] 700: electronic device; 701: processor; 702: memory. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0093] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. The exemplary first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0094] The following, in conjunction with the accompanying drawings, describes in detail the super-depth-of-field image generation method, the three-dimensional reconstruction method and the 3D camera provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0095] The super depth of field image generation method provided in the embodiment of the present application can be executed by a 3D camera or a functional module or functional entity in a 3D camera that can implement the super depth of field image generation method. The super depth of field image generation method provided in the embodiment of the present application is described below using a 3D camera as an example of the execution subject.

[0096] Figure 1 FIG. 1 is a flow chart of a method for generating a super depth of field image provided by some embodiments of the present application. Figure 1 As shown, the super depth of field image generation method includes: step 110, step 120, step 130, step 140 and step 150.

[0097] Step 110: adjust the focal length of the target lens to obtain a set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, and each of the images to be processed includes the same target object.

[0098] The focal length of the target lens refers to the distance from the optical center of the target lens to the imaging plane (such as the photosensitive element of a 3D camera). The depth of field refers to the range of distances in front and behind the target object in which a clear image can be obtained in the imaging system. By adjusting the focal length of the target lens, the depth of field of the captured image can be affected. It should be noted that the depth of field is also affected by other factors, including: aperture, shooting distance, image quality requirements, etc.

[0099] Optionally, in the process of adjusting the focal length of the target lens, images of the target object are captured, and the captured images are integrated to obtain a set of images to be processed. The set of images to be processed includes at least two images taken at different focal lengths, and these images are all for the same target object. It should be noted that each image in the set of images to be processed has a clear imaging effect at the corresponding focal length. When the images in the set of images to be processed can cover all areas of interest of the target object, it is helpful for the subsequently generated super depth of field image to achieve clear imaging in the full range of the target object.

[0100] Step 120: Determine key point features of the image set to be processed based on a key point detection algorithm.

[0101] The key point features of the to-be-processed image set refer to a set of key point features of each to-be-processed image in the to-be-processed image set.

[0102] Optionally, a key point (also called a feature point) refers to a specific location in an image with significant changes, such as a corner point, an edge, or a texture mutation point. A corner point refers to the place where two or more edges in an image intersect, usually marking the location where the structure in the image changes dramatically, an edge refers to the location where the grayscale value in the image changes dramatically, usually corresponding to the outline of the target object, and a texture mutation point refers to the location where the grayscale value or color changes suddenly in the texture area of ​​the image. The key point feature is used to describe the image features near the key point. Optionally, the key point feature is a feature descriptor of the key point. A descriptor is a feature vector containing local information of the key point.

[0103] It can be understood that the key points can remain stable under different image transformation conditions (such as rotation, scaling and translation). Even if the image undergoes transformation, the key points can still be detected. In different images, for the same object or scene, the key points can be repeatedly detected.

[0104] Step 130: Match the key points of the image set to be processed based on the key point features to obtain matched key points.

[0105] It can be understood that by matching the key point features of each image to be processed in the image set to determine the matching key points, this process usually involves calculating the similarity between the key point features and determining the correspondence between them according to certain matching criteria. The matched key points represent the same physical position or feature in different images, which is an important basis for subsequent image alignment and depth of field fusion processes.

[0106] Step 140: Calculate a transformation matrix based on the matched key points, and perform spatial domain transformation on the set of images to be processed based on the transformation matrix to obtain a plurality of aligned images.

[0107] It is understandable that based on the matched key points, the transformation matrix required to transform different images into a common coordinate system can be calculated. The transformation matrix generally includes geometric transformation parameters such as rotation, translation and scaling. Based on the calculated transformation matrix, the image set to be processed is transformed in the spatial domain to achieve spatial domain unification, including applying the transformation matrix to each pixel in the image, thereby transforming the image from the original coordinate system to the new coordinate system, so that the key points of each image to be processed in the image set to be processed overlap in spatial position, and a set of aligned images is obtained.

[0108] Optionally, an image to be processed in the set of images to be processed is selected as a reference image, and used as a reference standard for aligning other images. Based on the matched key point set, the transformation matrix between the other images in the set of images to be processed and the reference image is calculated, and the other images in the set of images to be processed are transformed in the spatial domain based on the transformation matrix.

[0109] It can be understood that the spatial domain is unified between the aligned images, which can be expressed as:

[0110]

[0111] Where f represents the spatial coordinate transformation, (x, y) is the position of a point in the reference image space domain, is the position of a point in the spatial domain of the image to be transformed, Equal to (x, y), I2(x, y) represents the pixel value of the reference image at (x, y), Represents the pixel value at (x, y) after the image to be transformed. In order to achieve alignment, it is necessary to satisfy The spatial domain unification process is actually to find the optimal spatial coordinate transformation relationship f that satisfies the minimization problem of the above formula through a mathematical optimization algorithm.

[0112] Specifically, the steps for optimizing and solving the optimal spatial coordinate transformation relationship are:

[0113] Select a suitable transformation model based on the spatial relationship of the image set to be processed. For example, if the changes between images mainly involve rotation, scaling and translation, then select the affine transformation model; if there are more complex perspective effect changes, then select the homography transformation model. In some embodiments, the homography transformation model is selected, and its matrix form is expressed as:

[0114]

[0115] Among them, h 11 、h 12 、h 21 、h 22 Indicates image rotation, scaling, etc., h 13、h 23 Indicates the image offset, h 31 、h 32 Represents perspective related information, x and y represent the position of a point in the image to be transformed, and Represents the position of the corresponding point in the reference image. Based on the matched key points, the parameters of the selected transformation model are solved by the least squares method or other optimization methods. The goal is to minimize the error between the corresponding matching points in the transformed image, so as to obtain an accurate transformation matrix. When the transformation model is a homography, at least four pairs of matched key point pairs (consisting of key points and matched key points) are usually required to solve the eight free parameters in the matrix. Through spatial domain transformation, each image is aligned in the coordinate system of the reference image, so that multiple images are consistent under the same spatial reference. This alignment process can effectively reduce the misalignment problem caused by perspective differences during image stitching, and ensure the precise docking of multiple images in terms of geometric structure.

[0116] Step 150: Perform depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image.

[0117] Optionally, by analyzing the depth of field information of multiple aligned images, the clear area is determined, and weights are assigned to the pixels of each aligned image according to the clarity. The images are then finely fused using weighted averaging or other fusion algorithms. Through the above fusion process, a full-range clear super-depth of field image is generated, which combines the depth of field of multiple aligned images.

[0118] In the above technical solution, multiple images of different depths of field containing the same target object are obtained by adjusting the focal length of the target lens, the key point features of these images are extracted and matched using a key point detection algorithm, the transformation matrix is ​​calculated to transform the images in the spatial domain, image alignment is achieved, and the aligned images are subjected to depth of field fusion to generate a super depth of field image. The super depth of field image has higher clarity and achieves a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded, and it can be applied to 3D cameras to improve the imaging performance of 3D cameras in various distances and scenes.

[0119] In one embodiment of the present application, determining key point features of the image set to be processed based on a key point detection algorithm includes:

[0120] Preprocessing each of the to-be-processed images in the to-be-processed image set to obtain each of the to-be-processed images that have been preprocessed;

[0121] Based on a key point detection algorithm, key point detection is performed on each of the pre-processed images to be processed to obtain a key point coordinate set corresponding to each of the images to be processed, wherein the key point coordinate set includes a plurality of key point coordinates;

[0122] Calculate the feature descriptor of each key point coordinate to obtain the key point feature;

[0123] The key point features are optimized to obtain the key point features of the image set to be processed.

[0124] Optionally, the preprocessing of the image to be processed includes steps such as denoising and histogram equalization to improve the contrast and detail expression of the image. In the case where the image to be processed is not a grayscale image, grayscale processing is required before the preprocessing of the image to be processed to convert the image to a grayscale image.

[0125] Key point detection is performed on each preprocessed image to be processed. The key point detection algorithm can be a scale-invariant feature transform (SIFT) algorithm, a speeded up robust features (SURF) algorithm, or an oriented fast and rotated brief (ORB) algorithm. The selected key point detection algorithm is applied to each preprocessed image to be processed to identify the key points in the image. For each image to be processed, the detected key point coordinates P are recorded respectively. i (x i ,y i ), and form the corresponding key point coordinate set, where x i and i Represents the coordinates of the key point in the two-dimensional coordinate system.

[0126] Optionally, the key point feature is a feature descriptor of the key point coordinates. The feature descriptor contains a feature vector of the local information of the key point, which is used to describe the image features near the key point coordinates. The calculation process of the feature descriptor includes the extraction of image gradient, direction and texture information, so that the feature descriptor has good discrimination and matching between different images. i , calculate its feature descriptor d i :

[0127] D i =descriptor(IMG,P i )

[0128] Among them, descriptor (IMG, P i) represents the key point P in the grayscale image IMG i The process of calculating feature descriptors.

[0129] In some embodiments, key point features are optimized, including operations such as normalization, feature scaling, and direction correction, to improve the accuracy and robustness of matching between key points. Through optimization processing, the impact of external factors such as lighting changes and noise interference on key point features can be reduced, so that the key point features maintain the maximum similarity between different images.

[0130] In the above technical scheme, the images to be processed are preprocessed, and the key points in each preprocessed image to be processed are identified and located based on the key point detection algorithm to form a corresponding key point coordinate set, and the feature descriptor of each key point is calculated to obtain the key point feature, and the key point feature is optimized to obtain the key point feature of the image set to be processed. The key point feature of the image set to be processed can be used to match the key points of the image set to be processed, thereby realizing image alignment and depth of field fusion to generate a super depth of field image.

[0131] In one embodiment of the present application, the key point detection is performed on each of the pre-processed images to be processed based on the key point detection algorithm to obtain a key point coordinate set corresponding to each of the images to be processed, including:

[0132] Down-sampling the pre-processed images to be processed to obtain first resolution images corresponding to the images to be processed;

[0133] Based on a key point detection algorithm, key point detection is performed on the first resolution image corresponding to each of the images to be processed to obtain a key point coordinate set corresponding to each of the first resolution images;

[0134] The key point coordinate sets corresponding to each of the first resolution images are up-sampled to obtain the key point coordinate sets corresponding to each of the to-be-processed images.

[0135] Figure 2 Schematic diagram of key point detection provided by some embodiments of the present application. Figure 2 As shown, downsampling is performed on each pre-processed image to be processed to reduce the image resolution, and a first resolution image corresponding to each image to be processed is generated. The key point detection algorithm is applied to the first resolution image corresponding to each image to be processed, and key point detection is performed to generate a key point coordinate set corresponding to each first resolution image. An upsampling operation is performed on the key point coordinate set corresponding to each first resolution image to generate a key point coordinate set of the original resolution, that is, a key point coordinate set corresponding to each image to be processed. It can be understood that the resolution of the first resolution image is smaller than that of the image to be processed, which greatly reduces the number of image pixels, thereby shortening the key point detection time.

[0136] In other words, for images with higher resolution to be processed, when calculating feature point coordinates and descriptors, the image can be downsampled to reduce its resolution; after generating low-resolution feature points, it can be upsampled to restore the resolution to the original size, and then the feature descriptors can be calculated to generate results with high-resolution feature information. This method can not only speed up feature calculation and improve the working efficiency of the target lens, but also ensure that the final feature information has high accuracy and clarity, thereby maintaining the quality of image analysis.

[0137] Downsample each image to be processed IMG to generate a corresponding first resolution image IMG low :

[0138] IMG low = resize(IMG, size(IMG) × λ)

[0139] Among them, λ is the image reduction ratio, size(IMG) is the image resolution acquisition function, and resize is the image resolution adjustment function.

[0140] For each first resolution image IMG low Key point detection is performed to obtain a key point coordinate set corresponding to the first resolution image, and the key point coordinate set corresponding to each first resolution image is upsampled, that is, multiple key point coordinates P low Mapping from the coordinate system of the first-resolution image back to the coordinate system of the original-resolution image:

[0141] P=resize(P low ,size(IMG)÷λ)

[0142] By upsampling, multiple key point coordinates P at the original resolution are obtained, that is, the key point coordinates P corresponding to each image to be processed, and finally a key point coordinate set corresponding to each image to be processed is generated.

[0143] In the above technical scheme, each pre-processed image to be processed is down-sampled to generate each first-resolution image with a lower resolution, which helps to reduce the calculation amount of key point detection and speed up the processing speed. Key point detection is performed on the first-resolution image based on the key point detection algorithm to obtain the key point coordinate set corresponding to the first-resolution image. The key point coordinate set corresponding to each first-resolution image is up-sampled to obtain the key point coordinate set corresponding to each image to be processed, thereby improving the efficiency of key point detection in the process of generating super depth of field images.

[0144] In one embodiment of the present application, matching the key points of the image set to be processed based on the key point features to obtain matched key points includes:

[0145] Based on the nearest neighbor matching algorithm, the similarity between the key point features of the image set to be processed is calculated to determine the best matching candidate for each key point, and a plurality of preliminary matching key points are obtained;

[0146] Based on a random sampling consistency algorithm, key points that do not meet the spatial geometric consistency among the multiple preliminary matched key points are eliminated to obtain the matched key points.

[0147] In order to achieve more robust and accurate feature matching based on key points, in some embodiments, a multi-step matching optimization strategy is adopted. First, the key point features extracted from the image are preliminarily matched using the nearest neighbor matching algorithm. In this stage, the similarities between the key point features are compared, and one or more best matching candidate points are found for each key point.

[0148] Exemplarily, based on the nearest neighbor matching algorithm, the similarity between the key point features of the image set to be processed is calculated, which can be expressed as:

[0149] Match pairs =NN_match(Descriptors i ,Descriptors j )

[0150] Among them, Descriptors i and Descriptors j They represent the key point features of two images to be processed in the image set to be processed. The NN_match function is used to calculate the nearest neighbor matching result Match between the key point features of the two images to be processed. pairs .

[0151] Since the initial match usually contains some wrong matching points, further optimization is needed to improve the reliability and accuracy of the match. pairs Contains multiple matching point pairs. For each key point, there may be multiple best matching candidates, that is, one key point may form a preliminary match with multiple other key points in another image to be processed.

[0152] In order to further optimize the preliminary matching results, the random sample consensus algorithm (RANSAC) is introduced. This algorithm randomly selects several groups of matching point pairs (consisting of key points and key points of preliminary matching) from the preliminary matching results iteratively, calculates the corresponding model parameters, and evaluates the spatial geometric consistency of these matching point pairs with the model. By screening the matching point pairs, the wrong matching points that do not meet the spatial geometric constraints can be effectively eliminated, thereby improving the matching accuracy.

[0153] In the RANSAC optimization process, the matching optimization step can be expressed as:

[0154] Optimized matches =RANSAC(Match pairs ,model)

[0155] Among them, the RANSAC function selects matching point pairs consistent with the assumed model based on the principles of random sampling and consistency verification. Model represents the model parameters used to evaluate matching consistency, such as affine transformation or perspective transformation model, etc. Optimized matches is the result of the random sampling consensus algorithm. Optionally, model is the homography transformation model.

[0156] Specifically, in each iteration of the RANSAC algorithm, a set of minimum matching point pairs (corresponding to the homography transformation model, 4 point pairs are required to estimate the parameters of the homography transformation model) are randomly selected from the preliminary matching results, and the error between all preliminary matching point pairs and the hypothetical model is calculated. The error can be measured based on the pixel coordinate difference (exemplarily, the Euclidean distance). By setting an error threshold, matching point pairs with errors less than or equal to the threshold are regarded as inliers, and matching point pairs with errors greater than the threshold are regarded as outliers.

[0157] The number of inliers that are consistent with the current hypothetical model is counted. The more inliers there are, the more consistent the model is with the actual data. The above iterative process is repeated until the model parameters containing the most consistent matching points are found. The matching point pairs selected by the RANSAC algorithm are used as the final optimized matching results, thereby achieving further optimization of the preliminary matching results.

[0158] Through this matching optimization scheme, it is possible to effectively screen out those matching points with real spatial correlation from a large number of preliminary matching points, thereby improving the accuracy and robustness of feature matching.

[0159] In the above technical scheme, by combining the nearest neighbor matching algorithm and the random sampling consistency algorithm, the matching of key points in the processed image set is realized. The similarity between the key point features is calculated based on the nearest neighbor matching algorithm to obtain the preliminary matched key points, and then the random sampling consistency algorithm is used for iterative screening to eliminate the erroneous matching points that do not conform to the spatial geometric consistency to obtain the optimized matched key points, which effectively improves the matching accuracy of the key points. The matched key points are used to calculate the transformation matrix, which can realize image alignment and depth of field fusion to generate super depth of field images.

[0160] Since the liquid lens is used to change the focal length for imaging, images with large focal length differences may be severely deformed, resulting in the inability to achieve accurate alignment when using key points for image registration due to the quality and quantity of key points. Therefore, in order to achieve more accurate image alignment under such conditions, a one-by-one registration method can be used.

[0161] In one embodiment of the present application, the step of calculating a transformation matrix based on the matched key points and performing a spatial domain transformation on the set of images to be processed based on the transformation matrix to obtain a plurality of aligned images includes:

[0162] Sorting the set of images to be processed according to focal length, and setting the image to be processed with the longest or shortest focal length as the first reference image;

[0163] Calculating, based on the matched key points, a transformation matrix of the first reference image and an adjacent image to be processed of the first reference image;

[0164] Based on the transformation matrix, transforming the adjacent image to be processed of the first reference image to obtain a second reference image;

[0165] Repeat the steps of calculating the transformation matrix based on the matched key points and transforming adjacent images to be processed based on the transformation matrix until all images to be processed in the set of images to be processed are transformed into the same coordinate system to obtain multiple aligned images.

[0166] Figure 3 FIG. 1 is a schematic diagram of sorting a set of images to be processed provided by some embodiments of the present application. Figure 3 As shown, optionally, in order to cope with the deformation caused by the focal length change during the liquid lens imaging process, the image set to be processed is sorted according to the length of the focal length, which helps to make the focal length change smoother and reduce the error accumulation during each transformation, thereby improving the stability and reliability of the transformation during the step-by-step alignment process.

[0167] Optionally, an image to be processed with the longest or shortest focal length is selected from the sorted set of images to be processed as the first reference image. Such an image can usually provide more stable reference information and help reduce initial alignment errors.

[0168] Based on the matched key points, the transformation matrix between the first reference image and its adjacent image to be processed is calculated, and more accurate transformation parameters can be obtained when the key points have good distribution and stability.

[0169] Using the calculated transformation matrix, the adjacent image to be processed is transformed into the same coordinate system as the first reference image. This process can be expressed as follows:

[0170] IMG′=H 1,2 ·IMG

[0171] Where IMG represents the pixel coordinates of the adjacent image to be processed of the first reference image, IMG′ represents the corresponding point of the adjacent image to be processed of the first reference image in the coordinate system of the first reference image, and H 1,2 is the transformation matrix;

[0172] The adjacent image to be processed of the transformed first reference image is set as the new second reference image, and the above process is repeated to calculate the transformation matrix between the new second reference image and its adjacent image to be processed, and perform spatial domain transformation. This process is continued, each time using the previous aligned image as the new reference image, calculating the transformation matrix between it and the next adjacent image to be processed, and transforming the adjacent images to be processed until all images to be processed are transformed into the same coordinate system.

[0173] By adopting a step-by-step alignment strategy, it is helpful to reduce the alignment error under conditions where the imaging focal length changes greatly. Through step-by-step alignment and transformation, the overall deformation of the image set to be processed can be controlled, and the accuracy of the geometric docking of the final image set can be improved. This can not only maximize the use of the local similarities between adjacent images to be processed, but also improve the accuracy and robustness of the overall registration.

[0174] In the above technical scheme, by selecting the reference image by sorting by focal length, gradually calculating the transformation matrix based on the matched key points and performing spatial domain transformation on the adjacent images to be processed, the alignment of multiple images in the image set to be processed is achieved, which effectively addresses the deformation problem caused by the change of lens focal length during the imaging process, improves the stability and reliability of the graphic alignment in the image set to be processed, and provides a basis for the subsequent depth of field fusion of the aligned images.

[0175] In one embodiment of the present application, the step of performing depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image includes:

[0176] Extracting features from each of the aligned images based on a guided filtering algorithm to generate an image region map containing high-frequency information;

[0177] Calculating a fusion weight of each of the aligned images based on the high-frequency information;

[0178] A fusion operation is performed on all the aligned images based on the fusion weight to generate a super depth of field image.

[0179] Optionally, feature extraction is performed on each aligned image based on a guided filtering algorithm. The guided filtering algorithm uses local gradient information of the image to maintain the edges and details of the image while suppressing noise, thereby generating an image region map containing high-frequency information. The high-frequency information reflects the details and edges in the image and is an important indicator of image clarity. The fusion weight of each aligned image is calculated based on the above high-frequency information.

[0180] Optionally, a mean filtering operation is performed on the aligned image to generate a corresponding mean filtered image. A difference image can be generated by calculating the characteristic difference between the aligned image and its mean filtered image. The difference image reflects the degree of change of the pixel in the image relative to its local environment. The characteristic difference refers to the difference between the original pixel value and its mean filtered pixel value at the same coordinate position, reflecting the degree of change of the pixel in the image relative to its local environment. The difference image can be obtained by the following formula:

[0181] D(x,y)=|I(x,y)-I mean (x,y)|

[0182] Where D(x,y) represents the pixel value of the difference image at the pixel position (x,y), and I(x,y) is the pixel value of the original aligned image.

[0183] Guided filtering is performed on the difference image to obtain an image area map containing high-frequency information. The clarity of the aligned image is calculated based on the feature difference in the image area map. The larger the feature difference, the more drastic the pixel change and the richer the image details, so the higher the image clarity.

[0184] The corresponding fusion weights are calculated based on the high-frequency information, giving higher weights to clear areas and lower weights to blurred areas. After the fusion weights of each aligned image are calculated, multiple aligned images are fused based on the fusion weights.

[0185] Specifically, the weight map is used to prioritize the clear areas in each aligned image, while the blurred areas are supplemented by the corresponding areas in other clear aligned images, so that the final super depth of field image maintains good clarity in all depth of field areas. The fusion process can be expressed as:

[0186]

[0187] Among them, I fusion Represents the pixel value of the fused super depth of field image at position (x, y), I i (x,y) is the pixel value of the ith aligned image at position (x,y), N is the total number of aligned images, and W i (x,y) is the fusion weight of the i-th aligned image at position (x,y).

[0188] Figure 4 A schematic diagram of the effect of super depth of field imaging results provided in some embodiments of the present application. Figure 4 The image on the left is a continuous shot of images at different focal lengths, and the image on the right is the super depth of field imaging result.

[0189] In the above technical scheme, the high-frequency information of each aligned image is extracted through a guided filtering algorithm to generate an image area map containing high-frequency information, and then the fusion weight is calculated based on the high-frequency information, and multiple aligned images are fused based on the fusion weight to generate a super depth of field image. The image clarity is evaluated through the high-frequency information, and higher weights are given to clear areas, which effectively supplements the blurred areas and improves the clarity of the super depth of field image in each depth of field area.

[0190] Figure 5 is a flow chart of a 3D reconstruction method provided by some embodiments of the present application. Figure 5 As shown, the three-dimensional reconstruction method includes: step 510, step 520 and step 530.

[0191] Step 510: Project a preset grating pattern onto the surface of the target object, and adjust the focal length of the target lens to obtain a set of reflected light intensity images.

[0192] In some embodiments, projecting a preset grating pattern onto the surface of the target object and adjusting the focal length of the target lens to obtain a set of reflected light intensity images includes:

[0193] Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process;

[0194] Controlling the target lens to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images;

[0195] Adjusting the focal length of the target lens to obtain a series of multiple phase-shifted images;

[0196] The series of multiple phase-shifted images constitute the reflected light intensity image set.

[0197] Optionally, a set of periodic grating patterns are projected onto the surface of the target object. These grating patterns are usually composed of stripes or bands and are designed with a specific frequency and periodic distribution to form a clearly visible projection pattern on the surface of the target object. The projection light intensity can be expressed by the following formula:

[0198] I(x)=I A +I B cos(wx+δ+Δφ)

[0199] Where I(x) represents the projected light intensity at position x, I A represents the mean component, I B represents the amplitude component, w represents the frequency, δ represents the initial phase value, and Δφ represents the phase shift of each projection.

[0200] During the projection process, the grating pattern is phase-shifted in four steps, the initial phase of the grating pattern is gradually changed (such as 0 degrees, 90 degrees, 180 degrees and 270 degrees), and the frequency of the grating pattern is changed at the same time. The gradual change of phase and frequency causes the grating pattern to produce different light intensity changes on the surface of the object, forming a set of reflected light intensity images with specific phase differences, which helps to better analyze the deformation of the grating pattern on the surface of the object. Optionally, the frequency of the grating pattern should select a higher frequency pattern under the premise that the camera can decode it. The target lens is controlled to collect the reflected light intensity image of the target object at each phase shift state and frequency to obtain a set of multiple phase shift images.

[0201] Adjust the focal length of the target lens to obtain a series of multiple phase-shifted images. Optionally, the target lens is a liquid lens. By adjusting the driving voltage or driving current of the liquid lens, its refractive power is changed to achieve imaging at different focal lengths, so that the phase-shifted images collected at different focal lengths can cover all detail areas of the object surface from near to far.

[0202] A series of multiple phase-shifted images obtained through the above steps constitute a set of reflected light intensity images, which include not only images in different phase-shifted states, but also images collected at different focal lengths, providing detailed information on the surface of the target object at different depths of field.

[0203] In the above technical scheme, a preset periodic grating pattern is projected onto the surface of the target object, and combined with the four-step phase shift method and dynamic adjustment of the lens focal length, a series of multiple phase shift images covering different phase shift states and depths of field are obtained to form a reflected light intensity image set. By combining the surface structured light projection system and the dynamic focusing capability of the liquid lens, the depth of field range of the camera is effectively expanded, providing a data basis for the subsequent generation of ultra-depth-of-field two-dimensional images based on the reflected light intensity image set and then generating a three-dimensional reconstruction model.

[0204] Step 520: using the reflected light intensity image set as the image set to be processed, executing the above-mentioned super depth of field image generation method to generate a super depth of field two-dimensional image;

[0205] Step 530: Calculate the depth information of the surface of the target object according to the deformation of the preset grating pattern, and generate a three-dimensional reconstruction model of the target object based on the depth information and the super-depth-of-field two-dimensional image.

[0206] According to the deformation of the preset grating pattern after being projected on the surface of the target object, the depth information of the object surface can be calculated using a related algorithm. This depth information reflects the distance or height difference of each point on the object surface relative to the reference plane. Combined with the super-depth of field two-dimensional image generated in step 520, which provides detailed information such as the texture and reflection intensity of the object surface, a three-dimensional reconstruction model of the target object is generated. The three-dimensional reconstruction model is a digital representation that accurately reproduces the geometric shape, surface texture and depth information of the target object.

[0207] In the above technical scheme, a preset grating pattern is projected onto the surface of the target object, and a set of reflected light intensity images is obtained by combining the four-step phase shift method and dynamic adjustment of the lens focal length to generate a super-depth-of-field two-dimensional image. The depth information is calculated based on the deformation of the grating pattern, and the three-dimensional reconstruction model of the target object is generated by fusion, thereby realizing the reconstruction of the three-dimensional model in combination with the super-depth-of-field two-dimensional image.

[0208] In one embodiment of the present application, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0209] The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises:

[0210] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0211] Unwrapping the main phase information to obtain an unwrapped phase;

[0212] The unwrapped phase is converted into a depth value to obtain the depth information.

[0213] It can be understood that the multiple super-depth phase-shift images refer to a series of images obtained by projecting a preset grating pattern onto the surface of the target object and combining the four-step phase-shift method and the super-depth image generation method during the three-dimensional reconstruction process.

[0214] Optionally, the depth information of the target object surface is calculated, and the main phase information of each pixel point of the multiple super-depth phase-shift images is extracted by calculating the multiple super-depth phase-shift images using a four-step phase-shift formula. The four-step phase-shift formula can be expressed as:

[0215]

[0216]

[0217] The main phase can be solved

[0218]

[0219] Among them, φ(x,y) represents the main phase, also called the main phase value, I1(x,y), I2(x,y), I3(x,y), and I4(x,y) represent the reflected light intensity images of the pixel point (x,y) under different phase shift states, respectively.

[0220] It can be understood that the range of the phase main value calculated by the inverse tangent function is [-π,π]. Although the main phase value is unique within a grating period, the phase value may be repeated in different periods. Therefore, in order to ensure that each point has a unique code when matching the phase value to avoid mismatching, the calculated phase main value needs to be unwrapped.

[0221] Based on the main phase value, unwrapping is performed to determine the period number n of the main phase value, and 2nπ is added to the main phase value to obtain a continuous absolute phase value, which is expressed as:

[0222] Φ(x,y)=φ(x,y)+2πn

[0223] where φ represents the unwrapping phase.

[0224] Optionally, in order to determine the period number n where the main phase value is located, a multi-frequency heterodyne formula is used to calculate multiple super-depth frequency-converted images. The core of the multi-frequency heterodyne method is to use the differences between different frequencies to gradually eliminate the ambiguity of the phase by calculating the phase difference between the frequencies.

[0225] Specifically, a base frequency is selected as a reference, and the main phase values ​​of other frequencies are converted to the phase space of the base frequency. By comparing the phase change patterns of different frequencies, the real period range of each main phase value can be determined, so as to solve the continuous phase, and then unwrap from low frequency to high frequency step by step, using the phase information of lower frequencies to unwrap the approximate absolute phase range. Subsequently, the phase information of higher frequencies is used to further refine the unwrapping results.

[0226] For example, for two groups of grating patterns with different frequencies, the frequencies are set to be f1 and f2, the wavelengths are set to be λ1 and λ2, and the corresponding main phase values ​​are φ1 and φ2. The main phase values ​​of other frequencies are converted to the phase space of the base frequency, expressed as:

[0227]

[0228] The period number n where the main phase value is obtained is expressed as:

[0229]

[0230] Here, florr[·] means rounding down.

[0231] Optionally, the unwrapped phase data is converted into depth values ​​based on a triangulation method.

[0232] In the above technical scheme, multiple super-depth of field phase-shift images are processed by the four-step phase shift method, the main phase information of the pixel points is extracted, and the phase is unwrapped step by step using the multi-frequency heterodyne method, and the unwrapped phase is converted into a depth value, thereby achieving high-resolution and high-precision acquisition of the depth information of the target object surface, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and providing a data basis for three-dimensional model reconstruction.

[0233] It should be understood that the 3D reconstruction method described in some embodiments of the present application is only for illustrative purposes, and the specific implementation is not limited thereto. The 3D reconstruction process may also use other appropriate methods or technical means, and no limitation is made to the specific solution herein.

[0234] The super depth of field image generation method provided in the embodiment of the present application can be executed by a 3D camera. In the embodiment of the present application, the super depth of field image generation method performed by a 3D camera is taken as an example to illustrate the 3D camera provided in the embodiment of the present application.

[0235] Figure 6 3D camera provided by some embodiments of the present application. Figure 6 As shown, the 3D camera 600 includes: an imaging module 601 and a control module 602, the imaging module includes a target lens 603, the control module 602 is communicatively connected to the imaging module 601, and the control module 602 is configured to: execute the super depth of field image generation method provided by the above-mentioned method embodiments.

[0236] In the above technical solution, the 3D camera includes an imaging module and a control module. The imaging module includes a target lens. The control module executes a super depth-of-field image generation method to generate a super depth-of-field image. The super depth-of-field image has higher clarity and achieves a comprehensive and clear presentation of the target object. Compared with single exposure shooting, the depth of field range of the target lens is expanded, and the imaging performance of the 3D camera in various distances and scenes is improved.

[0237] Optionally, the 3D camera 600 further includes: a projection module 604, the projection module 604 is communicatively connected with the control module 602, and the control module 602 is further configured as follows:

[0238] Control the projection module 604 to project a preset grating pattern onto the surface of the target object, and adjust the focal length of the target lens 603 to control the target lens 603 to collect a set of reflected light intensity images;

[0239] Taking the reflected light intensity image set as the image set to be processed, executing the super-depth-of-field image generation method according to any one of claims 1 to 6 to generate a super-depth-of-field two-dimensional image;

[0240] According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

[0241] In the above technical solution, the 3D camera also includes a projection module. The control module controls the projection module to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens to obtain a set of reflected light intensity images, thereby generating a super-depth-of-field two-dimensional image, and calculating the depth information based on the deformation of the grating pattern, and fusing it to generate a three-dimensional reconstruction model of the target object, thereby realizing three-dimensional model reconstruction combined with the super-depth-of-field two-dimensional image.

[0242] Optionally, the projection module 604 includes a surface structured light projection system 605, and the control module 602 controls the projection module 604 to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens, and controls the target lens to collect a reflected light intensity image set, including:

[0243] Controlling the surface structured light projection system 605 to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process;

[0244] Controlling the target lens 603 to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images;

[0245] Adjusting the focal length of the target lens 603 to obtain a series of multiple phase-shifted images;

[0246] The series of multiple phase-shifted images constitute the reflected light intensity image set.

[0247] In the above technical scheme, the 3D camera projection module includes a surface structured light projection system. The control module controls the surface structured light projection system to project a preset periodic grating pattern onto the surface of the target object, and performs a four-step phase shift on the grating pattern, dynamically adjusts the focal length of the target lens, and obtains a series of multiple phase-shifted images covering different phase shift states and depths of field, forming a set of reflected light intensity images. By combining the dynamic focusing capability of the surface structured light projection system and the liquid lens, the depth of field range of the camera is effectively expanded, providing a data basis for the subsequent generation of ultra-depth-of-field two-dimensional images based on the reflected light intensity images and then generating a three-dimensional reconstruction model.

[0248] Optionally, the target lens 603 is a liquid lens, and adjusting the focal length of the target lens includes: adjusting a driving voltage or a driving current of the liquid lens to change the refractive power of the liquid lens.

[0249] Adjusting the focal length of a liquid lens is achieved by adjusting the drive voltage or drive current applied to the lens. This adjustment method directly affects the liquid inside the liquid lens, changing its shape or refractive index, and thus changing the diopter of the lens. Diopter is a measure of the focusing ability of a lens and is directly related to the focal length: the shorter the focal length, the higher the diopter; the longer the focal length, the lower the diopter. By precisely controlling the drive voltage or current, the focal length of the liquid lens can be flexibly adjusted to adapt it to different shooting needs or environmental conditions.

[0250] In the above technical solution, the 3D camera uses a liquid lens as the target lens, and changes the refractive power of the liquid lens by precisely adjusting the driving voltage or current, thereby achieving flexible adjustment of the focal length to adapt to different shooting requirements or environmental conditions.

[0251] Optionally, the super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images;

[0252] The control module calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including:

[0253] According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel;

[0254] Unwrapping the main phase information to obtain an unwrapped phase;

[0255] The unwrapped phase is converted into a depth value to obtain the depth information.

[0256] In the above technical solution, the 3D camera control module processes multiple ultra-depth of field phase-shift images through a four-step phase shift method, extracts the main phase information of the pixel points, and uses a multi-frequency heterodyne method to gradually unwrap the phase, and converts the unwrapped phase into a depth value, thereby achieving high-resolution and high-precision acquisition of the depth information on the surface of the target object, effectively solving the phase ambiguity problem, improving the accuracy and reliability of depth calculation, and realizing high-precision three-dimensional reconstruction in complex scenes.

[0257] The 3D camera provided in the embodiment of the present application can implement each process implemented in the embodiment of the super depth of field image generation method or the embodiment of the three-dimensional reconstruction method. To avoid repetition, they will not be described here.

[0258] In some embodiments, Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the program is executed by the processor 701, each process of the above-mentioned super-depth-of-field image generation method embodiment or the three-dimensional reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0259] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0260] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned super-depth of field image generation method embodiment or the three-dimensional reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0261] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0262] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned super-depth of field image generation method or three-dimensional reconstruction method when executed by a processor.

[0263] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0264] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned super-depth of field image generation method embodiment or the three-dimensional reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0265] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0266] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0267] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0268] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0269] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0270] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for generating a super depth of field image, characterized in that: include: Adjusting the focal length of the target lens to obtain a set of images to be processed, wherein the set of images to be processed includes a plurality of images to be processed with different depths of field, each of the images to be processed includes the same target object; Determining key point features of the image set to be processed based on a key point detection algorithm; Based on the key point features, matching the key points of the image set to be processed to obtain matched key points; Based on the matched key points, a transformation matrix is ​​calculated, and based on the transformation matrix, a spatial domain transformation is performed on the set of images to be processed to obtain a plurality of aligned images; Perform depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image.

2. The method for generating a super depth of field image according to claim 1, characterized in that: The determining of key point features of the image set to be processed based on a key point detection algorithm includes: Preprocessing each of the to-be-processed images in the to-be-processed image set to obtain each of the to-be-processed images that have been preprocessed; Based on a key point detection algorithm, key point detection is performed on each of the pre-processed images to be processed to obtain a key point coordinate set corresponding to each of the images to be processed, wherein the key point coordinate set includes a plurality of key point coordinates; Calculate the feature descriptor of each key point coordinate to obtain the key point feature; The key point features are optimized to obtain the key point features of the image set to be processed.

3. The method for generating a super depth of field image according to claim 2, characterized in that: The key point detection algorithm is used to detect key points of the pre-processed images to obtain a set of key point coordinates corresponding to the images to be processed, including: Down-sampling the pre-processed images to be processed to obtain first resolution images corresponding to the images to be processed; Based on a key point detection algorithm, key point detection is performed on the first resolution image corresponding to each of the images to be processed to obtain a key point coordinate set corresponding to each of the first resolution images; The key point coordinate sets corresponding to each of the first resolution images are up-sampled to obtain the key point coordinate sets corresponding to each of the to-be-processed images.

4. The method for generating a super depth of field image according to claim 1, characterized in that: The step of matching the key points of the image set to be processed based on the key point features to obtain matched key points includes: Based on the nearest neighbor matching algorithm, the similarity between the key point features of the image set to be processed is calculated to determine the best matching candidate for each key point, and a plurality of preliminary matching key points are obtained; Based on a random sampling consistency algorithm, key points that do not meet the spatial geometric consistency among the multiple preliminary matched key points are eliminated to obtain the matched key points.

5. The method for generating a super depth of field image according to claim 1, characterized in that: The step of calculating a transformation matrix based on the matched key points and performing spatial domain transformation on the set of images to be processed based on the transformation matrix to obtain a plurality of aligned images includes: Sorting the set of images to be processed according to focal length, and setting the image to be processed with the longest or shortest focal length as the first reference image; Calculating, based on the matched key points, a transformation matrix of the first reference image and an adjacent image to be processed of the first reference image; Based on the transformation matrix, transforming the adjacent image to be processed of the first reference image to obtain a second reference image; Repeat the steps of calculating the transformation matrix based on the matched key points and transforming adjacent images to be processed based on the transformation matrix until all images to be processed in the set of images to be processed are transformed into the same coordinate system to obtain multiple aligned images.

6. The method for generating a super depth of field image according to claim 1, characterized in that: The step of performing depth-of-field fusion on the multiple aligned images to generate a super-depth-of-field image includes: Extracting features from each of the aligned images based on a guided filtering algorithm to generate an image region map containing high-frequency information; Calculating a fusion weight of each of the aligned images based on the high-frequency information; A fusion operation is performed on all the aligned images based on the fusion weight to generate a super depth of field image.

7. A three-dimensional reconstruction method, characterized in that: include: Projecting a preset grating pattern onto the surface of a target object, and adjusting the focal length of the target lens, to obtain a set of reflected light intensity images; Taking the reflected light intensity image set as the image set to be processed, executing the super-depth-of-field image generation method according to any one of claims 1 to 6 to generate a super-depth-of-field two-dimensional image; According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super depth of field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The method of projecting a preset grating pattern onto the surface of a target object and adjusting the focal length of a target lens to obtain a reflected light intensity image set includes: Projecting a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shifts on the grating patterns during the projection process; Controlling the target lens to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images; Adjusting the focal length of the target lens to obtain a series of multiple phase-shifted images; The series of multiple phase-shifted images constitute the reflected light intensity image set.

9. The three-dimensional reconstruction method according to claim 8, characterized in that: The super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images; The step of calculating the depth information of the surface of the target object according to the deformation of the preset grating pattern comprises: According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel; Unwrapping the main phase information to obtain an unwrapped phase; The unwrapped phase is converted into a depth value to obtain the depth information.

10. A 3D camera, characterized in that: include: An imaging module and a control module, wherein the imaging module includes a target lens, the control module is communicatively connected to the imaging module, and the control module is configured to: execute the super-depth of field image generation method as described in any one of claims 1-6.

11. The 3D camera according to claim 10, characterized in that: Also includes: A projection module, wherein the projection module is communicatively connected to the control module, and the control module is further configured as follows: Controlling the projection module to project a preset grating pattern onto the surface of the target object, adjusting the focal length of the target lens, and controlling the target lens to collect a set of reflected light intensity images; Taking the reflected light intensity image set as the image set to be processed, executing the super-depth-of-field image generation method according to any one of claims 1 to 6 to generate a super-depth-of-field two-dimensional image; According to the deformation of the preset grating pattern, the depth information of the surface of the target object is calculated, and based on the depth information and the super-depth-of-field two-dimensional image, a three-dimensional reconstruction model of the target object is generated.

12. The 3D camera according to claim 11, characterized in that: The projection module includes a surface structured light projection system, and the control module controls the projection module to project a preset grating pattern onto the surface of the target object, and adjusts the focal length of the target lens, and controls the target lens to collect a reflected light intensity image set, including: Controlling the surface structured light projection system to project a set of periodic grating patterns onto the surface of the target object, and performing four-step phase shift on the grating patterns during the projection process; Controlling the target lens to collect the reflected light intensity image of the target object in each phase shift state to obtain a set of multiple phase shift images; Adjusting the focal length of the target lens to obtain a series of multiple phase-shifted images; The series of multiple phase-shifted images constitute the reflected light intensity image set.

13. The 3D camera according to any one of claims 10 to 12, characterized in that: The target lens is a liquid lens, and adjusting the focal length of the target lens includes: adjusting a driving voltage or a driving current of the liquid lens to change the diopter of the liquid lens.

14. The 3D camera according to claim 12, characterized in that: The super-depth-of-field two-dimensional image is a plurality of super-depth-of-field phase-shift images; The control module calculates the depth information of the surface of the target object according to the deformation of the preset grating pattern, including: According to the intensity changes of the grating images with different phases, a four-step phase shift method is used to calculate the multiple super-depth phase shift images to extract the main phase information of each pixel; Unwrapping the main phase information to obtain an unwrapped phase; The unwrapped phase is converted into a depth value to obtain the depth information.

Images