Image processing method and apparatus, and electronic device

Abstract

Provided in the embodiments of the present application are an image processing method and apparatus, and an electronic device. The method comprises: obtaining images collected by an image collection device in different illumination modes when a lens is located at different optical axis positions; on the basis of the obtained images, generating a super depth-of-field depth map; on the basis of the obtained images, generating a photometric stereo normal vector map, and on the basis of the photometric stereo normal vector map, generating a photometric stereo depth map; and on the basis of a photometric stereo normal vector in the photometric stereo normal vector map, correcting depth information in the same depth-of-field region in the super depth-of-field depth map, and on the basis of a photometric stereo depth value in the photometric stereo depth map, correcting depth information in a low-texture region in the super depth-of-field depth map. Since detailed depth information in the same depth-of-field region can be corrected by means of a photometric stereo normal vector, and missing depth information in a low-texture region can be restored by means of a photometric stereo depth value, a super depth-of-field depth map which can fully reflect the depth details of a collection field of view can be obtained.

Description

Image processing method, device and electronic equipment

[0001] The present application claims priority to the Chinese patent application No. 202410853550.1, filed on June 27, 2024, and entitled "Image processing method, device and electronic equipment", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, in particular to an image processing method, device and electronic equipment. BACKGROUND

[0003] The super-depth-of-field microscope is a microscope with extremely high resolution and depth perception capability, which can perform 3D reconstruction on the observed target and output the 3D depth map of the collection field of view. Specifically, the super-depth-of-field microscope can collect multiple images when the lens is located at different positions on the optical axis, synthesize these images according to the difference in sharpness between the images, and generate a super-depth-of-field depth map, which reconstructs the 3D depth map of the collection field of view.

[0004] However, for a small object, it can be completely located within the depth of field of the super-depth-of-field microscope due to its small size. In this case, even if the position of the lens on the optical axis is changed, the object is still completely located within the depth of field of the super-depth-of-field microscope, and the images collected by the super-depth-of-field microscope are all images in which the entire object is clear. The difference in sharpness between the collected images is not significant, and thus the super-depth-of-field depth map generated according to the difference in sharpness between the images cannot well reflect the depth details of the object. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device and electronic equipment to obtain a super-depth-of-field depth map that can well reflect the depth details of the collection field of view. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide an image processing method, which comprises:

[0007] obtaining images collected by an image collection device when the lens is located at different positions on the optical axis under different illumination modes;

[0008] generating a super-depth-of-field depth map according to the difference in sharpness between the images in a first image group corresponding to different illumination modes, wherein the first image group comprises images collected when the lens is located at different positions on the optical axis under the same illumination mode;

[0009] generating a photometric stereo normal vector map based on the obtained images;

[0010] generate a photometric stereo depth map based on the photometric stereo normal map;

[0011] correct the depth information in a same depth region in the hyper-z depth map according to a photometric stereo normal in the photometric stereo normal map, and correct the depth information in a low-texture region in the hyper-z depth map according to a photometric stereo depth value in the photometric stereo depth map, wherein the same depth region is a region in the hyper-z depth map in a non-perpendicular region that is in a same depth range, a hyper-z normal corresponding to a hyper-z depth value in the non-perpendicular region is not perpendicular to the hyper-z depth map, and a variation parameter corresponding to a hyper-z depth value in the low-texture region is less than a preset variation parameter threshold.

[0012] Optionally, the generating the photometric stereo normal map based on the obtained images comprises:

[0013] for each first image group, fusing images in the first image group to generate a corresponding all-in-focus image of the first image group, and generating a photometric stereo normal map based on light intensity differences between the corresponding all-in-focus images of the first image groups;

[0014] or,

[0015] for each second image group, generating a corresponding photometric stereo normal map of the second image group based on light intensity differences between images in the second image group, and fusing the photometric stereo normal maps corresponding to the second image groups at different optical axis positions to generate the photometric stereo normal map, wherein the second image group comprises images in the obtained images that are captured when the lens is at a same optical axis position.

[0016] Optionally, the correcting the depth information in the same depth region in the hyper-z depth map according to the photometric stereo normal in the photometric stereo normal map, and the correcting the depth information in the low-texture region in the hyper-z depth map according to the photometric stereo depth value in the photometric stereo depth map comprise:

[0017] for a same depth region, obtaining a hyper-z normal corresponding to each hyper-z depth value in the same depth region, selecting a target normal corresponding to the hyper-z depth value from the obtained hyper-z normal and a photometric stereo normal corresponding to the hyper-z depth value in a first region, and correcting the hyper-z depth value in the same depth region based on the selected target normal, wherein the first region is a region in the photometric stereo normal map corresponding to the same depth region;

[0018] for a low-texture region, correcting a hyper-z depth value in the low-texture region based on a photometric stereo depth value in a second region, wherein the second region is a region in the photometric stereo depth map corresponding to the low-texture region

[0019] Optionally, after the obtaining the images captured by the image acquisition device at different positions of the optical axis under different illumination modes, the method further comprises:

[0020] obtaining a confidence map corresponding to the super-depth-of-field depth map, wherein a pixel value in the confidence map represents a confidence of a super-depth-of-field depth value in the super-depth-of-field depth map;

[0021] determining a same-depth region and a low-texture region in the super-depth-of-field depth map according to the confidence map, wherein a confidence of a super-depth-of-field depth value in the same-depth region is greater than a preset confidence threshold, and a confidence of a super-depth-of-field depth value in the low-texture region is not greater than the preset confidence threshold.

[0022] Optionally, the correcting the super-depth-of-field depth value in the same-depth region based on the selected target normal vector comprises:

[0023] determining a target super-depth-of-field depth value that minimizes a first difference, wherein the first difference is a difference between a gradient corresponding to the super-depth-of-field depth value in the same-depth region and a tilt parameter corresponding to a target normal vector corresponding to the super-depth-of-field depth value, and the tilt parameter corresponding to the target normal vector is a ratio between a component of the target normal vector in an image coordinate system and a component of the target normal vector in an optical axis direction;

[0024] correcting the super-depth-of-field depth value in the same-depth region to the target super-depth-of-field depth value.

[0025] Optionally, the determining the target super-depth-of-field depth value that minimizes the first difference comprises:

[0026] determining the target super-depth-of-field depth value that minimizes the first difference according to the following expression:

[0027] wherein L represents a loss function between a gradient corresponding to the super-depth-of-field depth value in the same-depth region and a tilt parameter corresponding to a target normal vector corresponding to the super-depth-of-field depth value, D x represents the gradient of the super-depth-of-field depth value at position (i,j) in the same-depth region in the x-axis direction, and D y(i,j) represents a gradient of the hyper-deep depth value at position (i,j) in the same depth-of-field region in the y-axis direction, N(i,j,0) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the x-axis direction, N(i,j,1) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the y-axis direction, and N(i,j,2) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the z-axis direction.

[0028] Optionally, before the target hyper-deep depth value that makes the first difference minimum is determined, the method further comprises:

[0029] For each hyper-deep depth value in the same depth-of-field region, a difference value between a gradient corresponding to the hyper-deep depth value and a tilt parameter corresponding to the target normal vector corresponding to the hyper-deep depth value is obtained.

[0030] Hyper-deep depth values with a difference value greater than a preset difference threshold or a gradient corresponding to the hyper-deep depth value greater than a preset gradient threshold are removed from the same depth-of-field region.

[0031] Optionally, the target normal vector corresponding to the hyper-deep depth value is selected from the obtained hyper-deep normal vector and the photometric stereo normal vector corresponding to the hyper-deep depth value in the first region, comprising:

[0032] A first included angle between the obtained hyper-deep normal vector and a first direction is calculated, and a second included angle between the photometric stereo normal vector corresponding to the hyper-deep depth value in the first region and the first direction is calculated, wherein the first direction is a direction along the optical axis direction towards the lens;

[0033] If the first included angle is greater than the second included angle, the obtained hyper-deep normal vector is determined as the target normal vector corresponding to the hyper-deep depth value;

[0034] If the first included angle is not greater than the second included angle, the photometric stereo normal vector corresponding to the hyper-deep depth value in the first region is determined as the target normal vector corresponding to the hyper-deep depth value.

[0035] Optionally, the hyper-deep depth value in the low-texture region is corrected based on the photometric stereo depth value in the second region, comprising:

[0036] The low-texture region is morphologically dilated to obtain a third region;

[0037] A difference region between the third region and the low-texture region is determined, and a depth mean and a depth standard deviation of the hyper-deep depth values in the difference region are calculated.

[0038] calculating a depth difference between the hyper-z depth values included in each depth pair, and determining a target depth pair corresponding to a depth difference greater than the depth standard deviation, wherein each depth pair includes one hyper-z depth value greater than the depth mean and one hyper-z depth value not greater than the depth mean in the difference region;

[0039] calculating a photometric stereo depth difference between the hyper-z depth values in the target depth pair and the corresponding target photometric stereo depth values in the photometric stereo depth map;

[0040] determining a size information corresponding to the target photometric stereo depth value as a ratio between the depth difference and the photometric stereo depth difference corresponding to the target depth pair;

[0041] correcting the hyper-z depth values in the low-texture region to the photometric stereo depth values in the second region according to the size information corresponding to the target photometric stereo depth value.

[0042] Optionally, the generating the hyper-z depth map according to the clarity difference between the images in the first image group corresponding to different illumination modes comprises:

[0043] generating a depth map corresponding to each first image group according to the clarity difference between the images in the first image group; and fusing the depth maps corresponding to the first image groups to obtain the hyper-z depth map.

[0044] or,

[0045] generating a clarity evaluation map corresponding to each image in the first image group by performing clarity analysis on each image in the first image group; stacking the clarity evaluation maps corresponding to each image according to the optical axis positions of the images to obtain a multi-dimensional clarity evaluation volume corresponding to the first image group; fusing the multi-dimensional clarity evaluation volumes corresponding to the first image groups to obtain a fused clarity evaluation volume; and generating the hyper-z depth map according to the fused clarity evaluation volume and a preset hyper-z reconstruction algorithm.

[0046] Optionally, the fusing the depth maps corresponding to the first image groups to obtain the hyper-z depth map comprises:

[0047] For each depth value in each depth map, a first depth range overlapping with an image depth range to which the depth value belongs is determined from a voting body depth range of a three-dimensional voting body corresponding to the depth map, and a unit position corresponding to the depth value in the three-dimensional voting body is voted in the first depth range with the confidence of the depth value as a voting value, wherein the each depth map is a depth map corresponding to each first image group, the unit position of the three-dimensional voting body in the horizontal direction one-to-one corresponds to the depth value in the depth map, and the three-dimensional voting body is preset with a plurality of voting body depth ranges in the height direction;

[0048] The voting results of the same unit positions and the same voting body depth ranges in the three-dimensional voting bodies corresponding to the depth maps are summed to obtain a target three-dimensional voting body;

[0049] For each unit position of the target three-dimensional voting body, a voting body depth range corresponding to a maximum voting result of the unit position is determined as a second depth range, and a weighting coefficient of a depth value corresponding to the unit position in each depth map is determined according to a voting value of the depth value in the second depth range;

[0050] The depth values in each depth map are weighted and calculated according to the weighting coefficients of the depth values in the depth map to obtain a hyper-z depth map.

[0051] Optionally, the photometric stereo depth map is generated based on the photometric stereo normal vector map, and the method comprises:

[0052] The photometric stereo normal vector map is reduced by a first preset multiple to obtain a first normal vector map;

[0053] A photometric stereo depth map that minimizes a second difference is determined as a first depth map, wherein the second difference is a difference between a gradient corresponding to a photometric stereo depth value in the photometric stereo depth map and an inclination parameter corresponding to a photometric stereo normal vector in the first normal vector map;

[0054] The first depth map is enlarged by a second preset multiple to obtain a second depth map, and the photometric stereo normal vector map is reduced to the same resolution as the second depth map to obtain a second normal vector map;

[0055] randomly determining a region of a preset size in the second depth map as an updating region, determining a partial normal vector graph corresponding to a target region in the second normal vector graph, wherein the target region is a center region or an edge region, determining a target photometric stereo depth value when a third difference is minimum, wherein the third difference is a difference between a gradient of a photometric stereo depth value in the target region and an inclination parameter of a photometric stereo normal vector in the partial normal vector graph, correcting the photometric stereo depth value in the target region to the target photometric stereo depth value to obtain a corrected second depth map, updating the second depth map to the corrected second depth map, and returning to the step of randomly determining a region of a preset size in the second depth map as an updating region until a quantity of the determined updating regions is equal to a preset quantity;

[0056] if a resolution of the second normal vector graph is less than a resolution of the photometric stereo normal vector graph, updating the first depth map to the corrected second depth map, resetting the quantity of the determined updating regions, and returning to the step of magnifying the first depth map by a second preset multiple to obtain a second depth map, and magnifying the photometric stereo normal vector graph to the same resolution as the second depth map to obtain a second normal vector graph;

[0057] if the resolution of the second normal vector graph is equal to the resolution of the photometric stereo normal vector graph, taking the corrected second depth map as a photometric stereo depth map.

[0058] Optionally, the determining the photometric stereo depth map when the second difference is minimum comprises:

[0059] determining the photometric stereo depth map when the second difference is minimum according to the following expression: J(z) = ∫∫((z x -p) 2 +(z y -q) 2 )dxdy+λ(∑|z x |+|z y |)

[0060] wherein J(z) represents a loss function between a gradient of a photometric stereo depth value in a photometric stereo depth map and an inclination parameter of a photometric stereo normal vector in a first normal vector graph, z x represents a gradient of the photometric stereo depth value in an x-axis direction, z y represents a gradient of the photometric stereo depth value in a y-axis direction, p = n x / n z , q = n y / n z , and n xdenotes the component of the photometric stereo normal vector in the x-axis direction, n y denotes the component of the photometric stereo normal vector in the y-axis direction, n z denotes the component of the photometric stereo normal vector in the z-axis direction, λ denotes the regularization constraint strength, |z x |z y |z x |z y |z

[0061] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0062] an image acquisition module configured to obtain images captured by an image acquisition device when a lens is located at different optical axis positions under different illumination modes;

[0063] an extended depth of field depth map generation module configured to generate an extended depth of field depth map according to differences in sharpness between images in a first image group corresponding to different illumination modes, wherein the first image group comprises images captured when the lens is located at different optical axis positions under the same illumination mode;

[0064] a normal vector map generation module configured to generate a photometric stereo normal vector map based on the obtained images;

[0065] a photometric stereo depth map generation module configured to generate a photometric stereo depth map based on the photometric stereo normal vector map;

[0066] a correction module configured to correct depth information in a same depth of field region in the extended depth of field depth map according to a photometric stereo normal vector in the photometric stereo normal vector map, and correct depth information in a low texture region in the extended depth of field depth map according to a photometric stereo depth value in the photometric stereo depth map, wherein the same depth of field region is a region in the extended depth of field depth map that is in a same depth of field range and is in a non-perpendicular region, an extended depth of field normal vector corresponding to an extended depth of field depth value in the non-perpendicular region is not perpendicular to the extended depth of field depth map, and a change parameter corresponding to the extended depth of field depth value in the low texture region is less than a preset change parameter threshold.

[0067] In a third aspect, an embodiment of the present application provides an electronic device, comprising:

[0068] a memory configured to store a computer program;

[0069] a processor configured to execute the program stored in the memory, and implement the method of any one of the first aspect.

[0070] In a fourth aspect, an embodiment of the present application provides an extended depth of field microscope, comprising:

[0071] a memory for storing a computer program;

[0072] a processor for executing the program stored on the memory to implement the method of any one of the first aspect.

[0073] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the method of any one of the first aspect.

[0074] In a sixth aspect, an embodiment of the present application provides a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the method of any one of the first aspect.

[0075] Embodiments of the present application have the following beneficial effects:

[0076] In the scheme provided by the embodiments of the present application, the electronic device can obtain images collected by the image acquisition device when the lens is located at different optical axis positions under different light modes; generate an ultra-deep depth map according to the definition differences between the images in the first image group corresponding to different light modes, wherein the first image group includes images collected when the lens is located at different optical axis positions under the same light mode; generate a photometric stereo normal vector map based on the obtained images; generate a photometric stereo depth map based on the photometric stereo normal vector map; correct the depth information in the same depth region in the ultra-deep depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correct the depth information in the low-texture region in the ultra-deep depth map according to the photometric stereo depth value in the photometric stereo depth map. The same depth region is a region in the non-perpendicular region in the ultra-deep depth map that is in the same depth range, and the ultra-deep normal vector corresponding to the ultra-deep depth value in the non-perpendicular region is not perpendicular to the ultra-deep depth map, and the change parameter corresponding to the ultra-deep depth value in the low-texture region is less than a preset change parameter threshold. Since the photometric stereo normal vector in the photometric stereo normal vector map can accurately reflect the subtle differences in the depth values in the collection field of view, and the photometric stereo depth value in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view, therefore, when the depth information in the same depth region is corrected according to the photometric stereo normal vector in the photometric stereo normal vector map, the detailed depth information in the same depth region can be corrected, and when the depth information in the low-texture region is corrected according to the photometric stereo depth value in the photometric stereo depth map, the missing depth information in the low-texture region can be restored, so that the three-dimensional topography of the low-texture region can be restored, and thus an ultra-deep depth map that can well reflect the depth details of the collection field of view can be obtained.

[0077] Of course, implementing any product or method of the present application does not necessarily require all the advantages described above to be achieved at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0078] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0079] FIG. 1 is a flow diagram of an image processing method according to an embodiment of the present application;

[0080] FIG. 2a is an optical shading rendering of a threaded structure;

[0081] FIG. 2b is a super-z depth map of the threaded structure;

[0082] FIG. 2c is an optical stereo depth map of the threaded structure;

[0083] FIG. 3 is a flow diagram of a super-z depth value correction method according to an embodiment of the present application;

[0084] FIG. 4 is a flow diagram of a specific super-z depth value correction method according to an embodiment of the present application;

[0085] FIG. 5 is a flow diagram of a super-z depth value rejection method according to an embodiment of the present application;

[0086] FIG. 6 is a flow diagram of another specific super-z depth value correction method according to an embodiment of the present application;

[0087] FIG. 7 is a flow diagram of a super-z depth map generation method according to an embodiment of the present application;

[0088] FIG. 8 is a flow diagram of another super-z depth map generation method according to an embodiment of the present application;

[0089] FIG. 9 is a flow diagram of a specific super-z depth map generation method according to an embodiment of the present application;

[0090] FIG. 10 is a flow diagram of a photometric stereo depth map generation method according to an embodiment of the present application;

[0091] FIG. 11 is a diagram of edge errors according to an embodiment of the present application;

[0092] FIG. 12 is a diagram of a photometric stereo depth map generation method according to an embodiment of the present application;

[0093] FIG. 13 is a flow diagram of a photometric stereo normal vector map generation method according to an embodiment of the present application;

[0094] FIG. 14 is a flow diagram of another photometric stereo normal vector map generation method according to an embodiment of the present application;

[0095] FIG. 15 is a structural schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0096] FIG. 16 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0097] For the purpose, technical solutions, and advantages of the present application to be clearer, further detailed description will be made below with reference to the drawings and embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0098] For a larger object, since its volume is large, it cannot be completely located within the depth of field range of the super-depth-of-field microscope, so the position of the lens of the super-depth-of-field microscope on the optical axis can be changed to collect multiple images, so that different parts of the object in different images are in a clear state. In this way, the super-depth-of-field microscope can generate a super-depth-of-field depth map according to the clarity difference between each image and the optical axis position corresponding to each image.

[0099] However, for a small object, since its volume is small, it can be completely located within the depth of field range of the super-depth-of-field microscope. In this case, even if the position of the lens on the optical axis is changed, the object is completely located within the depth of field range of the super-depth-of-field microscope or is completely not located within the depth of field range of the super-depth-of-field microscope. In this way, the images collected by the super-depth-of-field microscope are all images in which the entire object is clear or all images in which the entire object is not clear. The image in which the entire object is not clear cannot be used to generate a super-depth-of-field depth map. For the image in which the entire object is clear, the clarity difference between images is not significant, and thus the super-depth-of-field depth map generated according to the clarity difference between images cannot well reflect the depth details of the object.

[0100] Therefore, in order to obtain a super-depth-of-field depth map that can well reflect the depth details of the collected field of view, an embodiment of the present application provides an image processing method, device, electronic device, super-depth-of-field microscope, computer readable storage medium, and computer program product. Hereinafter, an image processing method provided by an embodiment of the present application will be introduced.

[0101] The image processing method provided by an embodiment of the present application can be applied to any electronic device that needs to generate a super-depth-of-field depth map, for example, can be a super-depth-of-field microscope, or a terminal device or a server, etc., which is not specifically limited here. In order to describe clearly, the following is referred to as an electronic device.

[0102] As shown in FIG. 1, an image processing method comprises:

[0103] S101, obtaining images collected by an image collection device when a lens is located at different optical axis positions under different illumination modes.

[0104] S102, generating a hyper-depth depth map according to the definition difference between images in a first image group corresponding to different illumination modes.

[0105] The first image group comprises images collected when the lens is located at different optical axis positions under the same illumination mode.

[0106] S103, generating a photometric stereo normal vector map based on the obtained images.

[0107] S104, generating a photometric stereo depth map based on the photometric stereo normal vector map.

[0108] S105, correcting the depth information in a same depth region in the hyper-depth depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correcting the depth information in a low-texture region in the hyper-depth depth map according to the photometric stereo depth value in the photometric stereo depth map.

[0109] The same depth region is a region in the hyper-depth depth map in a non-perpendicular region and in the same depth range, the hyper-depth normal vector corresponding to the hyper-depth depth value in the non-perpendicular region is not perpendicular to the hyper-depth depth map, and the change parameter corresponding to the hyper-depth depth value in the low-texture region is less than a preset change parameter threshold.

[0110] In the scheme provided by the embodiments of the present application, the electronic device can obtain images collected by the image collection device when the lens is located at different optical axis positions under different illumination modes; generate a hyper-depth depth map according to the definition difference between images in a first image group corresponding to different illumination modes, wherein the first image group includes images collected when the lens is located at different optical axis positions under the same illumination mode; generate a photometric stereo normal vector map based on the obtained images; generate a photometric stereo depth map based on the photometric stereo normal vector map; correct the depth information in a same depth of field region in the hyper-depth depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correct the depth information in a low-texture region in the hyper-depth depth map according to the photometric stereo depth value in the photometric stereo depth map. The same depth of field region is a region in the hyper-depth depth map that is in the same depth of field range in a non-perpendicular region, the hyper-depth normal vector corresponding to the hyper-depth depth value in the non-perpendicular region is not perpendicular to the hyper-depth depth map, and the change parameter corresponding to the hyper-depth depth value in the low-texture region is less than a preset change parameter threshold. Since the photometric stereo normal vector in the photometric stereo normal vector map can accurately reflect the subtle difference of the depth value in the collection field of view, and the photometric stereo depth value in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view, when the depth information in the same depth of field region in the hyper-depth depth map is corrected according to the photometric stereo normal vector in the photometric stereo normal vector map, the detail depth information in the same depth of field region can be corrected, and when the depth information in the low-texture region in the hyper-depth depth map is corrected according to the photometric stereo depth value in the photometric stereo depth map, the missing depth information in the low-texture region can be restored, so that the three-dimensional topography of the low-texture region can be restored, and thus a hyper-depth depth map that can well reflect the depth details of the collection field of view can be obtained.

[0111] When the electronic device is a hyper-depth microscope, it has an image collection function by itself. In this case, the hyper-depth microscope can control the lens to move in the optical axis direction and collect images at each optical axis position under different illumination modes, thereby obtaining images collected when the lens is located at different optical axis positions under different illumination modes.

[0112] When the electronic device is a terminal device or a server, it can be in communication connection with the image collection device. The image collection device can move in the optical axis direction and collect images at each optical axis position based on the control instruction sent by the electronic device or the manual operation of the user under different illumination modes, and then the image collection device can send the collected images to the electronic device, so that the electronic device obtains images collected by the image collection device when the lens is located at different optical axis positions under different illumination modes.

[0113] Of course, the electronic device can also receive images collected by the image collection device at different light axis positions of the lens under different illumination modes manually input by the user, which is also possible, and is not specifically limited here.

[0114] Different illumination modes can be achieved by changing the illumination direction of the light source or turning on the lighting units in different parts of the light source. The illumination modes can include full illumination, upper left, lower left, upper right, lower right, up, down, left, right, etc. Of course, the illumination modes here are only examples, and the specific types of illumination modes can be set according to specific use requirements, which are not specifically limited here.

[0115] In an embodiment, because the ring-shaped light source is characterized by its ability to produce a clear edge profile effect. The coaxial light source is characterized by its ability to provide uniform and parallel light, which can clearly illuminate the microstructure and details of the sample surface, and the design of the coaxial light source can reduce shadows and reflections. In low magnification observation, it is usually necessary to quickly and accurately identify the overall morphology and edge features of the sample. In high magnification observation, more attention is paid to the fine structure and detail information on the sample surface.

[0116] Therefore, in order to obtain more accurate images, when collecting at low magnification, i.e. when the image collection device collects images at a light axis position relatively far from the collection field of view, the collection field of view can be illuminated with different illumination modes based on the ring-shaped light source to highlight the edge information in the collection field of view. When collecting at high magnification, i.e. when the image collection device collects images at a light axis position relatively close to the collection field of view, the collection field of view can be illuminated with different illumination modes based on the coaxial light source to highlight the fine structure in the collection field of view.

[0117] After the electronic device obtains the images, it can group the images according to the illumination modes to obtain a first image group corresponding to each illumination mode, wherein the images in each first image group are images collected under the same illumination mode.

[0118] For each first image group, the electronic device can perform sharpness analysis on each image in the first image group to determine the sharpness difference between the images, and then the electronic device can generate a depth map corresponding to the first image group according to the sharpness difference between the images in the first image group. Thus, the electronic device can generate a depth map corresponding to each first image group, i.e. generate a depth map corresponding to each illumination mode. Then the electronic device can fuse the depth maps to generate a hyper-zoom depth map.

[0119] In addition, after determining the sharpness difference between the images in the first image group, the electronic device can also not generate the depth map corresponding to the first image group, but can fuse the analysis results of the sharpness difference between the images in the first image group corresponding to different illumination modes to obtain a fused analysis result of the sharpness difference, and then generate the hyper-depth depth map based on the fused analysis result of the sharpness difference. The specific way of generating the hyper-depth depth map according to the sharpness difference between the images can adopt the existing hyper-depth depth map generation algorithm, which is not limited here.

[0120] As shown in FIGS. 2a-2c, for the micro thread structure shown in FIG. 2a, the hyper-depth depth map shown in FIG. 2b cannot well reflect the depth details of the object, while for the photometric stereo depth map shown in FIG. 2c, it can well reflect the detailed structure of the thread area. Therefore, after the electronic device obtains the image, in addition to generating the hyper-depth depth map based on the image, it can also generate the photometric stereo normal vector map based on the light intensity difference between the images obtained, and then generate the photometric stereo depth map based on the photometric stereo normal vector map.

[0121] For the region (low-texture region) in the hyper-depth depth map where the change parameter corresponding to the hyper-depth depth value is less than the preset change parameter threshold, the low-texture region actually lacks relevant depth information, and the photometric stereo depth value of each pixel point in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view. Therefore, the electronic device can correct the depth information in the low-texture region in the hyper-depth depth map according to the photometric stereo depth value in the photometric stereo depth map to restore the missing depth information in the low-texture region, thereby restoring the three-dimensional topography of the low-texture region.

[0122] For the region (same depth region) in the non-perpendicular region in the hyper-depth depth map that is in the same depth range, the same depth region has relevant depth information, but the depth information of some pixel points can be more accurate in photometric stereo geometry. Therefore, the electronic device can correct the depth information in the same depth region in the hyper-depth depth map according to the photometric stereo normal vector in the photometric stereo normal vector map to correct the detailed depth information in the same depth region. The non-perpendicular region in the hyper-depth depth map refers to a region that is not located at the edge position of the object, which can be determined by whether the hyper-depth normal vector corresponding to the hyper-depth depth value is perpendicular to the hyper-depth depth map. If the hyper-depth normal vector corresponding to the hyper-depth depth value is not perpendicular to the hyper-depth depth map, it is determined that the hyper-depth depth value is located in the non-perpendicular region.

[0123] It should be noted that the execution sequence of steps S102 and S103 shown in FIG. 1 is only an example, and the electronic device can also execute steps S102 and S103 simultaneously, or execute step S103 first and then execute step S102, which is not limited here.

[0124] In the scheme provided by the embodiments of the present application, since the photometric stereo normal vector in the photometric stereo normal vector map can accurately reflect the subtle differences of the depth values in the collection field of view, and the photometric stereo depth value in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view, when the photometric stereo normal vector in the photometric stereo normal vector map is used to correct the depth information in the same depth-of-field region in the super-depth depth map, the detailed depth information in the same depth-of-field region can be corrected, and when the photometric stereo depth value in the photometric stereo depth map is used to correct the depth information in the low-texture region in the super-depth depth map, the missing depth information in the low-texture region can be restored, so that the three-dimensional topography of the low-texture region is restored, and thus the super-depth depth map that can well reflect the depth details of the collection field of view can be obtained.

[0125] As an implementation manner of the embodiments of the present application, after step S101, the method can further include:

[0126] obtaining a confidence map corresponding to the super-depth depth map.

[0127] Since not all the depth values in the photometric stereo depth map need to be optimized to the depth values in the super-depth depth map, before the electronic device corrects the super-depth depth values in the super-depth depth map, it needs to determine which super-depth depth values in the super-depth depth map are relatively accurate and which super-depth depth values are less accurate, that is, to determine which pixel points in the super-depth depth map are located in the same depth-of-field region and which pixel points are located in the low-texture region. Then, the electronic device can adopt different correction methods for the super-depth depth values in different regions.

[0128] Therefore, when the electronic device generates the super-depth depth map according to the clarity difference between images in the first image group corresponding to different light modes, it can also generate a confidence map corresponding to the super-depth depth map, wherein the pixel value in the confidence map can represent the confidence of the super-depth depth value in the super-depth depth map.

[0129] After obtaining the confidence map, the electronic device can determine, for each pixel point in the super-depth depth map, whether the confidence of the super-depth depth value is greater than a preset confidence threshold according to the confidence map, if yes, it represents that the accuracy of the super-depth depth value is high, and if no, it represents that the accuracy of the super-depth depth value is high.

[0130] In this way, the electronic device determines, according to the confidence map and the preset confidence threshold, a region in which the confidence of the hyper-depth value in the hyper-depth map is greater than the preset confidence threshold as a same depth region, and a region in which the confidence of the hyper-depth value in the hyper-depth map is not greater than the preset confidence threshold as a low-texture region.

[0131] Correspondingly, as shown in FIG. 3, the step S105 can include:

[0132] S301, for the same depth region, obtaining a hyper-depth normal vector corresponding to each hyper-depth value in the same depth region, and selecting a target normal vector corresponding to the hyper-depth value from the obtained hyper-depth normal vector and a photometric stereo normal vector corresponding to the hyper-depth value in the first region; and correcting the hyper-depth value in the same depth region based on the selected target normal vector.

[0133] For the same depth region in which the confidence of the hyper-depth value is greater than the preset confidence threshold, because the accuracy of the hyper-depth value in the region is higher, the electronic device can correct the hyper-depth value in the same depth region in a selective updating manner, specifically:

[0134] The electronic device can determine the first region corresponding to the same depth region in the photometric stereo normal vector map, and then for each hyper-depth value in the same depth region, the electronic device can determine the hyper-depth normal vector corresponding to the hyper-depth value through the difference between the hyper-depth value and each hyper-depth value around the hyper-depth value. Meanwhile, the electronic device can also determine the photometric stereo normal vector corresponding to the hyper-depth value from the first region.

[0135] Then the electronic device can select the optimal normal vector from the obtained hyper-depth normal vector and the photometric stereo normal vector as the target normal vector corresponding to the hyper-depth value through a preset evaluation manner.

[0136] In an implementation, the preset evaluation manner can be that: a direction along the optical axis direction towards the lens is set as a first direction, after obtaining the hyper-depth normal vector corresponding to the hyper-depth value and the photometric stereo normal vector, the electronic device can calculate a first included angle between the obtained hyper-depth normal vector and the first direction, and a second included angle between the photometric stereo normal vector and the first direction. If the first included angle is greater than the second included angle, the electronic device can determine the obtained hyper-depth normal vector as the target normal vector corresponding to the hyper-depth value; if the first included angle is not greater than the second included angle, the electronic device can determine the photometric stereo normal vector as the target normal vector corresponding to the hyper-depth value.

[0137] Because the depth value is used to reflect the depth information in the collection field of view, and the normal vector is used to reflect the direction of the target surface in the collection field of view, in an ideal case, for the same position in the collection field of view, the depth information at the position and the direction of the target surface at the position have a certain corresponding relationship, therefore, after obtaining the target normal vector corresponding to the hyper-depth depth value, the electronic device can correct the hyper-depth depth value in the same depth-of-field region based on the selected target normal vector.

[0138] In S302, for the low-texture region, the hyper-depth depth value in the low-texture region is corrected based on the photometric stereo depth value in the second region.

[0139] For the low-texture region with the confidence of the hyper-depth depth value being not more than the preset confidence threshold, because the accuracy of the hyper-depth depth value in the region is low, it is basically impossible to use, but for the region, the photometric stereo depth value in the photometric stereo depth map is relatively accurate, therefore, for the hyper-depth depth value in the low-texture region, the electronic device can determine the second region corresponding to the low-texture region in the photometric stereo depth map, and then correct the hyper-depth depth value in the low-texture region based on the photometric stereo depth value in the second region.

[0140] In the scheme provided by the embodiments of the present application, the electronic device can divide the hyper-depth depth map into the same depth-of-field region and the low-texture region according to the confidence map corresponding to the hyper-depth depth map, and then adopt different correction methods for the hyper-depth depth value in the same depth-of-field region and the low-texture region, so that the relatively accurate depth information in the hyper-depth depth map can be retained, the correction efficiency can be improved, and the inaccurate depth information in the hyper-depth depth map can be quickly corrected.

[0141] As an implementation manner of the embodiments of the present application, as shown in FIG. 4, the above correction of the hyper-depth depth value in the same depth-of-field region based on the selected target normal vector can include:

[0142] In S401, the target hyper-depth depth value that makes the first difference minimum is determined.

[0143] The gradient corresponding to the hyper-z-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-z-depth value substantially reflect the same index. The tilt parameter corresponding to the target normal vector is the ratio between the component of the target normal vector in the image coordinate system and the component of the target normal vector in the optical axis direction. For example, taking the x-axis and y-axis of the image coordinate system as the x-axis and y-axis of the three-dimensional coordinate system, and taking the optical axis direction of the image acquisition device as the z-axis of the three-dimensional coordinate system, in this case, the tilt parameter corresponding to the target normal vector can include: the ratio between the component of the target normal vector in the x-axis direction and the component in the z-axis direction, and the ratio between the component of the target normal vector in the y-axis direction and the component in the z-axis direction.

[0144] Therefore, after the electronic device determines the target normal vector corresponding to the hyper-z-depth value, the electronic device can calculate the difference value between the gradient corresponding to the hyper-z-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-z-depth value for each hyper-z-depth value in the same depth-of-field region, and then construct the following expression according to the difference value corresponding to each hyper-z-depth value in the same depth-of-field region, and then determine the target hyper-z-depth value that minimizes the first difference, i.e., the target hyper-z-depth value that minimizes the difference between the gradient corresponding to the hyper-z-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-z-depth value in the same depth-of-field region.

[0145] wherein L represents the loss function between the gradient corresponding to the hyper-z-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-z-depth value in the same depth-of-field region, D x D(i,j) represents the gradient of the hyper-z-depth value at position (i,j) in the same depth-of-field region in the x-axis direction, D y D(i,j) represents the gradient of the hyper-z-depth value at position (i,j) in the same depth-of-field region in the y-axis direction, N(i,j,0) represents the component of the target normal vector corresponding to the hyper-z-depth value at position (i,j) in the same depth-of-field region in the x-axis direction, N(i,j,1) represents the component of the target normal vector corresponding to the hyper-z-depth value at position (i,j) in the same depth-of-field region in the y-axis direction, and N(i,j,2) represents the component of the target normal vector corresponding to the hyper-z-depth value at position (i,j) in the same depth-of-field region in the z-axis direction.

[0146] The x-axis and y-axis are coordinate axes on the image plane, and correspond to the horizontal arrangement direction and vertical arrangement direction of the pixel points in the image, respectively. The z-axis is a coordinate axis perpendicular to the two-dimensional coordinate system composed of the x-axis and y-axis, and corresponds to the optical axis direction of the image acquisition device.

[0147] In an embodiment, when the electronic device determines the target hyper-depth-of-field value based on the above expression such that the difference between the gradient corresponding to the hyper-depth-of-field value in the same depth-of-field region and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth-of-field value is the smallest, the electronic device can perform iteration based on a preset threshold, in particular:

[0148] The electronic device can determine the loss value according to the above expression, and then determine whether the loss value is greater than the preset threshold. If it is greater, it indicates that the difference between the current hyper-depth-of-field value and the accurate hyper-depth-of-field value is large, and adjustment is needed. Therefore, in this case, the electronic device can update the gradient corresponding to the hyper-depth-of-field value in the same depth-of-field region according to the obtained loss value and a preset parameter adjustment method, such as a stochastic gradient descent method, a momentum method, etc., and then determine the loss value again according to the above expression.

[0149] If it is not greater, it indicates that the difference between the current hyper-depth-of-field value and the accurate hyper-depth-of-field value is small, and no further adjustment is needed. In this case, the electronic device can integrate the updated gradient corresponding to the hyper-depth-of-field value to determine the updated hyper-depth-of-field value, and then determine the updated hyper-depth-of-field value as the target hyper-depth-of-field value that makes the above difference the smallest.

[0150] In another embodiment, when the electronic device determines the target hyper-depth-of-field value based on the above expression such that the difference between the gradient corresponding to the hyper-depth-of-field value in the same depth-of-field region and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth-of-field value is the smallest, the electronic device can perform iteration based on a preset number of times, in particular:

[0151] The electronic device can determine the loss value according to the above expression, and then update the gradient corresponding to the hyper-depth-of-field value in the same depth-of-field region according to the obtained loss value and a preset parameter adjustment method, such as a stochastic gradient descent method, a momentum method, etc., and then determine the loss value again according to the above expression. Until the number of updates reaches the preset number of times, at this time, the electronic device can integrate the gradient corresponding to the hyper-depth-of-field value obtained by the last update to determine the updated hyper-depth-of-field value, and then determine the updated hyper-depth-of-field value as the target hyper-depth-of-field value that makes the above difference the smallest.

[0152] S402, correct the hyper-depth-of-field value in the same depth-of-field region to the target hyper-depth-of-field value.

[0153] After obtaining the target hyper-depth-of-field value that makes the difference between the gradient corresponding to the hyper-depth-of-field value in the same depth-of-field region and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth-of-field value the smallest, the electronic device can correct the hyper-depth-of-field value in the same depth-of-field region to the target hyper-depth-of-field value.

[0154] In the scheme provided by the embodiments of the present application, the electronic device can calculate the difference between the gradient corresponding to the hyper-depth value in the same depth-of-field region and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth value, then iteratively update based on the difference, and finally correct the target hyper-depth value corresponding to the hyper-depth value in the same depth-of-field region when the difference is the smallest. In this way, global optimization can be achieved, ensuring that the hyper-depth value correction is more consistent and accurate in the entire image, balancing the correction needs of different regions, avoiding local optimal solution, finding a depth value closer to the global optimal solution, and iteratively updating to gradually approach the true value, improving the accuracy and stability of the correction. Therefore, the accuracy and reliability of the hyper-depth map are improved.

[0155] As an embodiment of the present application, as shown in FIG. 5, before the above step S401, the above method can further include:

[0156] S501, for each hyper-depth value in the same depth-of-field region, obtaining the difference value between the gradient corresponding to the hyper-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth value.

[0157] Since there may be a boundary parallel to the optical axis direction, i.e. a vertical boundary, in the field of view, for the normal vector at the vertical boundary, the component in the optical axis direction is close to 0. Taking the above expression for determining the target hyper-depth value as an example, for the normal vector at the vertical boundary, the component N(i,j,2) in the optical axis direction is close to 0, so the above expression becomes NaN, and an effective numerical result cannot be obtained.

[0158] Therefore, before updating the hyper-depth value, the electronic device can determine whether the hyper-depth value corresponding to the difference value and the gradient is the hyper-depth value corresponding to the N(i,j,2) close to 0.

[0159] The difference value corresponding to the hyper-depth value can be determined according to the difference between the gradient corresponding to the hyper-depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-depth value. Taking the above expression for determining the target hyper-depth value as an example, for each hyper-depth value, the electronic device can substitute the gradient corresponding to the hyper-depth value and the target normal vector corresponding to the hyper-depth value into the expression (D x (i,j)-N(i,j,0) / N(i,j,2)) 2 +(D y (i,j)-N(i,j,1) / N(i,j,2)) 2 to obtain the difference value corresponding to the hyper-depth value.

[0160] S502, remove the hyper-depth value corresponding to the difference value greater than the preset difference threshold or the gradient greater than the preset gradient threshold from the same depth region.

[0161] After the electronic device determines the difference value and the gradient corresponding to the hyper-depth value, it can determine whether the difference value corresponding to the hyper-depth value is greater than the preset difference threshold, and determine whether the gradient corresponding to the hyper-depth value is greater than the preset gradient threshold. If the result of one of the above two determination processes is yes, the hyper-depth value is removed from the same depth region, that is, the hyper-depth value is not considered when determining the target hyper-depth value.

[0162] Taking the expression of the loss function L above as an example, for a hyper-depth value, if max(|D x |,|D y |) is greater than 2ΔZ, where 2ΔZ is the preset gradient threshold, and ΔZ represents the distance between adjacent optical axis positions, or (D x (i,j)-N(i,j,0) / N(i,j,2)) 2 +(D y (i,j)-N(i,j,1) / N(i,j,2)) 2 is greater than the preset difference threshold, the hyper-depth value is removed from the same depth region.

[0163] In the scheme provided by the embodiments of the present application, before the electronic device performs hyper-depth value updating, the hyper-depth value at the vertical boundary can be removed from the same depth region by the difference value and the gradient corresponding to the hyper-depth value, and the subsequent updating process will not be performed according to the hyper-depth value, thereby avoiding the influence of the hyper-depth value on the updating process.

[0164] As an embodiment of the present application, as shown in FIG. 6, the above-mentioned correction of the hyper-depth value in the low-texture region based on the photometric stereo depth value in the second region can include:

[0165] S601, performing morphological dilation on the low-texture region to obtain a third region.

[0166] After the electronic device obtains the low-texture region, the low-texture region can be subjected to a morphological dilation operation according to the pixel value of the pixel point in the low-texture region, thereby obtaining a third region. The morphological dilation operation can be performed based on an existing dilation algorithm, which is not limited here.

[0167] S602, determine a difference region between the third region and the low-texture region, and calculate a depth mean and a depth standard deviation of the hyper-depth values in the difference region.

[0168] After obtaining the low-texture region and the third region, the electronic device can determine a difference region between the low-texture region and the third region, and then for each hyper-depth value in the difference region, the electronic device can calculate a depth mean and a depth standard deviation of the hyper-depth value.

[0169] S603, for each depth pair, calculate a depth difference between the hyper-depth values included in the depth pair, and determine a target depth pair corresponding to a depth difference greater than the depth standard deviation.

[0170] After obtaining the depth mean, the hyper-depth values in the difference region can be divided into a first type of hyper-depth values greater than the depth mean and a second type of hyper-depth values not greater than the depth mean, and the electronic device can pair each hyper-depth value in the first type of hyper-depth values with each hyper-depth value in the second type of hyper-depth values to obtain each depth pair. That is, each depth pair includes one hyper-depth value greater than the depth mean and one hyper-depth value not greater than the depth mean in the difference region.

[0171] Further, the electronic device can calculate a depth difference between the hyper-depth values in each depth pair, compare the depth difference with the obtained depth standard deviation, and thus determine a target depth pair corresponding to a depth difference greater than the depth standard deviation.

[0172] S604, for each target depth pair, calculate a photometric stereo depth difference between the target photometric stereo depth values corresponding to the hyper-depth values in the target depth pair in the photometric stereo depth map.

[0173] After obtaining the target depth pair, the electronic device can determine the target photometric stereo depth values corresponding to the hyper-depth values in the target depth pair in the photometric stereo depth map, and then calculate the difference between the target photometric stereo depth values to obtain the photometric stereo depth difference corresponding to the target depth pair.

[0174] S605, determine the ratio between the depth difference corresponding to the target depth pair and the photometric stereo depth difference as the size information corresponding to the target photometric stereo depth value.

[0175] S606, according to the size information corresponding to the target photometric stereo depth value, correct the hyper-depth values in the low-texture region to the photometric stereo depth values in the second region.

[0176] After obtaining the target depth difference value and the parallax stereo depth difference value, the electronic device can determine the ratio between the depth difference value and the parallax stereo depth difference value as the size information s corresponding to the target parallax stereo depth value.

[0177] The electronic device can also calculate the deviation Bias between the depth average of the corrected hyper-depth depth value in the same depth-of-field region and the depth average of the original hyper-depth depth value.

[0178] After obtaining the size information s and the deviation Bias, the electronic device can correct the hyper-depth depth value in the low-texture region to (s*parallax stereo depth value+Bias) based on the parallax stereo depth value in the second region.

[0179] In addition, the electronic device can also correct the hyper-depth depth value in the difference region through alpha blending. Specifically, for each hyper-depth depth value in the difference region, the electronic device can determine the hyper-depth depth value in the same depth-of-field region (excluding the difference region) closest to it, determine the Euclidean distance d1 between the two, determine the hyper-depth depth value in the low-texture region closest to it, and determine the Euclidean distance d2 between the two. Then, the hyper-depth depth value in the difference region is corrected to [d1 / (d1+d2)*(s*parallax stereo depth value+Bias)+d2 / (d1+d2)*hyper-depth depth value].

[0180] In the scheme provided by the embodiments of the present application, for the low-texture region, the depth information corresponding to the parallax stereo depth value is relatively accurate, but the size information corresponding to the parallax stereo depth value is inaccurate. The electronic device can restore the size information corresponding to the parallax stereo depth value by calculating the difference between the depth difference value and the parallax stereo depth difference value, thereby accurately mapping the parallax stereo depth value to the hyper-depth depth map and implementing correction of the hyper-depth depth map.

[0181] As an implementation manner of the embodiments of the present application, when generating the hyper-depth depth map according to the clarity difference between images in the first image group corresponding to different illumination modes, the electronic device can use the generation manner shown in FIG. 7 or FIG. 8.

[0182] As shown in FIG. 7, for each first image group, the electronic device can generate the depth map corresponding to the first image group according to the clarity difference between images in the first image group. Then, the depth maps corresponding to the respective first image groups are fused to obtain the hyper-depth depth map.

[0183] As shown in FIG. 8, for each first image group, the electronic device can also perform sharpness analysis on each image in the first image group, generate a sharpness evaluation map corresponding to each image, stack the sharpness evaluation map corresponding to each image according to the optical axis position corresponding to each image, and obtain a multi-dimensional sharpness evaluation body corresponding to the first image group; fuse the multi-dimensional sharpness evaluation bodies corresponding to the respective first image groups to obtain a fused sharpness evaluation body; and generate an ultra-DOF depth map according to the fused sharpness evaluation body and a preset ultra-DOF reconstruction algorithm.

[0184] The manner of generating an ultra-DOF depth map as shown in FIG. 7 will be described in detail in subsequent embodiments, and will not be described here.

[0185] For the manner of generating an ultra-DOF depth map as shown in FIG. 8, specifically:

[0186] For each first image group, the electronic device can perform sharpness analysis on each image in the first image group, and generate a sharpness evaluation map corresponding to each image.

[0187] For example, under the light mode A, the image acquisition device respectively acquires images and under the light mode B, the image acquisition device respectively acquires and under the light mode C, the image acquisition device respectively acquires images and

[0188] After the electronic device acquires the above images, the electronic device can take the images and as a first image group corresponding to the light mode A, and then perform sharpness analysis on each image in the first image group, and then determine the sharpness evaluation map corresponding to the image the sharpness evaluation map corresponding to the image and the sharpness evaluation map corresponding to the image Similarly, the electronic device can take the images and as a first image group corresponding to the light mode B, and determine the sharpness evaluation maps and and take the images and as a first image group corresponding to the light mode C, and determine the sharpness evaluation maps and

[0189] When electronic devices perform image sharpness analysis and determine the sharpness evaluation map corresponding to the image, they can calculate the sharpness index corresponding to each pixel in the image based on existing sharpness indices and corresponding sharpness index calculation methods. Then, based on each pixel and its corresponding sharpness index, the sharpness evaluation map corresponding to the image can be generated.

[0190] After obtaining multiple sharpness evaluation images corresponding to multiple images in the first image group, the electronic device can place these images at corresponding positions based on the optical axis positions of the images, aligning identical pixels in each image, thereby stacking the multiple sharpness evaluation images to obtain a multi-dimensional sharpness evaluation volume corresponding to the first image group. In this multi-dimensional sharpness evaluation volume, one pixel corresponds to multiple sharpness values.

[0191] For example, electronic devices obtain images The corresponding sharpness evaluation chart image The corresponding sharpness evaluation chart and images The corresponding sharpness evaluation chart Then, assuming that the lens positions corresponding to Z1, Z2, and Z3 gradually approach the field of view, the electronic device can determine the acquisition field of view based on the Z-axis positions corresponding to these multiple images. Place it on the top layer, Placed in the middle layer, Place it on the bottom layer, and and Align the same pixels in the image to obtain the multi-dimensional sharpness evaluation volume corresponding to the lighting mode.

[0192] For the same location, when the image acquisition device moves its lens at different optical axis positions, the change trend in the sharpness of the acquired image should follow a certain pattern. For example, for the same pixel, the sharpness of that pixel acquired by the image acquisition device at different optical axis positions should only have one maximum value, and the change curve should be relatively obvious. Furthermore, the sharpness of that pixel should be relatively high within a certain range of optical axis positions, and so on.

[0193] Therefore, the electronic device can determine the confidence of a pixel point in the multi-dimensional sharpness evaluation body according to the difference between the plurality of sharpness values corresponding to the pixel point in the multi-dimensional sharpness evaluation body, so as to obtain a confidence map corresponding to the multi-dimensional sharpness evaluation body.

[0194] In this way, after obtaining the multi-dimensional sharpness evaluation bodies corresponding to the respective illumination modes, the electronic device can fuse the multi-dimensional sharpness evaluation bodies according to the confidence maps corresponding to the respective multi-dimensional sharpness evaluation bodies, so as to obtain a fused sharpness evaluation body.

[0195] In an implementation, after obtaining the multi-dimensional sharpness evaluation body, the electronic device can generate, for each pixel point in the multi-dimensional sharpness evaluation body, a sharpness curve corresponding to the pixel point according to the plurality of sharpness values corresponding to the pixel point in the multi-dimensional sharpness evaluation body, where the Y axis of the sharpness curve can be the sharpness value, and the X axis can be the index information of the sharpness map corresponding to the sharpness value.

[0196] Further, the electronic device can calculate the unimodality, the distinctness, and the maximum peak width of the sharpness curve corresponding to the pixel point, and then determine whether the sharpness curve satisfies the preset unimodality condition, the distinctness condition, and the maximum peak width condition. If yes, the confidence corresponding to the pixel point is set to 1; if not, the confidence corresponding to the pixel point is set to 0. In this way, the electronic device can obtain a confidence map corresponding to the multi-dimensional sharpness evaluation body.

[0197] Specifically, for the unimodality index, after obtaining the sharpness curve corresponding to the pixel point, the electronic device can regard the sharpness value Φ in the curve that is greater than as a peak, and count the number n peak of peaks in the curve, where Φ x,y,max represents the maximum sharpness value in the curve, Φ x,y,min represents the minimum sharpness value in the curve, and Φ x,y,mean represents the average value of the sharpness values in the curve.

[0198] Further, the electronic device can count the number w peak of peaks in the maximum sharpness value region in the sharpness curve, where the maximum sharpness value region is a region in which the maximum peak in the curve is located, and the sharpness values in the region are all greater than

[0199] Finally, the electronic device can determine whether n peak -w peak <1 is established, and if yes, it is determined that the sharpness satisfies the preset unimodality condition.

[0200] For the distinguishability index, the electronic device can determine whether w is greater than a preset threshold value, and if so, determine that the sharpness curve has a sufficiently large distinguishability and satisfies the preset distinguishability condition.

[0201] For the maximum peak width index, the electronic device can determine whether the value of w peak is located between the product of the first preset parameter and the number of images in the image sequence corresponding to the multi-dimensional sharpness evaluation body (i.e., the first image group corresponding to the multi-dimensional sharpness evaluation body), and the product of the second preset parameter and the number of images in the image sequence corresponding to the multi-dimensional sharpness evaluation body, and if so, determine that the sharpness curve satisfies the preset maximum peak width condition.

[0202] In an embodiment, when the electronic device fuses each multi-dimensional sharpness evaluation body according to the confidence map corresponding to each multi-dimensional sharpness evaluation body, the following steps can be used for fusion:

[0203] a. According to the preset confidence threshold evaluation, the overall confidence level of the object is evaluated, and a high confidence number threshold is set based on the overall confidence level.

[0204] First, by a preset confidence threshold, each confidence in the confidence map is divided into three levels of high, medium, and weak. The specific confidence threshold can be determined according to the actual application and the characteristics of the data, which is not limited here.

[0205] Second, the electronic device sums the number of high confidence in each confidence map, and calculates the ratio between the number of high confidence and the number of groups of the first image group to determine the overall confidence level highconflevel of each pixel.

[0206] Third, the electronic device pre-sets the corresponding relationship between highconflevel and high confidence number threshold highthre in different value ranges, so that the electronic device can determine the corresponding highthre according to the above corresponding relationship after obtaining highconflevel. Wherein, highthre is used to represent how many high confidences are required for a pixel point to be considered as having high enough confidence.

[0207] In an example, the corresponding relationship between highconflevel and high confidence number threshold highthre in different value ranges can be as follows:

[0208] Wherein, m represents the number of confidence maps.

[0209] b. Determine whether each pixel point belongs to the highlight region according to the gray scale value corresponding to the pixel point in each image.

[0210] For each image in the same image sequence, the gray scale value of each pixel point in the gray scale image corresponding to the image is determined. Then for each pixel point, if the number of gray scale values reaching 255 in each gray scale image is greater than a preset threshold, it is preliminarily determined that the pixel point is located in the highlight region, thereby obtaining the highlight region image corresponding to each image sequence;

[0211] After obtaining the highlight region image corresponding to each image sequence, the electronic device can further determine, for each pixel point, the number of times the pixel point belongs to the highlight region in the highlight region images corresponding to each image sequence, and if the number of times the pixel point belongs to the highlight region in the highlight region images corresponding to each image sequence is greater than a preset threshold max(1, 0.2xm), it is finally determined that the pixel point is located in the highlight region, thereby obtaining the target highlight region image.

[0212] c. Fuse each multi-dimensional sharpness evaluation body according to the confidence map corresponding to each multi-dimensional sharpness evaluation body, the high confidence number threshold highthre, and the target highlight region image.

[0213] For each pixel point, the maximum confidence maxconf of the pixel point in each confidence map is determined, and the sharpness value in the multi-dimensional sharpness evaluation body corresponding to the confidence map where the maximum confidence maxconf is located is determined as the estimated height depthtmp corresponding to the pixel point. In addition, it can also be judged whether the pixel point is located in the highlight region according to the target highlight region image.

[0214] If maxconf belongs to the high confidence level, the electronic device further determines the confidence of the pixel point in which confidence map belongs to the high confidence level, and determines the number n of these confidence maps. Then the electronic device can average the sharpness values of the same height in the multi-dimensional sharpness evaluation bodies corresponding to these confidence maps to obtain each target sharpness value corresponding to the pixel point.

[0215] And the electronic device can also judge whether n is not less than highthre, and whether the difference between the index of the maximum sharpness value in the multi-dimensional sharpness evaluation bodies corresponding to these confidence maps is less than a preset threshold, if so, keep the confidence corresponding to the pixel point unchanged; if not, reduce the confidence corresponding to the pixel point.

[0216] In addition, if the pixel point does not belong to the highlight region in the highlight region map corresponding to the image sequence corresponding to the maximum confidence maxconf, and the confidence of the pixel point is still determined to remain at maxconf according to the above determination process, the electronic device marks the pixel point as belonging to the non-highlight region in the target highlight map.

[0217] If maxconf belongs to the medium confidence level, the electronic device further determines the confidence of the pixel point in which confidence maps belongs to the medium confidence level, and determines the number n of the confidence maps. Then the electronic device can average the same height sharpness values in the multi-dimensional sharpness evaluation body corresponding to the confidence maps to obtain each target sharpness value corresponding to the pixel point.

[0218] And the electronic device can also judge whether n is not less than highthre, judge whether the difference between the indexes of the maximum sharpness values in the multi-dimensional sharpness evaluation body corresponding to the confidence maps is less than a preset threshold, and judge whether the pixel point does not belong to the highlight region in the highlight region map corresponding to the image sequence corresponding to the maximum confidence maxconf, if so, keep the confidence corresponding to the pixel point unchanged; if not, reduce the confidence corresponding to the pixel point.

[0219] If maxconf belongs to the low confidence level, the electronic device directly takes each height sharpness value in the multi-dimensional sharpness evaluation body corresponding to the confidence map where the maximum confidence maxconf is located as each target sharpness value corresponding to the pixel point, and takes the average of the confidence in all confidence maps as the confidence corresponding to the pixel point.

[0220] In an embodiment, in order to ensure that an accurate multi-dimensional sharpness evaluation body can be obtained, for each image, the electronic device can determine a plurality of target sharpness evaluation maps corresponding to the image according to a plurality of sharpness indicators, and then for a plurality of images in the same image sequence, fuse the sharpness evaluation maps corresponding to the plurality of images generated according to the same sharpness indicator to obtain a target multi-dimensional sharpness evaluation body corresponding to the sharpness indicator, and finally fuse a plurality of target multi-dimensional sharpness evaluation bodies corresponding to a plurality of sharpness indicators to obtain a multi-dimensional sharpness evaluation body corresponding to the image sequence.

[0221] Here, the GRA (Gradient-based operators, based on the clarity of the gradient operator), LAP (Laplacian-based operators, based on the clarity of the Laplacian operator) and STA (Statistics-based operators, based on the clarity of the statistical operator) three clarity indicators are generated respectively The clarity evaluation map corresponding to the image is taken as an example to introduce the above process:

[0222] For each image in the image sequence, the electronic device can calculate the GRA clarity evaluation map corresponding to the image according to the following formula

[0223] Where, I x and I y respectively represent the first-order derivative of the image in the X and Y directions.

[0224] The electronic device can calculate the LAP clarity evaluation map corresponding to the image according to the following formula

[0225] Where, ΔI represents the Laplace transform value of the image.

[0226] The electronic device can calculate the STA clarity evaluation map corresponding to the image according to the following formula

[0227] Where, L v represents the variance of the image block in the neighborhood window, represents the average variance of the image block in the neighborhood window.

[0228] After the electronic device obtains the and corresponding to each image in the image sequence, it can stack the corresponding to each image to obtain the GRA multi-dimensional clarity evaluation body corresponding to the image sequence, stack the corresponding to each image to obtain the LAP multi-dimensional clarity evaluation body corresponding to the image sequence, and stack the corresponding to each image to obtain the STA multi-dimensional clarity evaluation body corresponding to the image sequence. In addition, the electronic device can also obtain the GRA confidence map corresponding to the GRA multi-dimensional clarity evaluation body, the LAP confidence map corresponding to the LAP multi-dimensional clarity evaluation body, and the STA confidence map corresponding to the STA multi-dimensional clarity evaluation body.

[0229] The manner in which the electronic device generates the multi-dimensional sharpness evaluation body according to the sharpness evaluation map and the manner in which the electronic device generates the confidence map corresponding to the evaluation body have been described in detail in the foregoing, and will not be described again here.

[0230] After obtaining the respective GRA multi-dimensional sharpness evaluation body, the LAP multi-dimensional sharpness evaluation body, and the STA multi-dimensional sharpness evaluation body, because the scales of the three sharpness evaluation indexes are inconsistent, before fusing the respective multi-dimensional sharpness evaluation bodies, the electronic device can first normalize the sharpness evaluation values in the respective multi-dimensional sharpness evaluation bodies. wherein Φ x,y (z) represents the original sharpness evaluation value at the position (x, y) in the multi-dimensional sharpness evaluation body, (z) represents the normalized sharpness evaluation value at the position (x, y) in the multi-dimensional sharpness evaluation body, sum z Φ x,y (z) represents the sum of the respective sharpness evaluation values at the position (x, y) in the multi-dimensional sharpness evaluation body.

[0231] In one manner, after obtaining the respective normalized multi-dimensional sharpness evaluation bodies, the electronic device can fuse the respective multi-dimensional sharpness evaluation bodies directly based on the confidence maps corresponding to the respective multi-dimensional sharpness evaluation bodies.

[0232] In another manner, after obtaining the respective normalized multi-dimensional sharpness evaluation bodies, the electronic device can perform a voting operation based on the confidence maps corresponding to the respective multi-dimensional sharpness evaluation bodies to determine the fusion weights between the respective multi-dimensional sharpness evaluation bodies, and then fuse the respective multi-dimensional sharpness evaluation bodies according to the determined weights.

[0233] Specifically, the electronic device can first determine the high-confidence region and the low-confidence region according to a preset confidence threshold, and then the electronic device can determine the voting values corresponding to the pixel points in the high-confidence region and the pixel points in the low-confidence region according to the correspondence between the preset confidence region types and the voting values. For example, the pixel points in the high-confidence region can correspond to a voting value of 1, and the pixel points in the low-confidence region can correspond to a voting value of 0.1 respectively.

[0234] The electronic device can construct a three-dimensional voting body corresponding to the multi-dimensional sharpness evaluation body, wherein each unit position of the three-dimensional voting body in the horizontal direction corresponds to a respective pixel point in the multi-dimensional sharpness evaluation body, and the three-dimensional voting body has a plurality of Z positions in the Z axis.

[0235] For each pixel point in the multi-dimensional sharpness evaluation body, the electronic device can vote around the Z position of the highest point of the sharpness curve in the three-dimensional voting body according to the voting value corresponding to the pixel point, to obtain a voting result corresponding to the multi-dimensional sharpness evaluation body.

[0236] For example, the multi-dimensional sharpness evaluation body is generated according to the images collected at Z1, Z2, Z3, Z4, Z5, and Z6. For a pixel point in the multi-dimensional sharpness evaluation body, the Z position corresponding to the pixel point is Z1, Z2, Z3, Z4, Z5, or Z6. Correspondingly, the multiple positions of the three-dimensional voting body corresponding to the multi-dimensional sharpness evaluation body on the Z axis can be Z1, Z2, Z3, Z4, Z5, or Z6.

[0237] On this basis, assuming that for a pixel point in the GRA multi-dimensional sharpness evaluation body, the Z position of the highest point in the sharpness curve corresponding to the pixel point is Z3, and the pixel point belongs to a high-confidence region, the electronic device can vote on Z2, Z3, and Z4 corresponding to the unit position corresponding to the pixel point on the three-dimensional voting body, with a voting value of 1.

[0238] Assuming that for the same pixel point in the LAP multi-dimensional sharpness evaluation body, the Z position of the highest point in the sharpness curve corresponding to the pixel point is Z5, and the pixel point belongs to a low-confidence region, the electronic device can vote on Z4, Z5, and Z6 corresponding to the unit position corresponding to the pixel point on the three-dimensional voting body, with a voting value of 0.1.

[0239] Assuming that for the same pixel point in the STA multi-dimensional sharpness evaluation body, the Z position of the highest point in the sharpness curve corresponding to the pixel point is Z4, and the pixel point belongs to a high-confidence region, the electronic device can vote on Z3, Z4, and Z5 corresponding to the unit position corresponding to the pixel point on the three-dimensional voting body, with a voting value of 1.

[0240] After obtaining the voting result corresponding to the multi-dimensional sharpness evaluation body, the electronic device can combine the voting results of each voting body to obtain a target voting body, then determine the Z position with the maximum voting value on the Z axis for each unit position in the target voting body, and further determine the fusion weight between each multi-dimensional sharpness evaluation body based on the voting weights of each voting body for the Z position with the maximum voting value.

[0241] In the scheme provided by the embodiments of the present application, the electronic device can generate the hyper-depth depth map by generating the depth map corresponding to each first image group first and then fusing the depth maps, or can generate the hyper-depth depth map by generating the fusion sharpness evaluation volume corresponding to each first image group first and then generating the hyper-depth depth map according to the fusion sharpness evaluation volume, thereby providing multiple hyper-depth depth maps for selection and meeting the user experience well.

[0242] As an implementation of the embodiments of the present application, as shown in FIG. 9, the above-mentioned fusing the depth map corresponding to each first image group to obtain the hyper-depth depth map can include:

[0243] S901, for each depth value in each depth map, determining a first depth range that overlaps with the image depth range to which the depth value belongs from the voting body depth range of the three-dimensional voting body corresponding to the depth map, and voting the unit position corresponding to the depth value in the three-dimensional voting body in the first depth range with the confidence of the depth value as the voting value.

[0244] After obtaining the depth map corresponding to each first image group, the electronic device can construct the three-dimensional voting body corresponding to the depth map for each depth map, where the unit position of the three-dimensional voting body in the horizontal direction corresponds to the depth value in the depth map one by one, and the three-dimensional voting body is pre-provided with multiple voting body depth ranges in the height direction.

[0245] Further, for each depth value in each depth map, the electronic device can determine which one of the pre-provided multiple image depth ranges includes the depth value, thereby determining the image depth range to which the depth value belongs, or can determine the image depth range composed of the depth value and other depth values within a certain threshold from the depth value as the image depth range to which the depth value belongs.

[0246] After obtaining the image depth range to which the depth value belongs, the electronic device can determine the first depth range that overlaps with the image depth range from the multiple voting body depth ranges of the three-dimensional voting body corresponding to the depth map. Then the electronic device can vote the unit position corresponding to the depth value in the three-dimensional voting body in the first depth range with the confidence corresponding to the depth value as the voting value.

[0247] For example, the depth map includes 4 pixels, the hyper-depth depth values of the 4 pixels are 4, 8, 7, and 12 respectively, and the confidence values are 1, 1, 0.5, and 0.5 respectively. The electronic device constructs a three-dimensional voting body for the depth map, the three-dimensional voting body has 4 unit positions A, B, C, and D in the horizontal direction, the 4 unit positions correspond to the 4 hyper-depth depth values respectively, and the three-dimensional voting body is preset with 3 voting body depth ranges [0, 5), [5, 10), and [10, 15] in the height direction.

[0248] The electronic device determines, according to the depth values and other depth values that have a difference with the depth values within a certain threshold, that the image depth range to which the hyper-depth depth value 4 belongs is [3, 5], the image depth range to which the hyper-depth depth value 8 belongs is [7, 9], the image depth range to which the hyper-depth depth value 7 belongs is [6, 8], and the image depth range to which the hyper-depth depth value 12 belongs is [11, 13].

[0249] For the hyper-depth depth value 4, [0, 5) and [5, 10) are the first depth ranges that have an overlap with the image depth range to which the depth value belongs, so the electronic device can vote for [0, 5) and [5, 10) at the unit position A, and the voting value is 1, that is, the voting values of [0, 5) and [5, 10) in the two voting body depth ranges at the unit position A are both 1.

[0250] Similarly, for the hyper-depth depth value 8, the electronic device can vote for [5, 10) at the unit position B, and the voting value is 1, that is, the voting value of [5, 10) in the voting body depth range at the unit position B is 1. For the hyper-depth depth value 7, the electronic device can vote for [5, 10) at the unit position C, and the voting value is 0.5, that is, the voting value of [5, 10) in the voting body depth range at the unit position C is 0.5. For the hyper-depth depth value 12, the electronic device can vote for [10, 15) at the unit position D, and the voting value is 0.5, that is, the voting value of [10, 15) in the voting body depth range at the unit position D is 0.5.

[0251] In an implementation manner, the depth map corresponding to the first image set obtained by the electronic device can not have a confidence map, in this case, if the light mode corresponding to the first image set is not a full light mode, a confidence map with all 1s can be taken as the confidence map corresponding to the depth map. If the light mode corresponding to the first image set is a full light mode, because the image collected in the full light mode is more accurate than the image collected in the non-full light mode, a confidence map with all 2s can be taken as the confidence map corresponding to the depth map to represent that the image collected in the full light mode has a higher credibility.

[0252] S902, sum the voting results of the same unit position and the same voting body depth range in the three-dimensional voting body corresponding to each depth map to obtain a target three-dimensional voting body.

[0253] After obtaining the three-dimensional voting body corresponding to each depth map, the electronic device can sum the voting results of the same unit position and the same voting body depth range in each three-dimensional voting body to obtain a target three-dimensional voting body.

[0254] For example, in the three-dimensional voting body corresponding to the first depth map, the voting values of the [0, 5) and [5, 10) voting body depth ranges at the unit position A are both 1, in the three-dimensional voting body corresponding to the second depth map, the voting value of the [0, 5) voting body depth range at the unit position A is 0.3, and in the three-dimensional voting body corresponding to the second depth map, the voting value of the [10, 15) voting body depth range at the unit position A is 0.6.

[0255] Then, after fusing the three three-dimensional voting bodies, in the target three-dimensional voting body obtained by fusion, the voting value of the [0, 5) voting body depth range at the unit position A is 1.3, the voting value of the [5, 10) voting body depth range is 1, and the voting value of the [10, 15) voting body depth range is 0.6.

[0256] S903, for each unit position of the target three-dimensional voting body, determine the voting body depth range corresponding to the maximum voting result of the unit position as a second depth range, and determine the weighting coefficient of the depth value corresponding to the unit position in each depth map according to the voting value of the second depth range contributed by the depth value corresponding to the unit position in each depth map.

[0257] After obtaining the target three-dimensional voting body, the electronic device can determine, for each unit position of the target three-dimensional voting body, the voting body depth range corresponding to the maximum voting result of the unit position as a second depth range.

[0258] For example, for the target three-dimensional voting body in the above example, for the unit position A in the target three-dimensional voting body, the voting value of the [0, 5) voting body depth range is the most, so [0, 5) can be determined as the second depth range corresponding to the unit position A.

[0259] After obtaining the second depth range corresponding to each unit position, the electronic device can take the ratio between the voting value of the second depth range in each depth map and the total voting value of the second depth range as the weighting coefficient of the depth value corresponding to the unit position in each depth map,

[0260] For example, for the target three-dimensional voting body in the above example, because the voting value of the first depth map for the voting body depth range [0, 5) of unit position A is 1, the voting value of the second depth map for the voting body depth range [0, 5) of unit position A is 0.3, and the voting value of the third depth map for the voting body depth range [0, 5) of unit position A is 0. Therefore, the weighted coefficients of the depth values corresponding to unit position A in the three depth maps are (1 / 1.3), (0.3 / 1.3) and (0 / 1.3) respectively.

[0261] S904, according to the weighted coefficients of the depth values in each depth map, the depth values in each depth map are weighted calculated to obtain a hyper-depth depth map.

[0262] After obtaining the weighted coefficients of the depth values in each depth map, the electronic device can perform weighted calculation on the depth values in each depth map according to the weighted coefficients of the depth values in each depth map to obtain a hyper-depth depth map.

[0263] In the scheme provided by the embodiment of the application, the electronic device can determine the three-dimensional voting bodies corresponding to each depth map, and then fuse each three-dimensional voting body to obtain a target three-dimensional voting body, and finally accurately determine the fusion weight between each depth map based on the target three-dimensional voting body, thereby improving the accuracy of fusing each depth map based on the fusion weight.

[0264] As an implementation manner of the embodiment of the application, as shown in FIG. 10, the above step S104 can include:

[0265] S1001, the photometric stereo vector map is reduced by a first preset multiple to obtain a first normal vector map.

[0266] Since the photometric stereo depth map is obtained by integrating the photometric stereo normal vectors in the photometric stereo normal vector map, there is a cumulative error in the photometric stereo depth map. On this basis, it is considered that the smaller the integration region is, the smaller the cumulative error should theoretically be.

[0267] Therefore, when generating the photometric stereo depth map based on the tilt parameters corresponding to the photometric stereo normal vectors in the photometric stereo normal vector map, the electronic device can first reduce the photometric stereo normal vector map by a first preset multiple to obtain a first normal vector map, so as to reduce the integration region.

[0268] S1002, the photometric stereo depth map that makes the second difference minimum is determined as the first depth map.

[0269] After obtaining the first normal vector map by reducing the photometric stereo normal vector map, the electronic device can determine the photometric stereo depth map that minimizes the difference between the gradient corresponding to the photometric stereo depth in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal vector in the first normal vector map, i.e., the second difference, as the first depth map based on a minimization algorithm.

[0270] Specifically, the electronic device can determine the photometric stereo depth map that minimizes the difference between the gradient corresponding to the photometric stereo depth value in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal vector in the first normal vector map according to the following expression:

[0271] J(z) = ∫∫((z x -p) 2 +(z y -q) 2 )dxdy

[0272] wherein J(z) represents the loss function between the gradient corresponding to the photometric stereo depth value in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal vector in the first normal vector map, z x represents the gradient of the photometric stereo depth value in the x-axis direction, z y represents the gradient of the photometric stereo depth value in the y-axis direction, p = n x / n z , q = n y / n z , n x represents the component of the photometric stereo normal vector in the x-axis direction, n y represents the component of the photometric stereo normal vector in the y-axis direction, n z represents the component of the photometric stereo normal vector in the z-axis direction.

[0273] In an embodiment, due to the existence of the vertical boundary, |n z | in the above expression tends to 0, and when the photometric stereo depth map is determined according to the above expression, the problem in the area shown in the upper right corner of FIG. 11 can occur. Therefore, when the photometric stereo depth map is determined according to the above expression, the electronic device can add the regularization of the gradient corresponding to the photometric stereo depth value in the above expression. That is, the photometric stereo depth map that minimizes the second difference, i.e., the photometric stereo depth map that minimizes the difference between the gradient corresponding to the photometric stereo depth value in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal vector in the first normal vector map, is determined according to the following optimized expression:

[0274] J(z) = ∫∫((z x -p) 2 +(zy q) 2 dxdy+λ(∑|z x |+|z y |)

[0275] wherein, λ represents a regularization constraint strength, |z x |, |z y | respectively represent absolute values of z x , z y .

[0276] S1003, the first depth map is enlarged by a second preset multiple to obtain a second depth map, and the photometric stereo normal vector map is reduced to the same resolution as the second depth map to obtain a second normal vector map.

[0277] S1004, a region of a preset size is randomly determined in the second depth map as a to-be-updated region, for a target region in the to-be-updated region, a partial normal vector map corresponding to the target region in the second normal vector map is determined; a target photometric stereo depth value that minimizes the third difference is determined; the photometric stereo depth value in the target region is corrected to the target photometric stereo depth value to obtain a corrected second depth map; the second depth map is updated to the corrected second depth map, and the step of randomly determining a region of a preset size in the second depth map as a to-be-updated region is returned until the number of to-be-updated regions determined is equal to a preset number.

[0278] Although the influence of cumulative error can be reduced by determining the first depth map through the reduced photometric stereo normal vector map, reducing the photometric stereo normal vector map will also cause the accuracy of the photometric stereo normal vector map to decrease, and further cause the accuracy of the first depth map obtained according to the first normal vector map to be relatively small. Therefore, in order to obtain a more accurate photometric stereo depth map, the electronic device can reduce the reduction multiple of the photometric stereo normal vector map, and then use the photometric stereo normal vector map with the reduced reduction multiple to optimize the photometric stereo depth value in the first depth map.

[0279] Specifically, the electronic device can first enlarge the first depth map by a second preset multiple to obtain a second depth map, and then determine the reduction multiple of the photometric stereo depth map according to the resolution of the second depth map, and further reduce the photometric stereo normal vector map to the same resolution as the second depth map according to the determined reduction multiple to obtain a second normal vector map. At this time, the second normal vector map is equivalent to being enlarged by the second preset multiple compared with the first normal vector map, but it is not directly enlarged on the basis of the first normal vector map, but is realized by reducing the reduction multiple of the photometric stereo normal vector map. The size of the second normal vector map is less than or equal to the size of the photometric stereo normal vector map.

[0280] After obtaining the second depth map, the electronic device can randomly determine a region of a preset size in the second depth map as the to-be-updated region, for example, the electronic device can randomly determine a pixel point in the second depth map, and then determine a region composed of the pixel point and a certain number of pixel points around the pixel point as the to-be-updated region.

[0281] After obtaining the to-be-updated region, the electronic device can update a central region in the to-be-updated region or update an edge region in the to-be-updated region, that is, the central region can be determined as a target region to achieve the update effect of fixed edge updating center, or the edge region can be determined as the target region to achieve the update effect of fixed center updating edge, which can be set according to specific use requirements and is not limited here.

[0282] For the target region in the to-be-updated region, the electronic device can determine a partial normal vector map corresponding to the target region in the second normal vector map, and then the electronic device can determine a target photometric stereo depth value that minimizes the difference between the gradient of the photometric stereo depth value in the target region and the inclination parameter of the photometric stereo normal vector in the partial normal vector map, that is, a target photometric stereo depth value that minimizes the third difference, wherein when determining the target photometric stereo depth value that minimizes the third difference, the electronic device can input relevant parameters into the expression corresponding to the second difference for calculation, which is not described here.

[0283] After obtaining the target photometric stereo depth value, the electronic device can correct the photometric stereo depth value in the target region to the target photometric stereo depth value, thereby updating the photometric stereo depth value in the target region in the to-be-updated region and obtaining a corrected second depth map.

[0284] Taking FIG. 12 as an example, after obtaining the second depth map, the electronic device can determine the region shown in FIG. 12 as the to-be-updated region from the second depth map, and then the electronic device can determine the central region shown by the black part in FIG. 12 as the target region in the to-be-updated region. Then the electronic device can perform the above updating process for the target region, correct the photometric stereo depth value in the target region to the target photometric stereo depth value, and then take the second depth map containing the corrected target region as the corrected second depth map.

[0285] After the electronic device performs the photometric stereo depth value updating process once, the electronic device can determine whether the number of to-be-updated regions that have been determined is equal to a preset number, that is, whether the number of times of updating the second depth map reaches a preset number.

[0286] If the number of the determined to-be-updated regions is less than the preset number, it indicates that the updating of the second depth map is not completed, in which case, the electronic device can update the second depth map to the corrected second depth map, and then return to the step of randomly determining a region of the preset size in the second depth map as a to-be-updated region, that is, to determine the to-be-updated region again in the corrected second depth map, and then perform the above photometric stereo depth value updating process on the newly determined to-be-updated region.

[0287] If the number of the determined to-be-updated regions is equal to the preset number, it indicates that the updating of the second depth map is completed, in which case, the electronic device can determine whether the resolution of the second normal vector map is less than the resolution of the photometric stereo normal vector map, and then select to perform step S1005 or step S1006 according to the determination result.

[0288] S1005, if the resolution of the second normal vector map is less than the resolution of the photometric stereo normal vector map, update the first depth map to the corrected second depth map, reset the number of the determined to-be-updated regions, and return to the step of enlarging the first depth map by the second preset multiple to obtain the second depth map, and reducing the photometric stereo normal vector map to the same resolution as the second depth map to obtain the second normal vector map.

[0289] S1006, if the resolution of the second normal vector map is equal to the resolution of the photometric stereo normal vector map, take the corrected second depth map as the photometric stereo depth map.

[0290] In addition to performing iterative updating on the photometric stereo depth value in the second depth map through step S1004, the electronic device can also perform iterative updating by constantly changing the size of the depth map, thereby realizing double iteration. Taking the first preset multiple as 16 times and the second preset multiple as 2 times as an example, the way of performing iterative updating by constantly changing the size of the depth map in the present application is described in detail as follows:

[0291] The electronic device can first reduce the photometric stereo normal vector map by 16 times to obtain a first normal vector map, and then obtain a depth map reduced by 16 times, i.e., a first depth map, according to the normal vector map reduced by 16 times. Then the electronic device can enlarge the above first depth map by 2 times to obtain a depth map reduced by 8 times, i.e., a second depth map, and then the electronic device can reduce the photometric stereo normal vector map to the same resolution as the second depth map to obtain a normal vector map reduced by 8 times, i.e., a second normal vector map, and finally the electronic device can perform iterative updating on the depth map reduced by 8 times according to the above step S1004 based on the normal vector map reduced by 8 times to obtain a corrected depth map reduced by 8 times.

[0292] In this case, the electronic device can update the first depth map to the corrected second depth map, reset the number of determined to-be-updated regions, and then return to the step of magnifying the first depth map by the second preset multiple to obtain the second depth map, and magnifying the photometric stereo normal vector map to the same resolution as the second depth map to obtain the second normal vector map.

[0293] That is, after obtaining the corrected depth map that is reduced by 8 times, the electronic device can magnify the corrected depth map that is reduced by 8 times by 2 times to obtain a depth map that is reduced by 4 times, and magnify the photometric stereo normal vector map to the same resolution as the depth map that is reduced by 4 times to obtain a normal vector map that is reduced by 4 times. Then, the electronic device can perform the iterative update of the step S1004 on the depth map that is reduced by 4 times based on the normal vector map that is reduced by 4 times to obtain a corrected depth map that is reduced by 4 times.

[0294] Similarly, after obtaining the corrected depth map that is reduced by 4 times, the electronic device can magnify the corrected depth map that is reduced by 4 times by 2 times to obtain a depth map that is reduced by 2 times, and magnify the photometric stereo normal vector map to the same resolution as the depth map that is reduced by 2 times to obtain a normal vector map that is reduced by 2 times. Then, the electronic device can perform the iterative update of the step S1004 on the depth map that is reduced by 2 times based on the normal vector map that is reduced by 2 times to obtain a corrected depth map that is reduced by 2 times.

[0295] After obtaining the corrected depth map that is reduced by 2 times, the electronic device can magnify the corrected depth map that is reduced by 2 times by 2 times to obtain a non-reduced-size depth map, and magnify the photometric stereo normal vector map to the same resolution as the non-reduced-size depth map. In this case, because the resolution of the non-reduced-size depth map is the same as that of the photometric stereo normal vector map, when the photometric stereo normal vector map is magnified to the same resolution as the non-reduced-size depth map, the electronic device can directly determine the photometric stereo normal vector map as the magnified result. Then, the electronic device can perform the iterative update of the step S1004 on the non-reduced-size depth map based on the photometric stereo normal vector map to obtain a corrected non-reduced-size depth map, which is used as the photometric stereo depth map.

[0296] In the scheme provided in the embodiments of the present application, the electronic device can reduce the integral region by reducing the normal vector map, thereby reducing the influence of the cumulative error. In addition, the electronic device can magnify the reduced normal vector map, and then update the depth map according to the magnified normal vector map, thereby improving the accuracy of the photometric stereo depth map on the basis of reducing the cumulative error.

[0297] As an implementation manner of the embodiment of the present application, the electronic device can adopt the generation manners shown in FIG. 13 or FIG. 14 when generating the photometric stereo normal vector map based on the obtained images.

[0298] As shown in FIG. 13, for each first image group, the electronic device can fuse the images in the first image group to generate a full-focus image corresponding to the first image group; and then generate the photometric stereo normal vector map according to the light intensity difference between the full-focus images corresponding to each first image group.

[0299] As shown in FIG. 14, after obtaining the images, the electronic device can also group the images according to the optical axis positions to obtain second image groups corresponding to the optical axis positions, wherein the images in each second image group are images collected when the lens is located at the same optical axis position. In this case, for each second image group, the electronic device can generate a photometric stereo normal vector map corresponding to the second image group according to the light intensity difference between the images in the second image group, and then the electronic device can fuse the photometric stereo normal vector maps corresponding to the second image groups corresponding to different optical axis positions to generate the photometric stereo normal vector map.

[0300] In the scheme provided by the embodiment of the present application, the electronic device can either generate the full-focus image corresponding to each first image group first, and then generate the photometric stereo normal vector map according to the light intensity difference between the full-focus images, or generate the photometric stereo normal vector map corresponding to each second image group first, and then fuse the photometric stereo normal vector maps to generate the photometric stereo normal vector map, thereby providing multiple photometric stereo normal vector map generation manners for selection, which can well meet the user's use experience.

[0301] As an implementation manner of the embodiment of the present application, after the step S105, the method can further include:

[0302] The super-DOF full-focus image is determined based on the corrected super-DOF depth map by at least one of the following three manners.

[0303] The first implementation manner is direct fusion. Specifically, each image collected by the image collection device when the lens is located at different optical axis positions in the full light mode is obtained, then for each pixel point, it is determined which image collected at which optical axis position is the clearest according to the super-DOF depth map, and then the pixel value of the pixel point in the image collected at the optical axis position is determined as the pixel value of the pixel point in the super-DOF full-focus image. After the pixel value of each pixel point in the super-DOF full-focus image is determined through the above manner, the super-DOF full-focus image is obtained.

[0304] The second embodiment is to remove the highlight reflection region. Specifically, considering that the region with highlight reflection in the full light mode is probably not a region with highlight reflection in other light mode(s), the image captured in the light mode(s) is the image content with the region.

[0305] Therefore, according to the first embodiment, the full focus images in different light modes are generated based on the corrected hyper-depth depth map, and each full focus image is converted from the RGB format to the HSV format. Then, for each pixel, the full focus images in different light modes are sorted by the V channel, thereby generating full focus images with different degrees of brightness.

[0306] Suppose that the full focus images in five light modes are generated, then after sorting, five full focus images with different degrees of brightness can be obtained. Specifically, suppose that for pixel A, the V values in the full focus images in the first to fifth light modes are 0.1, 0.5, 0.3, 0.7 and 0.2 respectively, and for pixel B, the V values in the full focus images in the first to fifth light modes are 0.4, 0.2, 0.3, 0.1 and 0.5 respectively. After sorting by the V channel, five full focus images with different degrees of brightness can be obtained.

[0307] Among the obtained full focus images with the highest brightness, the HSV value of pixel A is the HSV value of pixel A in the full focus image in the fourth light mode (because the V value of pixel A in the full focus image in the fourth light mode is the largest, which is 0.7), and the HSV value of pixel B is the HSV value of pixel A in the full focus image in the fifth light mode (because the V value of pixel B in the full focus image in the fifth light mode is the largest, which is 0.5). Among the obtained full focus images with the lowest brightness, the HSV value of pixel A is the HSV value of pixel A in the full focus image in the first light mode (because the V value of pixel A in the full focus image in the first light mode is the largest, which is 0.1), and the HSV value of pixel B is the HSV value of pixel A in the full focus image in the fourth light mode (because the V value of pixel B in the full focus image in the fourth light mode is the largest, which is 0.1).

[0308] Finally, according to the actual use requirement, a full focus image with a certain degree of brightness can be selected from the full focus images with different degrees of brightness, and the full focus image is subjected to contrast enhancement (such as gamma enhancement, histogram equalization, etc.), thereby obtaining a hyper-depth full focus image.

[0309] The third implementation manner is based on a normal vector rendering. Specifically, in some scenarios, more attention is paid to the change of a 3D microstructure, such as a scratch and some other defects, and a generated depth map is of an overall structure, although the microstructure is also reflected, but cannot be intuitively reflected. However, a normal vector map can easily distinguish the microstructure. Therefore, the normal vector map corresponding to the corrected hyper-deep depth map can be re-rendered in eight directions, that is, up, down, left, right, top-left, bottom-left, top-right, and bottom-right. In the re-rendering process, the colors are all white. The texture details reflected in the re-rendered maps are all 3D structure information, and the structure information in different directions can be viewed through the re-rendered maps in different directions.

[0310] The normal vector hyper-focusing map can be obtained in two ways: a. the normal vector hyper-focusing map is obtained in different light modes according to the second manner, and then the normal vector hyper-focusing map is obtained according to the photometric stereo geometry method based on the normal vector hyper-focusing maps in different light modes; b. the normal vector hyper-focusing map is obtained according to the photometric stereo geometry method based on different light axis positions, and then the normal vector hyper-focusing map is obtained according to the first manner.

[0311] In the scheme provided in the embodiments of the present application, after obtaining the corrected hyper-deep depth map, the electronic device can generate a hyper-deep hyper-focusing map based on the corrected hyper-deep depth map, so as to restore the image details. In addition, the electronic device can select a suitable scheme from the "direct fusion", "removing the highlight reflection area", and / or "normal vector rendering" modes to generate the hyper-deep hyper-focusing map according to the specific application scenarios and requirements, thereby improving the flexibility of the scheme.

[0312] Corresponding to the image processing method, the embodiments of the present application also provide an image processing device, which will be introduced below.

[0313] As shown in FIG. 15, an image processing device includes:

[0314] The image acquisition module 1510 is configured to obtain images collected by the image collection device when the lens is located at different light axis positions in different light modes.

[0315] The hyper-deep depth map generation module 1520 is configured to generate a hyper-deep depth map based on the clarity difference between the images in the first image group corresponding to different light modes, where the first image group includes images collected when the lens is located at different light axis positions in the same light mode.

[0316] The normal vector map generation module 1530 is configured to generate a photometric stereo normal vector map based on the obtained images.

[0317] The photometric stereo depth map generation module 1540 is configured to generate a photometric stereo depth map based on the photometric stereo normal vector map;

[0318] The correction module 1550 is configured to correct the depth information in a same depth range in the hyper-z depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correct the depth information in a low-texture region in the hyper-z depth map according to the photometric stereo depth value in the photometric stereo depth map, wherein the same depth range is a range in a non-perpendicular region in the hyper-z depth map, and the hyper-z depth value in the non-perpendicular region corresponds to a hyper-z normal vector that is not perpendicular to the hyper-z depth map, and the change parameter corresponding to the hyper-z depth value in the low-texture region is less than a preset change parameter threshold.

[0319] In the scheme provided by the embodiments of the present application, the electronic device can obtain images collected by the image collection device when the lens is located at different optical axis positions under different light modes; generate a hyper-z depth map according to the definition difference between images in a first image group corresponding to different light modes, wherein the first image group includes images collected when the lens is located at different optical axis positions under the same light mode; generate a photometric stereo normal vector map based on the obtained images; generate a photometric stereo depth map based on the photometric stereo normal vector map; correct the depth information in a same depth range in the hyper-z depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correct the depth information in a low-texture region in the hyper-z depth map according to the photometric stereo depth value in the photometric stereo depth map. The same depth range is a range in a non-perpendicular region in the hyper-z depth map, and the hyper-z depth value in the non-perpendicular region corresponds to a hyper-z normal vector that is not perpendicular to the hyper-z depth map, and the change parameter corresponding to the hyper-z depth value in the low-texture region is less than a preset change parameter threshold. Since the photometric stereo normal vector in the photometric stereo normal vector map can accurately reflect the subtle difference of the depth value in the collection field of view, and the photometric stereo depth value in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view, when the depth information in the same depth range in the hyper-z depth map is corrected according to the photometric stereo normal vector in the photometric stereo normal vector map, the detail depth information in the same depth range can be corrected, and when the depth information in the low-texture region in the hyper-z depth map is corrected according to the photometric stereo depth value in the photometric stereo depth map, the missing depth information in the low-texture region can be restored, so that the three-dimensional topography of the low-texture region can be restored, and thus the hyper-z depth map that can well reflect the depth details of the collection field of view can be obtained.

[0320] As an implementation manner of the embodiments of the present application, the apparatus can further include:

[0321] The confidence map generation module is configured to obtain a confidence map corresponding to the hyper-depth depth map, wherein a pixel value in the confidence map represents a confidence of a hyper-depth value in the hyper-depth depth map.

[0322] The region determination module is configured to determine, according to the confidence map, a same-depth region and a low-texture region in the hyper-depth depth map, wherein the confidence of the hyper-depth value in the same-depth region is greater than a preset confidence threshold, and the confidence of the hyper-depth value in the low-texture region is not greater than the preset confidence threshold.

[0323] The correction module 1550 can include:

[0324] The first correction unit is configured to, for the same-depth region, obtain a hyper-depth normal vector corresponding to each hyper-depth value in the same-depth region, select a target normal vector corresponding to the hyper-depth value from the obtained hyper-depth normal vector and a photometric stereo normal vector corresponding to the hyper-depth value in a first region, and correct the hyper-depth value in the same-depth region based on the selected target normal vector, wherein the first region is a region corresponding to the same-depth region in a photometric stereo normal vector map.

[0325] The second correction unit is configured to, for the low-texture region, correct the hyper-depth value in the low-texture region based on a photometric stereo depth value in a second region, wherein the second region is a region corresponding to the low-texture region in a photometric stereo depth map.

[0326] As an implementation manner of the embodiment of the present application, the first correction unit can include:

[0327] The target hyper-depth value determination subunit is configured to determine a target hyper-depth value that minimizes a first difference, wherein the first difference is a difference between a gradient corresponding to the hyper-depth value in the same-depth region and an inclination parameter corresponding to the target normal vector corresponding to the hyper-depth value, and the inclination parameter corresponding to the target normal vector is a ratio between a component of the target normal vector in an image coordinate system and a component of the target normal vector in an optical axis direction; and the first correction subunit is configured to correct the hyper-depth value in the same-depth region to the target hyper-depth value.

[0328] As an implementation manner of the embodiment of the present application, the target hyper-depth value determination subunit can be specifically configured to determine the target hyper-depth value that minimizes the first difference according to the following expression:

[0329] wherein L represents a loss function between the gradient corresponding to the hyper-depth value in the same-depth region and the inclination parameter corresponding to the target normal vector corresponding to the hyper-depth value, and D x(i,j) represents a gradient of the hyper-deep depth value at position (i,j) in the same depth-of-field region in the x-axis direction, D y (i,j) represents a gradient of the hyper-deep depth value at position (i,j) in the same depth-of-field region in the y-axis direction, N(i,j,0) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the x-axis direction, N(i,j,1) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the y-axis direction, and N(i,j,2) represents a component of the target normal vector corresponding to the hyper-deep depth value at position (i,j) in the same depth-of-field region in the z-axis direction.

[0330] As an implementation form of the embodiment of the present application, the first correction unit can further include:

[0331] a difference value determination sub-unit, configured to, before determining the target hyper-deep depth value that makes the first difference minimum, obtain, for each hyper-deep depth value in the same depth-of-field region, a difference value between the gradient corresponding to the hyper-deep depth value and the tilt parameter corresponding to the target normal vector corresponding to the hyper-deep depth value;

[0332] a hyper-deep depth value elimination sub-unit, configured to eliminate, from the same depth-of-field region, the hyper-deep depth value whose difference value is greater than a preset difference threshold value or whose gradient is greater than a preset gradient threshold value.

[0333] As an implementation form of the embodiment of the present application, the first correction unit can further include:

[0334] an angle calculation sub-unit, configured to calculate a first included angle between the obtained hyper-deep normal vector and a first direction, and calculate a second included angle between the photometric stereo normal vector corresponding to the hyper-deep depth value in the first region and the first direction, wherein the first direction is a direction along the optical axis direction towards the lens;

[0335] a target normal vector determination sub-unit, configured to, when the first included angle is greater than the second included angle, determine the obtained hyper-deep normal vector as the target normal vector corresponding to the hyper-deep depth value; and when the first included angle is not greater than the second included angle, determine the photometric stereo normal vector corresponding to the hyper-deep depth value in the first region as the target normal vector corresponding to the hyper-deep depth value.

[0336] As an implementation form of the embodiment of the present application, the second correction unit can include:

[0337] an expansion sub-unit, configured to perform morphological expansion on the low-texture region to obtain a third region;

[0338] a difference region determination sub-unit, configured to determine a difference region between the third region and the low-texture region, and calculate a depth mean and a depth standard deviation of the hyper-depth depth values in the difference region;

[0339] a target depth pair determination sub-unit, configured to, for each depth pair, calculate a depth difference value between the hyper-depth depth values included in the depth pair, and determine a target depth pair with a corresponding depth difference value greater than the depth standard deviation, wherein each depth pair includes one hyper-depth depth value greater than the depth mean and one hyper-depth depth value not greater than the depth mean in the difference region;

[0340] a photometric stereo depth difference value determination sub-unit, configured to, for each target depth pair, calculate a photometric stereo depth difference value between the target photometric stereo depth values corresponding to the hyper-depth depth values in the target depth pair in the photometric stereo depth map;

[0341] a size information determination sub-unit, configured to determine a size information corresponding to the target photometric stereo depth values as a ratio between the depth difference value and the photometric stereo depth difference value corresponding to the target depth pair;

[0342] a second correction sub-unit, configured to correct the hyper-depth depth values in the low-texture region to the photometric stereo depth values in the second region according to the size information corresponding to the target photometric stereo depth values.

[0343] As an implementation manner of the embodiment of the present application, the hyper-depth depth map generation module 1520 can be specifically used for:

[0344] generating a depth map corresponding to each first image group according to the sharpness difference between images in the first image group; and fusing the depth maps corresponding to the first image groups to obtain the hyper-depth depth map;

[0345] or,

[0346] performing sharpness analysis on each image in each first image group to generate a sharpness evaluation map corresponding to each image, stacking the sharpness evaluation maps corresponding to each image according to the optical axis positions corresponding to each image to obtain a multi-dimensional sharpness evaluation volume corresponding to the first image group, fusing the multi-dimensional sharpness evaluation volumes corresponding to the first image groups to obtain a fused sharpness evaluation volume, and generating the hyper-depth depth map according to the fused sharpness evaluation volume and a preset hyper-depth reconstruction algorithm.

[0347] As an implementation manner of the embodiment of the present application, the hyper-depth depth map generation module 1520 can include:

[0348] The voting unit is configured to determine, for each depth value in each depth map, a first depth range that overlaps with an image depth range to which the depth value belongs, from a voting body depth range of a three-dimensional voting body corresponding to the depth map, vote, with a confidence degree of the depth value as a voting value, a unit position corresponding to the depth value in the three-dimensional voting body in the first depth range, wherein each depth map is a depth map corresponding to each first image group, a unit position of the three-dimensional voting body in a horizontal direction is one-to-one corresponding to a depth value in the depth map, and the three-dimensional voting body is preset with a plurality of voting body depth ranges in a height direction.

[0349] The target three-dimensional voting body determination unit is configured to sum up voting results of the same unit position and the same voting body depth range in the three-dimensional voting bodies corresponding to the depth maps to obtain a target three-dimensional voting body.

[0350] The weighting coefficient determination unit is configured to determine, for each unit position of the target three-dimensional voting body, a voting body depth range corresponding to a maximum voting result of the unit position as a second depth range, and determine a weighting coefficient of a depth value corresponding to the unit position in each depth map according to a voting value of the depth value in the second depth range.

[0351] The super-depth-of-field depth map generation unit is configured to perform weighted calculation on the depth values in the depth maps according to the weighting coefficients of the depth values in the depth maps to obtain a super-depth-of-field depth map.

[0352] As an implementation manner of an embodiment of the present application, the photometric stereo depth map generation module 1540 can include:

[0353] The first normal vector map generation unit is configured to downscale the photometric stereo normal vector map by a first preset multiple to obtain a first normal vector map.

[0354] The first depth map determination unit is configured to determine a photometric stereo depth map that makes a second difference minimum as a first depth map, wherein the second difference is a difference between a gradient corresponding to a photometric stereo depth value in the photometric stereo depth map and an inclination parameter corresponding to a photometric stereo normal vector in the first normal vector map.

[0355] The image magnification unit is configured to magnify the first depth map by a second preset multiple to obtain a second depth map, and downscale the photometric stereo normal vector map to the same resolution as the second depth map to obtain a second normal vector map.

[0356] The second depth map updating unit is configured to randomly determine a region of a preset size in the second depth map as a to-be-updated region, determine, for a target region in the to-be-updated region, a partial normal vector graph in the second normal vector graph corresponding to the target region, wherein the target region is a central region or an edge region, determine a target photometric stereo depth value that minimizes a third difference, wherein the third difference is a difference between a gradient of a photometric stereo depth value in the target region and an inclination parameter of a photometric stereo normal vector in the partial normal vector graph, correct the photometric stereo depth value in the target region to the target photometric stereo depth value to obtain a corrected second depth map, update the second depth map to the corrected second depth map, and return to the step of randomly determining a region of a preset size in the second depth map as a to-be-updated region until the number of the determined to-be-updated regions is equal to a preset number.

[0357] The iterative updating unit is configured to, when the resolution of the second normal vector graph is less than the resolution of the photometric stereo normal vector graph, update the first depth map to the corrected second depth map, reset the number of the determined to-be-updated regions, return to the step of magnifying the first depth map by the second preset multiple to obtain the second depth map, and reducing the photometric stereo normal vector graph to the same resolution as the second depth map to obtain the second normal vector graph.

[0358] The photometric stereo depth map generating unit is configured to, when the resolution of the second normal vector graph is equal to the resolution of the photometric stereo normal vector graph, take the corrected second depth map as the photometric stereo depth map.

[0359] As an implementation manner of the embodiment of the present application, the first depth map determining unit can be specifically configured to:

[0360] determine the photometric stereo depth map that minimizes the second difference according to the following expression: J(z) =∫∫((z x -p) 2 +(z y -q) 2 )dxdy+λ(∑|z x |+|z y |)

[0361] wherein J(z) represents a loss function between a gradient of a photometric stereo depth value in a photometric stereo depth map and an inclination parameter of a photometric stereo normal vector in the first normal vector graph, z x represents a gradient of the photometric stereo depth value in an x-axis direction, z y represents a gradient of the photometric stereo depth value in a y-axis direction, p = n x / n z , q = n y / n z , and n xn x represents a component of the photometric stereo normal vector in the x-axis direction, n y n y represents a component of the photometric stereo normal vector in the y-axis direction, n z n z represents a component of the photometric stereo normal vector in the z-axis direction, λ represents a regularization constraint strength, and |z x |z y |z x |z y |z

[0362] As an implementation of an embodiment of the present application, the normal vector map generation module 1530 can be specifically used for:

[0363] for each first image group, fusing images in the first image group to generate a full-focus image corresponding to the first image group; and generating a photometric stereo normal vector map according to light intensity differences between the full-focus images corresponding to the respective first image groups;

[0364] or,

[0365] for each second image group, generating a photometric stereo normal vector map corresponding to the second image group according to light intensity differences between images in the second image group; and fusing the photometric stereo normal vector maps corresponding to the second image groups corresponding to different optical axis positions to generate a photometric stereo normal vector map, wherein the second image group includes images obtained from the images and captured when the lens is at the same optical axis position.

[0366] The present application also provides an electronic device, as shown in FIG. 16, which includes:

[0367] a memory 1601 for storing a computer program;

[0368] a processor 1602 for executing the program stored in the memory 1601 to implement the image processing method described in any of the above embodiments.

[0369] The electronic device described above can also include a communication bus and / or a communication interface, and the processor 1602, the communication interface, and the memory 1601 can communicate with each other through the communication bus.

[0370] In the scheme provided by the embodiments of the present application, the electronic device can obtain images collected by the image collection device when the lens is located at different optical axis positions under different illumination modes; generate a hyper-depth depth map according to the definition difference between images in a first image group corresponding to different illumination modes, wherein the first image group includes images collected when the lens is located at different optical axis positions under the same illumination mode; generate a photometric stereo normal vector map based on the obtained images; generate a photometric stereo depth map based on the photometric stereo normal vector map; correct the depth information in a same depth region in the hyper-depth depth map according to the photometric stereo normal vector in the photometric stereo normal vector map, and correct the depth information in a low-texture region in the hyper-depth depth map according to the photometric stereo depth value in the photometric stereo depth map. The same depth region is a region in the hyper-depth depth map in a non-perpendicular region that is in the same depth range, the hyper-depth normal vector corresponding to the hyper-depth depth value in the non-perpendicular region is not perpendicular to the hyper-depth depth map, and the change parameter corresponding to the hyper-depth depth value in the low-texture region is less than a preset change parameter threshold. Since the photometric stereo normal vector in the photometric stereo normal vector map can accurately reflect the subtle difference of the depth value in the collection field of view, and the photometric stereo depth value in the photometric stereo depth map can reflect the overall three-dimensional topography of the collection field of view, when the depth information in the same depth region is corrected according to the photometric stereo normal vector in the photometric stereo normal vector map, the detailed depth information in the same depth region can be corrected, and when the depth information in the low-texture region is corrected according to the photometric stereo depth value in the photometric stereo depth map, the missing depth information in the low-texture region can be restored, so that the three-dimensional topography of the low-texture region can be restored, and thus the hyper-depth depth map that can well reflect the depth details of the collection field of view can be obtained.

[0371] The embodiments of the present application also provide a hyper-depth microscope, comprising:

[0372] a memory for storing a computer program;

[0373] a processor for executing the program stored on the memory to implement the image processing method of any of the above embodiments.

[0374] The hyper-depth microscope can further comprise a communication bus and / or a communication interface, and the processor, the communication interface and the memory can communicate with each other through the communication bus.

[0375] The communication bus mentioned in the above electronic device / ultra-deep field microscope can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0376] The communication interface is used for communication between the above electronic device / ultra-deep field microscope and other devices.

[0377] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0378] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0379] In another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the image processing method in any of the above embodiments.

[0380] In another embodiment provided in the present application, a computer program product containing instructions is also provided, and when the computer program product is executed on a computer, the computer is caused to execute the image processing method in any of the above embodiments.

[0381] In the embodiments described above, all or some of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or some of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded into and executed by a computer, all or some of the processes or functions according to the embodiments described in the specification are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a solid state disk (SSD) and the like.

[0382] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0383] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, electronic device, super-depth microscope, computer storage medium and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0384] The above description is only the preferred embodiment of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that, The method includes: To obtain images captured by an image acquisition device under different lighting conditions when the lens is positioned at different optical axes; Based on the difference in sharpness between images in the first image group corresponding to different lighting modes, a super depth map is generated, wherein the first image group includes: images acquired when the lens is located at different optical axis positions under the same lighting mode; Based on the obtained image, a photometric stereo normal map is generated; Based on the aforementioned photometric stereo normal vector map, a photometric stereo depth map is generated; Based on the photometric stereo normal vector in the photometric stereo normal vector map, the depth information in the same depth region of the super depth of field map is corrected. Based on the photometric stereo depth value in the photometric stereo depth map, the depth information in the low-texture region of the super depth of field map is corrected. The same depth region is a region in the non-vertical region of the super depth of field map that is within the same depth range. The super depth of field normal vector corresponding to the super depth of field depth value in the non-vertical region is not perpendicular to the super depth of field map. The change parameter corresponding to the super depth of field depth value in the low-texture region is less than a preset change parameter threshold.

2. The method according to claim 1, characterized in that, The step of generating a photometric stereo normal map based on the obtained image includes: For each first image group, the images in the first image group are fused to generate a full-focus image corresponding to the first image group; based on the light intensity differences between the full-focus images corresponding to each first image group, a photometric stereo normal vector map is generated. or, For each second image group, a photometric stereo normal vector map corresponding to the second image group is generated based on the light intensity difference between the images in the second image group. The photometric stereo normal vector maps corresponding to the second image groups at different optical axis positions are fused to generate a photometric stereo normal vector map. The second image group includes: images acquired when the lens is located at the same optical axis position in the obtained images.

3. The method according to claim 1, characterized in that, The step of correcting the depth information in the same depth-of-field region of the super-depth-of-field depth map based on the photometric stereo normal vector in the photometric stereo normal vector map, and correcting the depth information in the low-texture region of the super-depth-of-field depth map based on the photometric stereo depth value in the photometric stereo depth map, includes: For the same depth of field region, obtain the hyperdepth of field normal vector corresponding to each hyperdepth of field depth value in the same depth of field region. From the obtained hyperdepth of field normal vector and the photometric stereo normal vector corresponding to the hyperdepth of field depth value in the first region, select the target normal vector corresponding to the hyperdepth of field depth value. Correct the hyperdepth of field depth value in the same depth of field region based on the selected target normal vector, wherein the first region is the region corresponding to the same depth of field region in the photometric stereo normal vector map. For low-texture areas, the super depth of field value in the low-texture areas is corrected based on the photometric stereo depth value in the second area, wherein the second area is the area in the photometric stereo depth map corresponding to the low-texture area.

4. The method according to claim 1, characterized in that, After obtaining images acquired by the image acquisition device under different lighting conditions when the lens is located at different optical axis positions, the method further includes: Obtain the confidence map corresponding to the super depth-of-field depth map, wherein the pixel value in the confidence map represents the confidence level of the super depth-of-field depth value in the super depth-of-field depth map; Based on the confidence map, the same depth-of-field region and low-texture region in the super-depth-of-field depth map are determined, wherein the confidence of the super-depth-of-field depth value in the same depth-of-field region is greater than a preset confidence threshold, and the confidence of the super-depth-of-field depth value in the low-texture region is not greater than the preset confidence threshold.

5. The method according to claim 3, characterized in that, The step of correcting the hyperdepth of field value in the same depth-of-field region based on the selected target normal vector includes: Determine the target super-depth of field value that minimizes the first difference, wherein the first difference is the difference between the gradient corresponding to the super-depth of field value in the same depth of field region and the tilt parameter corresponding to the target normal vector corresponding to the super-depth of field value, and the tilt parameter corresponding to the target normal vector is the ratio between the component of the target normal vector in the image coordinate system and the component of the target normal vector in the optical axis direction. The hyperdepth of field value in the same depth of field region is corrected to the target hyperdepth of field value.

6. The method according to claim 5, characterized in that, Determining the target depth-of-field value that minimizes the first difference includes: Determine the target depth-of-field value that minimizes the first difference using the following expression: Where L represents the loss function between the gradient corresponding to the hyperdepth of field value and the tilt parameter corresponding to the target normal vector corresponding to the hyperdepth of field value in the same depth of field region, and D x (i,j) represents the gradient of the hyperdepth of field value at position (i,j) within the same depth of field region along the x-axis. y (i,j) represents the gradient of the hyperdepth of field value at position (i,j) in the same depth of field region along the y-axis, N(i,j,0) represents the component of the target normal vector corresponding to the hyperdepth of field value at position (i,j) in the same depth of field region along the x-axis, N(i,j,1) represents the component of the target normal vector corresponding to the hyperdepth of field value at position (i,j) in the same depth of field region along the y-axis, and N(i,j,2) represents the component of the target normal vector corresponding to the hyperdepth of field value at position (i,j) in the same depth of field region along the z-axis.

7. The method according to claim 5, characterized in that, Before determining the target depth-of-field value that minimizes the first difference, the method further includes: For each hyperdepth of field value in the same depth of field region, the difference between the gradient corresponding to the hyperdepth of field value and the tilt parameter corresponding to the target normal vector corresponding to the hyperdepth of field value is obtained. Remove out any out-of-field depth values ​​from the same depth-of-field region that have a difference value greater than a preset difference threshold or an out-of-field depth value whose gradient is greater than a preset gradient threshold.

8. The method according to claim 3, characterized in that, The step of selecting the target normal vector corresponding to the hyper-depth-of-field value from the obtained hyper-depth-of-field normal vector and the photometric stereo normal vector corresponding to the hyper-depth-of-field value in the first region includes: Calculate the first angle between the obtained hyper-depth-of-field normal vector and the first direction, and calculate the second angle between the photometric stereo normal vector corresponding to the hyper-depth-of-field depth value in the first region and the first direction, wherein the first direction is: the direction towards the lens along the optical axis; If the first included angle is greater than the second included angle, the obtained hyper-depth of field normal vector is determined as the target normal vector corresponding to the hyper-depth of field depth value; If the first included angle is not greater than the second included angle, the photometric stereo normal vector corresponding to the hyper-depth of field value in the first region is determined as the target normal vector corresponding to the hyper-depth of field value.

9. The method according to claim 3, characterized in that, The method of correcting the super depth of field value in the low-texture region based on the photometric stereo depth value in the second region includes: Morphological dilation of the low-texture region yields the third region; Determine the difference region between the third region and the low-texture region, and calculate the depth mean and depth standard deviation of the super-depth of field depth values ​​in the difference region; For each depth pair, calculate the depth difference between the hyperdepth of field depth values ​​included in the depth pair, and determine the target depth pair whose depth difference is greater than the depth standard deviation. Each depth pair includes a hyperdepth of field depth value greater than the mean depth value and a hyperdepth of field depth value not greater than the mean depth value in the difference region. For each target depth pair, calculate the photometric stereo depth difference between the super-depth of field values ​​in that target depth pair and the corresponding target photometric stereo depth values ​​in the photometric stereo depth map; The ratio between the depth difference corresponding to the target depth and the photometric stereo depth difference is determined as the size information corresponding to the target photometric stereo depth value; Based on the size information corresponding to the target photometric depth value, the super depth of field value in the low-texture region is corrected to the photometric depth value in the second region.

10. The method according to any one of claims 1-9, characterized in that, The step of generating a super-depth-of-field map based on the sharpness differences between images in the first image group corresponding to different lighting modes includes: For each first image group, a depth map corresponding to the first image group is generated based on the difference in sharpness between the images in the first image group; the depth maps corresponding to each first image group are fused to obtain a super depth map. or, For each first image group, a sharpness analysis is performed on each image in the first image group to generate a sharpness evaluation map corresponding to each image. Based on the optical axis position corresponding to each image, the sharpness evaluation maps corresponding to each image are stacked to obtain a multi-dimensional sharpness evaluation volume corresponding to the first image group. The multi-dimensional sharpness evaluation volumes corresponding to each first image group are fused to obtain a fused sharpness evaluation volume. Based on the fused sharpness evaluation volume and a preset super-depth-of-field reconstruction algorithm, a super-depth-of-field depth map is generated.

11. The method according to claim 10, characterized in that, The step of fusing the depth maps corresponding to each first image group to obtain a super-depth-of-field depth map includes: For each depth value in each depth map, a first depth range that overlaps with the image depth range to which the depth value belongs is determined from the depth range of the voting body of the corresponding 3D voting body in the depth map. The confidence level of the depth value is used as the voting value, and the unit position in the 3D voting body corresponding to the depth value is voted on in the first depth range. Here, each depth map is the depth map corresponding to each first image group, the unit position of the 3D voting body in the horizontal direction corresponds one-to-one with the depth value in the depth map, and the 3D voting body has multiple preset voting body depth ranges in the height direction. The target 3D voting volume is obtained by summing the voting results of the same unit position and the same depth range of the voting volume in the 3D voting volume corresponding to each depth map. For each unit position of the target three-dimensional voting volume, the voting volume depth range corresponding to the maximum voting result of that unit position is determined as the second depth range. Based on the voting value contributed by the depth value corresponding to that unit position in each depth map to the second depth range, the weighting coefficient of the depth value corresponding to that unit position in each depth map is determined. The depth values ​​in each depth map are weighted according to the weighting coefficients of the depth values ​​in each depth map to obtain the super-depth of field depth map.

12. The method according to any one of claims 1-9, characterized in that, The step of generating a photometric stereo depth map based on the photometric stereo normal vector map includes: The photometric stereo normal map is reduced by a first preset factor to obtain the first normal map; A photometric stereo depth map that minimizes the second difference is determined as the first depth map, wherein the second difference is the difference between the gradient corresponding to the photometric stereo depth value in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal in the first normal map; The first depth map is magnified by a second preset factor to obtain a second depth map, and the photometric stereo normal map is reduced to the same resolution as the second depth map to obtain a second normal map; In the second depth map, a region of a preset size is randomly selected as the region to be updated. For the target region in the region to be updated, a partial normal vector map corresponding to the target region is determined in the second normal vector map, wherein the target region is a central region or an edge region. A target photometric stereo depth value that minimizes the third difference is determined, wherein the third difference is the difference between the gradient corresponding to the photometric stereo depth value in the target region and the tilt parameter corresponding to the photometric stereo normal vector in the partial normal vector map. The photometric stereo depth value in the target region is corrected to the target photometric stereo depth value to obtain a corrected second depth map. The second depth map is updated to the corrected second depth map, and the step of randomly selecting a region of a preset size as the region to be updated in the second depth map is returned until the number of determined regions to be updated is equal to the preset number. If the resolution of the second normal vector map is less than the resolution of the photometric stereo normal vector map, update the first depth map to the corrected second depth map, reset the number of identified regions to be updated, and return to the steps of enlarging the first depth map by a second preset factor to obtain the second depth map, and shrinking the photometric stereo normal vector map to the same resolution as the second depth map to obtain the second normal vector map. If the resolution of the second normal vector map is equal to the resolution of the photometric stereo normal vector map, the corrected second depth map is used as the photometric stereo depth map.

13. The method according to claim 12, characterized in that, Determining the photometric stereo depth map that minimizes the second difference includes: The photometric stereo depth map that minimizes the second difference is determined using the following expression: J(z)=∫∫((z x -p) 2 +(with y -q) 2 )dxdy+λ(∑|z x |+|with y |) Where J(z) represents the loss function between the gradient corresponding to the photometric stereo depth value in the photometric stereo depth map and the tilt parameter corresponding to the photometric stereo normal vector in the first normal vector map, z x The z-axis represents the gradient of the photometric stereo depth value along the x-axis. y p = n represents the gradient of the photometric depth value along the y-axis. x / n z q = n y / n z n x n represents the component of the photometric solid normal vector along the x-axis. y n represents the component of the photometric solid normal vector along the y-axis. z Let |z| represent the component of the photometric solid normal vector along the z-axis, and λ represent the regularization constraint strength. x |、|z y | represent z respectively x z y The absolute value of.

14. An image processing apparatus, characterized in that, The device includes: The image acquisition module is used to acquire images captured by the image acquisition device under different lighting conditions when the lens is located at different optical axis positions; The super depth-of-field depth map generation module is used to generate a super depth-of-field depth map based on the sharpness difference between images in the first image group corresponding to different lighting modes. The first image group includes images acquired when the lens is located at different optical axis positions under the same lighting mode. The normal vector map generation module is used to generate a photometric stereo normal vector map based on the obtained image; A photometric stereo depth map generation module is used to generate a photometric stereo depth map based on the photometric stereo normal vector map. The correction module is used to correct the depth information in the same depth region of the super depth of field map based on the photometric stereo normal vector in the photometric stereo normal vector map, and to correct the depth information in the low-texture region of the super depth of field map based on the photometric stereo depth value in the photometric stereo depth map. The same depth region refers to regions within the same depth range in the non-vertical region of the super depth of field map, the super depth of field normal vector corresponding to the super depth of field depth value in the non-vertical region is not perpendicular to the super depth of field map, and the change parameter corresponding to the super depth of field depth value in the low-texture region is less than a preset change parameter threshold.

15. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-13.

16. A super depth-of-field microscope, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-13.

18. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method according to any one of claims 1-13.

Images