Defocus depth measurement method, device, electronic device and storage medium

Abstract

This invention relates to a method, apparatus, electronic device, and storage medium for measuring defocus depth, belonging to the field of computer vision 3D spatial measurement. The invention utilizes camera parameters of a focus-overlay camera and defocused images to determine the correlation between radiosity and depth information, and incorporates this correlation into a first objective function. By minimizing the first objective function to estimate the depth information of the scene, the influence of radiosity on the depth measurement process can be reduced, thereby improving the accuracy of the defocus depth measurement method.

Description

Background Art

[0002] In recent years, the application of computer vision has become more and more common, such as autonomous driving, virtual reality, etc. These technologies are inseparable from the acquisition of three-dimensional information. Depth measurement methods are divided into active depth measurement and passive measurement methods. Passive depth measurement mainly uses traditional optical imaging technology to collect scene images, and analyzes and calculates the texture, clarity and other information in the collected images to reconstruct the depth information in the scene. Passive depth measurement technology mainly includes focus depth method, defocus depth method and binocular vision method.

[0003] In traditional defocus depth measurement methods, data fidelity items are constructed mainly based on the defocused images captured of the scene to obtain depth measurement results. The depth information measurement process is easily affected by the radiosity in the scene, resulting in reduced accuracy of the depth measurement results. Summary of the invention

[0004] In order to solve the defect that the existing defocus depth measurement method is easily affected by the radiometry in the scene, resulting in reduced accuracy of the depth measurement result, the present invention provides a defocus depth measurement method, device, electronic device and medium.

[0005] In a first aspect, in order to solve the above technical problem, the present invention provides a defocus depth measurement method, comprising:

[0006] Using a focus stacking camera to collect a preset number of defocused image sequences of a scene, wherein each defocused image sequence includes a plurality of defocused images focused at different depths in the scene;

[0007] For each defocused image, based on the camera parameters of the focus stacking camera and the defocused image, a first association relationship corresponding to the defocused image is determined by a first formula, wherein the first formula represents an association relationship among the camera parameters, the defocused image, the depth information, and the radiosity, and the first association relationship is a first formula when the camera parameters and the defocused image in the first formula are known;

[0008] Based on the defocused image sequence, a second correlation relationship is determined by a second formula, wherein the second formula characterizes the correlation relationship between the defocused image sequence, the original image characteristic value of the defocused image sequence, and the depth information, and the second correlation relationship is the second formula when the defocused image sequence in the second formula is known;

[0009] For each defocused image, based on the second association relationship and the first association relationship, determine a first objective function corresponding to the defocused image, wherein the first objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the depth information;

[0010] Determine first depth information according to each first objective function, wherein the first depth information includes depth information when the weighted sum value in each first objective function takes a minimum value;

[0011] Based on the first depth information, a measurement result of the depth information of the scene is obtained.

[0012] The beneficial effects of the present invention are as follows: the correlation between radiometry and depth information is determined by utilizing the camera parameters of a focus stacking camera and a defocused image, and the correlation is introduced into a first objective function. By minimizing the first objective function to estimate the depth information of the scene, the influence of radiometry on the depth information measurement process can be reduced, thereby improving the accuracy of the defocus depth measurement method.

[0013] Furthermore, the camera parameters of the focus stacking camera include focal length and aperture value, and the first formula is expressed as follows:

[0014]

[0015] Among them, E bz (s) represents the radiance, s represents the depth information, A represents the aperture value, I i (y,x) represents the defocused image, F i represents the focal length, x and y represent the 2D pixel coordinates in the defocused image, and F represents the F-norm.

[0016] The beneficial effect of adopting the above-mentioned improved solution is that the correlation between the radiometry and the depth information determined based on the first formula is added to the first objective function, which can effectively reduce the influence of the radiometry on the measurement result of the depth information.

[0017] Further, based on the first depth information, a measurement result of the depth information of the scene is obtained, including:

[0018] For each defocused image, determining a defocus blur kernel image corresponding to the defocused image based on first depth information corresponding to the defocused image;

[0019] Determine the defocus blur kernel image corresponding to each defocus image as a defocus blur kernel image sequence corresponding to the defocus image sequence;

[0020] For each defocused image, the defocused image and the camera parameters corresponding to the defocused image are input into a third formula to obtain the target radiometry corresponding to the defocused image, wherein the third formula represents the correlation between the radiometry, the defocused image and the camera parameters;

[0021] Based on the defocused image sequence and the defocused blur kernel image sequence, a third association relationship is determined by a fourth formula, wherein the fourth formula characterizes the association relationship among the original image characteristic value, the defocused image sequence, the defocused blur kernel image sequence and the all-focused image sequence, and the third association relationship is the fourth formula when the defocused image sequence and the defocused blur kernel image sequence in the fourth formula are known;

[0022] For each defocused image, based on the target radiance corresponding to the defocused image and the third association relationship, a second objective function corresponding to the defocused image is determined, wherein the second objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the fully focused image;

[0023] Determine a target fully focused image sequence according to each second objective function, wherein the target fully focused image sequence includes a fully focused image when the weighted sum value in each second objective function takes a minimum value;

[0024] Based on the first depth information and the target all-focus image sequence, a measurement result of the depth information of the scene is obtained.

[0025] The beneficial effect of adopting the above-mentioned improved solution is that the first depth information obtained can be optimized by using the target all-focus image sequence, thereby making the measurement result of the depth information closer to the real depth information of the scene.

[0026] Furthermore, the camera parameters of the focus stacking camera include focal length and aperture value, and the third formula is:

[0027]

[0028] Among them, E bc (I) represents the radiance, A represents the aperture value, I i (y,x) represents the defocused image, F i represents the focal length, and x and y represent the two-dimensional pixel coordinates in the defocused image.

[0029] The beneficial effect of adopting the above-mentioned improved solution is: adding the target radiometry determined based on the third formula to the second objective function to reduce the influence of radiometry on the full-focus image sequence acquisition process, thereby reducing the influence of radiometry on the depth measurement process.

[0030] Further, based on the first depth information and the target all-focus image sequence, a measurement result of the depth information of the scene is obtained, including:

[0031] Inputting the defocused image sequence, the target fully focused image sequence, and the defocus blur kernel image sequence into a fifth formula to obtain a target image energy of the target fully focused image sequence, wherein the fifth formula represents the correlation relationship between the image energy of the fully focused image sequence, the defocused image sequence, the defocus blur kernel image sequence, and the fully focused image sequence;

[0032] For each defocused image, based on the second association relationship, the target image energy, and the first association relationship corresponding to the defocused image, a third objective function corresponding to the defocused image is determined, wherein the third objective function represents the association relationship between the original image characteristic value, the weighted sum value between the radiance and the image energy, and the depth information;

[0033] Determine second depth information according to each third objective function, wherein the second depth information includes depth information when the weighted sum value in each third objective function takes a minimum value;

[0034] The second depth information is determined as a measurement result of the depth information of the scene.

[0035] The beneficial effect of adopting the above-mentioned improved scheme is: the target image energy of the target fully focused image sequence is introduced into the first objective function to obtain the third objective function, and the first depth information is optimized by solving the second depth information through the third objective function, thereby improving the accuracy of the depth information measurement result.

[0036] Furthermore, the focus stacking camera includes a liquid crystal lens, the liquid crystal lens includes a liquid crystal layer and two device alignment films, the liquid crystal layer contains multi-walled carbon nanotubes, and the liquid crystal layer is arranged between the two device alignment films.

[0037] The beneficial effect of adopting the above-mentioned improvement scheme is that the use of a liquid crystal layer containing multi-walled carbon nanotubes can improve the imaging depth of field of the focus stacking camera, thereby increasing the applicable scenarios of the defocus depth measurement method.

[0038] Further, the liquid crystal layer is prepared by the following method:

[0039] Mixing multi-walled carbon nanotubes and ethanol to obtain a first mixture;

[0040] Using an ultrasonic vibrator to vibrate the first mixture;

[0041] Mixing the first mixture after the oscillation treatment with liquid crystal to obtain a second mixture;

[0042] Using an ultrasonic vibrator to vibrate the second mixture;

[0043] Using an oven to remove ethanol from the second mixture after the shaking treatment;

[0044] Repeat the shaking treatment and ethanol removal operation for a preset number of times on the second mixture to obtain a third mixture;

[0045] The third mixture is poured into the liquid crystal box at a first preset temperature, and the liquid crystal box is sealed when the temperature of the liquid crystal box reaches a second preset temperature to obtain a liquid crystal layer, wherein the first preset temperature is greater than the second preset temperature.

[0046] The beneficial effect of adopting the above-mentioned improvement scheme is: using ethanol as a dispersing solvent and through multiple oscillation treatments, the multi-walled carbon nanotubes can be evenly dispersed in the liquid crystal layer, thereby ensuring the imaging effect of the liquid crystal lens doped with the multi-walled carbon nanotubes.

[0047] In a second aspect, the present invention provides a defocus depth measurement device, comprising:

[0048] An acquisition module, used for collecting a preset number of defocused image sequences of a scene using a focus stacking camera, wherein each defocused image sequence includes a plurality of defocused images focused at different depths in the scene;

[0049] A first processing module is used to determine, for each defocused image, a first association relationship corresponding to the defocused image based on camera parameters of a focus stacking camera and the defocused image by a first formula, wherein the first formula represents an association relationship among the camera parameters, the defocused image, the depth information, and the radiosity, and the first association relationship is a first formula in which the camera parameters and the defocused image in the first formula are known;

[0050] A second processing module is used to determine a second correlation relationship based on the defocused image sequence by a second formula, wherein the second formula represents the correlation relationship between the defocused image sequence, the original image characteristic value of the defocused image sequence, and the depth information, and the second correlation relationship is the second formula when the defocused image sequence in the second formula is known;

[0051] A third processing module is used to determine, for each defocused image, a first objective function corresponding to the defocused image based on the second association relationship and the first association relationship corresponding to the defocused image, wherein the first objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the depth information;

[0052] A fourth processing module, configured to determine first depth information according to each first objective function, wherein the first depth information includes depth information when a weighted sum value in each first objective function takes a minimum value;

[0053] The measurement module is used to obtain a measurement result of the depth information of the scene based on the first depth information.

[0054] In a third aspect, the present invention provides a computer-readable storage medium having instructions stored therein. When the instructions are executed on a terminal device, the terminal device executes all or part of the steps of the defocus depth measurement method described in the first aspect.

[0055] In a fourth aspect, the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and running on the processor, wherein when the processor executes the program, all or part of the steps of the defocus depth measurement method described in the first aspect are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic flow chart of a defocus depth measurement method provided by an embodiment of the present invention;

[0057] Figure 2 A schematic structural diagram of a defocus depth measurement device provided by an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following examples are provided to further explain and supplement the present invention and do not constitute any limitation to the present invention.

[0060] A defocus depth measurement method according to an embodiment of the present invention is described below with reference to the accompanying drawings.

[0061] Reference Figure 1 As shown, the present invention provides a defocus depth measurement method, including the following steps S1 to S6.

[0062] In step S1, a preset number of defocused image sequences are collected for a scene using a focus stacking camera, wherein each defocused image sequence includes a plurality of defocused images focused at different depths in the scene.

[0063] It can be understood that the focus stacking camera has a fixed imaging depth of field, which refers to a specific object distance range. The subject within the specific object distance range can find a corresponding focus plane on the imaging plane of the focus stacking camera. The image of the subject on the focus plane is clear, while the image on other imaging planes is blurred.

[0064] In this embodiment, the focal length of the focus stacking camera is adjusted to capture two-dimensional images of the same scene on different imaging planes of the focus stacking camera, thereby obtaining defocused images focused on different depths in the scene, wherein the scene includes at least one object, and each object should be located within the imaging depth of field of the focus stacking camera, so that the depth information of the scene can be obtained based on the two-dimensional images on different imaging planes, and then the object distance corresponding to each object can be obtained based on the depth information.

[0065] Optionally, in one embodiment, the focus stacking camera includes a liquid crystal lens, the liquid crystal lens includes a liquid crystal layer and two device orientation films, the liquid crystal layer contains multi-walled carbon nanotubes, and the liquid crystal layer is disposed between the two device orientation films.

[0066] The focal stacking camera consists of a liquid crystal lens (LC) doped with multi-walled carbon nanotubes (MWCNT), an image sensor (CCD), and a lens with a fixed focal length.

[0067] In this embodiment, the focal length range is determined according to the imaging depth of field of the focus stacking camera, the focal length is adjusted in steps based on a preset focal length adjustment within the focal length range, and after each focal length adjustment, a defocused image is captured of the scene, thereby obtaining a single defocused image sequence.

[0068] The above steps are repeated based on the same focus adjustment step and focus range until a preset number of defocused image sequences are obtained.

[0069] As a possible implementation, the focus adjustment step is set to meet Where Δf represents the focal length adjustment step, f max Indicates the maximum focal length of the liquid crystal lens, f min Indicates the minimum focal length of the liquid crystal lens.

[0070] Optionally, in one embodiment, the liquid crystal layer is prepared by the following method:

[0071] Mixing multi-walled carbon nanotubes and ethanol to obtain a first mixture;

[0072] Using an ultrasonic vibrator to vibrate the first mixture;

[0073] Mixing the first mixture after the oscillation treatment with liquid crystal to obtain a second mixture;

[0074] Using an ultrasonic vibrator to vibrate the second mixture;

[0075] Using an oven to remove ethanol from the second mixture after the shaking treatment;

[0076] Repeat the shaking treatment and ethanol removal operation for a preset number of times on the second mixture to obtain a third mixture;

[0077] The third mixture is poured into the liquid crystal box at a first preset temperature, and the liquid crystal box is sealed when the temperature of the liquid crystal box reaches a second preset temperature to obtain a liquid crystal layer, wherein the first preset temperature is greater than the second preset temperature.

[0078] Among them, the liquid crystal lens doped with multi-walled carbon nanotubes can be divided into three layers: the upper layer is a glass substrate, an ITO film and a polyimide layer, the middle layer is a liquid crystal layer doped with multi-walled carbon nanotubes, and the lower layer is a polyimide layer, an ITO film and a glass substrate, and the liquid crystal layer is arranged between the two polyimide layers.

[0079] Exemplarily, the process of preparing a multi-walled carbon nanotube-doped lens includes:

[0080] A single circular hole electrode pattern (ITO film) with a diameter of 2 mm was engraved on the ITO upper electrode by UV photolithography and wet hydrochloric acid (HCL); a device orientation film (polyimide layer) was formed by coating a polyimide (PI) layer on the inner surfaces of two glass substrates and rubbing along the x direction, which could induce a 20% pretilt angle of the liquid crystal molecules in the liquid crystal layer; the two device orientation films could be sandwiched by glass lens microspheres as spacers to form an empty liquid crystal box.

[0081] An appropriate amount of multi-walled carbon nanotubes was put into ethanol to obtain a first mixture, which was continuously oscillated for 3.5 hours using an ultrasonic vibrator. The first mixture after oscillation was then mixed with liquid crystal in a glass container to obtain a second mixture, which was further oscillated for 30 minutes.

[0082] Then, the second mixture after the oscillation treatment is placed in an oven at 100 degrees Celsius for 30 minutes (ethanol removal operation). In order to volatilize the ethanol to the greatest extent, the above operation is repeated many times so that the multi-walled carbon nanotubes are uniformly doped into the liquid crystal to reduce the agglomeration phenomenon. Among them, ethanol is used as a dispersing solvent, which can effectively make the multi-walled carbon nanotubes uniformly dispersed in the liquid crystal.

[0083] At 80° C. (the first preset temperature), the liquid crystal material is poured into the empty liquid crystal box by capillary action. After cooling to room temperature (the second preset temperature), the liquid crystal box is sealed with AB glue.

[0084] A fixed-focal-length lens is placed between the multi-walled carbon nanotube-doped liquid crystal lens and the CCD. The frequency and voltage of the square wave are set and added to the electrodes of the liquid crystal lens to adjust the focal length of the liquid crystal lens. The light emitted by the target object will be captured by the CCD camera placed behind the liquid crystal lens and the fixed-focal-length lens, thereby obtaining a two-dimensional image sequence with good lighting.

[0085] In step S2, for each defocused image, based on the camera parameters of the focus stacking camera and the defocused image, a first association relationship corresponding to the defocused image is determined through a first formula, wherein the first formula characterizes the association relationship between the camera parameters, the defocused image, the depth information and the radiosity, and the first association relationship is the first formula when the camera parameters and the defocused image in the first formula are known.

[0086] Optionally, in one embodiment, the camera parameters of the focus stacking camera include focal length and aperture value, and the first formula is expressed as follows:

[0087]

[0088] Among them, E bz (s) represents the radiance, s represents the depth information, A represents the aperture value, I i (y,x) represents the defocused image, I i (y,x) can be I1(y,x),...I n (y, x), n is the total number of defocused images in a preset number of defocused image sequences, F i represents the focal length, x and y represent the 2D pixel coordinates in the defocused image, and F represents the F-norm.

[0089] In step S3, based on the defocused image sequence, a second association relationship is determined by a second formula, wherein the second formula characterizes the association relationship between the defocused image sequence, the original image characteristic values ​​of the defocused image sequence and the depth information, and the second association relationship is the second formula when the defocused image sequence in the second formula is known.

[0090] It can be understood that the original image characteristic values ​​of the defocused image sequence are used to describe the image features of the defocused image sequence related to depth, and the image features may include texture features and spatial relationship features of each defocused image.

[0091] As a possible implementation manner, the preset number of defocused image sequences includes a first defocused image sequence and a second defocused image sequence, and both the first defocused image sequence and the second defocused image sequence should satisfy the relationship I b (y) = ∫h σ (y,x)I(x)dx, where x∈N y , N y is the defocused image sequence I in the fully focused image sequence I(y) b The set of pixels that contribute to the defocus blur of (y), the blur kernel h σ (y,x) is the Gaussian convolution model, σ(y) is the depth-related blur, x and y are the two-dimensional coordinates of the pixel, and b represents the sequence group, which is A or B here.

[0092] According to the above Gaussian convolution model, the second formula can be expressed as:

[0093]

[0094] in:

[0095]

[0096]

[0097] Among them, E d (s) represents the original image characteristic value, I A (y) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, ||·|| is the Euclid norm, H(·) is the step function, Δσ(y) represents the depth-dependent defocus blur amount, and x and y represent two-dimensional pixel coordinates.

[0098] It should be noted that the correlation between the first defocused image sequence, the second defocused image sequence and the depth-dependent defocus blur amount can be expressed as follows:

[0099]

[0100] Among them, I A (x) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, π represents pi, x and y represent two-dimensional pixel coordinates, σ A (y),σ B (y) represents the depth-related blur amount corresponding to the first defocused image sequence and the second defocused image sequence respectively.

[0101] The depth-dependent blur amount corresponding to pixel y in the defocused image sequence is σ(y)=γ1b(y), where b(y) represents the blur radius, F1 represents the focal length, D represents the aperture diameter, ν represents the image distance, s(y) represents the depth information, and γ1 represents the calibration coefficient.

[0102] In step S4, for each defocused image, a first objective function corresponding to the defocused image is determined based on the second association relationship and the first association relationship, wherein the first objective function characterizes the association relationship between the weighted sum of the original image characteristic value and the radiance and the depth information.

[0103] As a possible implementation, the first objective function is expressed as follows:

[0104]

[0105] Among them, E d(s) is the data fidelity item, which is used to characterize the characteristic value of the original image; E bz (s) is the imaging-related regularization term, used to characterize the radiance; E m (s) is the total variation regularization term (smoothness term), is the spatial gradient of the defocused image, s(y) is the depth information; μ is the weight factor.

[0106] It can be understood that the total variation regularization term and the imaging-related regularization term are prior information. The total variation regularization term is used to smooth the estimated depth information, and the imaging-related regularization term is used to implement brightness consistency constraints on the estimated depth information.

[0107] In step S5, first depth information is determined according to each first objective function, wherein the first depth information includes depth information when the weighted sum value in each first objective function takes a minimum value.

[0108] For example, Indicates the first depth information, E d (s) represents the data fidelity term, E bz (s) represents the imaging related regularization term, E m (s) represents the smoothness term, s represents the depth information, μ is a weight factor used to balance the data fidelity term and the regularization term.

[0109] In step S6, a measurement result of the depth information of the scene is obtained based on the first depth information.

[0110] Optionally, in one embodiment, the process of obtaining the measurement result of the depth information of the scene based on the first depth information includes:

[0111] For each defocused image, determining a defocus blur kernel image corresponding to the defocused image based on first depth information corresponding to the defocused image;

[0112] Determine the defocus blur kernel image corresponding to each defocus image as a defocus blur kernel image sequence corresponding to the defocus image sequence;

[0113] For each defocused image, the defocused image and the camera parameters corresponding to the defocused image are input into a third formula to obtain the target radiometry corresponding to the defocused image, wherein the third formula represents the correlation between the radiometry, the defocused image and the camera parameters;

[0114] Based on the defocused image sequence and the defocused blur kernel image sequence, a third association relationship is determined by a fourth formula, wherein the fourth formula characterizes the association relationship among the original image characteristic value, the defocused image sequence, the defocused blur kernel image sequence and the all-focused image sequence, and the third association relationship is the fourth formula when the defocused image sequence and the defocused blur kernel image sequence in the fourth formula are known;

[0115] For each defocused image, based on the target radiance corresponding to the defocused image and the third association relationship, a second objective function corresponding to the defocused image is determined, wherein the second objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the fully focused image;

[0116] Determine a target fully focused image sequence according to each second objective function, wherein the target fully focused image sequence includes a fully focused image when the weighted sum value in each second objective function takes a minimum value;

[0117] Based on the first depth information and the target all-focus image sequence, a measurement result of the depth information of the scene is obtained.

[0118] In this embodiment, the depth-related blur amount corresponding to the defocused image is determined based on the first depth information, and the defocus blur kernel image corresponding to the defocused image is:

[0119]

[0120] Among them, h σ (y, x) represents the defocus blur kernel image, σ(y) represents the depth-related blur amount, π represents pi, and x and y represent two-dimensional pixel coordinates.

[0121] It can be understood that the defocus blur kernel image represents the blur kernel corresponding to a single defocus image, and the defocused image can be represented as the convolution of its corresponding defocus blur kernel image and the fully focused image. The depth-related blur amount represents the degree of blur of the defocused image focused at a specific scene depth relative to the fully focused image (clear image).

[0122] Optionally, the fourth formula is expressed as follows:

[0123]

[0124] Where I represents the all-focus image sequence, E d (I) represents the original image characteristic value, I A (y) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, x and y represent two-dimensional pixel coordinates, h σA (y,x) represents the defocus blur kernel image sequence corresponding to the first defocus image sequence, h σB(y, x) represents the defocus blur kernel image sequence corresponding to the second defocus image sequence.

[0125] As a possible implementation, the second objective function is expressed as follows:

[0126] E d (I)+λE x (I)+κE bc (I)

[0127] Among them, E d (I) is the data fidelity item, which is used to characterize the characteristic value of the original image; E bc (I) is the imaging-related regularization term, used to characterize the radiance; E x (I) is a natural sparse prior, is the gradient operator, I(y) is the all-focus image sequence; λ and κ are the set regularization item weights, which are used to balance various data items (such as data fidelity items, imaging-related regularization items, etc.).

[0128] It should be noted that in the process of parameter estimation, the use of natural sparse priors can greatly reduce the number of parameters and complexity while maintaining accuracy; the imaging-related regularization term is used to achieve brightness consistency constraints on the estimation of the fully focused image.

[0129] For example, represents the target all-focus image sequence, E d (I) represents the data fidelity term, E bc (I) represents the imaging related regularization term, E x (I) represents the natural sparse prior, I represents the fully focused image sequence, and λ and κ are the weights of the regularization term.

[0130] Optionally, in one embodiment, the camera parameters of the focus stacking camera include focal length and aperture value, and the third formula is:

[0131]

[0132] Among them, E bc (I) represents the radiance, A represents the aperture value, and I i (y,x) represents the defocused image, F i represents the focal length, and x and y represent the two-dimensional pixel coordinates in the defocused image.

[0133] Optionally, in one embodiment, the process of obtaining a measurement result of the depth information of the scene based on the first depth information and the target all-focus image sequence includes:

[0134] Inputting the defocused image sequence, the target fully focused image sequence, and the defocus blur kernel image sequence into a fifth formula to obtain a target image energy of the target fully focused image sequence, wherein the fifth formula represents the correlation relationship between the image energy of the fully focused image sequence, the defocused image sequence, the defocus blur kernel image sequence, and the fully focused image sequence;

[0135] For each defocused image, based on the second association relationship, the target image energy, and the first association relationship corresponding to the defocused image, a third objective function corresponding to the defocused image is determined, wherein the third objective function represents the association relationship between the original image characteristic value, the weighted sum value between the radiance and the image energy, and the depth information;

[0136] Determine second depth information according to each third objective function, wherein the second depth information includes depth information when the weighted sum value in each third objective function takes a minimum value;

[0137] The second depth information is determined as a measurement result of the depth information of the scene.

[0138] The original image characteristic values ​​of the defocused image sequence are used to describe the image features of the defocused image sequence related to depth; the image energy of the fully focused image sequence can be a spectrum obtained by Fourier transforming the fully focused image sequence, and the spectrum is used to characterize the energy distribution of the image. For example, if there are more dark points in the spectrum, it means that the image is soft, otherwise the image must be sharp.

[0139] In this embodiment, the target image energy is:

[0140]

[0141] Among them, E f (s) represents the target image energy, I represents the target full focus image sequence, I A (y) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, x and y represent two-dimensional pixel coordinates, h σA (y,x) represents the defocus blur kernel image sequence corresponding to the first defocus image sequence, h σB (y, x) represents the defocus blur kernel image sequence corresponding to the second defocus image sequence.

[0142] As a possible implementation, the third objective function is expressed as follows:

[0143] α(E d (s)+νE f (s))+γE m (s)+ζE bz (s)

[0144] Among them, E d (s) represents the data fidelity term, E bz (s) represents the imaging related regularization term, E m (s) represents the smoothness term, E f (s) represents the target image energy, α represents the weight factor, ν represents the weight coefficient, and γ and ζ represent the regularization parameters.

[0145] It can be understood that the fully focused image sequence is a bridge to associate depth information with the defocused image sequence. Therefore, by adding the image energy of the fully focused image sequence as an energy constraint on the depth estimation result to the first objective function, the accuracy of the estimated first depth information is optimized.

[0146] For example, Indicates the second depth information, E d (s) represents the data fidelity term, E bz (s) represents the imaging related regularization term, E m (s) represents the smoothness term, E d (s) represents the data fidelity term, α represents the weight factor, ν represents the weight coefficient, γ and ζ represent the regularization parameters, and s represents the depth information.

[0147] As a possible implementation, an alternating direction method (ADM) is used to iteratively solve the second depth information.

[0148] The defocus depth measurement method provided in the above embodiment uses the camera parameters of the focus stacking camera and the defocused image to determine the correlation between the radiometry and the depth information, and introduces the correlation into the first objective function. By minimizing the first objective function, the influence of the radiometry on the first depth information acquisition process can be reduced, thereby improving the accuracy of the defocus depth measurement method.

[0149] In the above embodiments, although the steps are numbered, such as S1, S2, etc., these are only specific embodiments given in the present application. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0150] like Figure 2 As shown, an embodiment of the present invention provides a defocus depth measurement device, comprising:

[0151] An acquisition module 20 is used to collect a preset number of defocused image sequences of a scene using a focus stacking camera, wherein each defocused image sequence includes a plurality of defocused images focused at different depths in the scene;

[0152] A first processing module 30 is used to determine, for each defocused image, a first correlation relationship corresponding to the defocused image based on camera parameters of the focus stacking camera and the defocused image by a first formula, wherein the first formula represents a correlation relationship among the camera parameters, the defocused image, the depth information, and the radiosity, and the first correlation relationship is a first formula in which the camera parameters and the defocused image in the first formula are known;

[0153] A second processing module 40 is used to determine a second correlation relationship based on the defocused image sequence by a second formula, wherein the second formula represents the correlation relationship between the defocused image sequence, the original image characteristic value of the defocused image sequence, and the depth information, and the second correlation relationship is the second formula when the defocused image sequence in the second formula is known;

[0154] A third processing module 50 is used to determine, for each defocused image, a first objective function corresponding to the defocused image based on the second association relationship and the first association relationship corresponding to the defocused image, wherein the first objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the depth information;

[0155] A fourth processing module 60 is used to determine first depth information according to each first objective function, wherein the first depth information includes depth information when the weighted sum value in each first objective function takes a minimum value;

[0156] The measuring module 70 is used to obtain a measurement result of the depth information of the scene based on the first depth information.

[0157] Optionally, the measurement module 70 is specifically used to determine, for each defocused image, a defocus blur kernel image corresponding to the defocused image based on the first depth information corresponding to the defocused image; determine the defocus blur kernel images corresponding to each defocused image as a defocus blur kernel image sequence corresponding to the defocused image sequence; for each defocused image, input the defocused image and the camera parameters corresponding to the defocused image into a third formula to obtain the target radiometry corresponding to the defocused image, wherein the third formula characterizes the correlation between the radiometry, the defocused image and the camera parameters; based on the defocused image sequence and the defocus blur kernel image sequence, determine the third correlation through a fourth formula, wherein the fourth formula characterizes the original image characteristic value, the defocused image sequence, and the defocus blur kernel image sequence. and the fully focused image sequence, the third association relationship is the fourth formula when the defocused image sequence and the defocus blur kernel image sequence in the fourth formula are known; for each defocused image, based on the target radiance corresponding to the defocused image and the third association relationship, the second objective function corresponding to the defocused image is determined, wherein the second objective function characterizes the association relationship between the weighted sum of the original image characteristic value and the radiance and the fully focused image; according to each second objective function, the target fully focused image sequence is determined, wherein the target fully focused image sequence includes the fully focused image when the weighted sum in each second objective function takes the minimum value; based on the first depth information and the target fully focused image sequence, the measurement result of the depth information of the scene is obtained.

[0158] Optionally, the measurement module 70 is also used to input the defocused image sequence, the target fully focused image sequence, and the defocused blur kernel image sequence into the fifth formula to obtain the target image energy of the target fully focused image sequence, wherein the fifth formula characterizes the correlation between the image energy of the fully focused image sequence, the defocused image sequence, the defocused blur kernel image sequence, and the fully focused image sequence; for each defocused image, based on the second correlation relationship, the target image energy, and the first correlation relationship corresponding to the defocused image, determine the third objective function corresponding to the defocused image, wherein the third objective function characterizes the correlation between the weighted sum value between the original image characteristic value, the radiance and the image energy and the depth information; determine the second depth information according to each third objective function, wherein the second depth information includes the depth information when the weighted sum value in each third objective function takes the minimum value; and determine the second depth information as the measurement result of the depth information of the scene.

[0159] An embodiment of the present invention provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the steps of the defocus depth measurement method in any of the above embodiments.

[0160] like Figure 3As shown, an electronic device 500 provided by an embodiment of the present invention includes a memory 510, a processor 520, and a program 530 stored in the memory 510 and running on the processor 520. When the processor 520 executes the program 530, the steps of the defocus depth measurement method of any of the above embodiments are implemented.

[0161] Among them, the electronic device 500 can be a computer, a mobile phone, etc. Correspondingly, its program 530 is a computer software or a mobile phone App, etc., and the above-mentioned parameters and steps in an electronic device 500 of the present invention can refer to the parameters and steps in the embodiment of the defocus depth measurement method above, and will not be repeated here.

[0162] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), and it can also be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0163] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0164] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for measuring defocus depth, characterized in that: include: Using a focus stacking camera to collect a preset number of defocused image sequences of a scene, wherein each of the defocused image sequences includes a plurality of defocused images focused at different depths in the scene; For each of the defocused images, based on the camera parameters of the focus stacking camera and the defocused image, determine a first association relationship corresponding to the defocused image by a first formula, wherein the first formula represents an association relationship among the camera parameters, the defocused image, the depth information, and the radiosity, and the first association relationship is a first formula in the case where the camera parameters and the defocused image in the first formula are known; Based on the defocused image sequence, a second association relationship is determined by a second formula, wherein the second formula characterizes an association relationship between the defocused image sequence, original image characteristic values ​​of the defocused image sequence, and depth information, and the second association relationship is a second formula in the case where the defocused image sequence in the second formula is known; For each of the defocused images, based on the second association relationship and the first association relationship, determine a first objective function corresponding to the defocused image, wherein the first objective function represents an association relationship between a weighted sum value between an original image characteristic value and a radiometry and depth information; Determine first depth information according to each of the first objective functions, wherein the first depth information includes depth information when a weighted sum value in each of the first objective functions takes a minimum value; Obtaining a measurement result of the depth information of the scene based on the first depth information; The camera parameters of the focus stacking camera include focal length and aperture value, and the first formula is expressed as follows: Among them, E bz (s) represents the radiance, s represents the depth information, A represents the aperture value, I i (y, x) represents the defocused image, F i represents the focal length, x and y represent the two-dimensional pixel coordinates in the defocused image, and F represents the F norm; The preset number of defocused image sequences includes a first defocused image sequence and a second defocused image sequence, and the second formula is expressed as follows: in: Among them, E d (s) represents the original image characteristic value, I A (y) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, ||·|| is the Euclid d norm, H(·) is a step function, and Δσ(y) represents the depth-dependent defocus blur amount.

2. The method according to claim 1, characterized in that The obtaining a measurement result of the depth information of the scene based on the first depth information includes: For each of the defocused images, determining a defocus blur kernel image corresponding to the defocused image based on first depth information corresponding to the defocused image; Determining the defocus blur kernel image corresponding to each of the defocus images as a defocus blur kernel image sequence corresponding to the defocus image sequence; For each of the defocused images, inputting the defocused image and the camera parameters corresponding to the defocused image into a third formula to obtain the target radiance corresponding to the defocused image, wherein the third formula represents the correlation between the target radiance, the defocused image and the camera parameters; Based on the defocused image sequence and the defocus blur kernel image sequence, a third association relationship is determined by a fourth formula, wherein the fourth formula characterizes the association relationship between the original image characteristic value, the defocused image sequence, the defocus blur kernel image sequence and the all-focused image sequence, and the third association relationship is the fourth formula when the defocused image sequence and the defocus blur kernel image sequence in the fourth formula are known; For each of the defocused images, based on the target radiance corresponding to the defocused image and the third association relationship, determine a second objective function corresponding to the defocused image, wherein the second objective function represents the association relationship between a weighted sum value between the original image characteristic value and the radiance and the fully focused image; Determine a target all-focused image sequence according to each of the second objective functions, wherein the target all-focused image sequence includes an all-focused image when the weighted sum value in each of the second objective functions takes a minimum value; Obtaining a measurement result of the depth information of the scene based on the first depth information and the target all-focus image sequence; The third formula is: Among them, E bc (I) represents the target radiance; The fourth formula is: Where I represents the all-focus image sequence, E d (I) represents the original image characteristic value, h σA (y, x) represents the defocus blur kernel image sequence corresponding to the first defocus image sequence, h σB (y, x) represents the defocus blur kernel image sequence corresponding to the second defocus image sequence.

3. The method according to claim 2, characterized in that The obtaining a measurement result of the depth information of the scene based on the first depth information and the target all-focus image sequence includes: Inputting the defocused image sequence, the target fully focused image sequence, and the defocus blur kernel image sequence into a fifth formula to obtain a target image energy of the target fully focused image sequence, wherein the fifth formula represents a correlation relationship among the image energy of the fully focused image sequence, the defocused image sequence, the defocus blur kernel image sequence, and the fully focused image sequence; For each of the defocused images, based on the second association relationship, the target image energy, and the first association relationship corresponding to the defocused image, determine a third objective function corresponding to the defocused image, wherein the third objective function represents an association relationship between a weighted sum value of an original image characteristic value, radiance, and image energy, and depth information; Determine second depth information according to each of the third objective functions, wherein the second depth information includes depth information when the weighted sum value in each of the third objective functions takes a minimum value; The second depth information is determined as a measurement result of the depth information of the scene.

4. The method according to any one of claims 1 to 3, characterized in that The focus stacking camera comprises a liquid crystal lens, which comprises a liquid crystal layer and two device orientation films. The liquid crystal layer contains multi-walled carbon nanotubes, and the liquid crystal layer is arranged between the two device orientation films.

5. The method according to claim 4, characterized in that The liquid crystal layer is prepared by the following method: Mixing multi-walled carbon nanotubes and ethanol to obtain a first mixture; Using an ultrasonic vibrator to vibrate the first mixture; Mixing the first mixture after the oscillation treatment with liquid crystal to obtain a second mixture; Using an ultrasonic vibrator to vibrate the second mixture; Using an oven to remove ethanol from the second mixture after the shaking treatment; Repeat the shaking treatment and the ethanol removal operation for a preset number of times on the second mixture to obtain a third mixture; The third mixture is poured into a liquid crystal box at a first preset temperature, and the liquid crystal box is sealed when the temperature of the liquid crystal box reaches a second preset temperature to obtain the liquid crystal layer, wherein the first preset temperature is greater than the second preset temperature.

6. A defocus depth measurement device, characterized in that: include: An acquisition module, configured to collect a preset number of defocused image sequences of a scene using a focus stacking camera, wherein each of the defocused image sequences includes a plurality of defocused images focused at different depths in the scene; A first processing module is used to determine, for each of the defocused images, a first association relationship corresponding to the defocused image based on camera parameters of the focus stacking camera and the defocused image by using a first formula, wherein the first formula represents an association relationship among camera parameters, defocused images, depth information, and radiosity, and the first association relationship is a first formula in which the camera parameters and the defocused image in the first formula are known; A second processing module is used to determine a second association relationship based on the defocused image sequence by a second formula, wherein the second formula represents an association relationship between the defocused image sequence, original image characteristic values ​​of the defocused image sequence, and depth information, and the second association relationship is a second formula in which the defocused image sequence in the second formula is known; A third processing module is used to determine, for each of the defocused images, a first objective function corresponding to the defocused image based on the second association relationship and the first association relationship corresponding to the defocused image, wherein the first objective function represents the association relationship between the weighted sum value between the original image characteristic value and the radiance and the depth information; a fourth processing module, configured to determine first depth information according to each of the first objective functions, wherein the first depth information includes depth information when a weighted sum value in each of the first objective functions takes a minimum value; A measuring module, configured to obtain a measurement result of the depth information of the scene based on the first depth information; The camera parameters of the focus stacking camera include focal length and aperture value, and the first formula is expressed as follows: Among them, E bz (s) represents the radiance, s represents the depth information, A represents the aperture value, I i (y, x) represents the defocused image, F i represents the focal length, x and y represent the 2D pixel coordinates in the defocused image, and F represents the F-norm: The preset number of defocused image sequences includes a first defocused image sequence and a second defocused image sequence, and the second formula is expressed as follows: in: Among them, E d (s) represents the original image characteristic value, I A (y) represents the first defocused image sequence, I B (y) represents the second defocused image sequence, ||·|| is the Euclid norm, H(·) is the step function, and Δσ(y) represents the depth-dependent defocus blur amount.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the steps of the defocus depth measurement method according to any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the defocus depth measurement method according to any one of claims 1 to 5 are implemented.

Images