A depth estimation method based on focused image and image processing device

Through superpixel segmentation and image registration, a focus evaluation curve is constructed to calculate the depth estimate, solving the problems of inconsistent reference space, inaccurate depth changes, and noise-sensitive in three-dimensional depth estimation, achieving more efficient and accurate depth estimation.

CN113344941BActive Publication Date: 2024-11-26SHENZHEN HUAHAN WEIYE TECH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202110529700.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-14
Publication Date
2024-11-26
Estimated Expiration
2041-05-14

AI Technical Summary

Technical Problem

The existing three-dimensional depth estimation methods have problems such as measurement errors caused by inconsistent reference space of image sequences, inaccurate changes in surface depth of objects, low algorithm efficiency, and noise sensitivity.

Method used

By acquiring multiple focused images with different focal lengths, superpixel segmentation and image registration, a focus evaluation curve is constructed, the depth estimation value of pixel points is calculated, the depth estimation image is formed, and the depth estimation image is optimized to improve accuracy.

Benefits of technology

The accuracy and noise resistance of three-dimensional depth estimation are improved, errors caused by inconsistency in the reference space of the image sequence are avoided, and computational complexity is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113344941B_ABST
    Figure CN113344941B_ABST
Patent Text Reader

Abstract

The present application relates to a depth estimation method and image processing device based on focused images, wherein the depth estimation method includes: acquiring an image sequence, performing superpixel segmentation on each focused image in the image sequence to obtain multiple superpixel regions; constructing a focus evaluation curve for each superpixel region; and calculating the depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image. On the one hand, the technical solution adopts superpixel segmentation, and the segmentation boundary of the segmented superpixel region can coincide with the region where the real object has a drastic depth change, which can avoid the problem of over-smoothing of the region with a large actual depth change in the prior art; on the other hand, the focus evaluation curve constructed by the technical solution has the advantages of a wide steep area width and a high clarity ratio, which can ensure the accuracy of depth estimation and the ability to resist noise interference.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a depth estimation method and an image processing device based on a focused image. Background Art

[0002] Three-dimensional depth estimation is crucial to computer vision applications. As an important technology in the field of computer vision, three-dimensional depth estimation based on two-dimensional images has been widely used in surveying and mapping, automated manufacturing, biomedicine, autonomous driving and other fields.

[0003] In order to achieve three-dimensional depth estimation, some technicians have proposed a method of depth from focus (DFF). The general processing process of this method can be summarized as follows: first, the object is deeply focused by changing parameters such as the focal length of the camera to obtain multiple focused images, and the focusing function curve of the pixels in the image is calculated according to a certain focusing function. Then, the depth estimation result of each pixel in the image is obtained; in some methods, it is further proposed to use linear weighting to improve the focusing function to compensate for the problem that the focus estimation of the general method is not accurate enough.

[0004] However, the three-dimensional depth estimation used in the prior art still has the following problems: (1) Changing the focal length of the camera for image acquisition cannot avoid the inconsistency of the reference space of the image sequence, and even using a telecentric lens will cause certain measurement errors; (2) The imaging area where the depth of the object surface changes dramatically is easily over-smoothed during the focusing function calculation process, resulting in inaccurate surface depth estimation; (3) Calculating the focusing function curve for each pixel on the image will lead to low algorithm efficiency, which is not conducive to the efficient computing requirements in practical applications; (4) When using the focusing function curve to calculate the depth estimation result, it is sensitive to noise interference, which seriously affects the result of three-dimensional depth estimation. Summary of the invention

[0005] The main technical problem solved by the present application is: how to improve the accuracy of three-dimensional depth estimation. To solve the above technical problem, the present application provides a depth estimation method based on a focused image and an image processing device.

[0006] According to the first aspect, an embodiment provides a depth estimation method based on a focused image, which includes: acquiring an image sequence, the image sequence including a plurality of focused images with different focal depths; performing superpixel segmentation on each focused image in the image sequence to obtain a plurality of superpixel regions; constructing a focus evaluation curve for each of the superpixel regions; and calculating a depth estimation value for each pixel point based on the focus evaluation curve to form a depth estimation image.

[0007] The step of performing superpixel segmentation on each focused image in the image sequence to obtain a plurality of superpixel regions comprises: forming each focused image in the image sequence into a composite image M by using a pixel gradient index. merge ; Initializing the seed points of the composite image according to a preset spacing d to obtain a plurality of first seed points, and selecting a pixel point with the smallest pixel gradient within the neighborhood of each of the first seed points as a second seed point;

[0008] Perform superpixel clustering step: for any pixel point p in the synthetic image, calculate the difference measure between the pixel point p and each surrounding second seed point s, and attribute the pixel point p to the second seed point corresponding to the minimum difference measure value; the difference measure value is expressed as

[0009]

[0010] Among them, sqrt() is the square root calculation function, (x p ,y p ) is the coordinate of pixel point p, (x s ,y s ) is the coordinate of the second seed point s, M merge represents the composite image, I represents the index image corresponding to the composite image, and N is the number of images in the image sequence; corresponding second seed point regions are formed according to the pixel points belonging to each second seed point, and the composite image is superpixel segmented using each second seed point region to form a plurality of superpixel regions.

[0011] Before performing superpixel segmentation on each focused image in the image sequence, an image registration step is also included. The image registration step includes: selecting a focused image in the image sequence as a reference image, and selecting any other focused image as an image to be registered, and calculating the derivative F of the image to be registered relative to the reference image. x 、F y 、F t , where x, y, and t are derivative parameters about the X-axis, Y-axis, and time T, respectively; construct the constraint equation F x u+F y v+F t = 0, and the motion vector u on the X axis = x j -x i and the motion vector v=y on the Y axis j -y i Substituting the preset spatial transformation relationship, we get

[0012]

[0013] The spatial transformation relationship is expressed as

[0014]

[0015] Calculate the transformation parameters a1, ..., a6 and configure the spatial transformation relationship, and perform spatial transformation on the image to be registered according to the configured spatial transformation relationship to transform the image sequence {M i} are transformed into the same reference space.

[0016] The constructing of the focus evaluation curve of each super pixel region includes: randomly sampling each super pixel region to obtain a plurality of sampling points; establishing a focus evaluation function model and expressing it as

[0017] F(x,y)=w1f1(x,y)+w2f2(x,y)+w3f3(x,y);

[0018] Among them, f1(), f2(), f3() are different types of focusing functions, (x, y) are the coordinates of any sampling point, w1, w2, w3 are weight coefficients to be configured; each sampling point is used to participate in the calculation of the focusing evaluation function model, and the focusing evaluation curve of each sampling point is constructed and expressed as {F i (x, y)}(i=0,1,…,N); For each sampling point constructed, the focus evaluation curve {F i (x,y)}, and set the loss function of the focus evaluation function model to

[0019]

[0020] Among them, F min (x,y)=min{F i (x,y)}, F max (x,y)=max{F i (x, y)} are the maximum and minimum values ​​of the focus evaluation curve at the sampling point (x, y), respectively, and W peak is the width of the sensitive area; the focus evaluation function model is trained, and the weight coefficients w1, w2, and w3 are calculated when the corresponding loss function converges, and the focus evaluation function model is configured using the calculated weight coefficients to obtain the focus evaluation curve of each sampling point.

[0021] The depth estimation value of each pixel is calculated according to the focus estimation curve to form a depth estimation image, including: for each pixel (x, y) in the superpixel area, calculating the pixel (x, y) on the focus estimation curve {F i The maximum value F in (x,y)}(i=0,1,…,N-1) max(x, y), determine the focused image corresponding to the maximum value in the image sequence {M i}(i=0,1…N-1) index value k; through three points (k-1,F k-1 (x,y))、(k,F k (x,y))、(k+1,F k+1 (x,y)) fits a quadratic curve and calculates the maximum position k of the quadratic curve in [k-1,k+1] max (x,y), using the position k of the maximum value max (x, y) Determine the depth estimate of the pixel (x, y) D(x, y) = 255 × k max (x, y) / N; obtain the depth estimation value of each pixel point in the superpixel area and form a depth estimation image.

[0022] After obtaining the depth estimation image, an image optimization step is included, and the image optimization step includes: performing difference calculation on each pixel point (i, j) in the depth estimation image and expressing it as

[0023]

[0024] Take the minimum difference calculation result min(|d c |,c=1,...,8) as d m , re-assign the depth estimation value D(i,j) of the pixel point (i,j) and obtain Using the assignment result Get a new depth estimation image.

[0025] According to the second aspect, an embodiment provides an image processing device, which includes: a camera, used to continuously capture images of a target scene and form an image sequence, wherein the image sequence includes a plurality of focused images with different focal depths; a processor, connected to the camera, used to process the image sequence according to the depth estimation method described in the first aspect above, to obtain a depth estimation image of the target scene; and a display, connected to the processor, used to display the focused image and / or the depth estimation image.

[0026] The processor includes: an image acquisition module, used to acquire the image sequence from the camera; a pixel segmentation module, used to perform super-pixel segmentation on each focused image in the image sequence to obtain a plurality of super-pixel regions; a focus evaluation module, used to construct a focus evaluation curve for each super-pixel region; and a depth estimation module, used to calculate a depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image.

[0027] According to the third aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement the depth estimation method described in the first aspect.

[0028] The beneficial effects of this application are:

[0029] According to the above-mentioned embodiment, a depth estimation method based on a focused image and an image processing device are provided, wherein the depth estimation method comprises: acquiring an image sequence; performing superpixel segmentation on each focused image in the image sequence to obtain a plurality of superpixel regions; constructing a focus evaluation curve for each superpixel region; and calculating the depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image. On the one hand, the technical solution adopts superpixel segmentation, and the segmentation boundary of the segmented superpixel region can coincide with the region where the depth of the real object undergoes a drastic depth change, thereby avoiding the problem of over-smoothing of the region with a large actual depth change in the prior art; on the other hand, the focus evaluation curve constructed by the technical solution has the advantages of a wide steep area width and a high clarity ratio compared to the focus curve calculated by the general method, and can ensure the accuracy of depth estimation and the ability to resist noise interference.

[0030] In the present application, since the optical flow method is used to perform image registration on each focused image in the image sequence before superpixel segmentation, not only can each focused image in the image sequence be converted to the same reference space, but also the error caused by the inconsistency of the reference space of the image sequence can be avoided.

[0031] In the present application, when constructing the focus evaluation curve of each superpixel area, the superpixel area is randomly sampled and only the focus evaluation function is calculated for the sampling points. Since the superpixel area has the characteristic of smooth curvature change, this method can greatly reduce the complexity of the calculation without affecting the accuracy.

[0032] In the present application, after obtaining the depth estimation image, the depth estimation image is also optimized. The estimated depth image is optimized through curvature filtering, so that the final depth estimation image has stronger anti-interference ability against noise, further improving the accuracy of three-dimensional depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Flow chart of the depth estimation method based on focused image in this application;

[0034] Figure 2 Flowchart for superpixel segmentation;

[0035] Figure 3 Flowchart for constructing a focus assessment curve;

[0036] Figure 4 A flow chart for forming a depth estimation image;

[0037] Figure 5 A flowchart of the steps for image registration;

[0038] Figure 6 Flowchart of steps for image optimization;

[0039] Figure 7a This is a schematic diagram of the results obtained when seed initialization and superpixel segmentation are performed with a larger spacing;

[0040] Figure 7b This is a schematic diagram of the results obtained by performing seed initialization and superpixel segmentation with a small spacing;

[0041] Figure 8 is a schematic diagram of the structure of the image processing device in this application;

[0042] Fig. 9 is a schematic diagram of the structure of the processor;

[0043] Fig.10a is the depth estimation map with noise interference;

[0044] Fig.10b is a depth estimation map with smoothed edges;

[0045] Fig.10c A depth estimation image obtained by processing the depth estimation method of the present application;

[0046] Fig.11 FIG. 1 is a schematic diagram of the structure of an image processing device in another embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are intended to enable the present application to be better understood. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, and methods. In some cases, some operations related to the present application are not shown or described in the specification, in order to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0048] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0049] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0050] The technical solution of the present application is described in detail below with reference to the embodiments.

[0051] Embodiment 1

[0052] Please refer to Figure 1 In this embodiment, a depth estimation method based on a focused image is disclosed, which includes steps 110-140, which are described below respectively.

[0053] Step 110, acquiring an image sequence, the image sequence including a plurality of focused images with different focal depths; in order to ensure the validity of each focused image, the object depth to be estimated should be within the focus depth range of the entire image sequence.

[0054] It should be noted that a camera or video camera can be used to capture the target scene, and the focal length can be adjusted by automatic zoom or manual zoom to obtain a set of image sequences containing different focus depths, and then the processor obtains the image sequence from the camera or video camera to perform further image processing. The target scene displayed in each focused image can be a physical object such as a road, a building, a tree, a person, or an object. The purpose of image processing on the focused image is to obtain the depth information of these physical objects in the target scene.

[0055] Step 120, superpixel segmentation is performed on each focused image in the image sequence to obtain multiple superpixel regions. It should be noted that superpixel segmentation refers to the process of subdividing a digital image into multiple image sub-regions (superpixels). Superpixels are small regions composed of a series of pixels with adjacent positions and similar characteristics such as color, brightness, and texture. Most of these small regions retain effective information for further image segmentation and generally do not destroy the boundary information of objects in the image. It can be understood that superpixels, as the basic unit of other image processing algorithms, can greatly reduce the data processing dimension and eliminate some abnormal pixels.

[0056] In this embodiment, in order to achieve superpixel segmentation of the image sequence, it is necessary to form each focused image into a composite image, and then perform superpixel segmentation on the composite image. For example, when performing superpixel segmentation on the composite image, the seed point (cluster center) can be initialized, the seed point can be reselected in the neighborhood of the seed point, and each pixel point in the neighborhood around each seed point can be assigned an attribution label. After the seed point area is determined, multiple superpixel areas are formed in the form of image segmentation. The implementation process of superpixel segmentation will be described in detail below.

[0057] Step 130, constructing a focus evaluation curve for each superpixel region. For example, a focus evaluation function model can be established to represent the focus curve of the pixels in the superpixel region, and the weight coefficient of the model can be configured by training the focus evaluation function model, and then the focus evaluation curve of the pixel point is obtained, and then the depth estimation operation can be performed on the pixel point.

[0058] Step 140 , calculating a depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image.

[0059] In this embodiment, see Figure 2 The above step 120 mainly involves the process of superpixel segmentation, which may specifically include steps 121-124, which are described as follows.

[0060] Step 121, each focused image in the image sequence is formed into a composite image M by pixel gradient indexing. merge .

[0061] In a specific embodiment, for the image sequence {M i}(i=0,1…N-1), contains N focused images, each focused image is represented by M i Then each focused image M can be calculated i The pixel gradient of each focused image is used to obtain the corresponding gradient image. And construct the gradient image sequence The gradient image is the focused image M i Next, in the gradient image sequence In the image sequence {M, the gradient images are compared and the focused image corresponding to the maximum pixel gradient at each pixel coordinate is obtained. i}, construct the index image I; that is, in the gradient image sequence The image sequence index with the largest gradient is obtained for each pixel position, and then an index image I is obtained. Then, the image sequence {M i} corresponds to the pixel value of the focused image at the corresponding pixel coordinate, and the synthetic image M is formed by pixel assignment. merge , the assignment process can be expressed as M merge (i,j)=M I(i,j) (i, j), where (i, j) is the pixel coordinate. Through the above process, the image sequence {M i Each focused image in}(i=0,1...N-1) forms a composite image M merge .

[0062] It should be noted that if the focused image is regarded as a two-dimensional discrete function, then the pixel gradient of the image is actually the derivative of this two-dimensional discrete function: G(i,j) = |d x (i,j)|+|d y (i,j)|,d x (i,j)=R(i+1,j)-R(i,j),d y (i,j)=R(i,j+1)-R(i,j), where R is the pixel value (such as grayscale value).

[0063] Step 122, initialize the seed points of the composite image according to the preset spacing d to obtain a plurality of first seed points, and select the pixel point with the smallest pixel gradient in the neighborhood of each first seed point as the second seed point.

[0064] In a specific implementation, when the seed points are initialized, the seed points can be evenly distributed in the synthetic image according to the set number of superpixels; assuming that the synthetic image has a total of X pixels, which are pre-divided into K superpixels of the same size, then the size of each superpixel is X / K, and the distance between adjacent first seed points is approximately d=sqrt(X / K); at this time, the first seed point area is a point area of ​​(k1*d, k2*d), where k1, k2 represent the index of the seed point row and column direction, and d represents the distance between adjacent first seed points. Reselect seed points in the n×n neighborhood (for example, 3×3 neighborhood) of each first seed point, obtain the gradient values ​​of all pixels in the neighborhood, and move the seed point to the place with the smallest gradient in the neighborhood, so as to obtain the second seed point; the purpose of this is to avoid the seed point falling on the contour boundary with a large gradient, so as not to affect the subsequent clustering effect.

[0065] The superpixel clustering process will be described in detail in the following steps 123 and 124.

[0066] Step 123, for any pixel point p in the synthetic image, calculate the difference measure between the pixel point p and the surrounding second seed points s, and attribute the pixel point p to the second seed point corresponding to the minimum difference measure value; for example, for each second seed point s, the pixel points in the area of ​​m×m with the second seed point s as the center are used to determine the attribution of the seed point (where m is a configurable parameter). The difference measure value involved here is expressed as

[0067]

[0068] Among them, sqrt() is the square root calculation function, (x p ,y p ) is the coordinate of pixel point p, (x s ,y s ) is the coordinate of the second seed point s, M merge represents a composite image, I represents the index image corresponding to the composite image, and N is the number of images in the image sequence.

[0069] Step 124, forming corresponding second seed point regions according to the pixel points belonging to each second seed point, and performing superpixel segmentation on the synthesized image using each second seed point region to form a plurality of superpixel regions.

[0070] In a specific embodiment, the position coordinates of each second seed point are recalculated based on the pixel points belonging to each second seed point, and the central expansion area is the second seed point area. The position coordinates of each second seed point are expressed as

[0071]

[0072] Among them, (x i ,y i ) is the coordinate of the pixel point belonging to the second seed point, n is the number of pixel points belonging to the second seed point, and (x′, y′) is the recalculated position coordinate of the second seed point.

[0073] Furthermore, in order to accurately determine the second seed point area, the superpixel clustering step (i.e., step 123 and step 124) can be performed again on the synthetic image using the position coordinates of the recalculated second seed points until the position change of each recalculated second seed point converges, or the calculation reaches a preset number of times (e.g., 10 times). In other words, the above steps 123 and 124 are repeated until the change in the position distance of the second seed point converges, or the number of iterations reaches a preset number of times. At this time, the position coordinates of each second seed point can be accurately obtained, and the central expansion area (e.g., m×m) is the second seed point area.

[0074] See also Figure 7a and Figure 7b , perform superpixel clustering on the image of the circuit board. Figure 7a The diagram is a schematic diagram of the results of seed point initialization and superpixel segmentation with a larger spacing. Since a larger spacing d is used for seed point initialization, not only does it lead to a relatively large area for each second seed point, but it also results in many irregular superpixel areas in superpixel segmentation. Each area represents the outline of the components on the circuit board. Figure 7b This is a schematic diagram of the results of seed point initialization and superpixel segmentation using a smaller spacing. Due to the use of a smaller spacing d for seed point initialization, the second seed point area is relatively small, the superpixel segmentation is more refined, and more irregular superpixel areas can be obtained. Each area is as compact and neat as a cell, and the neighborhood features are easier to express. The compactness and contour retention of the generated superpixels are ideal.

[0075] It can be understood that the superpixel segmentation method used here allows the segmentation boundaries of the segmented superpixel areas to coincide with areas where the depth of real objects undergoes drastic changes, thereby avoiding the problem of over-smoothing of areas with large actual depth changes in the prior art.

[0076] In this embodiment, see Figure 3 The above-mentioned step 130 mainly involves the process of constructing a focus evaluation curve, which may specifically include steps 131-134, which are described as follows.

[0077] Step 131, randomly sample each superpixel region to obtain multiple sampling points. Since the curvature between pixels in a single superpixel region segmented by superpixel is relatively smooth, each superpixel region can be randomly sampled, and then only the focus evaluation curve is calculated for the sampled points, so as to further estimate the depth of the sampled points, and the remaining points are obtained by interpolation calculation using the sampled points.

[0078] In a specific embodiment, a random array with a number of n and a value ranging from 0 to 1 can be obtained by a mixed remainder method: Where n = S SuperPixel / 25, S SuperPixel represents the area of ​​the current superpixel region. Then, the superpixel region can be indexed as R i *S SuperPixel The points are recorded as sampling points, so that the focusing evaluation function can be calculated for the sampling points.

[0079] Step 132, establish a focus evaluation function model and express it as

[0080] F(x,y)=w1f1(x,y)+w2f2(x,y)+w3f3(x,y);

[0081] Among them, f1(), f2(), f3() are different types of focusing functions, (x, y) are the coordinates of any sampling point, and w1, w2, w3 are weight coefficients to be configured.

[0082] In a specific embodiment, the three focusing functions of EOG, SML and FSWM can be integrated through the Boosting method in ensemble learning to establish a focusing evaluation function model, which has advantages such as higher clarity and wider sensitive area width compared with the existing general methods.

[0083] The focused evaluation function model of Boosting is expressed as

[0084] F boosting (x) = w1*f EOG (x)+w2*f SML (x)+w3*f FSWM (x);

[0085] Among them, the EOG function is expressed as The SML function is expressed as The FSWM function is expressed as

[0086] In the FSWM function, the parameters are

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Among them, med{} is the function of taking the median.

[0094] Step 133, using each sampling point to participate in the calculation of the focus evaluation function model, constructing a focus evaluation curve for each sampling point and expressing it as {F i (x, y)}(i=0,1,…,N-1). For each sampling point constructed, the focus evaluation curve {F i (x,y)}, and set the loss function of the focus evaluation function model to

[0095]

[0096] Among them, F min(x,y)=min{F i (x,y)}, F max (x,y)=max{F i (x, y)} are the maximum and minimum values ​​of the focus evaluation curve at the sampling point (x, y), respectively, and W peak is the width of the sensitive area. For example, the sensitive area can be the area where the focus curve function value is greater than 0.1×(f max -f min )+f min The length of the region, f max 、f min Represent the maximum and minimum values ​​of the focusing curve respectively.

[0097] Step 134, training the focus evaluation function model, calculating the weight coefficients w1, w2, w3 when the corresponding loss function converges, configuring the focus evaluation function model using the calculated weight coefficients and obtaining the focus evaluation curve for each sampling point.

[0098] In a specific embodiment, in order to configure the focus evaluation function model through training, the weight coefficients w1, w2, and w3 may be updated using a gradient descent method, so that Until convergence (the subscript l in the formula is the number of the weight coefficient). After obtaining the weight coefficients w1, w2, and w3, substituting them into the focus evaluation function model can obtain the complete expression of F(x, y) = w1f1(x, y) + w2f2(x, y) + w3f3(x, y), and then the expression of the focus evaluation curve can be further obtained {F i (x,y)}(i=0,1,…,N-1).

[0099] It can be understood that when constructing the focus evaluation curve of each superpixel area, the superpixel area is randomly sampled and only the focus evaluation function is calculated for the sampling points. Since the superpixel area has the characteristic of smooth curvature change, this method can greatly reduce the complexity of the calculation without affecting the accuracy.

[0100] In this embodiment, see Figure 4 The above-mentioned step 140 mainly involves the process of forming a depth estimation image, which may specifically include steps 141-143, which are described as follows.

[0101] Step 141, for each pixel point (x, y) in the superpixel region, calculate the pixel point (x, y) on the focus evaluation curve {F i The maximum value F in (x,y)}(i=0,1,…,N-1) max (x, y), determine the focused image corresponding to the maximum value in the image sequence {M i}(i=0,1…N-1).

[0102] Step 142, through three points (k-1, F k-1 (x,y))、(k,F k (x,y))、(k+1,F k+1 (x,y)) fits a quadratic curve and calculates the maximum position k of the quadratic curve in [k-1,k+1] max (x,y), using the position k of the maximum value max (x,y) Determine the depth estimate of the pixel (x,y) D(x,y) = 255*k max (x,y) / N.

[0103] Step 143: Obtain the depth estimation value of each pixel point in the superpixel area and form a depth estimation image. The depth estimation image formed at this time can be represented as P.

[0104] It can be understood that, since the constructed focus evaluation curve has the advantages of a wide steep area and a high clarity ratio compared to the focus curve calculated by a general method, the accuracy of depth estimation and the ability to resist noise interference can be guaranteed.

[0105] In this embodiment, before performing superpixel segmentation on each focused image in the image sequence, an image registration step is also included. Figure 5 The image registration step may specifically include steps 151-153, which are described as follows.

[0106] Step 151, in the image sequence {M i Select a focused image from the image set (i=0, 1…N-1) as the reference image, and select any other focused image as the image to be registered, and calculate the derivative F of the image to be registered relative to the reference image. x 、F y 、F t , where x, y, and t are the derivative parameters about the X-axis, Y-axis, and time T, respectively.

[0107] Step 152: According to the brightness constant constraint principle, the constraint equation F can be constructed. x u+F y v+F t = 0, and the motion vector u on the X axis = x j -x i and the motion vector v=y on the Y axis j -y i Substituting the preset spatial transformation relationship, we get

[0108]

[0109] Among them, the spatial transformation relationship is expressed as

[0110]

[0111] Then it can be understood that the matrix can be configured by only finding a1, ..., a6 in the formula Thus, the spatial transformation relationship between the image to be registered and the reference image is calculated.

[0112] Step 153: According to the formula in step 152, the transformation parameters a1, ..., a6 can be calculated. Then, the spatial transformation relationship can be configured using these transformation parameters. The image to be registered can be spatially transformed according to the configured spatial transformation relationship. i After the remaining focused images in}(i=0,1…N-1) are spatially transformed, the image sequence {M i} are transformed into the same reference space.

[0113] It can be understood that, since the optical flow method is used to perform image registration on each focused image in the image sequence before superpixel segmentation, not only can each focused image in the image sequence be converted to the same reference space, but also the error caused by the inconsistency of the reference space of the image sequence can be avoided.

[0114] In this embodiment, after obtaining the depth estimation image, an image optimization step may also be included, so see Figure 6 The image optimization step may specifically include steps 161-163, which are described as follows.

[0115] Step 161, the difference is calculated for each pixel (i, j) in the depth estimation image P and expressed as

[0116]

[0117] It should be noted that it is necessary to traverse the depth estimation map P and perform the above calculation for each pixel point (i, j).

[0118] Step 162, take the minimum difference calculation result min(|d c |,c=1,...,8) as d m , re-assign the depth estimate D(i,j) of the pixel (i,j), then we can get

[0119] Step 163, using the assignment result Get a new depth estimation image, the new depth estimation image can be expressed as The depth estimation image It is the depth estimation image after curvature optimization.

[0120] It can be understood that after obtaining the depth estimation image, the depth estimation image is also optimized. The estimated depth image is optimized through curvature filtering, so that the final depth estimation image has stronger anti-interference ability to noise, further improving the performance of three-dimensional depth estimation.

[0121] In this embodiment, the above steps can be described as: a set of image sequences with different focus depths are acquired by zooming an imaging device such as a camera or a video camera, and all the focused images in the image sequence are registered to ensure that all the focused images in the image sequence are in the same reference space. Then, the image sequence is segmented by superpixels to obtain the segmentation results of the superpixel region. Next, each superpixel region is sampled based on the obtained superpixel region segmentation results, and the focus evaluation function is calculated for each sampling point to obtain a focus evaluation curve, and then the depth estimation value of the sampling point is estimated according to the focus evaluation curve, and the initial depth estimation results of all pixel points are obtained by interpolation, thereby forming an initial depth estimation image. Finally, the initial depth estimation image is optimized to obtain the final depth estimation image.

[0122] Embodiment 2

[0123] Based on the depth estimation method based on the focused image disclosed in the first embodiment, this embodiment discloses an image processing device. Figure 8 The image processing device 2 includes a camera 21, a processor 22 and a display 23.

[0124] The camera 21 is used to continuously capture images of the target scene and form an image sequence, where the image sequence includes a plurality of focused images with different focal lengths and depths.

[0125] The processor 22 is connected to the camera 21 and is used to process the image sequence according to the depth estimation method disclosed in the first embodiment to obtain a depth estimation image of the target scene. It can be understood that the processor 22 can be a CPU, GPU, FPGA, microcontroller or digital integrated circuit with data processing function, as long as it can implement the depth estimation method implemented in the above steps 110-140 according to its own logic instructions.

[0126] The display 23 is connected to the processor 22 and is used to display the focused image and / or the depth estimation image. It is understood that the display 23 can be a screen with an image display function, which can display the focused image and the depth estimation image separately or together, and the specific screen type and display layout are not limited.

[0127] In this embodiment, please refer to Fig. 9The processor 22 specifically includes an image acquisition module 22-1, a pixel segmentation module 22-2, a focus evaluation module 22-3 and a depth estimation module 22-4, which are described as follows.

[0128] The image acquisition module 22 - 1 can communicate with the camera 21 to acquire an image sequence from the camera 21 .

[0129] The pixel segmentation module 22-2 is connected to the image acquisition module 22-1 and is used to perform super-pixel segmentation on each focused image in the image sequence to obtain multiple super-pixel regions. For example, the pixel segmentation module 22-2 divides the image sequence {M i Each focused image in}(i=0,1...N-1) forms a composite image M merge ; Initialize the seed points of the synthetic image according to the preset spacing d to obtain multiple first seed points, and select the pixel point with the smallest pixel gradient in the neighborhood of each first seed point as the second seed point; Next, perform the superpixel clustering step, for any pixel point p in the synthetic image, calculate the difference measure between the pixel point p and the surrounding second seed points s, and assign the pixel point p to the second seed point corresponding to the minimum difference measure; According to the pixel points assigned to each second seed point, corresponding second seed point areas are formed respectively, and each second seed point area is used to perform superpixel segmentation on the synthetic image to form multiple superpixel areas. The functions of the pixel segmentation module 22-2 can be specifically referred to steps 121-124 in Example 1, which will not be repeated here.

[0130] The focus evaluation module 22-3 is connected to the pixel segmentation module 22-2 and is used to construct a focus evaluation curve for each superpixel region. For example, the focus evaluation module 22-3 randomly samples each superpixel region to obtain multiple sampling points; establishes a focus evaluation function model and expresses it as F(x, y) = w1f1(x, y) + w2f2(x, y) + w3f3(x, y); uses each sampling point to participate in the calculation of the focus evaluation function model, constructs a focus evaluation curve for each sampling point and expresses it as {F i (x, y)}(i=0,1,…,N-1); For each sampling point constructed, the focus evaluation curve {F i (x,y)}, and set the loss function of the focus evaluation function model to The focus evaluation function model is trained, and when the corresponding loss function converges, the weight coefficients w1, w2, and w3 are calculated. The focus evaluation function model is configured using the calculated weight coefficients to obtain the focus evaluation curve of each sampling point. The functions of the focus evaluation module 22-3 can be specifically referred to steps 131-134 in the first embodiment, which will not be described here.

[0131] The depth estimation module 22-4 is connected to the focus evaluation module 22-3 and is used to calculate the depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image. For example, the depth estimation module 22-4 calculates the depth of the pixel point (x, y) on the focus evaluation curve {F i The maximum value F in (x,y)}(i=0,1,…,N-1) max (x, y), determine the focused image corresponding to the maximum value in the image sequence {M i}(i=0,1…N-1) index value k; through three points (k-1,F k-1 (x,y))、(k,F k (x,y))、(k+1,F k+1 (x,y)) fits a quadratic curve and calculates the maximum position k of the quadratic curve in [k-1,k+1] max (x,y), using the position k of the maximum value max (x,y) Determine the depth estimate of the pixel (x,y) D(x,y) = 255*k max (x, y) / N; obtain the depth estimation value of each pixel point in the superpixel area and form a depth estimation image. The function of the depth estimation module 22-4 can be specifically referred to steps 141-143 in the first embodiment, which will not be repeated here.

[0132] for example Fig.10a , Fig.10b , Fig.10c The circuit board shown has some electronic components that make up the circuit. Fig.10a In order to obtain a depth estimation image with noise interference using a general method without using the depth estimation method of the present application, the noise influence is more prominent, so that users cannot Fig.10a Get the effective depth information of the circuit board. Fig.10b Without using the depth estimation method of the present application, the depth estimation image with smoothed edges obtained by the general method can identify the outlines of most electronic components, but the depth information is not obvious, resulting in low accuracy of depth estimation. Fig.10c The depth estimation image obtained by the depth estimation method of the present application can eliminate the similar Fig.10a The noise interference in 10a can also avoid the situation where the edge in 10b is transitionally smoothed, thereby obtaining a depth estimation image with rich depth information and high depth estimation accuracy.

[0133] In this embodiment, the image processing device 2 can realize the function of estimating the depth of the target scene. Since the processor 22 adopts the depth estimation method designed in steps 110-140, it is beneficial to make the segmentation boundaries of the segmented superpixel area coincide with the area where the depth of the real object undergoes drastic changes through superpixel segmentation, thereby avoiding the problem of excessive smoothing of areas with large actual depth changes in the prior art; in addition, it is also beneficial to construct a focus evaluation curve with a wide steep area width and a high clarity ratio, thereby ensuring the accuracy of depth estimation and the ability to resist noise interference.

[0134] Embodiment 3

[0135] Please refer to Fig.11 This embodiment discloses an image processing device. The image processing device 3 mainly includes a memory 31 and a processor 32.

[0136] The main components of the image processing device 3 are a memory 31 and a processor 32. The memory 31 is a computer-readable storage medium, mainly used to store a program, which may be a program code corresponding to the depth estimation method in the first embodiment. The processor 32 is connected to the memory 31 and is used to execute the program stored in the memory 31 to implement the depth estimation method. The functions implemented by the processor 32 can refer to the processor 22 in the second embodiment, and will not be described in detail here.

[0137] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0138] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A depth estimation method based on focused images, It is characterized in that include: Acquire an image sequence, wherein the image sequence includes a plurality of focused images at different focal depths; Forming each focused image in the image sequence into a composite image through pixel gradient indexing, and performing superpixel segmentation on the composite image to obtain a plurality of superpixel regions; Constructing a focus evaluation curve for each superpixel region; The depth estimation value of each pixel point is calculated according to the focus evaluation curve to form a depth estimation image, and the depth estimation value of each pixel point is determined according to the position of the maximum value of the quadratic curve in [k-1, k+1]; the quadratic curve is obtained by fitting three points centered on the index value, where k is the index value, and the maximum value of the pixel point in the focus evaluation curve is calculated to determine the index value of the focused image corresponding to the maximum value in the image sequence.

2. The depth estimation method according to claim 1, It is characterized in that The composite image is subjected to superpixel segmentation to obtain a plurality of superpixel regions, including: Initializing the seed points of the composite image according to a preset spacing d to obtain a plurality of first seed points, and selecting a pixel point with the smallest pixel gradient within a neighborhood of each of the first seed points as a second seed point; Perform the superpixel clustering step: For any pixel point p in the synthetic image, the difference measurement value between the pixel point p and each surrounding second seed point s is calculated, and the pixel point p is assigned to the second seed point corresponding to the minimum difference measurement value; the difference measurement value is expressed as Among them, sqrt() is the square root calculation function, (x p ,y p ) is the coordinate of pixel point p, (x s ,y s ) is the coordinate of the second seed point s, M merge represents the composite image, I represents the index image corresponding to the composite image, and N is the number of images in the image sequence; Corresponding second seed point regions are formed according to the pixel points belonging to each second seed point, and the synthesized image is segmented into superpixels using each of the second seed point regions to form a plurality of superpixel regions.

3. The depth estimation method according to claim 1, It is characterized in that The forming of each focused image in the image sequence into a composite image by using pixel gradient index comprises: For the image sequence {M i }(i=0,1…N-1), calculate each focused image M i The pixel gradient of And construct the gradient image sequence In the gradient image sequence In the image sequence {M, the gradient images are compared and the focused image corresponding to the maximum pixel gradient at each pixel coordinate is obtained. i }, construct index image I; According to each index value in the index image I, the image sequence {M i } corresponds to the pixel value of the focused image at the corresponding pixel coordinate, and the synthetic image M is formed by pixel assignment. merge And the assignment process is represented by M merge (i,j)=M I(i,j) (i,j), where (i,j) is the pixel coordinate.

4. The depth estimation method according to claim 2, It is characterized in that The forming corresponding second seed point regions according to the pixel points belonging to each second seed point respectively includes: The position coordinates of each second seed point are recalculated based on the pixel points belonging to each second seed point; the position coordinates of each second seed point are expressed as Among them, (x i ,y i ) is the coordinate of the pixel point belonging to the second seed point, n is the number of pixel points belonging to the second seed point, and (x′, y′) is the recalculated position coordinate of the second seed point; The superpixel clustering step is performed again on the synthetic image using the recalculated position coordinates of the second seed points until the recalculated position change of each second seed point reaches convergence or the calculation reaches a preset number of times.

5. The depth estimation method according to claim 1, It is characterized in that Before performing superpixel segmentation on the synthetic image, an image registration step is also included, and the image registration step includes: Select a focused image in the image sequence as the reference image, and select any other focused image as the image to be registered, and calculate the derivative F of the image to be registered relative to the reference image x 、F y 、F t , where x, y, and t are derivative parameters about the X-axis, Y-axis, and time T, respectively; Construct constraint equation F x u+F y v+F t = 0, and the motion vector u on the X axis = x j -x i and the motion vector v=y on the Y axis j -y i Substituting the preset spatial transformation relationship, we get The spatial transformation relationship is expressed as Calculate the transformation parameter a 1 , …, a 6 The spatial transformation relationship is configured, and the image to be registered is spatially transformed according to the configured spatial transformation relationship to transform the image sequence {M i } are transformed into the same reference space.

6. The depth estimation method according to claim 1, It is characterized in that The constructing of the focus evaluation curve of each superpixel region comprises: Randomly sampling each of the superpixel regions to obtain a plurality of sampling points; The focus evaluation function model is established and expressed as F(x,y)=w 1 f 1 (x,y)+w 2 f 2 (x,y)+w 3 f 3 (x,y); Among them, f 1 (), f 2 (), f 3 () are different types of focusing functions, (x, y) are the coordinates of any sampling point, w 1 、w 2 、w 3 All are weight coefficients to be configured; Each sampling point is used to participate in the calculation of the focus evaluation function model, and a focus evaluation curve of each sampling point is constructed and expressed as {F i (x,y)}(i=0,1,…,N-1); For each sampling point, the focus evaluation curve {F i (x,y)}, and set the loss function of the focus evaluation function model to Among them, F min (x,y)=min{F i (x,y)}, F max (x,y)=max{F i (x, y)} are the maximum and minimum values ​​of the focus evaluation curve at the sampling point (x, y), respectively, and W peak is the width of the sensitive area; The focus evaluation function model is trained, and the weight coefficient w is calculated when the corresponding loss function converges. 1 、w 2 、w 3 , the focus evaluation function model is configured using the calculated weight coefficient and a focus evaluation curve of each sampling point is obtained.

7. The depth estimation method according to claim 6, It is characterized in that The step of calculating the depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image includes: For each pixel point (x, y) in the superpixel region, calculate the pixel point (x, y) on the focus evaluation curve {F i The maximum value F in (x,y)}(i=0,1,…,N-1) max (x, y), determine the focused image corresponding to the maximum value in the image sequence {M i }(i=0,1…N-1); Through the three points (k-1, F k-1 (x,y))、(k,F k (x,y))、(k+1,F k+1 (x,y)) fits a quadratic curve and calculates the maximum position k of the quadratic curve in [k-1,k+1] max (x,y), using the position k of the maximum value max (x,y) Determine the depth estimate of the pixel (x,y) D(x,y) = 255*k max (x,y) / N; The depth estimation value of each pixel point in the superpixel area is obtained and a depth estimation image is formed.

8. The depth estimation method according to any one of claims 1 to 7, It is characterized in that After obtaining the depth estimation image, an image optimization step is included, and the image optimization step includes: The difference is calculated for each pixel (i, j) in the depth estimation image and expressed as Take the minimum difference calculation result min(|d c |,c=1,...,8) as d m , re-assign the depth estimation value D(i,j) of the pixel point (i,j) and obtain Using the assignment result Get a new depth estimation image.

9. An image processing device, It is characterized in that include: A camera, used to continuously capture images of a target scene and form an image sequence, wherein the image sequence includes a plurality of focused images with different focal depths; a processor, connected to the camera, and configured to process the image sequence according to the depth estimation method according to any one of claims 1 to 8 to obtain a depth estimation image of the target scene; A display is connected to the processor and is used to display the focused image and / or the depth estimation image.

10. The image processing device according to claim 9, It is characterized in that The processor comprises: An image acquisition module, used for acquiring the image sequence from the camera; A pixel segmentation module, used for performing super-pixel segmentation on each focused image in the image sequence to obtain a plurality of super-pixel regions; A focus evaluation module, used to construct a focus evaluation curve for each superpixel area; The depth estimation module is used to calculate the depth estimation value of each pixel point according to the focus evaluation curve to form a depth estimation image.

11. A computer-readable storage medium, It is characterized in that The medium stores a program, which can be executed by a processor to implement the depth estimation method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Depth estimation method based on image definition difference

    CN102903098A

  • Super pixel-based target reconstruction method

    CN108038905A

  • Depth estimation method and depth estimation device for multi-view image

    CN108510536A

  • Focusing depth measurement method and device

    CN111983828A