An anisotropic deep image inpainting method fusing dual-modal inverse determinism

Abstract

The application discloses a kind of anisotropic depth image repair methods of fusion bimodal inverse determinacy, wherein method includes: input aligning color chart and original depth image;Color space of color chart is converted from RGB to Lab, and Lab image is converted into gray chart;Pseudo-background chart is generated by segmenting original depth chart through triangular threshold method;Original depth chart is analyzed by connected domain, and invalid point pixel coordinates in the inside corresponding connected hole are obtained by marking connected hole;Invalid point pixel in next connected hole is processed in order;Based on fusion bimodal inverse deterministic score index, the repair priority of all invalid points in current hole is evaluated;The highest invalid point of repair priority is repaired by establishing filling mechanism based on anisotropic cost;Until all invalid points are filled, complete depth image is output.The application enhances the robustness of algorithm on the basis of improving depth repair precision, and reduces processing duration.

Description

Technical Field

[0001] This invention belongs to the field of machine vision image processing technology, specifically an anisotropic depth image restoration method that integrates dual-modal inverse determinism. Background Technology

[0002] In recent years, depth information has been widely applied in research fields such as 3D reconstruction, autonomous driving, semantic segmentation, and facial recognition. With the continuous development of depth sensor technology, a large number of inexpensive consumer-grade depth cameras have emerged. However, the scene depth information acquired by these cameras often contains numerous holes and missing edges. There are two main reasons for this phenomenon. Firstly, when there are objects occluding the scene, or special situations such as transparent, non-reflective materials, the depth sensor may fail to correctly receive the returned signal. Secondly, the limitations of the device itself can also cause the sensor to capture depth images with limited resolution, and the edges are usually very blurry. Therefore, depth image inpainting is one of the key problems in the field of 3D vision.

[0003] Numerous scholars have researched and contributed various methods for hole repair in depth images. Among mainstream traditional methods, these can be divided into two categories based on whether or not they utilize color images. The latter only considers the depth image, using the continuity of the object's distribution in three-dimensional space to predict the depth values ​​of invalid points. However, holes often appear in the transition area between the foreground and background, where depth values ​​vary significantly. Predicting the true depth solely based on depth data is prone to errors. While these methods have some advantages in real-time performance, they are generally not accurate or reliable enough. The former primarily utilizes the continuity of color components to assist in predicting the true depth values ​​of invalid points. These methods typically have higher accuracy, but the color information used is generally limited to color differences, and they still lack robustness to the effects of uneven lighting. Due to the uncertainty of hole location and size, and the high demands on real-time performance, depth image repair remains an unresolved and open problem. Summary of the Invention

[0004] In view of the problems existing in the above-mentioned restoration algorithms, this invention provides an anisotropic depth image restoration method that integrates bimodal inverse determinism. The robustness of the algorithm is increased by using inverse determinism scores to evaluate restoration priorities, and the accuracy of the algorithm is improved by predicting depth values ​​using an anisotropic cost function.

[0005] An anisotropic depth image inpainting method integrating dual-modal inverse determinism specifically includes the following steps:

[0006] Step 1: Input aligned color image and original depth image, where aligned color image and original depth image mean that the pixel coordinates of the two images correspond one-to-one; convert the color space of the color image from RGB to Lab, and then convert the Lab image into a grayscale image; segment the original depth image using the triangular thresholding method to generate a pseudo background image; perform connected component analysis on the original depth image, mark connected holes, and obtain the pixel coordinates of invalid points inside the corresponding connected holes;

[0007] Step 2: Process invalid pixels in the next connected hole in sequence. The invalid pixels are pixels with a depth value of 0 in the original depth map.

[0008] Step 3: Evaluate the repair priority of all invalid points in the current hole based on the score index of fused bimodal inverse determinism;

[0009] Step 4: Establish an anisotropic cost filling mechanism to repair the highest priority invalid points;

[0010] Step 5: Determine whether all invalid points in the current hole have been repaired; if not, return to step 3; if yes, proceed to step 6.

[0011] Step 6: Determine whether all connected holes in the depth map have been repaired; if not, return to step 2; if yes, proceed to step 7.

[0012] Step 7: Output the repair result as a complete depth image.

[0013] Furthermore, step 1 specifically includes:

[0014] Use the cvtColor function in OpenCV4 to convert the color space of the color image from RGB to Lab; then convert the Lab image to a grayscale image G. Lab Grayscale image G Lab The expression is as follows:

[0015] G Lab = 0.2 × L + 0.4 × a + 0.4 × b

[0016] Where L, a, and b represent the L component, a component, and b component images in the Lab image, respectively;

[0017] The original depth image is analyzed to obtain the corresponding depth histogram. The depth value s in the depth histogram is used as the horizontal axis coordinate of the image, and the frequency h(s) of the depth value is used as the vertical axis coordinate of the image. The coordinate point corresponding to the maximum frequency in the depth histogram is denoted as p1, and the coordinate point corresponding to the minimum frequency on the other side is denoted as p2. The line segment r connects p1 and p2, and the equation of the line segment r is as follows:

[0018] αx+βy+λ=0

[0019] Where α, β, and λ represent the coefficients of the linear equation. Calculate the pseudo-distance from each coordinate point (s, h(s)) to line segment r between the maximum value and the boundary value on the other side. The pseudo distance Positively correlated with actual distance. (Take...) The maximum depth value corresponds to the threshold T. The expression is as follows:

[0020]

[0021] The original depth image is segmented according to a threshold T. All pixels with depth values ​​less than or equal to T are set to 255, and pixels with depth values ​​greater than the threshold are set to 0, thus generating a pseudo background image.

[0022] The original depth image is obtained by assigning a value of 255 to all non-zero pixels. Then, the 8-neighborhood connectivity of the binary image is analyzed using the connectedComponents function in OpenCV4. Connecting holes are marked and the pixel coordinates of invalid points inside the corresponding connecting holes are obtained. An unprocessed list is created to record the coordinate information of invalid points.

[0023] Furthermore, step 3 specifically includes:

[0024] For an invalid pixel n, its repair priority is negatively correlated with the fused bimodal inverse deterministic score U(n), meaning that the invalid pixel has the highest priority when U(n) is at its lowest. The expression for U(n) is as follows:

[0025] U(n) = γ × N(n) + δ × E(n)

[0026] Here, γ and δ represent the balance factors of the two parts, and γ+δ=1 in general.

[0027] N(n) represents the inverse deterministic score calculated based on the number of valid neighborhood points, where valid points are pixels in the depth map with a depth value that is not 0. The expression for N(n) is as follows:

[0028] N(n) = e -num(q) +e -num(b)

[0029] Where num(·) represents the number of ·, q represents the valid points in the 8-neighborhood of invalid point n, and b represents the pseudo-background points that belong to q and have a value of 0 in the pseudo-background graph;

[0030] E(n) represents the inverse deterministic score calculated based on the depth of the effective neighborhood points and the color information entropy. The formula for E(n) is as follows:

[0031] E(n)=Ed (n)+E c (n)

[0032] Among them, E d (n) represents the effective point depth information entropy, expressed as follows:

[0033]

[0034] in, Let represent the i-th non-repeating value among the valid depth values ​​of the neighborhood, m represent the total number of non-repeating depth values, and log2(·) refers to the mathematical expression for finding how many times 2 (·) is raised to the power of 2. This means that a solution was found among all valid points. The probability, The expression is as follows:

[0035]

[0036] Among them, fd i express The total number of times it appears in the valid points;

[0037] E c (n) represents the entropy of the effective point color information, expressed as follows:

[0038]

[0039] in, This indicates that the effective neighborhood points are located in the grayscale image G. Lab The u-th unique grayscale value in the image, where h represents the total number of unique grayscale pixels. This indicates that a positive result was found among all valid points. The probability, The expression is as follows:

[0040]

[0041] Among them, fc u express The total number of times it appears in the valid points.

[0042] Furthermore, step 4 specifically includes:

[0043] For an invalid pixel n, an anisotropic cost function D(n) is established to measure the distance relationship between the neighborhood valid and invalid pixels. The valid pixel depth value corresponding to the minimum D(n) is assigned to the invalid pixel, and the coordinates of the repaired pixel are deleted from the unprocessed list, thus completing the filling of the current invalid pixel. The expression for D(n) is as follows:

[0044]

[0045] w k This represents the weight of each cost factor. Typically, w1 + w2 + w3 + w4 = 1, dis k (n,q) represents the k-th cost factor; dis1(n,q) represents the distance cost factor; dis2(n,q) represents the color cost factor; dis3(n,q) and dis4(n,q) represent the gradient magnitude cost factor and the gradient phase cost factor, respectively.

[0046] This invention evaluates the repair priority of invalid points based on a score index that integrates dual-modal inverse determinism, and fills in the depth value based on an anisotropic cost function.

[0047] Compared to mainstream traditional methods, the use of connected holes in this invention reduces the computational cost of single-order priority calculations. The robustness of the algorithm is enhanced by quantifying the inverse determinism of invalid points using effective neighborhood points and bimodal information entropy. The accuracy of the restoration results is improved by fully utilizing image information through an anisotropic cost function. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method of the present invention;

[0049] Figure 2 In the image, 2a is the Aloe color image from the Middlebury Databases, 2b is the corresponding Lab image, and 2c is the grayscale image G. Lab .

[0050] Figure 3 In the image, 3a is the original depth map of Aloe from Middlebury Databases, and 3b is the processed pseudo-background map.

[0051] Figure 4 In the image, 4a is a binary image of the hole in the original depth image, and 4b is a visualization of its connected components.

[0052] Figure 5 The images show the results of restoring the Aloe image from the Middlebury Databases using the restoration algorithm presented in this paper. In the images, 5a represents the color image of Aloe, 5b represents the original depth image of Aloe, and 5c represents the restored complete depth map.

[0053] Figure 6 This image shows the result of the algorithm of this invention repairing a test image from NYU Depth Dataset V2. In the image, 6a represents the color image, 6b represents the original depth image, and 6c represents the repaired complete depth map. Detailed Implementation

[0054] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention. For ease of understanding, specific details of the embodiments are included, but these should be considered merely exemplary partial embodiments and not intended to limit the scope of the invention. Based on the embodiments of the present invention, those skilled in the art should understand that other embodiments obtained without inventive effort are within the scope of protection of the present invention. To avoid redundancy and ambiguity, well-known functions or structures are not specifically described in the following description.

[0055] like Figure 1 As shown, an anisotropic depth image inpainting method integrating dual-modal inverse determinism includes the following steps:

[0056] S1: Input an aligned color image and an original depth image, wherein the aligned color image and the original depth image refer to two images whose pixel coordinates correspond one-to-one; convert the color space of the color image from RGB to Lab, and then convert the Lab image into a grayscale image; segment the original depth image using the triangular thresholding method to generate a pseudo background image; perform connected component analysis on the original depth image, mark connected holes, and obtain the pixel coordinates of invalid points inside the corresponding connected holes;

[0057] S2: Process invalid pixel points in the next connected hole in sequence. The invalid pixel point is a pixel point with a depth value of 0 in the original depth map.

[0058] S3: Evaluate the repair priority of all invalid points in the current hole based on the score index of fused dual-modal inverse determinism;

[0059] S4: Establish an anisotropic cost filling mechanism to repair the highest priority invalid points;

[0060] S5: Determine whether all invalid points in the current hole have been repaired; if not, return to execute S3; if yes, execute S6.

[0061] S6: Determine whether all connected holes in the depth map have been repaired; if not, return to execute S2; if yes, execute S7.

[0062] S7: Output the repair result as a complete depth image.

[0063] Furthermore, S1 specifically includes:

[0064] Use the cvtColor function in OpenCV4 to convert the color space of the color image from RGB to Lab; then convert the Lab image to a grayscale image G. Lab ;like Figure 2 As shown: Figure 2 'a' represents the RGB color image. Figure 2 b represents the Lab image. Figure 2c represents the grayscale image G. Lab Grayscale image G Lab The conversion formula is as follows:

[0065] G Lab = 0.2 × L + 0.4 × a + 0.4 × b

[0066] Where L, a, and b represent the L component, a component, and b component images in the Lab image, respectively;

[0067] The original depth image is analyzed to obtain the corresponding depth histogram. The depth value s in the depth histogram is used as the horizontal axis coordinate of the image, and the frequency h(s) of the depth value is used as the vertical axis coordinate of the image. The coordinate point corresponding to the maximum frequency in the depth histogram is denoted as p1, and the coordinate point corresponding to the minimum frequency on the other side is denoted as p2. The line segment r connects p1 and p2, and the equation of the line segment r is as follows:

[0068] αx+βy+λ=0

[0069] Where α, β, and λ represent the coefficients of the linear equation. Calculate the pseudo-distance from each coordinate point (s, h(s)) to line segment r between the maximum value and the boundary value on the other side. The pseudo distance Positively correlated with actual distance. (Take...) The maximum depth value corresponds to the threshold T. The expression is as follows:

[0070]

[0071] The original depth image is segmented according to a threshold T. Pixels with depth values ​​less than T are set to 255, and pixels with depth values ​​greater than the threshold are set to 0, thus generating a pseudo-background image. Figure 3 As shown: Figure 3 'a' represents the original depth image. Figure 3 b represents a pseudo background image;

[0072] The original depth map is modified by assigning a value of 255 to all non-zero pixels to obtain a binary image. Then, the 8-neighborhood connectivity of the binary image is analyzed using the `connectedComponents` function in OpenCV4. Figure 4 As shown, Figure 4 'a' represents the binary image of the hole in the original depth image; Figure 4 b represents its connected component visualization map; mark the connected holes and obtain the pixel coordinates of invalid points inside the corresponding connected holes, and create an unprocessed list to record the coordinate information of invalid points;

[0073] Furthermore, S3 specifically includes:

[0074] For an invalid pixel n, its repair priority is negatively correlated with the fused bimodal inverse deterministic score U(n), meaning that the invalid pixel has the highest priority when U(n) is at its lowest. The expression for U(n) is as follows:

[0075] U(n) = γ × N(n) + δ × E(n)

[0076] Here, γ and δ represent the balance factors of the two parts, and γ+δ=1 in general.

[0077] N(n) represents the inverse deterministic score calculated based on the number of valid neighborhood points, where valid points are pixels in the depth map with a depth value that is not 0. The expression for N(n) is as follows:

[0078] N(n) = e -num(q) +e -num(b)

[0079] Where num(·) represents the number of ·, q represents the valid points in the 8-neighborhood of invalid point n, and b represents the pseudo-background points that belong to q and have a value of 0 in the pseudo-background graph;

[0080] E(n) represents the inverse deterministic score calculated based on the depth of the effective neighborhood points and the color information entropy. The formula for E(n) is as follows:

[0081] E(n)=E d (n)+E c (n)

[0082] Among them, E d (n) represents the effective point depth information entropy, expressed as follows:

[0083]

[0084] in, Let represent the i-th non-repeating value among the valid depth values ​​of the neighborhood, m represent the total number of non-repeating depth values, and log2(·) refers to the mathematical expression for finding how many times 2 (·) is raised to the power of 2. This means that a solution was found among all valid points. The probability, The expression is as follows:

[0085]

[0086] Among them, fd i express The total number of times it appears in the valid points;

[0087] E c (n) represents the entropy of the effective point color information, expressed as follows:

[0088]

[0089] in, This indicates that the effective neighborhood points are located in the grayscale image G. Lab The u-th unique grayscale value in the image, where h represents the total number of unique grayscale pixels. This indicates that a positive result was found among all valid points. The probability, The expression is as follows:

[0090]

[0091] Among them, fc u express The total number of times it appears in the valid points.

[0092] Furthermore, S4 specifically includes:

[0093] For an invalid pixel n, an anisotropic cost function D(n) is established to measure the distance relationship between the neighborhood valid and invalid pixels. The valid pixel depth value corresponding to the minimum D(n) is assigned to the invalid pixel, and the coordinates of the repaired pixel are deleted from the unprocessed list, thus completing the filling of the current invalid pixel. The expression for D(n) is as follows:

[0094] k = 1, 2, 3, 4

[0095] w k This represents the weight of each cost factor. Typically, w1 + w2 + w3 + w4 = 1, dis k (n,q) represents the k-th cost factor;

[0096] dis1(n,q) represents the distance cost factor, and its expression is as follows:

[0097]

[0098] dis2(n,q) represents the color cost factor, and its expression is as follows:

[0099]

[0100] Where c(n) and c(q) represent invalid and valid points in image G, respectively. Lab grayscale values ​​in;

[0101] dis3(n,q) and dis4(n,q) represent the gradient magnitude cost factor and gradient phase cost factor, respectively. The gradient grad is defined as the first-order partial derivatives of the image along the x-axis and y-axis. Invalid points n in image G... Lab The gradient grad(n) expression in the code is as follows:

[0102]

[0103] Where n1, n2, n3, and n4 represent the coordinates of point n in its four-neighborhood (up, down, left, right), respectively, and c(n1), c(n2), c(n3), and c(n4) represent the coordinates of the four points in image G. Lab The grayscale value in the image.

[0104] The magnitude m(n) at point n is defined as:

[0105]

[0106] Phase angle at point n Defined as:

[0107]

[0108] The expressions for dis3(n,q) and dis4(n,q) are as follows:

[0109]

[0110]

[0111] The process continues until all invalid points in the depth map are filled, and the repaired result is output as a complete depth image.

[0112] Figure 5 This is the result of restoring an Aloe image from the Middlebury Databases using the restoration algorithm presented in this paper. Figure 5 As shown, Figure 5 'a' represents the color image of Aloe. Figure 5 b represents Aloe's original depth image. Figure 5 c represents the repaired complete depth map.

[0113] Figure 6 This is an image showing the effect of the algorithm of this invention repairing a test image from NYU Depth Dataset V2. Figure 6 As shown: Figure 6 'a' represents its color image. Figure 6 b represents its original depth image. Figure 6 c represents the repaired complete depth map.

[0114] pass Figure 5 and Figure 6 The results show that the depth image repaired using the method of this invention accurately repairs large-area holes while preserving good contour data and geometric integrity.

[0115] Finally, it should be noted that the specific embodiments described above are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made depending on application requirements or other reasons. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An anisotropic depth image inpainting method integrating dual-modal inverse determinism, characterized in that... Includes the following steps: Step 1: Input the aligned color image and the original depth image, where the aligned color image and the original depth image mean that the pixel coordinates of the two images correspond one-to-one; convert the color space of the color image from RGB to Lab, and then convert the Lab image into a grayscale image; A pseudo background image is generated by segmenting the original depth map using the triangular thresholding method; connected component analysis is performed on the original depth map to mark connected holes and obtain the pixel coordinates of invalid points inside the corresponding connected holes; Step 2: Process invalid pixels in the next connected hole in sequence. The invalid pixels are pixels with a depth value of 0 in the original depth map. Step 3: Evaluate the repair priority of all invalid points in the current hole based on the score index of the fused bimodal inverse determinism, characterized in that the score index of the fused bimodal inverse determinism is determined by the following function: For an invalid pixel n, its repair priority is negatively correlated with the fused bimodal inverse deterministic score U(n). That is, when U(n) is the lowest, the invalid pixel has the highest priority. The expression for U(n) is as follows: U(n) = γ × N(n) + δ × E(n) Where γ and δ represent the balance factors of the two parts, γ+δ=1, and N(n) represents the inverse deterministic score calculated based on the number of valid neighborhood points, where valid points are pixels in the depth map with a depth value that is not 0. The expression for N(n) is as follows: N(n)=e -num(q) +e -num(b) Where num(·) represents the number of ·, q represents the valid points in the 8-neighborhood of the invalid point n, and b represents the pseudo-background points that belong to q and have a value of 0 in the pseudo-background graph. E(n) represents the inverse deterministic score calculated based on the depth of the effective neighborhood points and the color information entropy. The formula for E(n) is as follows: E(n)=E d (n)+E c (n) Among them, E d (n) represents the effective point depth information entropy, expressed as follows: in, Let represent the i-th unique value among the valid depth values ​​of the neighborhood, and m represent the total number of unique depth values. Indicates base 2 The logarithm of express The probability of it appearing in the valid points. The expression is as follows: Among them, fd i express The total number of times it appears in the valid points. E c (n) represents the entropy of the effective point color information, expressed as follows: in, This indicates that the effective neighborhood points are located in the grayscale image G. Lab The u-th unique grayscale value in the dataset, where h represents the total number of unique grayscale values. Indicates base 2 The logarithm of express The probability of it appearing in the valid points. The expression is as follows: Among them, fc u express The total number of times it appears in the valid points; Step 4: Establish an anisotropic cost filling mechanism to repair the highest priority invalid points, characterized in that the anisotropic cost filling mechanism is determined by the following function: For an invalid pixel n, an anisotropic cost function D(n) is established to measure the distance relationship between the neighborhood valid and invalid pixels. The valid pixel depth value corresponding to the minimum D(n) is assigned to the invalid pixel to complete the filling of the current invalid pixel. The expression of D(n) is as follows: Among them, w k This represents the weight of each cost factor, where w1+w2+w3+w4=1, dis k (n,q) represents the k-th cost factor; dis1(n,q) represents the distance cost factor; dis2(n,q) represents the color cost factor; dis3(n,q) and dis4(n,q) represent the gradient magnitude cost factor and the gradient phase cost factor, respectively; Step 5: Determine whether all invalid points in the current hole have been repaired; if not, proceed to step 3; if repaired, continue to step 6. Step 6: Determine if all connected holes in the depth map have been repaired; if not, proceed to Step 2; if repaired, continue to Step 7. Step 7: Output the repair result as a complete depth image.

2. The anisotropic depth image inpainting method fused with dual-modal inverse determinism as described in claim 1, wherein the Lab image is converted into a grayscale image as described in step 1, characterized in that, G Lab The expression is as follows: G Lab =0.2×L+0.4×a+0.4×b Where L, a, and b represent the L component, a component, and b component images in the Lab image, respectively; Generating a pseudo background image by segmenting the original depth map using the triangular thresholding method specifically includes: The original depth image is analyzed to obtain the corresponding depth histogram. The depth value s in the depth histogram is used as the horizontal axis coordinate of the image, and the frequency h(s) of the depth value is used as the vertical axis coordinate of the image. The coordinate point corresponding to the maximum frequency in the depth histogram is denoted as p1, and the coordinate point corresponding to the minimum frequency on the other side is denoted as p2. The line segment r connects p1 and p2, and the equation of the line segment r is as follows: αx+βy+λ=0 Where α, β, and λ represent the coefficients of the linear equation, and the pseudo distance from each coordinate point (s, h(s)) to line segment r is calculated between the maximum value and the boundary value on the other side. The pseudo distance Positively correlated with the actual distance, take The maximum depth value corresponds to the threshold T. The expression is as follows: The original depth image is segmented according to a threshold T. All pixels with depth values ​​less than T are set to 255, and pixels with depth values ​​greater than the threshold are set to 0, thus generating a pseudo background image.

Images