Depth Map Processing Method, Apparatus, Computer Device, and Storage Medium

Abstract

The present application relates to a depth map processing method, apparatus, computer device, and storage medium. The method includes: obtaining an original image and a depth image of the original image; the original image being a color image corresponding to the depth image; performing edge detection processing on the original image to obtain an image edge of the original image, and performing edge detection processing on the depth image to obtain a depth edge of the depth image; performing edge matching on the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges as an error image of the depth image; and performing correction processing on the error image of the depth image to obtain a corrected depth image. Using this method can improve the correction efficiency of correcting the depth image and reduce the workload of correction.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a depth map processing method, apparatus, computer device, and storage medium. Background Art

[0002] A depth map is an image or image channel containing information related to the distance from the surface of a scene object to the viewpoint. In recent years, the application scope of depth maps has become increasingly wide, not limited to applications such as depth of field simulation of cameras in the two-dimensional image field, and applications such as three-dimensional modeling, robot obstacle avoidance, augmented reality, and three-dimensional vehicle navigation in the three-dimensional image field. Existing depth estimation methods (such as stereo matching, deep learning, and depth cameras, etc.) are easily affected by factors such as lack of texture, occlusion problems, noise, and light source problems, resulting in a reduction in the accuracy of the depth map and causing the depth information to not match the actual image.

[0003] Although common depth map correction methods (such as bilateral filtering and guided filtering) can smooth the depth map and improve the accuracy of the depth map, however, these methods slide a fixed-size frame on the image to correct the image, and have limited correction effects on depth errors with a large correction range, and require more time.

[0004] Therefore, the existing depth map correction methods have problems of consuming more time and having a large workload. Summary of the Invention

[0005] Based on this, it is necessary to provide a depth map processing method, apparatus, computer device, and storage medium for the technical problem that the above-mentioned depth map correction method consumes more time and has a large workload.

[0006] A depth map processing method, the method includes:

[0007] Obtain an original image and a depth image of the original image; the original image is a color image corresponding to the depth image;

[0008] Perform edge detection processing on the original image to obtain an image edge of the original image, and perform edge detection processing on the depth image to obtain a depth edge of the depth image;

[0009] Perform edge matching on the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges as an error image of the depth image;

[0010] Perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0011] In one embodiment, the performing edge detection processing on the original image to obtain the image edge of the original image includes:

[0012] Perform image segmentation processing on the original image to obtain a segmented image; the segmented image includes a plurality of segmentation regions;

[0013] Perform edge detection processing on the segmented image, extract the detected pixel points, and sequentially connect each pixel point to obtain the image edge of the original image.

[0014] In one embodiment, the performing correction processing on the error image of the depth image to obtain a corrected depth image includes:

[0015] Determine the image identifier of the error image;

[0016] According to the image identifier, perform corresponding correction processing on the error image to obtain a corrected depth image.

[0017] In one embodiment, the image identifier includes: an internal error image and an edge error image;

[0018] The determining the image identifier of the error image includes:

[0019] If the error image is located within the segmentation region where the error image is located, then determine that the image identifier of the error image is an internal error image;

[0020] If the error image is adjacent to at least two segmentation regions, then determine that the image identifier of the error image is an edge error image.

[0021] In one embodiment, the performing corresponding correction processing on the error image according to the image identifier to obtain a corrected depth image includes:

[0022] If the image identifier of the error image is an internal error image, then determine the surface identifier of the segmentation region where the internal error image is located, and perform corresponding correction processing on the internal error image according to the surface identifier;

[0023] If the image identifier of the error image is an edge error image, then determine target pixel points from the adjacent region of the edge error image; obtain the depth value of the target pixel points, and use the depth value of the target pixel points to correct the edge error image.

[0024] In one embodiment, the surface identifier of the segmentation region includes a planar identifier and a non-planar identifier;

[0025] Determining the surface identifier of the segmentation region where the internal error image is located includes:

[0026] Obtaining the coordinates and corresponding depth values of each pixel point in the segmentation region where the internal error image is located;

[0027] Performing planar detection processing on the segmentation region where the internal error image is located according to the coordinates and depth values of each pixel point, to obtain the surface identifier of the segmentation region where the internal error image is located.

[0028] In one embodiment, performing corresponding correction processing on the internal error image according to the surface identifier includes:

[0029] If the surface identifier of the segmentation region where the internal error image is located is a planar identifier, obtaining the depth values of each pixel point in the segmentation region where the internal error image is located; calculating the average value of each depth value, and using the average value to correct the internal error image;

[0030] If the surface identifier of the segmentation region where the internal error image is located is a non-planar identifier, using a linear interpolation correction model to correct the internal error image.

[0031] In one embodiment, if the image identifier of the error image is an edge error image, determining a target pixel point from the adjacent region of the edge error image includes:

[0032] If the image identifier of the error image is an edge error image, determining the corresponding position of the edge error image in the original image;

[0033] Obtaining the color values of each candidate pixel point in the adjacent region of the corresponding position and the color average value of the edge error image;

[0034] Calculating the color difference between the color value of each candidate pixel point and the color average value of the edge error image, and determining a target pixel point from each candidate pixel point according to the color difference.

[0035] In one embodiment, calculating the color difference between the color value of each candidate pixel point and the color average value of the edge error image, and determining a target pixel point from each candidate pixel point according to the color difference includes:

[0036] Sorting each candidate pixel point according to the numerical order of the color difference to obtain a candidate pixel point sequence;

[0037] Obtain the average depth value of the adjacent region and obtain the depth value of the first candidate pixel in the candidate pixel sequence; the first candidate pixel represents the pixel with the smallest difference in color value from the color value of the edge error image;

[0038] Determine whether the first candidate pixel meets a preset recognition condition according to the depth value of the first candidate pixel and the average depth value of the adjacent region. If the first candidate pixel meets the recognition condition, use the first candidate pixel as the target pixel;

[0039] If the first candidate pixel does not meet the recognition condition, obtain a second candidate pixel from the candidate pixel sequence as the new first candidate pixel, and return to the step of obtaining the depth value of the first candidate pixel in the candidate pixel sequence.

[0040] In one embodiment, the determining whether the first candidate pixel meets a preset recognition condition according to the depth value of the first candidate pixel and the average depth value of the adjacent region includes:

[0041] Calculate the depth difference between the depth value of the first candidate pixel and the average depth value;

[0042] If the depth difference exceeds the difference threshold, determine that the first candidate pixel does not meet the recognition condition; if the depth difference does not exceed the difference threshold, determine that the first candidate pixel meets the recognition condition.

[0043] In one embodiment, after determining that the first candidate pixel does not meet the recognition condition, it further includes:

[0044] Eliminate the first candidate pixel to obtain a new candidate pixel sequence, and determine whether the new candidate pixel sequence is empty;

[0045] If the new candidate pixel sequence is not empty, obtain the second candidate pixel as the new first candidate pixel;

[0046] If the new candidate pixel sequence is empty, use the pixel with the smallest depth difference between the depth value and the average depth value in the candidate pixel sequence as the target pixel.

[0047] A depth map processing device, the device includes:

[0048] An image acquisition module, configured to acquire an original image and a depth image of the original image; the original image is a color image corresponding to the depth image;

[0049] An edge detection module, configured to perform edge detection processing on the original image to obtain the image edges of the original image, and perform edge detection processing on the depth image to obtain the depth edges of the depth image;

[0050] An edge matching module, configured to perform edge matching between the image edges of the original image and the depth edges of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image;

[0051] A correction module, configured to perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0052] A computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0053] Obtain an original image and the depth image of the original image; the original image is a color image corresponding to the depth image;

[0054] Perform edge detection processing on the original image to obtain the image edges of the original image, and perform edge detection processing on the depth image to obtain the depth edges of the depth image;

[0055] Perform edge matching between the image edges of the original image and the depth edges of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image;

[0056] Perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0057] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0058] Obtain an original image and the depth image of the original image; the original image is a color image corresponding to the depth image;

[0059] Perform edge detection processing on the original image to obtain the image edges of the original image, and perform edge detection processing on the depth image to obtain the depth edges of the depth image;

[0060] Perform edge matching between the image edges of the original image and the depth edges of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image;

[0061] Perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0062] The above-mentioned depth map processing method, device, computer device, and storage medium obtain the original image and the depth image of the original image, perform edge detection processing on the original image to obtain the image edges of the original image, and perform edge detection processing on the depth image to obtain the depth edges of the depth image. Further, the image edges of the original image are edge-matched with the depth edges of the depth image to obtain the image formed by the unmatched depth edges as the error image of the depth image. Finally, only the error image of the depth image is corrected to obtain the corrected depth image. This method matches the depth edges of the depth image with the image edges of the original image, can accurately determine the inaccurate depth edges in the depth image as error edges, and uses the image formed by the error edges as the error image of the depth image. Then, by only correcting the error image, the correction of the depth image can be realized without detecting each area of the depth image with a fixed-size frame, reducing unnecessary correction time. Therefore, the correction efficiency of the depth image is improved, the workload of correction is reduced, and it is more suitable for correcting depth images in a larger range. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 FIG. is an application scenario diagram of the depth map processing method in an embodiment;

[0064] Figure 2 FIG. is a structural block diagram of the image processing device in an embodiment;

[0065] Figure 3 FIG. is a schematic flowchart of the depth map processing method in an embodiment;

[0066] Figure 4 FIG. a is a schematic diagram of the depth edges of the depth image in an embodiment;

[0067] Figure 4 FIG. b is a schematic diagram of the image edges of the original image in an embodiment;

[0068] Figure 4 FIG. c is a schematic diagram of the error image of the depth image in an embodiment;

[0069] Figure 5 FIG. is a schematic flowchart of the target pixel point determination step in an embodiment;

[0070] Figure 6 FIG. is a schematic flowchart of obtaining the depth map to be corrected in an embodiment;

[0071] Figure 7 FIG. is a schematic flowchart of correcting the error image in an embodiment;

[0072] Figure 8It is a structural block diagram of a depth map processing device in an embodiment;

[0073] Figure 9 It is an internal structure diagram of a computer device in an embodiment. Specific embodiments

[0074] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] The depth map processing method provided by the present application can be applied to an application environment as Figure 1 shown. This application environment includes a depth map processing system, which includes a camera and depth map generation module and an image processing device. Among them, the image processing device includes a depth map correction module and a processor. Among them, the camera and depth map generation module interacts with the image processing device through a network. Among them,

[0076] The camera and depth map generation module is used to capture an original image and generate a depth image;

[0077] The image processing device is used to obtain the original image and the depth image of the original image from the camera and depth map generation module. The original image is a color image corresponding to the depth image; the image processing device is also used to perform edge detection processing on the original image to obtain the image edge of the original image, and perform edge detection processing on the depth image to obtain the depth edge of the depth image; match the image edge of the original image with the depth edge of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image; perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0078] Further, as Figure 2 shown, the image processing device further includes a depth edge extraction module 202, an image segmentation module 204, an image edge extraction module 206, an edge inconsistency detection module 208, and an error image calibration module 210. Among them,

[0079] The depth edge extraction module 202 is used to perform edge detection on the depth image to extract the depth edge of the depth image; the image segmentation module 204 is used to perform image segmentation processing on the image; the image edge extraction module 206 is used to perform edge detection on the original image to extract the image edge of the original image; the edge inconsistency detection module 208 is used to match the image edge of the original image with the depth edge of the depth image; the error image calibration module 210 is used to calibrate the image identifier of the error image.

[0080] In one embodiment, as Figure 3 shown, a depth map processing method is provided. Taking the image processing device in Figure 1 as an example, the method includes the following steps:

[0081] Step S302, obtaining an original image and a depth image of the original image; the original image is a color image corresponding to the depth image.

[0082] Among them, the original image can represent an image directly captured by a camera device, and each pixel in the original image is composed of three primary color components R, G, and B. Among them, the original image and the corresponding depth image are registered, and there is a one-to-one mapping relationship between the pixel points of the two images.

[0083] Among them, the depth image represents an image containing information related to the surface distance of the scene object from the viewpoint, and each pixel value of the depth image represents the actual distance of the sensor from the object. The depth image can be generated by methods such as stereo image matching, depth camera, or deep learning.

[0084] Step S304, performing edge detection processing on the original image to obtain the image edge of the original image, and performing edge detection processing on the depth image to obtain the depth edge of the depth image.

[0085] In a specific implementation, the processor in the image processing device performs edge detection processing on the original image, extracts the detected pixel points, and sequentially connects each pixel point to obtain the image edge of the original image. Similarly, the processor is also used to perform edge detection processing on the depth image to obtain the depth edge of the depth image. For example, as Figure 4 shown in a, it is a schematic diagram of the depth edge of the depth image. The thick line part in the figure can represent the obtained depth edge of the depth image. As Figure 4 shown in b, it is a schematic diagram of the image edge of the original image. The thick line in the figure can represent the image edge of the original image.

[0086] More specifically, the steps for performing edge detection on the depth image include: using Gaussian filtering to reduce the noise of the depth image, then calculating the gradient direction and gradient intensity of each pixel point in the depth image, and judging whether each pixel is a pixel point of a strong edge through a set threshold, that is, extracting the pixel points with obvious intensity changes detected, and sequentially connecting each pixel point to obtain the depth edge of the depth image.

[0087] Step S306, performing edge matching on the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image.

[0088] In a specific implementation, the processor compares the image edge of the original image with the depth edge of the depth image according to the corresponding relationship between the pixel points of the depth image and the original image, removes the overlapping area between the original image and the depth image, and leaves the depth edge that does not match the original image. The image formed by the unmatched depth edge is used as the error image of the depth image. For example, as Figure 4 shown in c, it is a schematic diagram of the error image of the depth image. The shaded area in the figure can represent the error image of the depth image. Then Figure 4 c includes 4 error images A ′ 、A″、B ′ and B″.

[0089] Furthermore, when performing edge matching between the image edge of the original image and the depth edge of the depth image, a tolerance interval value can be preset in advance. When the distance difference between the depth edge and the image edge is obtained based on the coordinates corresponding to each pixel point of the image edge of the original image and the coordinates corresponding to each pixel point of the depth edge of the depth image, if the distance difference is within the preset distance difference range, it can be determined that the depth edge and the image edge match, so as to reduce the comparison time of the depth image and the original image, reduce the workload, and improve the correction rate of the depth image.

[0090] Step S308, perform correction processing on the error image of the depth image to obtain the corrected depth image.

[0091] In a specific implementation, since the error image may exist inside the corresponding area of the original image, may exist outside the corresponding area of the original image, or may be adjacent to the corresponding area of the original image, after obtaining the error image of the depth image, the error image can be classified and calibrated first, and the corresponding correction method can be used to correct the error image according to the image identifier corresponding to the error image, so as to obtain the corrected depth image.

[0092] In the above depth map processing method, by obtaining the original image and the depth image of the original image, where the original image is a color image corresponding to the depth image, edge detection processing is performed on the original image to obtain the image edges of the original image, and edge detection processing is performed on the depth image to obtain the depth edges of the depth image; the image edges of the original image are edge-matched with the depth edges of the depth image to obtain the image formed by the unmatched depth edges as the error image of the depth image; correction processing is performed on the error image of the depth image to obtain the corrected depth image. This method matches the depth edges of the depth image with the image edges of the original image, filters out the inaccurate depth edges in the depth image through edge consistency detection as error edges, and uses the image formed by the error edges as the error image of the depth image. Then, only by correcting the error image can the correction of the depth image be achieved, without detecting each area of the depth image with a fixed-size frame, reducing unnecessary correction time. Thus, the correction efficiency of the depth image is improved, the workload of correction is reduced, and it is more suitable for correcting depth images in a larger range.

[0093] In one embodiment, the above step S304 further includes: performing image segmentation processing on the original image to obtain the segmented image; the segmented image includes multiple segmentation regions; performing edge detection processing on the segmented image, extracting the detected pixel points and connecting each pixel point in sequence to obtain the image edges of the original image.

[0094] In specific implementation, after obtaining the original image, the original image can be first subjected to image segmentation processing by the image segmentation module 204 to obtain the segmented image. Among them, the image segmentation method can be semantic segmentation or instance segmentation, which is not limited here. For example, in Figure 4 b, the original image is segmented into two segmentation regions A and B through image segmentation. After obtaining the segmented image, edge detection is performed on the segmented image by the image edge extraction module 206, and the pixel points of the strong edges in the segmented image are extracted and each pixel point is connected in sequence to obtain the image edges of the original image.

[0095] In this embodiment, by performing segmentation processing on the original image, a segmented image including multiple segmentation regions is obtained, which is convenient for subsequently determining the image identifier of the error image of the depth image according to the segmentation region.

[0096] In one embodiment, the above step S308 includes: determining the image identifier of the error image; performing corresponding correction processing on the error image according to the image identifier to obtain the corrected depth image.

[0097] Further, in one embodiment, determining the image identifier of an error image includes: if the error image is located within the segmentation region where the error image is located, determining that the image identifier of the error image is an internal error image; if the error image is adjacent to at least two segmentation regions, determining that the image identifier of the error image is an edge error image.

[0098] Among them, the image identifier includes an internal error image and an edge error image. An internal error image refers to an error image that is located within a certain segmentation region and is not adjacent to other segmentation regions, while an edge error image refers to an error image that is adjacent to at least two segmentation regions.

[0099] For example, in Figure 4 c, error images A ′ and B ′ are both adjacent to segmentation regions A and B. Therefore, error images A ′ and B ′ can be calibrated as edge error images. And error image A″ is located inside segmentation region A and is not adjacent to segmentation region B. Therefore, error image A″ can be calibrated as an internal error image. Similarly, error image B″ is located inside segmentation region B and is not adjacent to segmentation region A. Therefore, error image B″ can be calibrated as an internal error image.

[0100] In a specific implementation, the position information of the error image can be obtained, for example, coordinates, and the position information of the error image is compared with the position information of each segmentation region to determine the positional relationship between the error image and each segmentation region. Thus, it is determined whether the error image is located within a certain segmentation region or is adjacent to at least two segmentation regions, thereby determining the image identifier of the error image.

[0101] In this embodiment, by obtaining the position information of the error image, the positional relationship between the error image and the segmentation region is obtained, and then the image identifier of the error image is determined, so as to perform a corresponding correction method on the error image according to the image identifier, further improving the accuracy and correction effect of the correction result of the depth map.

[0102] In one embodiment, the step of performing corresponding correction processing on the error image according to the image identifier to obtain the corrected depth image includes: if the image identifier of the error image is an internal error image, determining the surface identifier of the segmentation region where the internal error image is located, and performing corresponding correction processing on the internal error image according to the surface identifier; if the image identifier of the error image is an edge error image, determining target pixel points from the adjacent regions of the edge error image; obtaining the depth values of the target pixel points, and using the depth values of the target pixel points to correct the edge error image.

[0103] Among them, the surface identifier includes a plane identifier and a non - plane identifier.

[0104] In a specific implementation, if Figure 4 A″ and B″ in c, and it is determined that the image identifier of the error image is an internal error image, then the segmentation area where the internal error image is located is used as the correction basis to correct the internal error image. More specifically, the plane identifier of the segmentation area where the internal error image is located can be determined, that is, it is determined whether the segmentation area is a plane or a non-plane, and according to the surface identifier of the segmentation area, corresponding correction methods are respectively used to correct the internal error image.

[0105] On the contrary, if Figure 4 A in c ′ and B ′ , and it is determined that the image identifier of the error image is an edge error image, then according to the corresponding relationship between the pixels of the original image and the depth image, the corresponding position of the edge error image is determined in the original image, multiple pixels and the color values of each pixel are obtained from the adjacent area of the corresponding position, and the target pixel is selected from each pixel according to the color values of each pixel. The depth value of the target pixel is obtained, and the edge error image is corrected with the depth value of the target pixel.

[0106] In this embodiment, corresponding correction methods are respectively used according to the image identifier of the error image to correct the error image, which improves the accuracy of correcting the error image, and further improves the accuracy of correcting the depth image. In addition, by calibrating and correcting the internal error image, the problems of inconsistent edge depth contours and incomplete internal depth information of the depth image can be solved simultaneously.

[0107] In one embodiment, the step of determining the surface identifier of the segmentation area where the internal error image is located includes: obtaining the coordinates and corresponding depth values of each pixel in the segmentation area where the internal error image is located; performing plane detection processing on the segmentation area where the internal error image is located according to the coordinates and depth values of each pixel to obtain the surface identifier of the segmentation area where the internal error image is located.

[0108] In this embodiment, by obtaining the coordinates and corresponding depth values of each pixel in the segmentation area where the internal error image is located, according to the coordinates and depth values of each pixel, the distribution information of the segmentation area is determined, and plane detection is performed on the segmentation area according to the distribution information to determine whether the segmentation area where the internal error image is located is a plane or a non-plane, so as to use the corresponding correction method according to the surface identifier of the segmentation area where the internal error image is located to correct the internal error image, improve the accuracy of correcting the internal error image, and further improve the accuracy of correcting the depth image.

[0109] In one embodiment, corresponding correction processing is performed on the internal error image according to the surface identifier, including: if the surface identifier of the segmentation region where the internal error image is located is a planar identifier, obtain the depth values of each pixel point in the segmentation region where the internal error image is located; calculate the average value of the depth values, and use the average value to correct the internal error image; if the surface identifier of the segmentation region where the internal error image is located is a non-planar identifier, use a linear interpolation correction model to correct the internal error image.

[0110] In specific implementation, if the surface identifier of the segmentation region where the internal error image is located is a planar identifier, it indicates that the depth values of each pixel point in the segmentation region where the internal error image is located are equal or linearly distributed. Therefore, the average value of the depth values of each pixel point in the segmentation region where the internal error image is located can be calculated, and the depth value of the internal error image is replaced with the average value of the depth values to achieve the correction of the internal error image.

[0111] If the surface identifier of the segmentation region where the internal error image is located is a non-planar identifier, it indicates that the depth value distribution of each pixel point in the segmentation region where the internal error image is located is relatively irregular. Therefore, the linear interpolation correction method can be used to correct the internal error image according to the gradient direction of the depth value.

[0112] In this embodiment, when the surface identifier of the segmentation region where the internal error image is located is a planar identifier, the average value of the depth values of the segmentation region is used to correct the internal error image. When the surface identifier of the segmentation region where the internal error image is located is a non-planar identifier, a linear interpolation correction model is used to correct the internal error image. By patching different internal error images with corresponding correction methods, the correction result of the internal error image can be greatly improved, and the matching degree between the corrected depth map and the original image can be improved.

[0113] In one embodiment, if the image identifier of the error image is an edge error image, target pixel points are determined from the adjacent region of the edge error image, including: if the image identifier of the error image is an edge error image, determine the corresponding position of the edge error image in the original image; obtain the color values of each candidate pixel point in the adjacent region of the corresponding position and the color average value of the edge error image; calculate the color difference between the color value of each candidate pixel point and the color average value of the edge error image, and determine the target pixel point from each candidate pixel point according to the color difference.

[0114] Further, as Figure 5 shown, in one embodiment, calculating the color difference between the color value of each candidate pixel point and the color average value of the edge error image, and determining the target pixel point from each candidate pixel point according to the color difference includes:

[0115] Step S502: Sort each candidate pixel point according to the numerical order of the color difference value to obtain a candidate pixel point sequence;

[0116] Step S504: Obtain the average depth value of the adjacent area and the depth value of the first candidate pixel point in the candidate pixel point sequence; The first candidate pixel point refers to the pixel point with the smallest difference between the color value and the color value of the edge error image.

[0117] Step S506: Determine whether the first candidate pixel point meets the preset recognition condition according to the depth value of the first candidate pixel point and the average depth value of the adjacent area. If the first candidate pixel point meets the recognition condition, use the first candidate pixel point as the target pixel point;

[0118] Step S508: If the first candidate pixel point does not meet the recognition condition, obtain the second candidate pixel point from the candidate pixel point sequence as the new first candidate pixel point, and return to the step of obtaining the depth value of the first candidate pixel point in the candidate pixel point sequence.

[0119] Further, in one embodiment, the above step S506 specifically includes: calculating the depth difference between the depth value of the first candidate pixel point and the average depth value; if the depth difference exceeds the difference threshold, it is determined that the first candidate pixel point does not meet the recognition condition; if the depth difference does not exceed the difference threshold, it is determined that the first candidate pixel point meets the recognition condition.

[0120] Among them, the preset recognition condition is that the absolute value of the difference between the depth value of the first candidate pixel point and the average depth value does not exceed the preset difference threshold.

[0121] In specific implementation, if the image identifier of the error image is an edge error image, first determine the corresponding position of the edge error image in the original image according to the coordinates corresponding to the boundary pixel points of the edge error image and the corresponding relationship between the pixel points of the depth image and the original image. Then, search for the adjacent area at this position in an interpolation manner, and screen out multiple candidate pixel points with the closest color space to the edge error image from the candidate pixel points in the adjacent area, and determine the target pixel point from the candidate pixel points. More specifically, the color difference can be calculated by obtaining the color values of each candidate pixel point and the color mean value of each pixel point in the edge error image, and calculating the difference between the color value of each candidate pixel point and the color mean value.

[0122] Further, sort the candidate pixel points in the order of the numerical values of the color differences to obtain a candidate pixel point sequence. Take the candidate pixel point with the smallest color difference in the candidate pixel point sequence as the first candidate pixel point, obtain the depth value of the first candidate pixel point, and obtain the average value of the depth values of the area near the corresponding position of the edge error image, denoted as the average depth value. Calculate the difference between the depth value of the first candidate pixel point and the average depth value as the depth difference. If the depth difference does not exceed the preset difference threshold, it is determined that the first candidate pixel point meets the preset recognition conditions, and the first candidate pixel point can be used as the target pixel point. Conversely, if the depth difference exceeds the preset difference threshold, it is determined that the first candidate pixel point does not meet the preset recognition conditions, then obtain the second candidate pixel point from the candidate pixel point sequence as the new first candidate pixel point, and return to step S504, and so on until a candidate pixel point that meets the preset recognition conditions is found.

[0123] In the above embodiment, when the image identifier of the error image is an edge error image, first determine the corresponding position of the edge error image in the original image, and find multiple candidate pixel points closest to the edge error image from the adjacent area of this corresponding position, and sort these candidate pixel points. By comparing the depth values of each candidate pixel point with the average depth value of the adjacent area, screen out the candidate pixel points that meet the preset recognition conditions as the target pixel points, so as to correct the edge error image according to the depth value of the target pixel points.

[0124] In one embodiment, after determining that the first candidate pixel point does not meet the recognition conditions, it further includes: removing the first candidate pixel point to obtain a new candidate pixel point sequence, and determining whether the new candidate pixel point sequence is empty; if the new candidate pixel point sequence is not empty, obtain the second candidate pixel point as the new first candidate pixel point; if the new candidate pixel point sequence is empty, take the pixel point with the smallest depth difference between the depth value and the average depth value in the candidate pixel point sequence as the target pixel point.

[0125] In this embodiment, after determining that the first candidate pixel point does not meet the preset recognition conditions, remove the first candidate pixel point from the candidate pixel point sequence to obtain a new candidate pixel point sequence, and determine whether the new candidate pixel point sequence is empty, so as to adopt corresponding processing methods according to the judgment result, which can improve the speed of determining the target pixel point.

[0126] It should be noted that the present application does not limit the repair method of the error image. The above repair methods such as linear interpolation repair, repair with the average depth value, and finding the target pixel point for repair by the difference method can also be repair methods such as inpainting (image completion algorithm) and deeplearning (deep learning).

[0127] To more clearly illustrate the technical solutions provided by the embodiments of the present application, the following will be combined with Figure 6 and Figure 7 to describe this solution. Figure 6 As shown in Figure 6 , it is a schematic flowchart of the process for obtaining a depth map to be corrected in an embodiment. The specific process of this method is as follows:

[0128] (1) For the original image, it is segmented by an image segmentation module to obtain a segmented image including multiple segmentation regions. The image edges of the segmented image are extracted by an image edge extraction module to obtain the image edges of the original image.

[0129] (2) For the depth image, edge extraction is performed by a depth edge extraction module to obtain the depth edges of the depth image.

[0130] (3) The edge inconsistency detection module performs edge matching on the image edges of the original image and the depth edges of the depth image, and takes the image formed by the unmatched depth edges as an error image.

[0131] (4) Determine the image identifier of the error image. The error image is calibrated by an error image calibration module, and the error image is used as the depth map to be corrected. The depth map to be corrected is corrected by a depth image correction module to obtain a corrected depth map.

[0132] As Figure 7 shown, it is a schematic flowchart of the process for correcting the error image in an embodiment, which specifically includes the following steps:

[0133] (5) Determine whether the depth map to be corrected (i.e., the error image) is an edge error image. If not, determine that the depth map to be corrected is an internal error image, and calculate the depth value distribution of the segmentation region where the internal error image is located; determine whether the segmentation region where the internal error image is located is a plane according to the depth value distribution. If so, use the average depth value of the segmentation region as the repair basis to correct the internal error image; if not, perform linear interpolation correction in the gradient direction of the depth values of the segmentation region to correct the internal error image.

[0134] (6) If it is determined that the depth map to be corrected is an edge error image, search for and sort the top n candidate pixels with the closest color space in the adjacent dx region to obtain a candidate pixel sequence.

[0135] (7) Calculate the average depth value of the segmentation region where it is located, and obtain the depth value of the currently ranked first candidate pixel. Then calculate the absolute value of the depth difference between the depth value of the currently ranked first candidate pixel (i.e., the first candidate pixel point) and the average depth value as the absolute difference.

[0136] (8) Compare the absolute difference with a preset difference threshold. If the absolute difference is less than the difference threshold, use the depth value of the currently top-ranked candidate pixel (i.e., the first candidate pixel point) as the basis for patching to correct the wrong image;

[0137] (9) If the absolute difference is not less than the difference threshold, remove the currently top-ranked candidate pixel to obtain a new candidate pixel sequence, and determine whether the new candidate pixel point sequence is empty. If it is not empty, return to step (7). If it is empty, use the depth value of the candidate pixel with the smallest absolute difference from the average depth value as the basis for patching to correct the wrong image.

[0138] It should be understood that although Figure 3 、 Figures 5 - 7 each step in the flowchart of Figure 3 、 Figures 5 - 7 is shown in sequence according to the arrow indication, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover,

[0139] In one embodiment, as Figure 8 shown, a depth map processing device is provided, including an image acquisition module 802, an edge detection module 804, an edge matching module 806, and a correction module 808, where:

[0140] The image acquisition module 802 is configured to acquire an original image and a depth image of the original image; the original image is a color image corresponding to the depth image;

[0141] The edge detection module 804 is configured to perform edge detection processing on the original image to obtain the image edge of the original image, and perform edge detection processing on the depth image to obtain the depth edge of the depth image;

[0142] The edge matching module 806 is configured to perform edge matching between the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges as the wrong image of the depth image;

[0143] The correction module 808 is configured to perform correction processing on the wrong image of the depth image to obtain a corrected depth image.

[0144] In one embodiment, the above-mentioned edge detection module 804 is further configured to perform image segmentation processing on the original image to obtain a segmented image; the segmented image includes a plurality of segmentation regions; perform edge detection processing on the segmented image, extract the detected pixel points, and sequentially connect each pixel point to obtain the image edge of the original image.

[0145] In one embodiment, the above-mentioned correction module 808 further includes:

[0146] An identification determination sub-module, configured to determine the image identification of the error image;

[0147] A correction sub-module, configured to perform corresponding correction processing on the error image according to the image identification to obtain a corrected depth image.

[0148] In one embodiment, the above-mentioned identification determination sub-module is further configured to determine that the image identification of the error image is an internal error image if the error image is within the segmentation region where the error image is located; determine that the image identification of the error image is an edge error image if the error image is adjacent to at least two segmentation regions.

[0149] In one embodiment, the above-mentioned correction sub-module is further configured to determine the surface identification of the segmentation region where the internal error image is located if the image identification of the error image is an internal error image, and perform corresponding correction processing on the internal error image according to the surface identification; if the image identification of the error image is an edge error image, determine a target pixel point from the adjacent region of the edge error image; obtain the depth value of the target pixel point, and use the depth value of the target pixel point to correct the edge error image.

[0150] In one embodiment, the above-mentioned correction sub-module is further configured to obtain the coordinates and corresponding depth values of each pixel point in the segmentation region where the internal error image is located; perform plane detection processing on the segmentation region where the internal error image is located according to the coordinates and depth values of each pixel point to obtain the surface identification of the segmentation region where the internal error image is located.

[0151] In one embodiment, the above-mentioned correction sub-module is further configured to obtain the depth values of each pixel point in the segmentation region where the internal error image is located if the surface identification of the segmentation region where the internal error image is located is a plane identification; calculate the average value of each depth value, and use the average value to correct the internal error image; if the surface identification of the segmentation region where the internal error image is located is a non-plane identification, use a linear interpolation correction model to correct the internal error image.

[0152] In one embodiment, the above-mentioned correction sub-module is further configured to, if the image identifier of the error image is an edge error image, determine the corresponding position of the edge error image in the original image; obtain the color values of each candidate pixel point in the adjacent area corresponding to the position and the color mean value of the edge error image; calculate the color difference between the color value of each candidate pixel point and the color mean value of the edge error image, and determine the target pixel point from each candidate pixel point according to the color difference.

[0153] In one embodiment, the above-mentioned correction sub-module is further configured to sort each candidate pixel point according to the numerical order of the color difference to obtain a candidate pixel point sequence; obtain the average depth value of the adjacent area and the depth value of the first candidate pixel point in the candidate pixel point sequence; the first candidate pixel point represents the pixel point with the smallest difference between the color value and the color value of the edge error image; determine whether the first candidate pixel point meets the preset recognition condition according to the depth value of the first candidate pixel point and the average depth value of the adjacent area. If the first candidate pixel point meets the recognition condition, the first candidate pixel point is used as the target pixel point; if the first candidate pixel point does not meet the recognition condition, obtain the second candidate pixel point from the candidate pixel point sequence as the new first candidate pixel point, and return to the step of obtaining the depth value of the first candidate pixel point in the candidate pixel point sequence.

[0154] In one embodiment, the above-mentioned correction sub-module is further configured to calculate the depth difference between the depth value of the first candidate pixel point and the average depth value; if the depth difference exceeds the difference threshold, it is determined that the first candidate pixel point does not meet the recognition condition; if the depth difference does not exceed the difference threshold, it is determined that the first candidate pixel point meets the recognition condition.

[0155] In one embodiment, the above-mentioned correction sub-module is further configured to remove the first candidate pixel point to obtain a new candidate pixel point sequence, and determine whether the new candidate pixel point sequence is empty; if the new candidate pixel point sequence is not empty, obtain the second candidate pixel point as the new first candidate pixel point; if the new candidate pixel point sequence is empty, use the pixel point with the smallest depth difference between the depth value in the candidate pixel point sequence and the average depth value as the target pixel point.

[0156] It should be noted that the depth map processing device of the present application corresponds one-to-one with the depth map processing method of the present application. The technical features and beneficial effects described in the embodiments of the above-mentioned depth map processing method are applicable to the embodiments of the depth map processing device. For specific content, reference can be made to the description in the method embodiments of the present application, which will not be repeated here. This is hereby declared.

[0157] In addition, each module in the above depth map processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0158] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a carrier network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a depth map processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0159] Those skilled in the art can understand that Figure 9 the structure shown in

[0160] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0161] Obtain an original image and the depth image of the original image; the original image is a color image corresponding to the depth image;

[0162] Perform edge detection processing on the original image to obtain the image edge of the original image, and perform edge detection processing on the depth image to obtain the depth edge of the depth image;

[0163] Perform edge matching between the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges, which is used as the error image of the depth image;

[0164] Perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0165] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0167] Obtain an original image and the depth image of the original image; the original image is a color image corresponding to the depth image;

[0168] Perform edge detection processing on the original image to obtain the image edge of the original image, and perform edge detection processing on the depth image to obtain the depth edge of the depth image;

[0169] Perform edge matching between the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges, which is used as the error image of the depth image;

[0170] Perform correction processing on the error image of the depth image to obtain a corrected depth image.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0173] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0174] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A depth map processing method, characterized in that The method includes: Obtaining an original image and a depth image of the original image; the original image is a color image corresponding to the depth image; Performing edge detection processing on the original image to obtain the image edge of the original image, and performing edge detection processing on the depth image to obtain the depth edge of the depth image; Performing edge matching on the image edge of the original image and the depth edge of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image; Determining an image identifier of the error image; the image identifier includes: an internal error image and an edge error image, the internal error image indicates that the error image is located within the segmentation region where the error image is located, and the edge error image indicates that the error image is adjacent to at least two segmentation regions obtained by segmenting the original image; Performing corresponding correction processing on the error image of the depth image according to the image identifier to obtain a corrected depth image.

2. The method according to claim 1, characterized in that, The performing edge detection processing on the original image to obtain the image edge of the original image includes: Performing image segmentation processing on the original image to obtain a segmented image; the segmented image includes a plurality of segmentation regions; Performing edge detection processing on the segmented image, extracting the detected pixel points and connecting the pixel points in sequence to obtain the image edge of the original image.

3. The method according to claim 1, wherein The performing corresponding correction processing on the error image of the depth image according to the image identifier to obtain a corrected depth image includes: If the image identifier of the error image is an internal error image, determining a surface identifier of the segmentation region where the internal error image is located, and performing corresponding correction processing on the internal error image according to the surface identifier; If the image identifier of the error image is an edge error image, determining target pixel points from the adjacent region of the edge error image; obtaining the depth value of the target pixel points, and using the depth value of the target pixel points to correct the edge error image.

4. The method according to claim 3, wherein The surface identifier of the segmentation region includes a plane identifier and a non-plane identifier; The determining the surface identifier of the segmentation region where the internal error image is located includes: Obtaining the coordinates and corresponding depth values of each pixel point in the segmentation region where the internal error image is located; Performing plane detection processing on the segmentation region where the internal error image is located according to the coordinates and depth values corresponding to each pixel point to obtain the surface identifier of the segmentation region where the internal error image is located.

5. The method according to claim 3, wherein The performing corresponding correction processing on the internal error image according to the surface identifier includes: If the surface identifier of the segmentation region where the internal error image is located is a plane identifier, obtaining the depth values of each pixel point in the segmentation region where the internal error image is located; calculating the mean value of each depth value, and using the mean value to correct the internal error image; If the surface identifier of the segmentation region where the internal error image is located is a non-plane identifier, correcting the internal error image by using a linear interpolation correction model.

6. The method according to claim 3, characterized in that If the image identifier of the error image is an edge error image, determining a target pixel point from adjacent regions of the edge error image includes: If the image identifier of the error image is an edge error image, determining the corresponding position of the edge error image in the original image; Obtaining the color values of each candidate pixel point in the adjacent region corresponding to the position and the color mean value of the edge error image; Calculating the color difference between the color value of each candidate pixel point and the color mean value of the edge error image, and determining a target pixel point from each candidate pixel point according to the color difference.

7. The method according to claim 6, characterized in that, The calculating the color difference between the color value of each candidate pixel point and the color mean value of the edge error image, and determining a target pixel point from each candidate pixel point according to the color difference includes: Sorting each of the candidate pixel points according to the numerical order of the color difference to obtain a candidate pixel point sequence; Obtaining the average depth value of the adjacent region and the depth value of the first candidate pixel point in the candidate pixel point sequence; the first candidate pixel point represents the pixel point with the smallest difference between the color value and the color value of the edge error image; Determining whether the first candidate pixel point meets a preset recognition condition according to the depth value of the first candidate pixel point and the average depth value of the adjacent region. If the first candidate pixel point meets the recognition condition, using the first candidate pixel point as the target pixel point; If the first candidate pixel point does not meet the recognition condition, obtaining a second candidate pixel point from the candidate pixel point sequence as a new first candidate pixel point, and returning to the step of obtaining the depth value of the first candidate pixel point in the candidate pixel point sequence.

8. The method according to claim 7, characterized in that The determining whether the first candidate pixel point meets a preset recognition condition according to the depth value of the first candidate pixel point and the average depth value of the adjacent region includes: Calculating the depth difference between the depth value of the first candidate pixel point and the average depth value; If the depth difference exceeds a difference threshold, determining that the first candidate pixel point does not meet the recognition condition; if the depth difference does not exceed the difference threshold, determining that the first candidate pixel point meets the recognition condition.

9. The method according to claim 8, characterized in that, After determining that the first candidate pixel point does not meet the recognition condition, it further includes: Removing the first candidate pixel point to obtain a new candidate pixel point sequence, and determining whether the new candidate pixel point sequence is empty; If the new candidate pixel point sequence is not empty, obtaining the second candidate pixel point as a new first candidate pixel point; If the new candidate pixel point sequence is empty, using the pixel point with the smallest depth difference between the depth value and the average depth value in the candidate pixel point sequence as the target pixel point.

10. A depth map processing device, characterized in that, The device includes: An image acquisition module, configured to acquire an original image and a depth image of the original image; the original image is a color image corresponding to the depth image; An edge detection module, configured to perform edge detection processing on the original image to obtain the image edges of the original image, and perform edge detection processing on the depth image to obtain the depth edges of the depth image; An edge matching module, configured to perform edge matching between the image edges of the original image and the depth edges of the depth image to obtain an image formed by the unmatched depth edges as the error image of the depth image; A correction module, configured to determine the image identifier of the error image, and perform corresponding correction processing on the error image of the depth image according to the image identifier to obtain a corrected depth image; the image identifier includes: an internal error image and an edge error image, where the internal error image indicates that the error image is located within the segmentation region where the error image is located, and the edge error image indicates that the error image is adjacent to at least two segmentation regions obtained by segmenting the original image.

11. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

Images