Depth image processing method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a method, apparatus, device, and storage medium for processing a depth image, which belongs to the field of artificial intelligence technology. The method includes: obtaining a color image and a depth image corresponding to the color image; extracting a first depth image block from the depth image based on a depth edge in the depth image, wherein the first depth image block is distributed along the depth edge; extracting a color image block corresponding to the first depth image block from the color image; based on the first depth image block and the color image block, performing edge refinement on the first depth image block to obtain a second depth image block, wherein the accuracy of the depth edge in the second depth image block is higher than the accuracy of the depth edge in the first depth image block; and replacing the first depth image block in the depth image with the second depth image block.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a depth image processing method, apparatus, device, and storage medium. Background Art

[0002] A depth image refers to an image that uses the distance (depth) values of each point in the scene captured by the image collector as pixel values.

[0003] In related technologies, a color image is processed by a depth estimation algorithm to obtain a corresponding depth image. However, since the depth image represents the depth value of the entire area in the scene, it is difficult to determine the accurate edge depth from the depth image when the proportion of edge pixels to all image pixels is low. Therefore, in order to confirm the edge position of different objects in the scene, it is necessary to confirm the edge depth in the depth image more accurately. Summary of the Invention

[0004] The embodiments of the present application provide a depth image processing method, apparatus, device, and storage medium that can improve the efficiency of optimizing depth edges in depth images. The technical solution is as follows:

[0005] In one aspect, an embodiment of the present application provides a method for processing a depth image, the method comprising:

[0006] Acquire a color image and a depth image corresponding to the color image;

[0007] extracting a first depth image block from the depth image based on a depth edge in the depth image, where the first depth image block is distributed along the depth edge;

[0008] Extracting a color image block corresponding to the first depth image block from the color image;

[0009] Based on the first depth image block and the color image block, perform edge thinning on the first depth image block to obtain a second depth image block, where precision of the depth edge in the second depth image block is higher than precision of the depth edge in the first depth image block;

[0010] The first depth image block in the depth image is replaced with the second depth image block.

[0011] On the other hand, an embodiment of the present application provides a method for processing a depth image, the method comprising:

[0012] Acquire a sample color image and a sample depth image corresponding to the sample color image;

[0013] extracting a first sample depth image block from the sample depth image based on a depth edge in the sample depth image, where the first sample depth image block is distributed along the depth edge;

[0014] extracting a sample color image block corresponding to the first sample depth image block from the sample color image;

[0015] Based on the first sample depth image block and the sample color image block, performing edge thinning on the first sample depth image block through a thinning network to obtain a second sample depth image block;

[0016] determining an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block;

[0017] The refinement network is trained based on the edge refinement loss.

[0018] On the other hand, an embodiment of the present application provides a depth image processing device, the device comprising:

[0019] An image acquisition module, configured to acquire a color image and a depth image corresponding to the color image;

[0020] an image block extraction module, configured to extract a first depth image block from the depth image based on a depth edge in the depth image, where the first depth image block is distributed along the depth edge;

[0021] The image block extraction module is further configured to extract a color image block corresponding to the first depth image block from the color image;

[0022] an edge refinement module, configured to perform edge refinement on the first depth image block based on the first depth image block and the color image block to obtain a second depth image block, wherein the precision of the depth edge in the second depth image block is higher than the precision of the depth edge in the first depth image block;

[0023] An image block replacement module is configured to replace the first depth image block in the depth image with the second depth image block.

[0024] On the other hand, an embodiment of the present application provides a depth image processing device, the device comprising:

[0025] An image acquisition module, configured to acquire a sample color image and a sample depth image corresponding to the sample color image;

[0026] an image block extraction module, configured to extract a first sample depth image block from the sample depth image based on a depth edge in the sample depth image, wherein the first sample depth image block is distributed along the depth edge;

[0027] The image block extraction module is further configured to extract a sample color image block corresponding to the first sample depth image block from the sample color image;

[0028] an edge refinement module, configured to perform edge refinement on the first sample depth image block through a refinement network based on the first sample depth image block and the sample color image block to obtain a second sample depth image block;

[0029] a loss determining module, configured to determine an edge thinning loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block;

[0030] A network training module is used to train the refinement network based on the edge refinement loss.

[0031] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the depth image processing method as described in the above aspect.

[0032] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the depth image processing method as described in the above aspects.

[0033] In another aspect, an embodiment of the present application provides a computer program product, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the depth image processing method provided in the above aspects.

[0034] In an embodiment of the present application, based on the depth edge in the depth image, a first depth image block is extracted from the first depth image and a corresponding color image block is extracted from the color image, so that based on the color image block and the first depth image block, the edge of the first depth image block is refined to obtain a second depth image block with higher depth edge accuracy than the first depth image block, and then the first depth image block in the depth image is replaced with the second depth image block to achieve optimization of the depth edge in the depth image. By adopting the solution provided in the embodiment of the present application, the depth edge is accurately refined by extracting the depth image block, thereby improving the optimization efficiency of the depth edge in the depth image and reducing the amount of computation in the process of refining the depth edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram showing an implementation scenario provided by an exemplary embodiment of the present application is shown;

[0036] Figure 2 is a flowchart of a depth image processing method provided by an exemplary embodiment of the present application;

[0037] Figure 3 is a flowchart of a depth image processing method provided by another exemplary embodiment of the present application;

[0038] Figure 4 is a flowchart of a depth image processing method provided by another exemplary embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a depth image processing process provided by an exemplary embodiment of the present application;

[0040] Figure 6 is a flowchart of a depth image processing method provided by another exemplary embodiment of the present application;

[0041] Figure 7 is a schematic diagram of a process for determining a reference depth image block provided by an exemplary embodiment of the present application;

[0042] Figure 8 is a schematic diagram of a refinement network training process provided by an exemplary embodiment of the present application;

[0043] Figure 9 is a structural block diagram of a depth image processing device provided by an exemplary embodiment of the present application;

[0044] Figure 10 is a structural block diagram of a depth image processing device provided by another exemplary embodiment of the present application;

[0045] Figure 11A schematic structural diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0047] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0049] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0050] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0051] The solutions provided in the embodiments of this application involve technologies such as artificial intelligence computer vision, which are specifically illustrated by the following embodiments.

[0052] Figure 1 A schematic diagram of an implementation scenario provided by an exemplary embodiment of the present application is shown.

[0053] The depth image processing method provided in the embodiments of the present application can be applied to products such as mobile robots, autonomous driving vehicles, robotic arms, and augmented reality (AR) devices.

[0054] For a mobile robot, during the movement of the mobile robot, the scene during the movement can be collected in real time through the camera component. By adopting the depth image processing method provided in the embodiment of the present application, the mobile robot can perform edge depth processing on the depth image corresponding to the collected scene image, determine the shape and distance of nearby obstacles in the scene, and make more reliable obstacle avoidance operations, thereby preventing the mobile robot from bumping into obstacles or repeatedly taking detours. At the same time, when the mobile robot encounters scenes such as steps and wooden boxes, it can quickly determine the edge with a large depth difference near the landing point and adjust the landing point, thereby avoiding situations such as stepping on air and slipping when the mobile robot lands.

[0055] Indicative, such as Figure 1 As shown, when the robot dog 110 encounters steps, it captures an image of the current step scene and performs depth edge processing on the depth image corresponding to the scene image, and then determines the edge of the step in the scene and the landing point based on the acquired depth information, thereby avoiding situations such as stepping on air and slipping when landing.

[0056] For autonomous vehicles, during the driving process, the vehicle uses a camera component to collect real-time information about road driving conditions. By adopting the depth image processing method provided in the embodiment of the present application, the autonomous vehicle can identify and judge the specific positions of surrounding vehicles in the road scene, thereby making reliable obstacle avoidance actions and improving the safety of the autonomous vehicle.

[0057] Indicative, such as Figure 1As shown, during the parking process, the autonomous driving vehicle 120 captures images of vehicles on the left and right sides in front, and performs depth edge processing on the depth images corresponding to the captured scene images, and then determines the appropriate parking position for the vehicle based on the acquired depth information.

[0058] For a robotic arm, during the process of grasping or moving highly complex objects, by adopting the depth image processing method provided in the embodiment of the present application, the efficiency of the robotic arm in recognizing the outline, size and spatial position of highly complex objects can be improved, which helps to grasp objects and apply force more accurately.

[0059] For AR scenes, depth edges with large errors will cause the reconstructed three-dimensional space to be blurred and distorted. The method provided in the embodiments of the present application can optimize the depth map of the image estimation corresponding to the virtual space, thereby improving the realism and immersion of the virtual space.

[0060] It should be noted that the processing of depth images and the training of the refinement network in the embodiments of the present application can be completed by a computer device (such as a personal computer) or by a server. The embodiments of the present application are not limited to this. The following embodiments only use the example of a computer device completing the processing of depth images and the training of the refinement network to illustrate.

[0061] Please refer to Figure 2 , which shows a flowchart of a depth image processing method provided by an exemplary embodiment of the present application. This embodiment uses the method applied to a computer device as an example for explanation. The method includes the following steps:

[0062] Step 201: Acquire a color image and a depth image corresponding to the color image.

[0063] Optionally, a color image refers to an image composed of multiple color channels, which can be acquired in real time by a computer device through an image collector, or can be an image pre-stored in a computer device. This embodiment of the present application does not limit this.

[0064] Optionally, the color image can be an RGB (RedGreenBlue) image, or a BGR (BlueGreenRed) image, or an HSV (HueSaturationValue) image, or a CMYK (CyanMagentaYellowBlack) image, or a color image in other color formats, which is not limited in the embodiments of the present application.

[0065] Optionally, the depth image refers to an image that uses the distance (depth) values of each point in the scene collected by the image collector as pixel values.

[0066] In one possible implementation, after the computer device acquires a color image, it can process the color image through a depth estimation algorithm to obtain a depth image corresponding to the color image. Optionally, the depth estimation algorithm can be a monocular depth estimation algorithm or a binocular depth estimation algorithm, which is not limited in this embodiment of the present application.

[0067] In one possible implementation, the computer device may process the color image using a binocular depth estimation algorithm. Optionally, the binocular depth estimation algorithm may be a cascade stereo matching algorithm, a fast stereo matching algorithm based on an adaptive aggregation network, a stereo matching algorithm based on a neural network structure search, and the like.

[0068] In one illustrative example, a computer device processes a color image using a binocular depth estimation algorithm. First, the computer device uses a binocular camera to capture two left and right views of the same scene at the same time. Then, based on these two views, the computer device infers the depth values corresponding to each pixel in the left view, thereby obtaining a depth image corresponding to the image captured by the left camera.

[0069] Step 202 : extracting a first depth image block from the depth image based on a depth edge in the depth image, where the first depth image block is distributed along the depth edge.

[0070] Optionally, the depth edge refers to the pixel area between the foreground and background in the depth image, and the difference in depth values corresponding to the pixels on both sides of the depth edge is greater than the depth value threshold. Optionally, the depth value threshold can be determined based on the depth values of each pixel in the actual depth image.

[0071] In one possible implementation, since the number of pixels corresponding to depth edges in the complete depth image is relatively small, there may be errors in the depth values corresponding to the pixels at the depth edge positions in the depth image obtained by the depth estimation algorithm, and directly performing edge refinement on the complete depth image may not be able to achieve more accurate refinement of the depth edges. Therefore, in order to improve the accuracy of the depth values corresponding to the pixels at the depth edges in the depth image, the computer device performs targeted refinement on the depth edges in the depth image.

[0072] In one possible embodiment, the computer device performs block processing on the depth edge based on the depth edge in the depth image, and extracts a first depth image block distributed along the depth edge from the depth image, so that the proportion of the number of pixel points corresponding to the depth edge in the first depth image block is significantly higher than the proportion of the number of pixel points corresponding to the depth edge in the complete depth image.

[0073] Optionally, the size of the first depth image block may be determined according to specific requirements for depth edge optimization, which is not limited in this embodiment of the present application.

[0074] In an illustrative example, the number of pixels corresponding to the depth edge in the complete depth image accounts for 1% of all pixels in the complete depth image. After determining the first depth image block along the depth edge, the number of pixels corresponding to the depth edge in the first depth image block accounts for 10% of all pixels in the first depth image block.

[0075] Step 203: extract a color image block corresponding to the first depth image block from the color image.

[0076] In a possible implementation, in order to adjust the depth edge in the depth image according to the edge information of each object in the real scene, the computer device needs to extract a color image block corresponding to the first depth image block from the color image based on the extracted first depth image block.

[0077] In one possible implementation, a computer device matches pixels of the color image and the depth image, and determines the pixels of the first depth image block in the depth image based on the position information of the first depth image block in the depth image, thereby extracting the corresponding color image block from the color image based on the pixel correspondence between the color image and the depth image.

[0078] In step 204 , based on the first depth image block and the color image block, edge refinement is performed on the first depth image block to obtain a second depth image block, wherein the accuracy of the depth edge in the second depth image block is higher than that in the first depth image block.

[0079] Optionally, edge refinement refers to the process of adjusting the originally thick depth edges in the depth image to thin depth edges. By adjusting the depth values corresponding to the pixel points at the depth edges in the depth image, the depth values at the depth edges in the depth image can be made closer to the foreground and background edge distribution of objects in the real scene.

[0080] In one possible embodiment, a computer device performs edge refinement on the first depth image block based on edge information of a real object in the color image block and a depth edge to be optimized in the first depth image block, adjusts the depth value corresponding to each pixel at the depth edge, and obtains a second depth image block, so that the accuracy of the depth edge in the second depth image block is higher than the accuracy of the depth edge in the first depth image block, wherein the accuracy of the depth edge refers to the degree of approximation between the depth value of each pixel at the depth edge in the depth image block and the distance from the object in the real scene to the image collector.

[0081] In a possible implementation, the computer device may estimate and adjust the depth value of each pixel at the depth edge according to the depth values corresponding to the pixels in the area adjacent to the depth edge.

[0082] Step 205: Replace the first depth image block in the depth image with the second depth image block.

[0083] In one possible embodiment, after completing the edge refinement of the first depth image block to obtain the second depth image block, the computer device replaces the first depth image block in the depth image with the second depth image block, thereby obtaining a depth image in which the depth edges are completely refined, and the accuracy of the depth edges in the replaced depth image is higher than the accuracy of the depth edges in the depth image before the replacement.

[0084] To sum up, in the embodiments of the present application, based on the depth edge in the depth image, a first depth image block is extracted from the first depth image and a corresponding color image block is extracted from the color image, so that based on the color image block and the first depth image block, the edge of the first depth image block is refined to obtain a second depth image block with higher depth edge accuracy than the first depth image block, and then the first depth image block in the depth image is replaced by the second depth image block to achieve optimization of the depth edge in the depth image. By adopting the solution provided in the embodiments of the present application, the depth edge is accurately refined by extracting the depth image block, thereby improving the optimization efficiency of the depth edge in the depth image and reducing the amount of computation in the process of refining the depth edge.

[0085] In a possible implementation, in order to more accurately refine the depth edge, the computer device performs edge refinement on the first depth image block through a refinement network. The following embodiment illustrates the training process of the refinement network.

[0086] Please refer to Figure 3 , which shows a flowchart of a depth image processing method provided by another exemplary embodiment of the present application. This embodiment uses the method applied to a computer device as an example for explanation. The method includes the following steps:

[0087] Step 301: Acquire a sample color image and a sample depth image corresponding to the sample color image.

[0088] Optionally, the sample color image may be a color image pre-stored in the computer device.

[0089] In a possible implementation, the computer device performs depth estimation on the sample color image using a depth estimation algorithm to obtain a sample depth image corresponding to the sample color image.

[0090] Optionally, the depth estimation algorithm may be a monocular depth estimation algorithm, a binocular depth estimation algorithm, or other depth estimation algorithms, which is not limited in the embodiments of the present application.

[0091] Step 302 : extracting a first sample depth image block from the sample depth image based on a depth edge in the sample depth image, where the first sample depth image block is distributed along the depth edge.

[0092] In one possible implementation, since the number of pixels corresponding to depth edges in the complete sample depth image is relatively small, there may be errors in the depth values corresponding to the pixels at the depth edge positions in the sample depth image obtained by the depth estimation algorithm, and directly performing edge refinement on the complete sample depth image may not be able to achieve more accurate refinement of the depth edges. Therefore, in order to improve the accuracy of the depth values corresponding to the pixels at the depth edges in the sample depth image, the computer device performs targeted refinement on the depth edges in the sample depth image.

[0093] In a possible implementation, the computer device performs block processing on the depth edge based on the depth edge in the sample depth image, and extracts a first sample depth image block distributed along the depth edge from the sample depth image.

[0094] Step 303: extract a sample color image block corresponding to the first sample depth image block from the sample color image.

[0095] In a possible implementation, in order to perform edge refinement on the first sample depth image block according to edge information of an object in a real scene, the computer device needs to determine a sample color image block corresponding to the first sample depth image block.

[0096] In one possible implementation, a computer device matches pixels of the sample color image and the sample depth image, and determines the pixels of the first sample depth image block in the sample depth image based on the position information of the first sample depth image block in the sample depth image, thereby extracting the corresponding sample color image block from the sample color image based on the pixel correspondence between the sample color image and the sample depth image.

[0097] Step 304 : Based on the first sample depth image block and the sample color image block, perform edge thinning on the first sample depth image block through a thinning network to obtain a second sample depth image block.

[0098] In one possible implementation, the computer device performs edge refinement on the first sample depth image block through a refinement network based on edge information of the real object in the sample color image block and the depth edge to be optimized in the first sample depth image block, adjusts the depth value corresponding to each pixel at the depth edge, and obtains the second sample depth image block.

[0099] In one possible implementation, a computer device inputs a first sample depth image block and a sample color image block into a refinement network, and the refinement network outputs an edge refinement adjustment amount for the first sample depth image block, thereby adjusting the depth edge in the first sample depth image block based on the edge refinement adjustment amount to obtain a second sample depth image block.

[0100] Step 305 : Determine an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block.

[0101] In one possible implementation, in order to improve the edge refinement accuracy of the refinement network for the first sample depth image block, the computer device uses the reference depth image block corresponding to the second sample depth image block as supervision of the refinement network, and determines the edge refinement loss based on the second sample depth image block and the reference depth image block.

[0102] Optionally, the edge refinement loss represents the difference between the depth value corresponding to each pixel in the second sample depth image block and the depth value corresponding to each pixel in the reference depth image block. The greater the difference, the greater the edge refinement loss.

[0103] Step 306: Train the refinement network based on the edge refinement loss.

[0104] In one possible embodiment, in order to improve the refinement accuracy of the refinement network for depth edges, the computer device trains the refinement network based on the edge refinement loss, so that the depth value corresponding to each pixel in the second sample depth image block is closer to the depth value corresponding to each pixel in the reference depth image block.

[0105] To summarize, in an embodiment of the present application, a first sample depth image block extracted from a sample depth image and a sample color image block extracted from a sample color image are input into a refinement network, and the first sample depth image block is edge-refined using the refinement network to obtain a second sample depth image block, thereby obtaining an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block, and the refinement network is trained with the edge refinement loss; by adopting the solution provided in an embodiment of the present application, the depth edge of the image is refined by extracting depth image blocks, thereby improving the efficiency of the refinement of the depth edge; by training the refinement network with the edge refinement loss, the accuracy of the depth edge refinement performed by the refinement network is improved.

[0106] In one possible implementation, in order to reduce unnecessary computational effort and achieve precise refinement of depth edges, the computer device may extract the depth edges using an edge extraction operator, and refine the first depth image block using a refinement network during the edge refinement process.

[0107] Please refer to Figure 4 , which shows a flowchart of a depth image processing method provided by another exemplary embodiment of the present application. This embodiment uses the method applied to a computer device as an example for explanation. The method includes the following steps:

[0108] Step 401: Acquire a color image and a depth image corresponding to the color image.

[0109] The specific implementation of this step can refer to step 201, and this embodiment will not be described in detail here.

[0110] Step 402: extract depth edges from the depth image based on an edge extraction operator.

[0111] In one possible implementation, since the edge conditions of objects in the scene are diverse, the interior of the objects may also correspond to different depth values in the depth image due to problems such as concave and convex shapes. Therefore, in order to reduce the amount of calculation and achieve accurate refinement of the depth edges, the computer device can use an edge extraction operator to extract depth edges with large depth differences in the depth image.

[0112] Optionally, the edge extraction operator may be a Sobel operator, a Laplacian operator, a Canny operator, etc., which is not limited in the embodiment of the present application.

[0113] In one possible implementation, the computer device determines a depth drop threshold according to the need for depth edge refinement, and based on the depth drop threshold, uses an edge extraction operator to extract the depth edges in the depth image, and extracts the depth edges whose depth drop is greater than the depth drop threshold.

[0114] In an illustrative example, the computer device sets the depth drop threshold to 1000 mm according to the need for depth edge refinement, so that the computer device uses the edge extraction operator to extract depth edges with a depth drop greater than 1000 mm.

[0115] Step 403: Create a candidate box covering the depth edge by sliding the window.

[0116] In a possible implementation, in order to achieve edge refinement processing for all depth edges, the computer device creates candidate boxes along the depth edges through a sliding window, and achieves full coverage of the depth edges through the candidate boxes.

[0117] Optionally, the size of the sliding window is the same as the size of the first depth image block.

[0118] In one possible embodiment, the computer device determines the sliding route of the sliding window with reference to each pixel point at the depth edge, and determines the size of the sliding window based on the set size of the first depth image block, and then aligns the center of the sliding window with the pixel point, so that the sliding window slides along the depth edge, creates a candidate box, and covers all pixel points corresponding to the depth edge through the candidate box.

[0119] Step 404 : Filter out a target box from the candidate boxes. The target box covers the depth edge, and the density of the target box is smaller than the density of the candidate boxes.

[0120] In one possible implementation, since there is inevitable overlap between candidate boxes created along the depth edge by sliding the window, in order to reduce the amount of computation and avoid repeated refinement of the same depth edge, the computer device screens the candidate boxes, removes redundant candidate boxes while covering all depth edges, and obtains the target box, so that the density of the target box is less than the density of the candidate box.

[0121] In one possible implementation, the computer device screens the candidate boxes using a non-maximum suppression (NMS) algorithm, and by setting an overlap threshold between the candidate boxes, screens the candidate boxes whose overlap is greater than the overlap threshold to obtain a target box, thereby removing redundant candidate boxes.

[0122] Step 405 : extracting a first depth image block from the depth image based on the position of the target box.

[0123] In a possible implementation, the computer device determines the pixels in the target box according to the position of the target box in the depth image, and extracts a first depth image block from the depth image, where the first depth image block is composed of all the pixels in the target box.

[0124] Step 406 : extracting a color image block corresponding to the first depth image block from the color image based on the position of the target box.

[0125] In one possible implementation, the computer device determines the pixel points corresponding to the target box position in the color image based on the position of the target box in the depth image and the pixel correspondence between the depth image and the color image, and extracts a color image block corresponding to the first depth image block from the color image, where the color image block is composed of all pixel points corresponding to the target box position in the color image.

[0126] Step 407 : scaling the first depth image block and the color image block based on the scaling ratio, wherein the scaled first depth image block and the color image block are used for edge refinement.

[0127] In one possible implementation, the number of pixels in the first depth image block can be adjusted by scaling the first depth image block at different ratios. When the first depth image block is enlarged, the number of pixels in the first depth image block also increases accordingly, so that the degree of refinement of the depth edge of the first depth image block performed by the computer device is higher.

[0128] In a possible implementation, the computer device scales the first depth image block and the color image block based on the determined scaling ratio, and then uses the scaled first depth image block and the color image block for edge refinement.

[0129] In one possible embodiment, the computer device can determine the scaling ratio based on the accuracy requirement of the current depth recognition scene. The scaling ratio is positively correlated with the accuracy requirement. The higher the precision requirement, the larger the determined scaling ratio. By scaling the image block, the number of pixels contained in the image block will also change accordingly. Therefore, when the first depth image block is enlarged, the computer device can refine the edge of more pixels at the depth edge.

[0130] In an illustrative example, the resolution of the first depth image block extracted by the computer device is 32×32 pixels. In order to more accurately refine the depth edge of the first depth image block, the computer device may adjust the resolution of the first depth image block to 128×128 pixels.

[0131] In step 408 , the first depth image block and the color image block are input into a refinement network to obtain a depth residual output by the refinement network. The depth residual is used to represent the depth adjustment amount of the pixel points in the first depth image block.

[0132] In one possible embodiment, a computer device inputs the scaled first depth image block and the color image block into a refinement network, and outputs a depth residual corresponding to each pixel in the first depth image block through the refinement network, wherein the depth residual is used to represent the depth adjustment amount of each pixel in the first depth image block.

[0133] In one possible embodiment, since the depth residual corresponds to each pixel in the first depth image block, in order to more accurately determine the depth adjustment amount of each pixel, the computer device needs to align the pixel points in the first depth image block with the pixel points in the color image block.

[0134] In one possible implementation, the pixels of the first depth image block correspond to a depth channel, and the pixels of the color image block correspond to three channels of red, green, and blue (RGB). The computer device performs channel alignment on the depth channel of the first depth image block and the color channel of the color image block, so that the pixels in the first depth image block can be aligned with the pixels in the color image block, and then the first depth image block and the color image block after channel alignment are input into the refinement network, and the aligned pixels are calculated and processed through the convolution layer in the refinement network, so as to obtain the depth residual corresponding to each pixel in the first depth image block output by the refinement network.

[0135] In one possible embodiment, since the error between the depth value of the non-edge area in the depth image obtained by the depth estimation algorithm and the true depth value is small, the computer device can use the depth value of the non-edge area near the depth edge in the first depth image block as a reference to determine the depth value corresponding to each pixel point at the depth edge, and thus determine the corresponding depth adjustment amount based on the current depth value of each pixel point at the depth edge.

[0136] Step 409 : Use the depth residual to refine the edge of the first depth image block to obtain a second depth image block.

[0137] In one possible implementation, the computer device uses the depth residual to refine the edges of the first depth image block, that is, based on the depth adjustment amount corresponding to each pixel point, the depth value of each pixel point in the first depth image block is adjusted to obtain the second depth image block.

[0138] Step 410 : performing scaling and restoration processing on the second depth image block based on the scaling ratio, wherein the second depth image after the scaling and restoration processing is used to replace the first depth image.

[0139] In one possible embodiment, since the resolution of the scaled first depth image block is different from the resolution of the depth image, in order to replace the first depth image block with the second depth image block, the computer device needs to perform corresponding scaling and restoration processing on the second depth image block based on the scaling ratio, so that the resolution of the second depth image block is the same as the resolution of the depth image.

[0140] In an illustrative example, when the resolution of the first depth image block is adjusted from 32×32 pixels to 128×128 pixels, after completing the edge refinement processing, the computer device needs to correspondingly adjust the resolution of the second depth image block from 128×128 pixels to 32×32 pixels.

[0141] Step 411: Replace the first depth image block in the depth image with the second depth image block.

[0142] The specific implementation of this step can refer to step 205, and this embodiment will not be described in detail here.

[0143] In the above embodiment, the depth edges with a depth difference greater than the depth difference threshold are extracted through the edge extraction operator, and on the premise of ensuring that the depth edges are covered, the candidate boxes are screened to obtain the target box, thereby avoiding repeated refinement of the depth edges and reducing the amount of calculation in the edge refinement process.

[0144] In addition, before the image block is input into the refinement network, the accuracy requirements of the depth recognition scene can be met as much as possible by adjusting the scaling ratio of the image block, and then the edge of the first depth image block is refined by the depth residual output by the refinement network, thereby improving the efficiency of the depth edge refinement.

[0145] Please refer to Figure 5 , which shows a schematic diagram of a depth image processing process provided by an exemplary embodiment of the present application. First, a computer device obtains a color image 502 and a depth image 501 corresponding to the color image 502, performs depth edge extraction on the depth image 501, and then extracts a first depth image block 503 along the depth edge. Based on the position information 505 of the first depth image block 503 in the depth image 501, a color image block 504 is extracted from the color image 502. Further, the first depth image block 503 and the color image block 504 are adjusted according to the scaling ratio. Based on the scaled first depth image block 503 and the color image block 504, the first depth image block 503 is subjected to depth edge refinement through a refinement network 506 to obtain a second depth image block 507.

[0146] In one possible embodiment, during the training of the refinement network, in order to improve the accuracy of the depth residual output by the refinement network, the computer device may use the first sample depth image block as the input of the refinement network at different scaling ratios, and use the reference depth image block obtained by training with the true depth image as a reference as the supervision of the refinement network.

[0147] Please refer to Figure 6 , which shows a flowchart of a depth image processing method provided by another exemplary embodiment of the present application. This embodiment uses the method applied to a computer device as an example for explanation. The method includes the following steps:

[0148] Step 601: Acquire a sample color image and a sample depth image corresponding to the sample color image.

[0149] The specific implementation of this step can refer to step 301, and this embodiment will not be described in detail here.

[0150] Step 602: extract depth edges from the sample depth image based on an edge extraction operator.

[0151] In one possible implementation, since the edge conditions of objects in the scene are diverse, the interior of the objects may also correspond to different depth values in the sample depth image due to problems such as concave and convex shapes. Therefore, in order to reduce the amount of calculation and achieve accurate refinement of the depth edges, the computer device can use an edge extraction operator to extract depth edges with large depth differences in the sample depth image.

[0152] In one possible implementation, the computer device determines a sample depth difference threshold according to the need for depth edge refinement, and based on the sample depth difference threshold, uses an edge extraction operator to extract the depth edges in the sample depth image, and extracts depth edges with a depth difference greater than the depth difference threshold.

[0153] Step 603: Create a candidate box covering the depth edge by sliding the window.

[0154] In a possible implementation, in order to achieve edge refinement processing for all depth edges, the computer device creates candidate boxes along the depth edges through a sliding window, and achieves full coverage of the depth edges through the candidate boxes.

[0155] Optionally, the size of the sliding window is the same as the size of the first sample depth image block.

[0156] In one possible implementation, the computer device determines the sliding route of the sliding window with reference to each pixel point at the depth edge, and determines the size of the sliding window based on the set size of the first sample depth image block, and then aligns the center of the sliding window with the pixel point, so that the sliding window slides along the depth edge, creates a candidate box, and covers all pixel points corresponding to the depth edge through the candidate box.

[0157] Step 604 : Filter out a target box from the candidate boxes. The target box covers the depth edge, and the density of the target box is smaller than the density of the candidate boxes.

[0158] The specific implementation of this step can refer to step 404, which will not be described in detail in this embodiment.

[0159] Step 605 : extracting a first sample depth image block from the sample depth image based on the position of the target box.

[0160] In one possible implementation, the computer device determines the pixels in the target box according to the position of the target box in the sample depth image, and extracts a first sample depth image block from the sample depth image, where the first sample depth image block is composed of all the pixels in the target box.

[0161] Step 606 : extracting a sample color image block corresponding to the first sample depth image block from the sample color image based on the position of the target box.

[0162] In one possible embodiment, the computer device determines the pixel points corresponding to the target box position in the sample color image based on the position of the target box in the sample depth image and the pixel correspondence between the sample depth image and the sample color image, and extracts the sample color image block corresponding to the first sample depth image block from the sample color image, where the sample color image block is composed of all pixel points corresponding to the target box position in the sample color image.

[0163] Step 607 : Scaling the first sample depth image block and the sample color image block based on at least one scaling ratio, wherein the scaled first sample depth image block and the sample color image block are input into a refinement network.

[0164] In one possible implementation, in order to enable the refinement network to meet the edge refinement processing requirements of first depth image blocks of different resolutions and improve the refinement efficiency of the refinement network for first depth image blocks of different resolutions, a computer device performs scaling processing on the first sample depth image block and the sample color image block based on at least one scaling ratio to obtain at least one set of scaled first sample depth image blocks and sample color image blocks, which are used to input into the refinement network for training the refinement network.

[0165] In a possible implementation, the computer device may determine at least one scaling ratio based on the resolution of the extracted first sample depth image block. Optionally, the scaling ratio may be 0.5 times, 1.5 times, 2 times, etc.

[0166] In an illustrative example, when the resolution of the extracted first sample depth image block is 64×64, the computer device can adjust the resolution of the first sample depth image block to 32×32, 96×96, and 128×128 based on three scaling ratios, thereby inputting the first sample depth image blocks and sample color image blocks at the four resolutions into the refinement network to train the refinement network.

[0167] Step 608 : performing channel alignment on the depth channel of the first sample depth image block and the color channel of the sample color image block.

[0168] In one possible implementation, since the depth residual output by the refinement network corresponds to each pixel in the first sample depth image block, in order to more accurately determine the depth adjustment amount of each pixel, the computer device needs to align the pixels in the first sample depth image block with the pixels in the sample color image block.

[0169] In one possible implementation, the pixels of the first sample depth image block correspond to a depth channel, and the pixels of the sample color image block correspond to three channels of red, green, and blue (RGB). The computer device aligns the depth channel of the first sample depth image block and the color channel of the sample color image block, thereby achieving alignment of the pixels in the first sample depth image block with the pixels in the sample color image block.

[0170] In step 609 , the first sample depth image block and the sample color image block after channel alignment are input into a refinement network to obtain a sample depth residual output by the refinement network. The depth residual is used to represent the depth adjustment amount of the pixel points in the first sample depth image block.

[0171] In one possible implementation, a computer device inputs the channel-aligned first sample depth image block and the sample color image block into a refinement network, and performs calculations on each aligned pixel point through a convolutional layer in the refinement network, thereby obtaining a depth residual corresponding to each pixel point in the first sample depth image block output by the refinement network, and the depth residual represents the depth adjustment amount of the pixel point in the first sample depth image block.

[0172] Step 610 : Use the sample depth residual to perform edge thinning on the first sample depth image block to obtain a second sample depth image block.

[0173] In one possible implementation, the computer device uses the depth residual to refine the edges of the first sample depth image block, that is, based on the depth adjustment amount corresponding to each pixel point, the depth value of each pixel point in the first sample depth image block is adjusted to obtain the second sample depth image block.

[0174] Step 611 : Generate a reference depth image block corresponding to the second sample depth image block based on the first sample depth image block and the true depth image.

[0175] In one possible implementation, since the true depth image is directly used as supervision for the refinement network, the refinement network may not be able to accurately distinguish between edge areas and non-edge areas. Without the need for depth optimization in the non-edge areas, it is impossible to learn how to process the depth of the non-edge areas. Therefore, in order to enable the refinement network to perform supervised training on the depth of the edge areas without changing the depth of the non-edge areas, the computer device generates a reference depth image block corresponding to the second sample depth image block based on the first sample depth image block and the true depth image. By using the reference depth image block as supervision for the refinement network, the refinement network can be helped to accurately distinguish between edge areas and non-edge areas.

[0176] In a possible implementation, the process of the computer device generating the reference depth image block corresponding to the second sample depth image block may include the following sub-steps:

[0177] 1. Extract the true depth image block corresponding to the first sample depth image block from the true depth image.

[0178] In one possible embodiment, in order to improve the accuracy of pixel correspondence between the first sample depth image block and the true depth image, the computer device extracts the true depth image block corresponding to the first sample depth image block from the true depth image based on the position information of the first sample depth image block in the sample depth image.

[0179] 2. Generate an edge mask based on the first sample depth image block, where the edge mask is used to characterize edge areas and non-edge areas.

[0180] In one possible embodiment, in order to accurately match pixel features of the first sample depth image block with the true depth image block, the computer device performs region judgment and marking on each pixel in the first sample depth image block, and generates an edge mask based on the first sample depth image block, which characterizes the edge area and the non-edge area.

[0181] Optionally, the computer device generates an edge mask based on the first sample depth image block, marking pixel points in an edge area of the first sample depth image block as 1, and marking pixel points in a non-edge area as 0.

[0182] In one possible embodiment, in order to more clearly mark each pixel point at the boundary between the edge area and the non-edge area in the first sample depth image block, the computer device performs edge dilation processing on the depth edge based on the edge position information of the depth edge in the first sample depth image block, and then marks the pixel points within a certain distance around the depth edge to obtain an edge mask that marks the edge area and the non-edge area.

[0183] Indicative, such as Figure 7 As shown, the computer device performs edge dilation processing on the depth edge of the first sample depth image block 701 and performs pixel marking on pixel points within a certain distance around the depth edge to obtain an edge mask 703.

[0184] 3. Generate foreground and background masks based on the true depth image block. The foreground and background masks are used to represent the foreground and background areas.

[0185] In one possible implementation, based on the characteristic that the boundary between edge areas and non-edge areas in the true depth image is relatively clear, a computer device marks the foreground and background of the true depth image block, distinguishes the foreground and background in the true depth image block, and thereby generates foreground and background masks based on the true depth image block, which represent the foreground area and the background area.

[0186] Optionally, the computer device generates foreground and background masks based on the true depth image block, marking the pixels in the foreground area of the true depth image as 0 and the pixels in the background area as 1.

[0187] In one possible implementation, in order to more accurately distinguish the foreground area and the background area in the true depth image, the computer device determines the foreground area and the background area by binarizing the depth values of the pixel points in the true depth image block, and marks the foreground and background to obtain foreground and background masks.

[0188] Indicative, such as Figure 7 As shown, the computer device performs binarization processing on the depth values of the pixels in the true depth image block 702 to determine the foreground area and the background area, and performs foreground and background marking to obtain foreground and background masks 704 .

[0189] 4. Based on the edge mask and the foreground and background masks, the depth values of the pixels at the depth edge are updated to obtain the reference depth image block.

[0190] Furthermore, the computer device updates the depth values of the pixels located at the depth edge based on the edge mask corresponding to the first sample depth image block and the foreground and background masks corresponding to the true depth image block, and updates the depth values of the pixels belonging to the depth edge of the foreground area and the pixel points belonging to the depth edge of the background area in the first sample depth image block, respectively, thereby obtaining a reference depth image block.

[0191] In one possible embodiment, the computer device generates a foreground edge mask and a foreground non-edge mask based on the edge mask and the foreground and background masks, according to the pixel markings of the edge area pixels and the non-edge area pixels in the first sample depth image block, and the markings of the foreground area pixels in the true value depth image block; further, the computer device updates the depth values of the pixels in the foreground edge area represented by the foreground edge mask based on the foreground non-edge depth of the foreground non-edge area represented by the foreground non-edge mask, wherein the foreground non-edge depth belongs to the first sample depth image block.

[0192] In one possible implementation, in order to make the updated depth value of the foreground edge area consistent with the distribution of the depth values of the overall pixel points in the depth image, for the foreground edge pixels within the foreground edge area, the computer device samples foreground non-edge pixels adjacent to the foreground edge pixels from the foreground non-edge area, thereby updating the depth value of the foreground edge pixels based on the depth value of the foreground non-edge pixels and the distance between the foreground non-edge pixels and the foreground edge pixels.

[0193] In one possible implementation, the computer device uses a bilinear interpolation algorithm to sample the depth values of four foreground non-edge pixels from the nearest adjacent foreground non-edge area along the four directions of the foreground edge pixel, and then performs a weighted summation on the four depth values based on the distance between the foreground edge pixel and the foreground non-edge pixel to obtain the depth value corresponding to the foreground edge pixel, thereby updating the depth value of the foreground edge pixel.

[0194] Indicative, such as Figure 7 As shown, the computer device generates a foreground edge mask 705 and a foreground non-edge mask 706 based on the edge mask 703 and the foreground and background masks 704, according to the pixel markings of the edge area pixels and the non-edge area pixels in the first sample depth image block 701, and the markings of the foreground area pixels in the true value depth image block 702.

[0195] In one possible embodiment, the computer device generates a background edge mask and a background non-edge mask based on the edge mask and the foreground and background masks, according to the pixel markings of the edge area pixels and the non-edge area pixels in the first sample depth image block, and the markings of the background area pixels in the true depth image block; further, the computer device updates the depth values of the pixels in the background edge area represented by the background edge mask based on the background non-edge depth of the background non-edge area represented by the background non-edge mask, wherein the background non-edge depth belongs to the first sample depth image block.

[0196] In one possible embodiment, in order to make the updated depth value of the background edge area consistent with the distribution of the depth value of the overall pixel points in the depth image, for the background edge pixels in the background edge area, the computer device samples background non-edge pixels adjacent to the background edge pixels from the background non-edge area, thereby updating the depth value of the background edge pixels based on the depth value of the background non-edge pixels and the distance between the background non-edge pixels and the background edge pixels.

[0197] In one possible implementation, the computer device uses a bilinear interpolation algorithm to sample the depth values of four background non-edge pixels from the nearest adjacent background non-edge area along the four directions of the background edge pixel, and then performs a weighted summation on the four depth values based on the distance between the background edge pixel and the background non-edge pixel to obtain the depth value corresponding to the background edge pixel, thereby updating the depth value of the background edge pixel.

[0198] Indicative, such as Figure 7 As shown, the computer device generates a background edge mask 707 and a background non-edge mask 708 based on the edge mask 703 and the foreground and background masks 704, according to the pixel markings of the edge area pixels and the non-edge area pixels in the first sample depth image block 701, and the markings of the background area pixels in the true value depth image block 702.

[0199] Indicative, such as Figure 7 As shown, the computer device updates the depth values of the pixels in the foreground edge area represented by the foreground edge mask 705 based on the foreground non-edge depth of the foreground non-edge area represented by the foreground non-edge mask 706; and updates the depth values of the pixels in the background edge area represented by the background edge mask 707 based on the background non-edge depth of the background non-edge area represented by the background non-edge mask 708, thereby obtaining a reference depth image block 709.

[0200] Step 612 : Determine an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block.

[0201] The specific implementation of this step can refer to step 305, and this embodiment will not be repeated here.

[0202] Step 613: Train a refinement network based on edge refinement losses at different scaling ratios.

[0203] In a possible implementation, in order to improve the refinement accuracy of the refinement network for the depth edge of the first sample depth image block at different scaling ratios, the computer device performs training supervision on the refinement network based on the edge refinement loss at different scaling ratios.

[0204] In the above embodiment, by extracting the first sample depth image block from the sample depth image, the first sample depth image block is made to contain a foreground and a background as much as possible, thereby reducing the computational complexity of the refinement network in performing depth edge refinement on the first sample depth image block; by inputting the first sample depth image block and the sample color image block under at least one scaling ratio into the refinement network, the depth edge refinement accuracy of the refinement network for first sample depth image blocks of different resolutions is improved.

[0205] In addition, by training and supervising the second sample depth image block output by the refinement network based on the reference depth image block generated based on the first sample depth image block and the true depth image block, the refinement network's different depth processing methods for edge areas and non-edge areas are optimized, and the refinement network is able to supervise the depth of edge areas without changing the depth of non-edge areas.

[0206] Please refer to Figure 8 , which shows a schematic diagram of a refinement network training process provided by an exemplary embodiment of the present application. First, a computer device obtains a sample color image 802 and a sample depth image 801 corresponding to the sample color image 802, performs depth edge extraction on the sample depth image 801, and then extracts a first sample depth image block 803 along the depth edge. Based on the position information 805 of the first sample depth image block 803 in the sample depth image 801, a sample color image block 804 is extracted from the sample color image 802. Further, the first sample depth image block 803 and the sample color image block 804 are adjusted according to the scaling ratio. Based on at least one set of the first sample depth image blocks 803 and the sample color image blocks 804 after scaling, the first sample depth image block 803 is subjected to depth edge refinement by a refinement network 806 to obtain a second sample depth image block 807. Based on the second sample depth image block 807 and the reference depth image block 808 corresponding to the second sample depth image block 807, an edge refinement loss is calculated to supervise the training of the refinement network 806.

[0207] Please refer to Figure 9, which shows a structural block diagram of a depth image processing device provided by an exemplary embodiment of the present application, the device includes:

[0208] An image acquisition module 901 is configured to acquire a color image and a depth image corresponding to the color image;

[0209] An image block extraction module 902 is configured to extract a first depth image block from the depth image based on a depth edge in the depth image, where the first depth image block is distributed along the depth edge;

[0210] The image block extraction module 902 is further configured to extract a color image block corresponding to the first depth image block from the color image;

[0211] an edge refinement module 903, configured to perform edge refinement on the first depth image block based on the first depth image block and the color image block to obtain a second depth image block, wherein the precision of the depth edge in the second depth image block is higher than the precision of the depth edge in the first depth image block;

[0212] The image block replacement module 904 is configured to replace the first depth image block in the depth image with the second depth image block.

[0213] Optionally, the edge refinement module 903 includes:

[0214] an output unit, configured to input the first depth image block and the color image block into a refinement network to obtain a depth residual output by the refinement network, wherein the depth residual is used to represent a depth adjustment amount for a pixel point in the first depth image block;

[0215] An edge refinement unit is configured to perform edge refinement on the first depth image block using the depth residual to obtain the second depth image block.

[0216] Optionally, the output unit is used to:

[0217] Performing channel alignment on the depth channel of the first depth image block and the color channel of the color image block;

[0218] The first depth image block and the color image block after channel alignment are input into the refinement network to obtain the depth residual output by the refinement network.

[0219] Optionally, before performing edge thinning on the first depth image block based on the first depth image block and the color image block to obtain the second depth image block, the apparatus further includes:

[0220] a scaling processing module, configured to perform scaling processing on the first depth image block and the color image block based on a scaling ratio, wherein the scaled first depth image block and the color image block are used for edge refinement;

[0221] After performing edge thinning on the first depth image block based on the first depth image block and the color image block to obtain the second depth image block, the apparatus further includes:

[0222] A scaling and restoration module is configured to perform scaling and restoration processing on the second depth image block based on the scaling ratio, wherein the second depth image after the scaling and restoration processing is used to replace the first depth image.

[0223] Optionally, the device further includes:

[0224] A requirement determination module is used to determine the accuracy requirement of the current depth recognition scenario;

[0225] The ratio determination module is configured to determine the scaling ratio based on the accuracy requirement, where the scaling ratio is positively correlated with the accuracy requirement.

[0226] Optionally, the image block extraction module 902 is configured to:

[0227] extracting the depth edge from the depth image based on an edge extraction operator;

[0228] Creating a candidate box covering the depth edge by sliding the window;

[0229] Filtering a target box from the candidate boxes, where the target box covers the depth edge and the density of the target box is less than the density of the candidate boxes;

[0230] Extracting the first depth image block from the depth image based on the position of the target box;

[0231] The image block extraction module 902 is further configured to:

[0232] Based on the position of the target box, the color image block corresponding to the first depth image block is extracted from the color image.

[0233] To sum up, in the embodiments of the present application, based on the depth edge in the depth image, a first depth image block is extracted from the first depth image and a corresponding color image block is extracted from the color image, so that based on the color image block and the first depth image block, the edge of the first depth image block is refined to obtain a second depth image block with higher depth edge accuracy than the first depth image block, and then the first depth image block in the depth image is replaced by the second depth image block to achieve optimization of the depth edge in the depth image. By adopting the solution provided in the embodiments of the present application, the depth edge is accurately refined by extracting the depth image block, thereby improving the optimization efficiency of the depth edge in the depth image and reducing the amount of computation in the process of refining the depth edge.

[0234] Please refer to Figure 10 , which shows a structural block diagram of a depth image processing device provided by an exemplary embodiment of the present application, the device includes:

[0235] An image acquisition module 1001 is configured to acquire a sample color image and a sample depth image corresponding to the sample color image;

[0236] An image block extraction module 1002 is configured to extract a first sample depth image block from the sample depth image based on a depth edge in the sample depth image, where the first sample depth image block is distributed along the depth edge;

[0237] The image block extraction module 1002 is further configured to extract a sample color image block corresponding to the first sample depth image block from the sample color image;

[0238] an edge refinement module 1003, configured to perform edge refinement on the first sample depth image block through a refinement network based on the first sample depth image block and the sample color image block to obtain a second sample depth image block;

[0239] a loss determining module 1004, configured to determine an edge thinning loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block;

[0240] The network training module 1005 is configured to train the refinement network based on the edge refinement loss.

[0241] Optionally, before determining the edge refinement loss based on the second sample depth image block and the reference depth image block corresponding to the second sample depth image block, the apparatus includes:

[0242] An image block generation module is configured to generate the reference depth image block corresponding to the second sample depth image block based on the first sample depth image block and a true depth image.

[0243] Optionally, the image block generation module includes:

[0244] An image block extraction unit, configured to extract a true depth image block corresponding to the first sample depth image block from the true depth image;

[0245] a mask generating unit, configured to generate an edge mask based on the first sample depth image block, wherein the edge mask is used to characterize an edge area and a non-edge area;

[0246] The mask generating unit is further configured to generate foreground and background masks based on the true depth image block, wherein the foreground and background masks are used to represent the foreground area and the background area;

[0247] The depth value updating unit is configured to update the depth value of the pixel points located at the depth edge based on the edge mask and the foreground and background masks to obtain the reference depth image block.

[0248] Optionally, the depth value updating unit is configured to:

[0249] generating a foreground edge mask and a foreground non-edge mask based on the edge mask and the foreground and background masks; updating the depth values of pixels in the foreground edge area represented by the foreground edge mask based on the foreground non-edge depth of the foreground non-edge area represented by the foreground non-edge mask, wherein the foreground non-edge depth belongs to the first sample depth image block;

[0250] Based on the edge mask and the foreground and background masks, a background edge mask and a background non-edge mask are generated; based on the background non-edge depth of the background non-edge area represented by the background non-edge mask, the depth value of the pixel point in the background edge area represented by the background edge mask is updated, wherein the background non-edge depth belongs to the first sample depth image block.

[0251] Optionally, the depth value updating unit is configured to:

[0252] For a foreground edge pixel point in the foreground edge area, sampling foreground non-edge pixel points adjacent to the foreground edge pixel point from the foreground non-edge area;

[0253] updating the depth value of the foreground edge pixel based on the depth value of the foreground non-edge pixel and the distance between the foreground non-edge pixel and the foreground edge pixel;

[0254] The depth value updating unit is further configured to:

[0255] For background edge pixels in the background edge area, sampling background non-edge pixels adjacent to the background edge pixels from the background non-edge area;

[0256] The depth value of the background edge pixel is updated based on the depth value of the background non-edge pixel and the distance between the background non-edge pixel and the background edge pixel.

[0257] Optionally, the mask generating unit is configured to:

[0258] Based on the edge position information of the depth edge in the first sample depth image block, edge dilation processing is performed on the depth edge to obtain the edge mask.

[0259] The depth values of the pixels in the true depth image block are binarized to obtain the foreground and background masks.

[0260] Optionally, before performing edge thinning on the first sample depth image block through a thinning network based on the first sample depth image block and the sample color image block to obtain the second sample depth image block, the apparatus further includes:

[0261] a scaling processing module, configured to perform scaling processing on the first sample depth image block and the sample color image block based on at least one scaling ratio, wherein the scaled first sample depth image block and the sample color image block are input into the refinement network;

[0262] The network training module 1005 is used to:

[0263] The refinement network is trained based on the edge refinement loss at different scaling ratios.

[0264] Optionally, the edge refinement module 1003 is configured to:

[0265] Performing channel alignment on the depth channel of the first sample depth image block and the color channel of the sample color image block;

[0266] Inputting the first sample depth image block and the sample color image block after channel alignment into the refinement network to obtain a sample depth residual output by the refinement network, where the depth residual is used to represent the depth adjustment amount of the pixel point in the first sample depth image block;

[0267] The edge of the first sample depth image block is refined by using the sample depth residual to obtain the second sample depth image block.

[0268] Optionally, the image block extraction module 1002 is configured to:

[0269] extracting the depth edge from the sample depth image based on an edge extraction operator;

[0270] Creating a candidate box covering the depth edge by sliding the window;

[0271] Filtering a target box from the candidate boxes, where the target box covers the depth edge and the density of the target box is less than the density of the candidate boxes;

[0272] Extracting the first sample depth image block from the sample depth image based on the position of the target box;

[0273] The image block extraction module 1002 is further configured to:

[0274] Based on the position of the target box, the sample color image block corresponding to the first sample depth image block is extracted from the sample color image.

[0275] To summarize, in an embodiment of the present application, a first sample depth image block extracted from a sample depth image and a sample color image block extracted from a sample color image are input into a refinement network, and the first sample depth image block is edge-refined using the refinement network to obtain a second sample depth image block, thereby obtaining an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block, and the refinement network is trained with the edge refinement loss; by adopting the solution provided in an embodiment of the present application, the depth edge of the image is refined by extracting depth image blocks, thereby improving the efficiency of the refinement of the depth edge; by training the refinement network with the edge refinement loss, the accuracy of the depth edge refinement performed by the refinement network is improved.

[0276] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0277] Please refer to Figure 11, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device 1100 can be implemented as a terminal or server in the above-mentioned embodiments. Specifically, the computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory 1102 and a read-only memory 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a basic input / output system (I / O system) 1106 that helps transmit information between various components within the computer, and a large-capacity storage device 1107 for storing an operating system 1113, application programs 1114, and other program modules 1115.

[0278] The basic input / output system 1106 includes a display 1108 for displaying information and an input device 1109 such as a mouse and keyboard for user input. The display 1108 and the input device 1109 are both connected to the central processing unit 1101 via an input / output controller 1110 connected to the system bus 1105. The basic input / output system 1106 may also include an input / output controller 1110 for receiving and processing input from a variety of other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1110 also provides output to a display screen, printer, or other types of output devices.

[0279] The mass storage device 1107 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1107 and its associated computer-readable media provide non-volatile storage for the computer device 1100. In other words, the mass storage device 1107 may include a computer-readable medium (not shown) such as a hard disk or drive.

[0280] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape cassette, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones. The above-mentioned system memory 1104 and mass storage device 1107 can be collectively referred to as memory.

[0281] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1101. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1101 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0282] According to various embodiments of the present application, the computer device 1100 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1100 may be connected to a network 1112 via a network interface unit 1111 connected to the system bus 1105. Alternatively, the network interface unit 1111 may be used to connect to other types of networks or remote computer systems (not shown).

[0283] An embodiment of the present application further provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the depth image processing method provided in the above embodiment.

[0284] Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drives (SSDs) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0285] The present invention provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the depth image processing method described in the above embodiment.

[0286] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0287] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for processing a depth image, characterized in that: The method comprises: Acquire a color image and a depth image corresponding to the color image; Extracting a depth edge from the depth image based on an edge extraction operator; creating a candidate box covering the depth edge through a sliding window; screening a target box from the candidate box, where the target box covers the depth edge and the density of the target box is less than the density of the candidate box; extracting a first depth image block from the depth image based on a position of the target box, where the first depth image block is distributed along the depth edge; Extracting, from the color image, a color image block corresponding to the first depth image block based on the position of the target box; Based on the first depth image block and the color image block, perform edge thinning on the first depth image block to obtain a second depth image block, where precision of the depth edge in the second depth image block is higher than precision of the depth edge in the first depth image block; The first depth image block in the depth image is replaced with the second depth image block.

2. The method according to claim 1, characterized in that The step of performing edge thinning on the first depth image block based on the first depth image block and the color image block to obtain a second depth image block includes: Inputting the first depth image block and the color image block into a refinement network to obtain a depth residual output by the refinement network, where the depth residual is used to represent a depth adjustment amount for a pixel point in the first depth image block; The first depth image block is subjected to edge thinning using the depth residual to obtain the second depth image block.

3. The method according to claim 2, characterized in that Inputting the first depth image block and the color image block into a refinement network to obtain a depth residual output by the refinement network includes: Performing channel alignment on the depth channel of the first depth image block and the color channel of the color image block; The first depth image block and the color image block after channel alignment are input into the refinement network to obtain the depth residual output by the refinement network.

4. The method according to claim 1, wherein Before performing edge thinning on the first depth image block based on the first depth image block and the color image block to obtain the second depth image block, the method further includes: performing scaling processing on the first depth image block and the color image block based on a scaling ratio, wherein the scaled first depth image block and the color image block are used for edge refinement; After performing edge thinning on the first depth image block based on the first depth image block and the color image block to obtain the second depth image block, the method further includes: Based on the scaling ratio, scaling and restoring processing is performed on the second depth image block, wherein the second depth image after scaling and restoring processing is used to replace the first depth image.

5. The method according to claim 4, characterized in that The method further comprises: Determine the accuracy requirements of the current depth recognition scenario; The scaling ratio is determined based on the accuracy requirement, and the scaling ratio is positively correlated with the accuracy requirement.

6. A method for processing a depth image, characterized in that: The method comprises: Acquire a sample color image and a sample depth image corresponding to the sample color image; Extracting a depth edge from the sample depth image based on an edge extraction operator; creating a candidate box covering the depth edge through a sliding window; screening a target box from the candidate box, wherein the target box covers the depth edge and the density of the target box is less than the density of the candidate box; extracting a first sample depth image block from the sample depth image based on the position of the target box, wherein the first sample depth image block is distributed along the depth edge; Based on the position of the target box, extracting a sample color image block corresponding to the first sample depth image block from the sample color image; Based on the first sample depth image block and the sample color image block, performing edge thinning on the first sample depth image block through a thinning network to obtain a second sample depth image block; determining an edge refinement loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block; The refinement network is trained based on the edge refinement loss.

7. The method according to claim 6, characterized in that Before determining the edge refinement loss based on the second sample depth image block and the reference depth image block corresponding to the second sample depth image block, the method includes: The reference depth image block corresponding to the second sample depth image block is generated based on the first sample depth image block and the true depth image.

8. The method according to claim 7, characterized in that The generating, based on the first sample depth image block and the true depth image, the reference depth image block corresponding to the second sample depth image block includes: Extracting a true depth image block corresponding to the first sample depth image block from the true depth image; generating an edge mask based on the first sample depth image block, wherein the edge mask is used to characterize an edge area and a non-edge area; Generate foreground and background masks based on the true depth image block, wherein the foreground and background masks are used to represent the foreground area and the background area; Based on the edge mask and the foreground and background masks, the depth values of the pixels located at the depth edge are updated to obtain the reference depth image block.

9. The method according to claim 8, characterized in that The updating of the depth values of the pixels located at the depth edge based on the edge mask and the foreground and background masks to obtain the reference depth image block includes: generating a foreground edge mask and a foreground non-edge mask based on the edge mask and the foreground and background masks; updating the depth values of pixels in the foreground edge area represented by the foreground edge mask based on the foreground non-edge depth of the foreground non-edge area represented by the foreground non-edge mask, wherein the foreground non-edge depth belongs to the first sample depth image block; Based on the edge mask and the foreground and background masks, a background edge mask and a background non-edge mask are generated; based on the background non-edge depth of the background non-edge area represented by the background non-edge mask, the depth value of the pixel point in the background edge area represented by the background edge mask is updated, wherein the background non-edge depth belongs to the first sample depth image block.

10. The method according to claim 9, characterized in that The updating of the depth value of the pixel point in the foreground edge area represented by the foreground edge mask based on the foreground non-edge depth of the foreground non-edge area represented by the foreground edge mask includes: For a foreground edge pixel point in the foreground edge area, sampling foreground non-edge pixel points adjacent to the foreground edge pixel point from the foreground non-edge area; updating the depth value of the foreground edge pixel based on the depth value of the foreground non-edge pixel and the distance between the foreground non-edge pixel and the foreground edge pixel; The updating of the depth value of the pixel point in the background edge area represented by the background edge mask based on the background non-edge depth of the background non-edge area represented by the background edge mask comprises: For background edge pixels in the background edge area, sampling background non-edge pixels adjacent to the background edge pixels from the background non-edge area; The depth value of the background edge pixel is updated based on the depth value of the background non-edge pixel and the distance between the background non-edge pixel and the background edge pixel.

11. The method according to claim 8, characterized in that The generating an edge mask based on the first sample depth image block includes: performing edge dilation processing on the depth edge based on edge position information of the depth edge in the first sample depth image block to obtain the edge mask; Generating foreground and background masks based on the true depth image block includes: The depth values of the pixels in the true depth image block are binarized to obtain the foreground and background masks.

12. The method according to claim 6, characterized in that Before performing edge thinning on the first sample depth image block through a thinning network based on the first sample depth image block and the sample color image block to obtain the second sample depth image block, the method further includes: performing scaling processing on the first sample depth image block and the sample color image block based on at least one scaling ratio, wherein the scaled first sample depth image block and the sample color image block are input into the refinement network; The training of the refinement network based on the edge refinement loss comprises: The refinement network is trained based on the edge refinement loss at different scaling ratios.

13. The method according to claim 6, characterized in that The step of performing edge thinning on the first sample depth image block through a thinning network based on the first sample depth image block and the sample color image block to obtain a second sample depth image block includes: Performing channel alignment on the depth channel of the first sample depth image block and the color channel of the sample color image block; Inputting the first sample depth image block and the sample color image block after channel alignment into the refinement network to obtain a sample depth residual output by the refinement network, where the depth residual is used to represent the depth adjustment amount of the pixel point in the first sample depth image block; The edge of the first sample depth image block is refined by using the sample depth residual to obtain the second sample depth image block.

14. A depth image processing device, characterized in that: The device comprises: An image acquisition module, configured to acquire a color image and a depth image corresponding to the color image; An image block extraction module is configured to extract a depth edge from the depth image based on an edge extraction operator; create a candidate box covering the depth edge through a sliding window; filter out a target box from the candidate box, wherein the target box covers the depth edge and has a density less than that of the candidate box; and extract a first depth image block from the depth image based on a position of the target box, wherein the first depth image block is distributed along the depth edge; The image block extraction module is further configured to extract a color image block corresponding to the first depth image block from the color image based on the position of the target box; an edge refinement module, configured to perform edge refinement on the first depth image block based on the first depth image block and the color image block to obtain a second depth image block, wherein the precision of the depth edge in the second depth image block is higher than the precision of the depth edge in the first depth image block; An image block replacement module is configured to replace the first depth image block in the depth image with the second depth image block.

15. A depth image processing device, characterized in that: The device comprises: An image acquisition module, configured to acquire a sample color image and a sample depth image corresponding to the sample color image; An image block extraction module is configured to extract a depth edge from the sample depth image based on an edge extraction operator; create a candidate box covering the depth edge through a sliding window; filter out a target box from the candidate box, wherein the target box covers the depth edge and has a density less than that of the candidate box; and extract a first sample depth image block from the sample depth image based on a position of the target box, wherein the first sample depth image block is distributed along the depth edge; The image block extraction module is further configured to extract, from the sample color image, a sample color image block corresponding to the first sample depth image block based on the position of the target box; an edge refinement module, configured to perform edge refinement on the first sample depth image block through a refinement network based on the first sample depth image block and the sample color image block to obtain a second sample depth image block; a loss determining module, configured to determine an edge thinning loss based on the second sample depth image block and a reference depth image block corresponding to the second sample depth image block; A network training module is used to train the refinement network based on the edge refinement loss.

16. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the depth image processing method according to any one of claims 1 to 5, or to implement the depth image processing method according to any one of claims 6 to 13.

17. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the depth image processing method according to any one of claims 1 to 5, or to implement the depth image processing method according to any one of claims 6 to 13.

18. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the depth image processing method described in any one of claims 1 to 5, or implements the depth image processing method described in any one of claims 6 to 13.

Images