Depth image completion method and system
Through the depth image completion method, RGB image feature extraction and fusion generate fine-grained feature maps, and combined with network models to generate depth completion images, solving the problem of poor depth map completion effect in the existing technology, achieving more accurate depth Graph details capture and quality improvement reduce the difficulty and cost of data acquisition.
Patent Information
- Application Number
- CN202410965966.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
The existing depth map completion method cannot accurately capture the complex structure and details of the depth map when the data is sparse and irregularly distributed, and requires additional RGB image data and complex synchronization and alignment processes, which increases the computational complexity, and It is difficult to make full use of RGB image data for label sample acquisition.
By acquiring RGB images for feature extraction, a coarse-grained feature map is obtained, and a fine-grained feature map is generated through feature fusion. A deep-completion image is generated by combining the network model. High-level semantic features are automatically extracted using a convolutional neural network, and a hollow convolution kernel is used for Feature fusion establishes a mapping relationship between depth values and categories, and reduces the difficulty and cost of data acquisition.
It significantly improves the details and accuracy of the depth map, avoids the blur and distortion problems caused by sparse data, reduces the difficulty and cost of data acquisition, and improves the overall quality of the depth map.
Smart Images

Figure CN118941608A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a depth image completion method and system. Background Art
[0002] In the field of computer vision and 3D reconstruction, the acquisition and processing of depth maps is one of the key technologies. Depth maps contain the distance information from each pixel in the scene to the camera and are widely used in applications such as autonomous driving, augmented reality (AR), virtual reality (VR), robot navigation, and 3D modeling. Currently, the main means of acquiring depth maps include lidar, depth cameras, and stereo vision systems. However, due to the influence of hardware costs, technical limitations, and environmental factors, the depth maps collected by these devices usually have the following problems: the distribution of data points is relatively sparse and cannot fully cover the entire scene; the depth map information is incomplete and lacks depth data in some areas; the acquisition process is easily affected by noise, resulting in errors and noise points in the depth map, affecting the overall accuracy and reliability.
[0003] In existing technical solutions, interpolation methods are used to estimate missing depth values by interpolating between known depth data points; the joint multimodal data method combines depth data and RGB images; and the deep learning method uses the original sparse depth map as input to generate a completed depth map. However, these methods also have certain shortcomings. For cases where data is sparse and irregularly distributed, the interpolation method cannot accurately capture the complex structure and details of the depth map, which can easily lead to blurring and distortion; the joint multimodal data method requires additional RGB image data, and the synchronization and alignment process is complicated, which increases the complexity of calculation and processing; in the deep learning method, it is difficult to obtain labeled samples, and the easily accessible RGB image data is not fully utilized.
[0004] In order to overcome these defects, the present application proposes a depth image completion method and system. Summary of the invention
[0005] The purpose of this application is to provide a depth image completion method and system, aiming to solve the problem of poor effect of existing depth map completion methods.
[0006] To achieve the above objectives, this application provides the following technical solutions:
[0007] The present application provides a depth image completion method, comprising:
[0008] Acquire an RGB image, perform feature extraction on the RGB image, and obtain a coarse-grained feature map;
[0009] Performing feature fusion on the coarse-grained feature map to obtain a fine-grained feature map;
[0010] Based on the fine-grained feature map and the network model, a depth completion image is generated.
[0011] Furthermore, in the step of acquiring an RGB image, performing feature extraction on the RGB image, and obtaining a coarse-grained feature map, the following steps are specifically included:
[0012] Preprocessing the RGB image, including but not limited to: noise removal and normalization;
[0013] Use convolutional neural networks to automatically extract high-level semantic features of preprocessed RGB images; through convolution and pooling operations, obtain different levels of features in RGB images, including coarse-grained global features;
[0014] Generate a coarse-grained feature map based on the extracted coarse-grained global features;
[0015] The obtained coarse-grained feature map is post-processed.
[0016] Furthermore, in the step of performing feature fusion on the coarse-grained feature map to obtain a fine-grained feature map, the following steps are specifically included:
[0017] The dilated convolution kernel is used to perform convolution operation on the input coarse-grained feature map to obtain a fine-grained feature map for channel-level superposition.
[0018] Furthermore, in the step of generating a depth completion image based on the fine-grained feature map and the network model, the following steps are specifically included:
[0019] The continuous depth values predicted by the network model are converted into corresponding depth categories, and a mapping relationship between the continuous depth values and the depth categories is established; the specific mapping relationship is as follows:
[0020] Calculate the corresponding category through continuous depth:
[0021]
[0022] Calculate the corresponding depth value through the current category:
[0023] depth = α + (β - α) × ( / K) 2
[0024] Where class is the depth category, depth is the continuous depth value, K is the number of categories, α is the minimum value among the continuous depth values, and β is the maximum value among the continuous depth values.
[0025] Furthermore, in the network model, the loss function is:
[0026]
[0027]
[0028] Where ρ and σ are the corresponding penalty weights; class(i,j) is the true value of the category corresponding to the pixel (,j) in the image; y(i,j,k) pre is the value of the kth channel at the pixel (,j) in the predicted result; y(i,j,k) pre is the value of the kth channel at the (,j) pixel in the labeled image; K is the number of categories.
[0029] The present application provides a depth image completion system, comprising:
[0030] Feature extraction module: obtains an RGB image, performs feature extraction on the RGB image, and obtains a coarse-grained feature map;
[0031] Feature fusion module: performs feature fusion on the coarse-grained feature map to obtain a fine-grained feature map;
[0032] Depth completion module: Generates a depth completion image based on the fine-grained feature map and the network model.
[0033] The present application provides a device, which includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a depth image completion method; the processor is used to execute the program instructions stored in the memory to implement a depth image completion.
[0034] The present application provides a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute a depth image completion method.
[0035] The present application provides a depth image completion method and system, which has the following beneficial effects:
[0036] (1) Compared with traditional interpolation methods, this application can more accurately capture the complex structure and details of the depth map, avoiding the blur and distortion problems caused by sparse and irregular data distribution;
[0037] (2) The sample data set required for this application only includes RGB images, which does not require additional complex equipment or high costs, reducing the difficulty and cost of data acquisition;
[0038] (3) Through feature extraction and fusion of convolutional neural networks, the complex texture and details in the image are accurately captured, and this information is effectively integrated into the depth completion process, which significantly improves the details and accuracy of the generated depth map. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of a flow chart of a depth image completion method according to Embodiment 1 of the present application;
[0040] Figure 2 A flowchart of a depth image completion method according to Embodiment 1 of the present application;
[0041] Figure 3 A schematic diagram of a mapping relationship between depth values and depth categories in Example 1 of the present application;
[0042] Figure 4 This is a schematic diagram of the structure of a depth image completion system according to Embodiment 2 of the present application;
[0043] Figure 5 This is a schematic diagram of the device structure of Example 3 of the present application;
[0044] Figure 6 This is a schematic diagram of the storage medium structure of Example 4 of the present application. DETAILED DESCRIPTION
[0045] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0047] Example 1
[0048] See also Figure 1 , is a flowchart of a depth image completion method according to Embodiment 1 of the present application; the steps include:
[0049] S1: Acquire an RGB image, perform feature extraction on the RGB image, and obtain a coarse-grained feature map.
[0050] In this embodiment, the RGB image is preprocessed, including but not limited to: noise removal and normalization. The high-level semantic features of the preprocessed RGB image are automatically extracted using a convolutional neural network; different levels of features in the RGB image are obtained through convolution and pooling operations, including coarse-grained global features. A coarse-grained feature map is generated based on the extracted coarse-grained global features. The obtained coarse-grained feature map is post-processed, such as smoothing and denoising, to improve the quality of the feature map.
[0051] Among them, the convolutional neural network structure used is shown in the following table.
[0052]
[0053] S2: Perform feature fusion on the coarse-grained feature map to obtain a fine-grained feature map.
[0054] In this embodiment, 3x3 dilated convolution kernels with dilation rates of 3, 6, and 9 are used to perform convolution operations on the input feature map of a scale of 40x30x1024, and the obtained three feature images of 40x30x256 are superimposed at the channel level.
[0055] S3: Generate a depth completion image based on the fine-grained feature map and the network model.
[0056] In this embodiment, the continuous depth values predicted by the network model are converted into corresponding depth categories, and a mapping relationship between the continuous depth values and the depth categories is established; the specific mapping relationship is as follows:
[0057] Calculate the corresponding category through continuous depth:
[0058]
[0059] Calculate the corresponding depth value through the current category:
[0060] depth = α + (β - α) × ( / K) 2
[0061] Where class is the depth category, depth is the continuous depth value, K is the number of categories, α is the minimum value among the continuous depth values, and β is the maximum value among the continuous depth values.
[0062] See also Figure 3 , is a schematic diagram of the mapping relationship between depth values and depth categories in Example 1 of the present application. K = 6, α = 0, β = 25. The design of the relationship is mainly to solve the specificity of different depth values to the same depth deviation, that is, the interval with a larger depth value should be less sensitive to the same depth deviation than the small interval.
[0063] The loss function is:
[0064]
[0065]
[0066] Where ρ and σ are the corresponding penalty weights; class(i,j) is the true value of the category corresponding to the pixel (,j) in the image; y(i,j,k) pre is the value of the kth channel at the pixel (,j) in the predicted result; y(i,j,k) preis the value of the kth channel at the (,j) pixel in the labeled image; K is the number of categories.
[0067] To summarize, in Example 1 of the present application, the acquired RGB image data is subjected to feature extraction to generate a coarse-grained feature map, the coarse-grained feature map is subjected to feature fusion to generate a fine-grained feature map, and the fine-grained feature map is input into the network model to generate the final depth completion image; by providing rich texture and edge information, accurately locating object boundaries, providing scene context, and enhancing data density, the details, accuracy, and overall quality of the depth map are significantly improved.
[0068] Example 2
[0069] See also Figure 4 , is a structural diagram of a depth image completion system in Example 2 of the present application; the specific contents include:
[0070] Feature extraction module: obtains an RGB image, performs feature extraction on the RGB image, and obtains a coarse-grained feature map;
[0071] Feature fusion module: performs feature fusion on the coarse-grained feature map to obtain a fine-grained feature map;
[0072] Depth completion module: Generates a depth completion image based on the fine-grained feature map and the network model.
[0073] Example 3
[0074] See also Figure 5 , is a schematic diagram of the device structure of Embodiment 3 of the present application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .
[0075] The memory 52 stores program instructions for implementing the above-mentioned depth image completion method.
[0076] The processor 51 is used to execute program instructions stored in the memory 52 to implement a depth image completion.
[0077] The processor 51 may also be referred to as a CPU (Central Processing Unit).
[0078] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0079] Example 4
[0080] See also Figure 6 , which is a schematic diagram of the structure of the storage medium of Example 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, a server, a mobile phone, a tablet, and other devices.
[0081] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0082] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.
[0083] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.
[0084] Of course, the present invention may have many other implementations. Based on this implementation, other implementations obtained by ordinary technicians in this field without any creative work are all within the scope of protection of the present invention.
Claims
1. A depth image completion method, characterized in that: include: Acquire an RGB image, perform feature extraction on the RGB image, and obtain a coarse-grained feature map; Performing feature fusion on the coarse-grained feature map to obtain a fine-grained feature map; Based on the fine-grained feature map and the network model, a depth completion image is generated.
2. A depth image completion method according to claim 1, characterized in that: The step of acquiring an RGB image, performing feature extraction on the RGB image, and obtaining a coarse-grained feature map specifically includes the following steps: Preprocessing the RGB image, including but not limited to: noise removal and normalization; Use convolutional neural networks to automatically extract high-level semantic features of preprocessed RGB images; through convolution and pooling operations, obtain different levels of features in RGB images, including coarse-grained global features; Generate a coarse-grained feature map based on the extracted coarse-grained global features; The obtained coarse-grained feature map is post-processed.
3. A depth image completion method according to claim 1, characterized in that: The step of fusing the coarse-grained feature map to obtain a fine-grained feature map specifically includes the following steps: The dilated convolution kernel is used to perform convolution operation on the input coarse-grained feature map to obtain a fine-grained feature map for channel-level superposition.
4. The depth image completion method according to claim 1, characterized in that: The step of generating a depth completion image based on the fine-grained feature map and the network model specifically includes the following steps: The continuous depth values predicted by the network model are converted into corresponding depth categories, and a mapping relationship between the continuous depth values and the depth categories is established; the specific mapping relationship is as follows: Calculate the corresponding category through continuous depth: Calculate the corresponding depth value through the current category: depth=α+(β-α)×(class / K) 2 Where class is the depth category, depth is the continuous depth value, K is the number of categories, α is the minimum value among the continuous depth values, and β is the maximum value among the continuous depth values.
5. The depth image completion method according to claim 1, characterized in that: In the network model, the loss function is: Where ρ and σ are the corresponding penalty weights; class(i,j) is the true value of the category corresponding to the pixel (i,j) in the image; y(i,j,k) pre is the value of the kth channel at the (i,j) pixel in the predicted result; y(i,j,k) pre is the value of the kth channel at the (i, j) pixel in the labeled image; K is the number of categories.
6. A system for a depth image completion method according to claim 1, characterized in that: include: Feature extraction module: obtains an RGB image, performs feature extraction on the RGB image, and obtains a coarse-grained feature map; Feature fusion module: performs feature fusion on the coarse-grained feature map to obtain a fine-grained feature map; Depth completion module: Generates a depth completion image based on the fine-grained feature map and the network model.
7. A device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a depth image completion method as described in any one of claims 1-5; and the processor is used to execute the program instructions stored in the memory to implement a depth image completion.
8. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute a depth image completion method as described in any one of claims 1 to 5.
