Methods, apparatus and equipment for acquiring depth images
By processing RGB and depth images using a color depth information fusion model, the problem of poor depth image quality is solved, and high-quality depth image generation is achieved.
Patent Information
- Application Number
- CN202310937667.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-27
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-07-27
AI Technical Summary
In existing technologies, the quality of depth images is poor, especially affected by environmental interference, weak textures, repetitive textures, and non-Lambertian regions, resulting in poor depth estimation accuracy.
A color depth information fusion model is adopted, which processes the original RGB image and depth image through a feature pyramid network and a depth image generation network. Combined with image masking and fusion processing, a high-quality target depth image is generated.
It effectively avoids the influence of environmental interference and areas with weak or repetitive textures, thus improving the quality and accuracy of depth images.
Smart Images

Figure CN117173232B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of three-dimensional reconstruction, video frame interpolation, and autonomous driving, and particularly to a method, apparatus, and device for acquiring depth images. Background Technology
[0002] Currently, the technology of constructing a 3D model of a 3D scene based on depth images of the scene is called 3D reconstruction. The higher the quality of the depth image, the higher the accuracy of the 3D model.
[0003] In related technologies, depth sensors or depth color cameras (such as multi-view or binocular cameras) can be used to acquire depth images of 3D scenes. However, depth sensors are easily affected by environmental interference, resulting in poor quality depth images. Depth color cameras typically perform depth estimation on their acquired RGB images to obtain depth images. Because depth estimation by depth color cameras is easily affected by weak textures, repetitive textures, and non-Lambert surfaces in the RGB images, the accuracy of depth estimation is poor, leading to low-quality depth images. Summary of the Invention
[0004] This invention provides a method, apparatus, and device for acquiring depth images, which addresses the shortcomings of poor depth image quality in the prior art and aims to improve the quality of depth images.
[0005] In a first aspect, the present invention provides a method for acquiring a depth image, comprising:
[0006] Acquire the original RGB image and original depth image of the target 3D scene;
[0007] The original RGB image is input into the feature pyramid network in the color depth information fusion model to obtain multiple image features;
[0008] The multiple image features and the original depth image are input into the depth image generation network in the color depth information fusion model to obtain the target depth image of the target 3D scene.
[0009] According to a method for acquiring a depth image provided by the present invention, the step of inputting the plurality of image features and the original depth image into a depth image generation network in the color depth information fusion model to obtain a target depth image of a target three-dimensional scene includes:
[0010] The first image feature among the multiple image features is input into the first-level depth image generation module of the depth image generation network to obtain the first depth image;
[0011] The second image feature and the first depth image from the plurality of image features are input into the second-level depth image generation module of the depth image generation network to obtain the second depth image;
[0012] The original depth image and the second depth image are input into the depth fusion module of the depth image generation network to obtain a fused depth image;
[0013] The third image feature from the plurality of image features and the fused depth image are input into the third-level depth image generation module to obtain the target depth image of the target 3D scene.
[0014] According to a method for acquiring a depth image provided by the present invention, the original depth image and the second depth image are input into the depth fusion module of the depth image generation network to obtain a fused depth image, comprising:
[0015] The depth fusion module upsamples the second depth image by a preset factor to obtain an upsampled depth image; the size of the upsampled depth image is the same as the size of the original depth image.
[0016] The original depth image is masked using the depth fusion module to obtain a masked image.
[0017] The depth fusion module performs a fusion process on the upsampled depth image, the mask image, and the original depth image to obtain the fused depth image.
[0018] According to a method for acquiring a depth image provided by the present invention, the depth fusion module performs image masking on the original depth image based on a mask image generation formula to obtain a mask image;
[0019] The formula for generating the mask image is:
[0020]
[0021] Where Mask represents the mask image, mask (x,y) d represents the mask value at pixel (x,y) in the masked image. (x,y) D represents the depth value at pixel (x,y) in the original depth image. raw This represents the original depth image.
[0022] According to a method for acquiring a depth image provided by the present invention, the depth fusion module performs fusion processing on the upsampled depth image, the mask image and the original depth image based on a fusion processing formula to obtain the fused depth image;
[0023] The fusion processing formula is as follows:
[0024]
[0025] Among them, D S This represents the upsampled depth image, -Mask represents the inverse mask image, and D F -mask indicates fused depth images (x,y) Indicates mask (x,y) The inverse mask value.
[0026] According to a method for acquiring a depth image provided by the present invention, the method further includes:
[0027] Perform the following operations on each sample's 3D scene:
[0028] Acquire multiple sample RGB images and sample depth images corresponding to each sample RGB image by multiple depth color cameras capturing images of the sample 3D scene;
[0029] For each sample RGB image, the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image are determined based on the sample RGB image;
[0030] The sample depth image is aligned with the sample RGB image using the preset built-in function to obtain the sample target depth image corresponding to the sample RGB image;
[0031] Based on the target depth image corresponding to each sample RGB image, and the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB images, a three-dimensional mesh model of the sample three-dimensional scene is constructed.
[0032] For each sample RGB image, based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image, depth rendering is performed on the mesh model to obtain the label depth image corresponding to the sample RGB image;
[0033] Based on the multiple sample RGB images and the label depth images corresponding to each sample RGB image, multiple training samples corresponding to the sample 3D scene are constructed, and the multiple training samples are different;
[0034] The color depth information fusion model is obtained by supervising the initial information fusion model through multiple training samples corresponding to the three-dimensional scenes of each sample.
[0035] According to a method for acquiring depth images provided by the present invention, the step of constructing multiple training samples corresponding to the three-dimensional scene of the samples based on the plurality of sample RGB images and the label depth images corresponding to each sample RGB image includes:
[0036] For each sample's 3D scene, the following operations are performed:
[0037] Based on a preset number of sample RGB images of the sample 3D scene, multiple RGB image sets are determined, each RGB image set including the preset number of sample RGB images, and the RGB image sets are different.
[0038] For an RGB image set, based on a preset number of sample RGB images in the RGB image set and the label depth images corresponding to each of the preset number of sample RGB images, a plurality of training samples are determined. The training samples include a preset number of sample RGB images and the label depth image corresponding to one of the sample RGB images of the preset number of sample RGB images.
[0039] Secondly, the present invention also provides a depth image acquisition device, comprising:
[0040] The acquisition module is used to acquire the original RGB image and original depth image of the target 3D scene;
[0041] The processing module is used to input the original RGB image into the feature pyramid network in the color depth information fusion model to obtain multiple image features;
[0042] The processing module is further configured to input the multiple image features and the original depth image into the depth image generation network in the color depth information fusion model to obtain the target depth image of the target three-dimensional scene.
[0043] According to the depth image acquisition device provided by the present invention, the processing module is further specifically used for:
[0044] The first image feature among the multiple image features is input into the first-level depth image generation module of the depth image generation network to obtain the first depth image;
[0045] The second image feature and the first depth image from the plurality of image features are input into the second-level depth image generation module of the depth image generation network to obtain the second depth image;
[0046] The original depth image and the second depth image are input into the depth fusion module of the depth image generation network to obtain a fused depth image;
[0047] The third image feature from the plurality of image features and the fused depth image are input into the third-level depth image generation module to obtain the target depth image of the target 3D scene.
[0048] According to the depth image acquisition device provided by the present invention, the processing module is further specifically used for:
[0049] The depth fusion module upsamples the second depth image by a preset factor to obtain an upsampled depth image; the size of the upsampled depth image is the same as the size of the original depth image.
[0050] The original depth image is masked using the depth fusion module to obtain a masked image.
[0051] The depth fusion module performs a fusion process on the upsampled depth image, the mask image, and the original depth image to obtain the fused depth image.
[0052] According to the present invention, a depth image acquisition device is provided, wherein the depth fusion module performs image masking on the original depth image based on a mask image generation formula to obtain a mask image;
[0053] The formula for generating the mask image is:
[0054]
[0055] Wherein, Mask represents the mask image, Mask (x,y) d represents the mask value at pixel (x, y) in the mask image. (x,y) D represents the depth value at pixel (x,y) in the original depth image. raw This refers to the original depth image.
[0056] According to a depth image acquisition device provided by the present invention, the depth fusion module performs fusion processing on the upsampled depth image, the mask image and the original depth image based on a fusion processing formula to obtain the fused depth image;
[0057] The fusion processing formula is as follows:
[0058]
[0059] Among them, D S The image represents the upsampled depth image, -Mask represents the inverse mask image, and D F The fused depth image is represented by -mask. (x,y) Indicates mask (x,y) The inverse mask value.
[0060] According to the depth image acquisition device provided by the present invention, the following operations are performed for each sample three-dimensional scene:
[0061] The acquisition module is also used to acquire multiple sample RGB images and sample depth images corresponding to each sample RGB image obtained by multiple depth color cameras capturing images of the sample 3D scene;
[0062] The processing module is also used to determine, based on the sample RGB image, the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image for each sample RGB image;
[0063] The processing module is also used to align the sample depth image with the sample RGB image using the preset built-in function to obtain the sample target depth image corresponding to the sample RGB image;
[0064] The processing module is also used to construct a three-dimensional mesh model of the sample three-dimensional scene based on the sample target depth image corresponding to each sample RGB image, the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image;
[0065] The processing module is also used to perform depth rendering on the mesh model for each sample RGB image based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image, so as to obtain the label depth image corresponding to the sample RGB image;
[0066] The processing module is also used to construct multiple training samples corresponding to the three-dimensional scene of the sample based on the multiple sample RGB images and the label depth image corresponding to each sample RGB image, wherein the multiple training samples are different;
[0067] The processing module is also used to perform supervised training on the initial information fusion model using multiple training samples corresponding to the three-dimensional scenes of each sample, so as to obtain the color depth information fusion model.
[0068] According to the depth image acquisition device provided by the present invention, the processing module is further specifically used for:
[0069] For each sample's 3D scene, the following operations are performed:
[0070] Based on a preset number of sample RGB images of the sample 3D scene, multiple RGB image sets are determined, each RGB image set including the preset number of sample RGB images, and the RGB image sets are different.
[0071] For an RGB image set, based on a preset number of sample RGB images in the RGB image set and the label depth images corresponding to each of the preset number of sample RGB images, a plurality of training samples are determined. The training samples include a preset number of sample RGB images and the label depth image corresponding to one of the sample RGB images of the preset number of sample RGB images.
[0072] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the depth image acquisition methods described above.
[0073] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the depth image acquisition methods described above.
[0074] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the depth image acquisition methods described above.
[0075] This invention provides a method, apparatus, and device for acquiring depth images. The original RGB image is input into the feature pyramid network in the color depth information fusion model to obtain multiple image features. The multiple image features and the original depth image are then input into the depth image generation network in the color depth information fusion model to obtain the target depth image of the target 3D scene. This method can avoid environmental interference and the influence of weak textures, repetitive textures, and non-Lambertian regions in the RGB image, thereby improving the quality of the target depth image. Attached Figure Description
[0076] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0077] Figure 1 This is a flowchart illustrating the method for acquiring depth images provided by the present invention;
[0078] Figure 2 This is a schematic diagram of the color depth information fusion model provided by the present invention;
[0079] Figure 3 This is a schematic diagram of the process for obtaining a fused depth image provided by the present invention;
[0080] Figure 4This is a schematic diagram of the process for obtaining the color depth information fusion model provided by the present invention;
[0081] Figure 5 This is a flowchart of the process for constructing a 3D scene of a sample, provided by the present invention.
[0082] Figure 6 This is a schematic diagram of the structure of the depth image acquisition device provided by the present invention;
[0083] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation
[0084] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0085] In related technologies, depth sensors are susceptible to environmental interference, resulting in poor quality of the acquired depth images. Depth color cameras typically perform depth estimation on their acquired RGB images to obtain depth images. However, during depth estimation, depth color cameras are easily affected by areas with weak textures, repetitive textures, and non-Lambertian regions, leading to poor quality of the resulting depth images.
[0086] In order to improve the quality of depth images, the inventors provide a method for acquiring depth images. In this method, the inventors design a color depth information fusion model to process the original RGB image and the original depth image to obtain a depth image. This method can avoid environmental interference and the influence of weak textures, repetitive textures and non-Lambertian areas in the RGB image.
[0087] The method for acquiring depth images according to the present invention is described below with reference to specific embodiments.
[0088] Figure 1 This is a schematic flowchart of the depth image acquisition method provided by the present invention. Figure 1 As shown, the method includes:
[0089] S11. Obtain the original RGB image and original depth image of the target 3D scene.
[0090] Optionally, the execution subject of the depth image acquisition method provided by the present invention can be an electronic device or a depth image acquisition device installed in the electronic device. The acquisition device can be implemented by a combination of software and / or hardware.
[0091] Electronic devices, such as personal computers or desktop computers.
[0092] The target 3D scene can be any real scene that needs to be reconstructed in 3D.
[0093] Optionally, images of the target 3D scene can be acquired using an RGB-D depth color camera to obtain the original RGB image and original depth image of the target 3D scene.
[0094] S12. Input the original RGB image into the Feature Pyramid Network (FPN) in the color depth information fusion model to obtain multiple image features.
[0095] S13. Input multiple image features and the original depth image into the depth image generation network in the color depth information fusion model to obtain the target depth image of the target 3D scene.
[0096] In this invention, the original RGB image is input into the feature pyramid network of the color depth information fusion model to obtain multiple image features. These multiple image features, along with the original depth image, are then input into the depth image generation network of the color depth information fusion model to obtain the target depth image of the target 3D scene. This not only avoids environmental interference and the influence of weak textures, repetitive textures, and non-Lambertian surfaces in the RGB image, but also accurately estimates the depth of the target 3D scene, improving the quality of the target depth image. Furthermore, in this invention, the input information of the color depth information fusion model includes the original RGB image and the original depth image; therefore, it is possible to obtain the target depth image based on the joint processing of the original RGB image and the original depth image, thereby improving the quality of the target depth image.
[0097] Based on the above embodiments, the following is combined with Figure 2 The color depth information fusion model is explained.
[0098] Figure 2 This is a schematic diagram of the color depth information fusion model provided by the present invention. Figure 2 As shown, the color depth information fusion model includes a feature pyramid network and a depth image generation network. The depth image generation network includes a first-level depth image generation module 21, a second-level depth image generation module 22, a depth fusion module 23, and a third-level depth image generation module 24.
[0099] The process of obtaining the target depth image through the color depth information fusion model will be explained below in conjunction with S21 and S25.
[0100] S21. Input the original RGB image into the feature pyramid network to obtain multiple image features.
[0101] For example, a feature pyramid network includes N levels, resulting in N image features.
[0102] It should be noted that, Figure 2 This explanation uses an example where N equals 3. When N=3, multiple image features include... Figure 2 The first image feature f1, the second image feature f2, and the third image feature f3 are shown.
[0103] S22. Input the first image feature f1 into the first-level depth image generation module 21 to obtain the first depth image.
[0104] Specifically, in the first-level depth image generation module 21, a differentiable homography warping operation is performed on the first image feature f1 to obtain a feature volume. The variance of the feature volume is calculated to obtain a cost volume. The cost volume is 3D convolved to obtain a depth probability map. The depth probability map is then subjected to regression processing to obtain the first depth image.
[0105] S23. Input the second image feature f2 and the first depth image into the second-level depth image generation module 22 to obtain the second depth image.
[0106] Specifically, in the second-level depth image generation module 22, a differentiable homography warping operation is performed on the second image feature f2 and the first depth image to obtain a feature volume. The variance of the feature volume is calculated to obtain a cost volume. The cost volume is 3D convolved to obtain a depth probability map. The depth probability map is then subjected to regression processing to obtain the second depth image.
[0107] S24. Input the original depth image and the second depth image into the depth fusion module of the depth image generation network to obtain the fused depth image D. F .
[0108] S25. Combine the third image feature f3 with the fused depth image D. F The image is input into the third-level depth image generation module 24 to obtain the target depth image of the target 3D scene.
[0109] Specifically, in the third-level depth image generation module 24, the third image feature f3 and the fused depth image D are processed. FPerform a differentiable homography warping operation to obtain a feature volume. Calculate the variance of the feature volume to obtain a cost volume. Perform 3D convolution on the cost volume to obtain a depth probability map. Perform regression processing on the depth probability map to obtain the target depth image.
[0110] Based on the above embodiments, the following is combined with Figure 3 This section explains the crimes committed by obtaining fused depth images.
[0111] Figure 3 This is a schematic diagram of the process for obtaining a fused depth image provided by the present invention. For example... Figure 3 As shown, the method includes:
[0112] S31. The second depth image is upsampled by a preset multiple through the depth fusion module to obtain an upsampled depth image. The size of the upsampled depth image is the same as the size of the original depth image.
[0113] For example, in Figure 2 Based on this, the preset multiplier is 2.
[0114] It should be noted that if the second depth image is replaced with the first depth image, the preset multiplier is 4.
[0115] S32. The original depth image is masked using the depth fusion module to obtain the masked image.
[0116] In some embodiments, the depth fusion module performs image masking on the original depth image based on the mask image generation formula to obtain a mask image.
[0117] The formula for generating a mask image is:
[0118]
[0119] Where Mask represents the mask image, mask (x,y) d represents the mask value at pixel (x,y) in the masked image. (x,y) D represents the depth value at pixel (x,y) in the original depth image. raw This represents the original depth image.
[0120] S33. The upsampled depth image, mask image and original depth image are fused using the depth fusion module to obtain the fused depth image.
[0121] In some embodiments, the depth fusion module performs fusion processing on the upsampled depth image, the mask image, and the original depth image based on a fusion processing formula to obtain a fused depth image.
[0122] The fusion processing formula is:
[0123]
[0124] Among them, D S This represents the upsampled depth image, -Mask represents the inverse mask image, and D F -mask indicates fused depth images (x,y) Indicates mask (x,y) The inverse mask value, where · represents the product of the depth value and the mask value at pixel (x,y).
[0125] For example, D raw Mask represents D raw The depth value at pixel (x,y) is multiplied by the mask value at pixel (x,y) in the Mask.
[0126] For example, D S ·(-Mask) represents D S The depth value at pixel (x,y) is multiplied by the inverse mask value at pixel (x,y) in (-Mask).
[0127] In this invention, since the original depth image contains invalid measurement regions such as holes, these invalid measurement regions cannot be used to construct the cost volume in S25 using a non-uniform sampling method. Therefore, the original depth image is supplemented by a second depth image through a depth fusion module to obtain a fused depth image D without invalid measurement regions. F This ensures that the cost body in S25 can be successfully constructed.
[0128] This invention also provides a method for supervised training of an initial information fusion model to obtain the aforementioned color depth information fusion model. The following is a combination of... Figure 4 The process of obtaining the color depth information fusion model is explained.
[0129] Figure 4 This is a schematic diagram of the process for obtaining a color depth information fusion model provided by the present invention. For example... Figure 4 As shown, the method includes:
[0130] S41. Acquire multiple sample RGB images and sample depth images corresponding to each sample RGB image obtained by multiple depth color cameras capturing images of the sample 3D scene.
[0131] Optionally, the total number of multiple depth color cameras can be X, where X is an integer greater than or equal to 2. For example, X can be 5.
[0132] The depth color camera can be an RGB-D depth color camera or other types of depth color cameras.
[0133] Optionally, multiple depth color cameras can be connected to a high-performance acquisition server, which controls the multiple depth color cameras to acquire images of the sample 3D scene at the same acquisition time.
[0134] Optionally, in order for multiple depth color cameras to obtain a larger common viewing area, a convergent acquisition method is used to acquire images of the sample 3D scene.
[0135] The total number of multiple depth color cameras and the total number of multiple sample RGB images can be the same.
[0136] The sample 3D scene is one of multiple sample 3D scenes.
[0137] The total number of sample 3D scenes can be Y, where Y is an integer greater than or equal to 2. For example, Y could be 56.
[0138] For each sample 3D scene in multiple sample 3D scenes, execute S41 to S46.
[0139] S42. For each sample RGB image, determine the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image based on the sample RGB image.
[0140] Specifically, the intrinsic parameters of the depth color camera are obtained by processing the sample RGB images using the Zhang Zhengyou calibration method.
[0141] Specifically, the Structure from Motion (SFM) process the sample RGB images to obtain the extrinsic parameters of the depth color camera.
[0142] S43. Align the sample depth image with the sample RGB image using a preset built-in function to obtain the sample target depth image corresponding to the sample RGB image.
[0143] Alternatively, the preset built-in functions can be built-in functions provided by a depth color camera (such as RealSense).
[0144] S44. Based on the target depth image corresponding to each sample RGB image, and the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB images, construct a three-dimensional mesh model of the sample three-dimensional scene.
[0145] The following is combined Figure 5 The construction of the 3D mesh model for the sample 3D scene is explained.
[0146] Figure 5 This is a flowchart illustrating the process of constructing a 3D scene for a sample, as provided by this invention. Figure 5As shown, it includes: multiple depth color cameras. For example, multiple depth color cameras include depth color cameras A, B, C, D, and E. Depth color cameras A, B, C, D, and E acquire a sample RGB image and a sample depth image corresponding to the sample 3D scene from their respective viewpoints.
[0147] Specifically, based on the intrinsic and extrinsic parameters of the depth color camera corresponding to each sample RGB image, the sample target depth images corresponding to each sample RGB image are fused into the volume data of the truncated signed distance function (TSDF); the volume data is extracted using the Marching Cube algorithm to obtain a special surface; and a three-dimensional mesh model of the sample three-dimensional scene is generated based on the special surface.
[0148] S45. For each sample RGB image, based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image, perform depth rendering on the mesh model to obtain the label depth image corresponding to the sample RGB image.
[0149] S46. Based on multiple sample RGB images and the label depth images corresponding to each sample RGB image, construct multiple training samples corresponding to the sample 3D scene. The multiple training samples are different.
[0150] In some embodiments, S46 specifically includes:
[0151] Perform the following operations on each sample's 3D scene:
[0152] Based on a preset number of sample RGB images of a sample 3D scene, multiple RGB image sets are determined. Each RGB image set includes a preset number of sample RGB images, and the RGB image sets are different.
[0153] For an RGB image set, multiple training samples are determined based on a preset number of sample RGB images and the label depth images corresponding to each of the preset number of sample RGB images. The training samples include a preset number of sample RGB images and the label depth image corresponding to one sample RGB image of the preset number of sample RGB images.
[0154] Optionally, the preset quantity can be Z, where Z can be an integer greater than or equal to 1. For example, Z can be 3.
[0155] For example, for a sample 3D scene, with Z = 3 and the total number of multiple sample RGB images of the sample 3D scene being 5 (i.e., X = 5, the total number of multiple depth color cameras and the total number of multiple sample RGB images can be the same), we can obtain 10 (i.e., A collection of RGB images.
[0156] For example, in the case of multiple sample RGB images including P1, P2, P3, P4 and P5, an RGB image set may include P1, P2 and P3, an RGB image set may include P2, P3 and P4, an RGB image set may include P1, P3 and P4, and so on.
[0157] Specifically, a training sample can be determined by a preset number of sample RGB images in the RGB image set and the label depth image corresponding to one sample RGB image in the preset number of sample RGB images.
[0158] For a set of RGB images, if the set of RGB images includes P1, P2 and P3, three training samples can be obtained. One training sample includes P1, P2, P3 and the label depth image corresponding to P1; another training sample includes P1, P2, P3 and the label depth image corresponding to P2; and yet another training sample includes P1, P2, P3 and the label depth image corresponding to P3.
[0159] S47. Supervised training is performed on the initial information fusion model using multiple training samples corresponding to the 3D scene of each sample to obtain the color depth information fusion model.
[0160] The initial information fusion model has the same structure as the color depth information fusion model, but the model parameters are different.
[0161] For the (i+1)th training sample, input a predetermined number of RGB images from the (i+1)th training sample into the information fusion model obtained from the i-th training to obtain the predicted depth image;
[0162] Calculate the loss value between the predicted depth image and the labeled depth image in the (i+1)th training sample. Adjust the model parameters of the information fusion model obtained in the i-th training based on the loss value to obtain the information fusion model obtained in the (i+1)th training.
[0163] In this invention, for each sample RGB image, depth rendering is performed on the mesh model based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB images, resulting in a label depth image corresponding to the sample RGB image, thus ensuring high quality of the label depth image. Furthermore, based on multiple sample RGB images and the label depth images corresponding to each sample RGB image, multiple training samples corresponding to the sample 3D scene are constructed. The initial information fusion model is then subjected to supervised training using these multiple training samples to obtain a color depth information fusion model, thereby improving the quality of the obtained color depth information fusion model.
[0164] The depth image acquisition apparatus provided by the present invention will be described below. The depth image acquisition apparatus described below can be referred to in correspondence with the depth image acquisition method described above.
[0165] Figure 6 This is a schematic diagram of the structure of the depth image acquisition device provided by the present invention. Figure 6 As shown, the depth image acquisition device includes:
[0166] The acquisition module 61 is used to acquire the original RGB image and the original depth image of the target 3D scene;
[0167] Processing module 62 is used to input the original RGB image into the feature pyramid network in the color depth information fusion model to obtain multiple image features;
[0168] The processing module 62 is further configured to input the multiple image features and the original depth image into the depth image generation network in the color depth information fusion model to obtain the target depth image of the target three-dimensional scene.
[0169] According to the depth image acquisition apparatus provided by the present invention, the processing module 62 is further specifically used for:
[0170] The first image feature among the multiple image features is input into the first-level depth image generation module of the depth image generation network to obtain the first depth image;
[0171] The second image feature and the first depth image from the plurality of image features are input into the second-level depth image generation module of the depth image generation network to obtain the second depth image;
[0172] The original depth image and the second depth image are input into the depth fusion module of the depth image generation network to obtain a fused depth image;
[0173] The third image feature from the plurality of image features and the fused depth image are input into the third-level depth image generation module to obtain the target depth image of the target 3D scene.
[0174] According to the depth image acquisition apparatus provided by the present invention, the processing module 62 is further specifically used for:
[0175] The depth fusion module upsamples the second depth image by a preset factor to obtain an upsampled depth image; the size of the upsampled depth image is the same as the size of the original depth image.
[0176] The original depth image is masked using the depth fusion module to obtain a masked image.
[0177] The depth fusion module performs a fusion process on the upsampled depth image, the mask image, and the original depth image to obtain the fused depth image.
[0178] According to the present invention, a depth image acquisition device is provided, wherein the depth fusion module performs image masking on the original depth image based on a mask image generation formula to obtain a mask image;
[0179] The formula for generating the mask image is:
[0180]
[0181] Wherein, Mask represents the mask image, mask (x,y) d represents the mask value at pixel (x, y) in the mask image. (x,y) D represents the depth value at pixel (x, y) in the original depth image. raw This refers to the original depth image.
[0182] According to the depth image acquisition device provided by the present invention, the depth fusion module performs fusion processing on the upsampled depth image, the mask image and the original depth image based on the fusion processing formula to obtain the fused image;
[0183] The fusion processing formula is as follows:
[0184]
[0185] Among them, D S This represents the upsampled depth image, -Mask represents the inverse mask image, and D F -mask indicates fused depth images (x,y) Indicates mask (x,y) The inverse mask value.
[0186] According to the depth image acquisition device provided by the present invention, the following operations are performed for each sample three-dimensional scene:
[0187] The acquisition module 61 is also used to acquire multiple sample RGB images and sample depth images corresponding to each sample RGB image obtained by multiple depth color cameras capturing images of the sample three-dimensional scene;
[0188] The processing module 62 is also used to determine, based on the sample RGB image, the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image for each sample RGB image;
[0189] The processing module 62 is further configured to align the sample depth image with the sample RGB image using the preset built-in function to obtain the sample target depth image corresponding to the sample RGB image;
[0190] The processing module 62 is also used to construct a three-dimensional mesh model of the sample three-dimensional scene based on the sample target depth image corresponding to each sample RGB image and the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB images;
[0191] The processing module 62 is also used to perform depth rendering on the mesh model for each sample RGB image based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image, so as to obtain the label depth image corresponding to the sample RGB image;
[0192] Processing module 62 is further configured to construct multiple training samples corresponding to the three-dimensional scene of the sample based on the multiple sample RGB images and the label depth image corresponding to each sample RGB image, wherein the multiple training samples are different;
[0193] The processing module 62 is also used to perform supervised training on the initial information fusion model using multiple training samples corresponding to the three-dimensional scenes of each sample, so as to obtain the color depth information fusion model.
[0194] According to the depth image acquisition apparatus provided by the present invention, the processing module 62 is further specifically used for:
[0195] For each sample's 3D scene, the following operations are performed:
[0196] Based on a preset number of sample RGB images of the sample 3D scene, multiple RGB image sets are determined, each RGB image set including the preset number of sample RGB images, and the RGB image sets are different.
[0197] For an RGB image set, based on a preset number of sample RGB images in the RGB image set and the label depth images corresponding to each of the preset number of sample RGB images, a plurality of training samples are determined. The training samples include a preset number of sample RGB images and the label depth image corresponding to one of the sample RGB images of the preset number of sample RGB images.
[0198] Figure 7 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. For example... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a depth image acquisition method. This method includes: acquiring an original RGB image and an original depth image of the target 3D scene; inputting the original RGB image into a feature pyramid network in a color-depth information fusion model to obtain multiple image features; and inputting the multiple image features and the original depth image into a depth image generation network in the color-depth information fusion model to obtain a target depth image of the target 3D scene.
[0199] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0200] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the depth image acquisition method provided by the above methods. The method includes: acquiring an original RGB image and an original depth image of a target three-dimensional scene; inputting the original RGB image into a feature pyramid network in a color depth information fusion model to obtain multiple image features; and inputting the multiple image features and the original depth image into a depth image generation network in a color depth information fusion model to obtain a target depth image of the target three-dimensional scene.
[0201] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for acquiring a depth image provided by the above methods. The method includes: acquiring an original RGB image and an original depth image of a target three-dimensional scene; inputting the original RGB image into a feature pyramid network in a color depth information fusion model to obtain multiple image features; and inputting the multiple image features and the original depth image into a depth image generation network in the color depth information fusion model to obtain a target depth image of the target three-dimensional scene.
[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for acquiring a depth image, characterized in that, include: Acquire the original RGB image and original depth image of the target 3D scene; The original RGB image is input into the feature pyramid network in the color depth information fusion model to obtain multiple image features; The process of inputting the multiple image features and the original depth image into the depth image generation network of the color depth information fusion model to obtain the target depth image of the target 3D scene includes: inputting a first image feature from the multiple image features into the first-level depth image generation module of the depth image generation network to obtain a first depth image; inputting a second image feature from the multiple image features and the first depth image into the second-level depth image generation module of the depth image generation network to obtain a second depth image; inputting the original depth image and the second depth image into the depth fusion module of the depth image generation network to obtain a fused depth image; and inputting a third image feature from the multiple image features into the depth image generation network to obtain a fused depth image. The image features and the fused depth image are input to the third-level depth image generation module to obtain the target depth image of the target 3D scene; the original depth image and the second depth image are input to the depth fusion module of the depth image generation network to obtain a fused depth image, including: upsampling the second depth image by a preset multiple through the depth fusion module to obtain an upsampled depth image, the size of the upsampled depth image being the same as the size of the original depth image; performing image masking on the original depth image through the depth fusion module to obtain a mask image; fusing the upsampled depth image, the mask image, and the original depth image through the depth fusion module to obtain the fused depth image; the depth fusion module performs image masking on the original depth image based on a mask image generation formula to obtain a mask image; the mask image generation formula is: ; in, This refers to the mask image. Represents the pixels in the mask image The mask value at that location, Represents the pixels in the original depth image Depth value at that location, The original depth image is represented; the depth fusion module performs fusion processing on the upsampled depth image, the mask image, and the original depth image based on a fusion processing formula to obtain the fused depth image; the fusion processing formula is: ; in, This refers to the upsampled depth image. express The inverse mask image, This represents the fused depth image. express The inverse mask value.
2. The method for acquiring a depth image according to claim 1, characterized in that, The method further includes: Perform the following operations on each sample's 3D scene: Acquire multiple sample RGB images and sample depth images corresponding to each sample RGB image by multiple depth color cameras capturing images of the sample 3D scene; For each sample RGB image, the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image are determined based on the sample RGB image; By using a preset built-in function, the sample depth image is aligned with the sample RGB image to obtain the sample target depth image corresponding to the sample RGB image; Based on the target depth image corresponding to each sample RGB image, and the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB images, a three-dimensional mesh model of the sample three-dimensional scene is constructed. For each sample RGB image, based on the intrinsic and extrinsic parameters of the depth color camera that acquired the sample RGB image, depth rendering is performed on the mesh model to obtain the label depth image corresponding to the sample RGB image; Based on the multiple sample RGB images and the label depth images corresponding to each sample RGB image, multiple training samples corresponding to the sample 3D scene are constructed, and the multiple training samples are different; The color depth information fusion model is obtained by supervising the initial information fusion model through multiple training samples corresponding to the three-dimensional scenes of each sample.
3. The method for acquiring a depth image according to claim 2, characterized in that, The construction of multiple training samples corresponding to the 3D scene of the samples based on the multiple sample RGB images and the label depth images corresponding to each sample RGB image includes: For each sample's 3D scene, the following operations are performed: Based on a preset number of sample RGB images of the sample 3D scene, multiple RGB image sets are determined, each RGB image set including the preset number of sample RGB images, and the RGB image sets are different. For an RGB image set, based on a preset number of sample RGB images in the RGB image set and the label depth images corresponding to each of the preset number of sample RGB images, a plurality of training samples are determined. The training samples include a preset number of sample RGB images and the label depth image corresponding to one of the sample RGB images of the preset number of sample RGB images.
4. A depth image acquisition device, characterized in that, include: The acquisition module is used to acquire the original RGB image and original depth image of the target 3D scene; The processing module is used to input the original RGB image into the feature pyramid network in the color depth information fusion model to obtain multiple image features; The processing module is further configured to input the plurality of image features and the original depth image into the depth image generation network in the color depth information fusion model to obtain a target depth image of the target 3D scene; the step of inputting the plurality of image features and the original depth image into the depth image generation network in the color depth information fusion model to obtain a target depth image of the target 3D scene includes: inputting a first image feature from the plurality of image features into a first-level depth image generation module of the depth image generation network to obtain a first depth image; inputting a second image feature from the plurality of image features and the first depth image into a second-level depth image generation module of the depth image generation network to obtain a second depth image; inputting the original depth image and the second depth image into a depth fusion module of the depth image generation network to obtain a fused depth image; and inputting the plurality of image features into a depth image generation network to obtain a target depth image. The third image feature and the fused depth image are input to the third-level depth image generation module to obtain the target depth image of the target 3D scene; the original depth image and the second depth image are input to the depth fusion module of the depth image generation network to obtain the fused depth image, including: upsampling the second depth image by a preset multiple through the depth fusion module to obtain an upsampled depth image, the size of the upsampled depth image being the same as the size of the original depth image; performing image masking on the original depth image through the depth fusion module to obtain a mask image; fusing the upsampled depth image, the mask image, and the original depth image through the depth fusion module to obtain the fused depth image; the depth fusion module performs image masking on the original depth image based on the mask image generation formula to obtain a mask image; the mask image generation formula is: ; in, This refers to the mask image. Represents the pixels in the mask image The mask value at that location, Represents the pixels in the original depth image Depth value at that location, The original depth image is represented; the depth fusion module performs fusion processing on the upsampled depth image, the mask image, and the original depth image based on a fusion processing formula to obtain the fused depth image; the fusion processing formula is: ; in, This refers to the upsampled depth image. express The inverse mask image, This represents the fused depth image. express The inverse mask value.
5. An electronic device, characterized in that, include: Memory, processor, and computer program stored in said memory and executable on said processor; When the processor executes the program, it implements the method for acquiring a depth image as described in any one of claims 1 to 3.
6. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for acquiring a depth image as described in any one of claims 1 to 3.
Citation Information
Patent Citations
Image processing method and device
CN114511778A
