A training method, a deep completion method, and related devices for a multi-scale deep completion network.

Abstract

This application relates to a training method for a depth completion network. The method includes: acquiring at least one frame of infrared image samples with the same target scene, sparse depth image samples, and at least four standard dense depth image samples at different scales as a training set; inputting the training set into an initial depth completion network including an encoder and a decoder to obtain at least four simulated dense depth images at different scales; and iteratively training the initial depth completion network using the at least four standard dense depth image samples at different scales, the at least four simulated dense depth images at different scales, and a preset loss function to obtain a target depth completion network. The training method of this application not only simplifies the implementation process but also significantly improves the generalization ability and computational efficiency of the depth completion network.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method, a depth completion method, and related apparatus for a multi-scale depth completion network. Background Technology

[0002] Depth completion is a key technology in the field of computer vision. Its main goal is to use deep learning algorithms to transform sparse depth maps captured by sensors into more detailed and dense depth maps. This technology is of great significance for many application areas such as autonomous driving, robot navigation, augmented reality, and 3D reconstruction.

[0003] Despite the progress made in deep completion techniques, existing methods still have some limitations. On the one hand, most current deep completion methods rely on fully convolutional networks and single-scale data processing, which often makes it difficult to fully capture and understand the multi-dimensional features in the training data. This limits their efficiency and effectiveness in handling deep completion problems. On the other hand, existing deep completion techniques often rely on RGB images as guiding information to improve the completion effect. However, the information provided by RGB images is limited by lighting conditions. In nighttime or indoor environments without light sources, the effectiveness of RGB images decreases significantly, making it difficult to provide useful feature information. Summary of the Invention

[0004] In view of this, embodiments of this application provide a training method, a deep completion method, and related apparatus for a multi-scale deep completion network, which can solve at least one technical problem in the related art.

[0005] In a first aspect, one embodiment of this application provides a training method for a multi-scale depth completion network, comprising: acquiring a training set including n image sample pairs, each sample pair including an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer; inputting the training set into an initial depth completion network including an encoder and a decoder to obtain at least four simulated dense depth images of different scales; wherein, the encoder is used to downsample and feature-stitch the infrared image sample with the same target scene and the sparse depth image sample to obtain at least four depth infrared feature maps of different scales, and the decoder is used to upsample the depth infrared feature maps of at least four different scales to obtain at least four simulated dense depth images of different scales; and iteratively training the initial depth completion network using at least four standard dense depth image samples of different scales, at least four simulated dense depth images of different scales, and a preset loss function to obtain a target depth completion network.

[0006] Secondly, one embodiment of this application provides a depth completion method, comprising: acquiring a sparse depth image and an infrared image; inputting the sparse depth image and the infrared image into a preset depth completion network, wherein the preset depth completion network is a target depth completion network obtained by the training method of the multi-scale depth completion network as described in the first aspect; downsampling the sparse depth image and the infrared image through the preset depth completion network to obtain at least four different scales of depth features and infrared features; stitching together depth features and infrared features with the same scale among the at least four different scales of depth features and infrared features to obtain four different scales of stitched features; and upsampling the four different scales of stitched features to obtain at least one dense depth image with the same scale as the original sparse depth image.

[0007] Thirdly, one embodiment of this application provides a training apparatus for a multi-scale depth completion network, comprising: a first image acquisition unit, configured to acquire a training set including n image sample pairs, each sample pair including an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer; a first image processing unit, configured to input the training set into an initial depth completion network including an encoder and a decoder to obtain at least four simulated dense depth images of different scales; wherein, the encoder is configured to downsample and feature-stitch the infrared image sample with the same target scene and the sparse depth image sample to obtain at least four depth infrared feature maps of different scales, and the decoder is configured to upsample the depth infrared feature maps of at least four different scales to obtain at least four simulated dense depth images of different scales; and a training execution unit, configured to iteratively train the initial depth completion network using at least four standard dense depth image samples of different scales, at least four simulated dense depth images of different scales, and a preset loss function to obtain a target depth completion network.

[0008] Fourthly, one embodiment of this application provides a depth completion device, including: a second image acquisition device, configured to acquire a sparse depth image and an infrared image, and input the sparse depth image and the infrared image into a preset depth completion network, wherein the preset depth completion network is an application... Figure 5 The target depth completion network obtained by the depth completion network training device shown; the second image processing unit is used to downsample the sparse depth image and infrared image through the preset depth completion network to obtain at least four different scales of depth features and infrared features, stitch together the depth features and infrared features with the same scale among the at least four different scales of depth features and infrared features to obtain four different scales of stitched features, and upsample the four different scales of stitched features to obtain at least one dense depth image with the same scale as the original sparse depth image.

[0009] Fifthly, one embodiment of this application provides a depth completion system, comprising: an acquisition device for acquiring infrared images and sparse depth images; and a terminal device for receiving the infrared images and sparse depth images, and processing the infrared images and sparse depth images according to the depth completion method described in the second aspect to obtain a dense depth map.

[0010] In a sixth aspect, one embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for a deep completion network as described in the first aspect embodiment; or implements the deep completion method as described in the second aspect embodiment.

[0011] Compared with the prior art, the beneficial effects of this application are as follows: This application downsamples and stitches sparse depth images and infrared images through different convolutional layers and stitching layers, then uses multiple decoding modules to upsample and predict dense depth maps of different scales, and constrains the training of the model in stages, finally generating a high-quality dense depth map with the same scale as the input sparse depth map. This method not only simplifies the implementation process, but also significantly improves the generalization ability and computational efficiency of the depth completion network. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the architecture of a deep completion system provided in an embodiment of this application;

[0014] Figure 2 This is a flowchart illustrating a training method for a multi-scale deep completion network according to an embodiment of this application;

[0015] Figure 3 This is a schematic diagram of the architecture of an initial deep completion network provided in an embodiment of this application;

[0016] Figure 4 This is a flowchart illustrating a deep completion method provided in an embodiment of this application;

[0017] Figure 5 This is a schematic diagram of the training device for a multi-scale deep completion network provided in another embodiment of this application;

[0018] Figure 6 This is a schematic diagram of the structure of a depth completion device provided in an embodiment of this application. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0021] The terms "one embodiment" or "some embodiments" described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] Furthermore, in the description of this application, "a plurality of" means two or more. The terms "first" and "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0024] Figure 1 This is a schematic diagram of the architecture of a depth completion system based on multi-scale features according to this application. The system includes an acquisition device 10 and a terminal device 20. The acquisition device 10 is used to acquire infrared images and sparse depth images. The terminal device 20 is electrically connected to the acquisition device 10 via wired or wireless means and is used to receive the infrared images and sparse depth images. The terminal device 20 processes the infrared images and sparse depth images according to the depth completion method provided in one or more embodiments of this application to obtain a dense depth map. It should be noted that in this application, the depth image is an image that includes distance information between the target scene and the acquisition device. For example, speckle images, phase images, parallax images, or depth images with distance information are all collectively referred to as depth images in this application.

[0025] In one embodiment, the acquisition device 10 includes an infrared camera and a depth camera for acquiring infrared images and depth images, respectively. It should be noted that the depth camera and the infrared camera can be integrated or independently configured; the depth camera can be one or more combinations of a structured light camera, a TOF camera, a binocular camera, or a multi-view camera, without limitation herein.

[0026] In another embodiment, the acquisition device 10 may consist of only a depth camera, which includes an infrared image sensor. This depth camera can be used not only to acquire sparse depth images but also to acquire infrared images. In this case, the acquisition device 10 can complete the acquisition of both sparse depth images and infrared images using only one depth camera, which is beneficial for miniaturization of the acquisition device 10 and reducing system costs. It should be noted that because infrared image sensors or infrared cameras have passive light-sensing characteristics, they are unaffected by factors such as lighting conditions, making them more suitable for dark or low-light environments, thereby achieving more accurate depth completion.

[0027] In one embodiment, the terminal device 20 includes, but is not limited to, devices with computing capabilities such as computers, tablets, servers, cloud computing, mobile phones, or wearable devices, and includes one or more processors ( Figure 1 (Only one is shown in the image). One or more processors can implement the various steps of the deep completion method provided in one or more embodiments of this application. Further, the processor includes a Central Processing Unit (CPU), and may also include other general-purpose processors, neural network chips, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0028] In one embodiment, the terminal device 20 further includes a memory and a computer program stored in the memory and executable on one or more processors, such as the program corresponding to the deep completion method provided in one or more embodiments of this application. Further, the memory includes internal storage units of the terminal device, such as the hard disk or memory of the terminal device; or includes external storage devices of the terminal device, such as plug-in hard disks, smart media cards (SMCs), secure digital (SD) cards, flash cards, etc., equipped on the terminal device; or may include both internal storage units and external storage devices of the terminal device. Additionally, the memory 101 is also used to store other programs and data required by the terminal device or to temporarily store data that has been output or will be output.

[0029] Those skilled in the art will understand that Figure 1 This is merely an example of a deep completion system and does not constitute a limitation on the system. A deep completion system may include more or fewer components than illustrated, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0030] Furthermore, one or more embodiments of this application provide a depth completion method based on a multi-scale depth completion network. To avoid the RGB image being limited by illumination and thus affecting the depth completion effect, this application designs a novel neural network architecture for a multi-scale depth completion network by combining infrared images and depth images. To ensure the high reliability of this multi-scale depth completion network, Figure 2 A training method for a multi-scale deep completion network according to this application includes:

[0031] S1: Obtain a training set consisting of n image sample pairs. Each sample pair includes an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer.

[0032] In one embodiment, by means of Figure 1 The acquisition device shown acquires multiple frames of infrared and depth images of the same scene to obtain infrared image samples and sparse depth image samples. The multiple frames of sparse depth image samples and infrared image samples are input into different teacher networks (such as Reset150) with highly complex network architectures to obtain at least four different scales of standard dense depth image samples corresponding to each frame of sparse depth image. The standard dense depth image samples are ground truth, which is the target value required for the output of the initial depth completion network.

[0033] It should be noted that multiple frames of infrared images, sparse depth images, and at least four different scales of dense depth images with the same scene as each frame of sparse depth image can be crawled from the Internet using crawling tools as n image sample pairs to construct a training set. This application does not impose any restrictions on this.

[0034] Furthermore, after acquiring the infrared image and the sparse depth image, the pixels in the infrared image and the sparse depth image need to be aligned. That is, each pixel in the sparse depth image can be found in the infrared image, which makes it easier to use the texture information of the infrared image to guide the sparse depth image for depth completion.

[0035] S2: Input the training set into an initial depth completion network including an encoder and a decoder to obtain at least four different scales of simulated dense depth images; wherein, the encoder is used to downsample and feature-stitch infrared image samples and sparse depth image samples with the same target scene to obtain at least four different scales of depth infrared feature maps, and the decoder is used to upsample at least four different scales of depth infrared feature maps to obtain at least four different scales of simulated dense depth images.

[0036] In one embodiment, the encoder includes at least a depth convolutional layer, an infrared convolutional layer, and a stitching layer, with the same number of samples as the standard dense depth image samples. The depth convolutional layer and the infrared convolutional layer perform feature aggregation on the depth image and the infrared image, respectively, to obtain a depth feature map and an infrared feature map. The stitching layer is used to stitch the depth feature map and the infrared feature map to obtain the depth infrared feature map.

[0037] Specifically, the encoder includes at least four depth convolutional layers, four infrared convolutional layers, and four stitching layers. The four cascaded depth convolutional layers and the four cascaded infrared convolutional layers are arranged in parallel and correspond one-to-one. Each depth convolutional layer is connected to each infrared convolutional layer through a stitching layer. When the sparse depth image and infrared image are input into the encoder and pass through the feature aggregation of the four depth convolutional layers and the four infrared convolutional layers respectively, a depth feature map and an infrared feature map of one scale are obtained at each depth convolutional layer and each infrared convolutional layer. The stitching layers between each depth convolutional layer and each infrared convolutional layer stitch together the depth feature map and infrared feature map of the corresponding scale to obtain at least four different scale depth infrared feature maps. The encoder network structure provided in this embodiment is simple. It not only ensures high efficiency of depth completion, but also enables the network model to capture richer details by extracting multi-scale features and stitching different features. This helps to complement the depth information of the sparse depth image and the texture information of the infrared image, increases the expressive power of features, and thus improves the accuracy of depth completion.

[0038] In another embodiment, the decoder includes a number of decoding modules equal to the number of scales of the standard dense depth image samples. These modules are used to receive depth infrared feature maps at four different scales, extract higher-level rich feature representations from the depth infrared features at different scales, and upsample the extracted rich features to obtain at least four different scales of simulated dense depth images. That is, when the initial depth completion network model is trained using standard dense depth images at four different scales as samples, the number of decoding modules is 4, and the four decoding modules receive and process depth infrared feature maps at different scales respectively. When the initial depth completion network model is trained using standard dense depth images at five different scales as samples, the number of decoding modules is 5, and so on. This application will not elaborate further.

[0039] Preferably, each decoding module includes at least one downsampling layer and one upsampling layer. The downsampling layer is used to extract features from the depth infrared features to obtain a hybrid feature map. The hybrid features in the hybrid feature map include one or more sets such as spatial features, depth features, texture features, rich features extracted by combining depth features and texture features, or semantic features. The upsampling layer is used to improve the spatial resolution of the hybrid feature map to restore image details and obtain a simulated dense depth image. The simulated dense depth image has at least the same scale as its corresponding standard dense depth image sample.

[0040] In addition, to improve the representational power of the deep completion network model, some ordinary convolutional layers that do not change the resolution of the feature map can be inserted into the deep convolutional layers, infrared convolutional layers in the encoder and / or the decoding module of the decoder. This helps to learn more complex feature levels while avoiding information loss caused by reducing resolution. This application does not impose any restrictions on this.

[0041] Figure 3 This is a schematic diagram of the architecture of an initial depth completion network model provided in this application. The initial depth completion network includes an encoder 30 and a decoder 40. The encoder 30 includes a first depth convolutional layer, a second depth convolutional layer, a third depth convolutional layer, and a fourth depth convolutional layer connected in series; a first infrared convolutional layer, a second infrared convolutional layer, a third infrared convolutional layer, and a fourth infrared convolutional layer connected in series; a first concatenation layer connecting the first depth convolutional layer and the first infrared convolutional layer; a second concatenation layer connecting the second depth convolutional layer and the second infrared convolutional layer; a third concatenation layer connecting the third depth convolutional layer and the third infrared convolutional layer; and a fourth concatenation layer connecting the fourth depth convolutional layer and the fourth infrared convolutional layer. Preferably, the above-mentioned convolutional layers of different depths or infrared convolutional layers are implemented with convolutional kernels of different sizes to reduce the spatial size of the feature maps, thereby reducing the number of parameters and computational load, while extracting important features; different concatenation layers are implemented with the Concat function, which allows the network to share information between different feature maps and enhances the feature representation capability of the network.

[0042] Specifically, the depth image and the infrared image are respectively input into the first depth convolutional layer and the first infrared convolutional layer in the encoder 30 for low-level feature extraction and dimensionality reduction to obtain the first-scale depth feature map and the first-scale infrared feature map, thereby reducing the amount of computation and the number of parameters while retaining more detailed feature information; the first-scale depth feature map and the first-scale infrared feature map are transmitted to the first stitching layer, which is used to perform channel stitching on the first-scale depth feature map and the first-scale infrared feature map to obtain the first-scale depth infrared feature map and send it to the decoder 40.

[0043] While the first-scale depth feature map and the first-scale infrared feature map are being transmitted to the first stitching layer, they are also being fed into the second depth convolutional layer and the second infrared convolutional layer, respectively. The second depth convolutional layer and the second infrared convolutional layer combine the low-level features from the first-scale depth feature map and the first-scale infrared feature map, respectively, and perform high-level feature extraction and downsampling based on the combined features to obtain the second-scale depth feature map and the second-scale infrared feature map. These second-scale depth feature maps and the second-scale infrared feature maps contain rich information, and the downsampling reduces model complexity, helping to prevent overfitting. Further, the second-scale depth feature map and the second-scale infrared feature map are being transmitted to the second stitching layer, which performs channel stitching on the second-scale depth feature map and the second-scale infrared feature map to obtain the second-scale depth infrared feature map, which is then sent to the decoder 40.

[0044] The third depth convolutional layer and the third infrared convolutional layer extract more abstract features from the second-scale depth feature map and the second-scale infrared feature map, respectively, and perform further dimensionality reduction. While reducing detailed information, they retain more robust feature representations, resulting in a third-scale depth feature map and a third-scale infrared feature map that focus more on local features. The third-scale depth feature map and the third-scale infrared feature map are then transmitted to the third stitching layer, which performs channel stitching on the third-scale depth feature map and the third-scale infrared feature map to obtain the third-scale depth infrared feature map, which is then sent to the decoder 40.

[0045] The third-scale depth feature map and the third-scale infrared feature map are respectively fed into the fourth depth convolutional layer and the fourth infrared convolutional layer for global feature extraction and significant dimensionality reduction to obtain the fourth-scale depth feature map and the fourth-scale infrared feature map. At this point, the size of the obtained feature maps is very small, the model focuses more on global features rather than local details, and the feature images have global scale invariance. The fourth-scale depth feature map and the fourth-scale infrared feature map are then transmitted to the fourth stitching layer, which performs channel stitching on the fourth-scale depth feature map and the fourth-scale infrared feature map to obtain the fourth-scale depth infrared feature map, which is then sent to the decoder 40.

[0046] This application uses deep convolutional layers and infrared convolutional layers in series, enabling each layer to help the model understand the input image data at different levels, thereby extracting useful features while maintaining computational efficiency. Furthermore, by channel-stitching the depth and infrared features output from different convolutional layers at different scales, it integrates features from different levels and scales that contain detailed, semantic, and complementary information, allowing the model to utilize multiple types of information simultaneously for more accurate depth completion, thereby enhancing the model's expressive power.

[0047] In one embodiment, the decoder 40 includes a first decoding module, a second decoding module, a third decoding module, and a fourth decoding module arranged in parallel, for receiving first-scale depth infrared feature maps, second-scale depth infrared feature maps, third-scale depth infrared feature maps, and fourth-scale depth infrared feature maps at different scales, respectively; the first decoding module includes a first downsampling layer and a first upsampling layer connected in series, the first downsampling layer is used to remove redundant information and enhance the representation ability of the retained features to obtain a first hybrid feature map, and the first upsampling layer is used to increase the spatial resolution of the first hybrid feature map to obtain a first-scale simulated dense depth image.

[0048] It should be noted that the second, third, and fourth decoding modules have the same architecture and function as the first decoding module, except that they respectively use downsampling and upsampling layers to process the second-scale depth infrared feature maps, the third-scale depth infrared feature maps, and the fourth-scale depth infrared feature maps to obtain second-scale simulated dense depth images, third-scale simulated dense depth images, and fourth-scale simulated dense depth images, thereby forming four scales of simulated dense depth images corresponding to the four different scales of standard dense depth image samples in the training set. Preferably, the downsampling layers in the different decoding modules of the decoder 40 can be implemented using max pooling and / or average pooling functions, and the upsampling layers can be implemented using transposed convolutions, to help the network model capture and utilize information at different levels, thereby improving the performance and accuracy of the network model.

[0049] S3: Iteratively train the initial depth completion network using at least four different scales of standard dense depth image samples, at least four different scales of simulated dense depth images, and a preset loss function to obtain the target depth completion network; wherein, the target depth completion network has the same architecture as the initial depth completion network, except that the parameters of each network layer are different.

[0050] After obtaining at least four different scales of simulated dense depth images, the pixel values ​​of the corresponding pixels of the at least four different scales of standard dense depth image samples, i.e., at least four different scales of simulated dense depth images, are input into a preset loss function to calculate the loss value. The initial depth completion network is then iteratively trained based on the loss value. If the calculated loss value is within the preset loss threshold range, the target depth completion network is obtained; otherwise, training continues until the loss value calculated based on the loss function is within the preset loss threshold range.

[0051] This application employs different loss function calculation methods at different training stages of the model. Before training, the total number of training iterations N and the model training iteration threshold M are determined. During the first M training iterations, the model is constrained using simulated dense depth images and standard dense depth image samples at different scales to obtain the first loss value, where M≤N, and M and N are both positive integers. After the model has been trained more than M times, the second loss value is calculated using only simulated dense depth images and standard dense depth image samples at specific scales. The target depth completion network is obtained based on whether the first and second loss values ​​are within the preset loss threshold range.

[0052] Preferably, obtaining the target depth completion network based on whether the first loss value and the second loss value are within the preset loss threshold range includes calculating the sum of the first loss value and the second loss value, determining whether the sum of the two loss values ​​is within the threshold range, or determining whether the first loss value and the second loss value are within the preset loss threshold range to determine whether to stop training. If they are within the preset loss threshold range, training is stopped; otherwise, the network model parameters are changed and a new round of training is carried out until the loss value is within the preset loss threshold range, thus obtaining the target depth completion network.

[0053] This embodiment constrains the model training process at different stages by using different loss functions, so that the model can fully understand information from multiple angles and perspectives, and focus on predicting a dense depth map of the same size as the input sparse depth image, thus obtaining an ideal dense depth map with excellent depth completion effect.

[0054] In one embodiment, the preset loss function is:

[0055]

[0056]

[0057] in, This represents the weighted value corresponding to the i-th scale image. This represents the loss value corresponding to the i-th scale image. This represents the pixel value corresponding to the j-th pixel in the i-th scale simulated dense depth image. This represents the pixel value corresponding to the j-th pixel in the dense depth image sample of the i-th scale standard; This represents the current number of training iterations for the network model, where M represents the threshold number of training iterations. It is represented as the loss value corresponding to the simulated dense depth image with the same scale as the sparse depth image of the input initial depth completion network.

[0058] Number of times the network model is trained Taking 50 iterations and M=20 as an example, in the first 20 training iterations, the training process of the model is constrained based on simulated dense depth images and standard dense depth image samples of different scales, which enhances its adaptability to depth maps of various scales. This can prevent the model from overfitting or learning invalid features during training, thereby ensuring the model's generalization ability.

[0059] In the later 30 iterations, the model focuses only on predicting simulated dense depth images with the same scale as the sparse depth image input to the initial depth completion network. This helps the model to focus more on optimizing and refining its prediction ability for depth maps at specific scales in the later stages of training, thereby improving the model's accuracy and robustness in practical applications.

[0060] Figure 4 According to the depth completion method provided in this application, the method includes:

[0061] S4: Acquire sparse depth images and infrared images, and input the sparse depth images and infrared images into a preset depth completion network, where the preset depth completion network is the application... Figure 1 The training method in this embodiment yields a target depth-completed network;

[0062] S5: Downsample the sparse depth image and infrared image using a preset depth completion network to obtain at least four different scales of depth features and infrared features. Then, stitch together the depth features and infrared features with the same scale among the at least four different scales of depth features and infrared features to obtain four different scales of stitched features. Finally, upsample the four different scales of stitched features to obtain at least one dense depth image with the same scale as the original sparse depth image.

[0063] based on Figure 1 The training method of the embodiment yields a deep completion network with a simple architecture, high depth completion efficiency, and is based on multi-scale features, enabling the deep completion network to make full use of limited existing input information and obtain a dense depth map with the best performance.

[0064] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0065] Figure 5 This is a schematic block diagram of a depth completion network training device provided in this application. The depth completion network training device includes: a first image acquisition unit 51, used to acquire a training set including n image sample pairs, each sample pair including an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer; and a first image processing unit 52, used to input the training set into an initial depth completion network including an encoder and a decoder to obtain simulated dense depth images of at least four different scales; wherein, the encoder is used to process the infrared image sample with the same target scene... The sparse depth image samples are downsampled and feature stitched to obtain at least four different scales of depth infrared feature maps. The decoder is used to upsample the at least four different scales of depth infrared feature maps to obtain at least four different scales of simulated dense depth images. The training execution unit 53 is used to iteratively train the initial depth completion network using at least four different scales of standard dense depth image samples, at least four different scales of simulated dense depth images, and a preset loss function to obtain the target depth completion network. The target depth completion network has the same architecture as the initial depth completion network, except that the parameters of each network layer are different.

[0066] Figure 6 This is a schematic block diagram of a depth completion device provided in this application. The depth completion device includes: a second image acquisition device 61, used to acquire a sparse depth image and an infrared image, and input the sparse depth image and the infrared image into a preset depth completion network, wherein the preset depth completion network is an application... Figure 5 The target depth completion network obtained by the depth completion network training device shown; the second image processing unit 62 is used to downsample the sparse depth image and infrared image through the preset depth completion network to obtain at least four different scale depth features and infrared features, stitch together the depth features and infrared features with the same scale among the at least four different scale depth features and infrared features to obtain four different scale stitched features, and upsample the four different scale stitched features to obtain at least one dense depth image with the same scale as the original sparse depth image.

[0067] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the deep completion network training method and / or the deep completion method embodiment.

[0070] One embodiment of this application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps in the deep completion network training method and / or the deep completion method embodiment.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can also be implemented by a computer program instructing related hardware, and a computer program can be stored in a computer-readable storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0073] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A training method for a multi-scale deep completion network, characterized in that, include: Obtain a training set consisting of n image sample pairs. Each sample pair includes an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer. The training set is input into an initial depth completion network including an encoder and a decoder to obtain at least four different scales of simulated dense depth images; wherein, the encoder is used to downsample and feature-stitch the infrared image samples and the sparse depth image samples to obtain at least four different scales of depth infrared feature maps, and the decoder is used to upsample the at least four different scales of depth infrared feature maps to obtain at least four different scales of simulated dense depth images. The encoder includes at least four deep convolutional layers, four infrared convolutional layers, and four splicing layers. The four cascaded deep convolutional layers and the four cascaded infrared convolutional layers are arranged in parallel and correspond one-to-one. Each deep convolutional layer and each infrared convolutional layer are connected through a splicing layer. When the sparse depth image and the infrared image are sequentially processed through four depth convolutional layers and four infrared convolutional layers for feature aggregation, a depth feature map and an infrared feature map of one scale are obtained in each depth convolutional layer and each infrared convolutional layer. The depth feature map and infrared feature map of the corresponding scale are then spliced ​​together by the splicing layer between each depth convolutional layer and each infrared convolutional layer to obtain the at least four different scales of depth infrared feature maps. The initial depth completion network is iteratively trained using at least four different scales of standard dense depth image samples, at least four different scales of simulated dense depth images, and a preset loss function to obtain the target depth completion network.

2. The training method as described in claim 1, characterized in that, The encoder includes at least a depth convolutional layer, an infrared convolutional layer, and a stitching layer, with the same number of samples as the standard dense depth image samples. The depth convolutional layer and the infrared convolutional layer perform feature aggregation on the depth image and the infrared image, respectively, to obtain a depth feature map and an infrared feature map. The stitching layer is used to stitch the depth feature map and the infrared feature map to obtain the depth infrared feature map.

3. The training method as described in claim 1, characterized in that, The decoder includes at least a number of decoding modules equal to the number of scales of the standard dense depth image samples, for receiving the depth infrared feature maps at four different scales respectively and extracting features from the depth infrared features at different scales respectively, and upsampling the extracted features to obtain the simulated dense depth images at at least four different scales.

4. The training method as described in claim 3, characterized in that, Each of the decoding modules includes at least one downsampling layer and one upsampling layer; wherein: The downsampling layer is used to extract features from the deep infrared features to obtain a hybrid feature map. The hybrid features in the hybrid feature map include one or more sets of spatial features, depth features, texture features, rich features extracted by combining depth features and texture features, or semantic features. The upsampling layer is used to improve the spatial resolution of the hybrid feature map to recover image details and obtain a simulated dense depth image, wherein the simulated dense depth image has at least the same scale as its corresponding standard dense depth image sample.

5. The training method according to any one of claims 1-4, characterized in that, The process of iteratively training the initial depth completion network using at least four different scales of standard dense depth image samples, at least four different scales of simulated dense depth images, and a preset loss function to obtain the target depth completion network includes: Determine the total number of training iterations N and the threshold M for the number of training iterations. During the first M training iterations of the model, use simulated dense depth images and standard dense depth image samples of different scales to constrain the model and obtain the first loss value, where M≤N and M and N are both positive integers. After the model has been trained more than M times, the second loss value is calculated using only simulated dense depth images and standard dense depth image samples at a specific scale. The target depth completion network is obtained based on whether the first loss value and the second loss value are within the preset loss threshold range.

6. The training method as described in claim 5, characterized in that, The preset loss function is: , ,in, This represents the weighted value corresponding to the i-th scale image. This represents the loss value corresponding to the i-th scale image. This represents the pixel value corresponding to the j-th pixel in the i-th scale simulated dense depth image. This represents the pixel value corresponding to the j-th pixel in the dense depth image sample of the i-th scale standard; This represents the current number of training iterations for the network model, where M represents the threshold number of training iterations. It is represented as the loss value corresponding to the simulated dense depth image with the same scale as the sparse depth image of the input initial depth completion network.

7. A depth completion method, characterized in that, include: Acquire sparse depth images and infrared images, and input the sparse depth images and infrared images into a preset depth completion network, wherein the preset depth completion network is a target depth completion network obtained by the training method of the multi-scale depth completion network as described in any one of claims 1-6. The sparse depth image and infrared image are downsampled by the preset depth completion network to obtain at least four different scales of depth features and infrared features. The depth features and infrared features with the same scale among the at least four different scales of depth features and infrared features are stitched together to obtain four different scales of stitched features. The stitched features of the four different scales are upsampled to obtain at least one dense depth image with the same scale as the original sparse depth image.

8. A multi-scale depth completion network training device, characterized in that, include: The first image acquisition unit is used to acquire a training set including n image sample pairs. Each sample pair includes an infrared image sample with the same target scene, a sparse depth image sample, and at least four standard dense depth image samples of different scales, where n≥1 and is an integer. The first image processing unit is used to input the training set into an initial depth completion network including an encoder and a decoder to obtain at least four different scales of simulated dense depth images; wherein, the encoder is used to downsample and feature-stitch the infrared image samples and the sparse depth image samples to obtain at least four different scales of depth infrared feature maps, and the decoder is used to upsample the at least four different scales of depth infrared feature maps to obtain at least four different scales of simulated dense depth images. The encoder includes at least four deep convolutional layers, four infrared convolutional layers, and four splicing layers. The four cascaded deep convolutional layers and the four cascaded infrared convolutional layers are arranged in parallel and correspond one-to-one. Each deep convolutional layer and each infrared convolutional layer are connected through a splicing layer. When the sparse depth image and the infrared image are sequentially processed through four depth convolutional layers and four infrared convolutional layers for feature aggregation, a depth feature map and an infrared feature map of one scale are obtained in each depth convolutional layer and each infrared convolutional layer. The depth feature map and infrared feature map of the corresponding scale are then spliced ​​together by the splicing layer between each depth convolutional layer and each infrared convolutional layer to obtain the at least four different scales of depth infrared feature maps. The training execution unit is used to iteratively train the initial depth completion network using the at least four standard dense depth image samples of different scales, the at least four simulated dense depth images of different scales, and a preset loss function to obtain the target depth completion network.

9. A depth completion device, characterized in that, include: The second image acquisition device is used to acquire sparse depth images and infrared images, and input the sparse depth images and infrared images into a preset depth completion network, wherein the preset depth completion network is a target depth completion network obtained by applying the depth completion network training device as described in claim 8. The second image processing unit is used to downsample the sparse depth image and infrared image through the preset depth completion network to obtain at least four different scales of depth features and infrared features, stitch together the depth features and infrared features with the same scale among the at least four different scales of depth features and infrared features to obtain four different scales of stitched features, and upsample the four different scales of stitched features to obtain at least one dense depth image with the same scale as the original sparse depth image.

10. A deep completion system, characterized in that, include: Acquisition equipment used to acquire infrared images and sparse depth images; A terminal device is used to receive an infrared image and a sparse depth image, and process the infrared image and the sparse depth image according to the depth completion method as described in claim 7 to obtain a dense depth map.

11. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the multi-scale deep completion network as described in any one of claims 1 to 6, or the deep completion method as described in claim 7.

Images