Method, apparatus, depth system, terminal, and storage medium for obtaining depth image

Abstract

This application is applicable to the field of image processing technology, and provides a method, apparatus, depth system, terminal, and storage medium for obtaining a depth image. Specifically, the method for obtaining the depth image includes: acquiring a color image and a sparse depth image of a target scene, extracting the color features and initial features of the color image, obtaining an initial dense depth image and initial hidden features based on the color features, initial features, and sparse depth image, and performing at least one iterative optimization operation to confirm the target hidden features based on the hidden features to be confirmed obtained in each iterative optimization operation, and performing depth estimation using the target hidden features to obtain the target dense depth image of the target scene. The embodiments of this application can improve the reliability of the obtained dense depth image.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to a method, device, depth system, terminal, and storage medium for obtaining a depth image. Background Art

[0002] In recent years, with the development of computer vision technology in fields such as autonomous driving, robotics, and AR applications, depth estimation has become a popular research and application field. Commonly used depth perception methods, such as structured light, TOF, binocular vision, lidar, etc., have become increasingly mature in technology after years of development and have been widely applied in multiple fields. However, these methods are restricted by cost and the technology itself and can only obtain reliable sparse depth point clouds or low-resolution depth maps. Therefore, in recent years, depth completion technology based on neural networks has received extensive attention.

[0003] Currently, how to effectively fuse RGB images and sparse depth images remains an open issue. Existing depth completion technologies often do not make good use of sparse depth images, and the obtained dense depth images lack reliability. Summary of the Invention

[0004] Embodiments of this application provide a method, device, depth system, terminal, and storage medium for obtaining a depth image, which can improve the reliability of the obtained dense depth image.

[0005] In a first aspect, an embodiment of this application provides a method for obtaining a depth image, including:

[0006] Obtain a color image and a sparse depth image of a target scene;

[0007] Extract the color feature and the initial feature of the color image, and obtain an initial dense depth image and an initial hidden feature according to the color feature, the initial feature, and the sparse depth image;

[0008] Use the color feature, the sparse depth image, and the initial hidden feature to perform at least one iterative optimization operation on the initial dense depth image, and confirm a target hidden feature according to the to-be-confirmed hidden feature obtained in each iterative optimization operation;

[0009] Perform depth estimation using the target hidden feature to obtain a target dense depth image of the target scene.

[0010] In a second aspect, an embodiment of this application provides a device for obtaining a depth image, including:

[0011] An image acquisition unit, configured to obtain a color image and a sparse depth image of a target scene;

[0012] An initial densification unit for extracting color features and initial features of the color image, and obtaining an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image;

[0013] An iterative optimization unit for performing at least one iterative optimization operation on the initial dense depth image by using the color features, the sparse depth image, and the initial hidden features, and confirming a target hidden feature according to the to-be-confirmed hidden features obtained by each iterative optimization operation;

[0014] A target densification unit for performing depth estimation by using the target hidden feature to obtain a target dense depth image of the target scene.

[0015] A third aspect of the embodiments of the present application provides a depth system, including a color module, a depth module, and the acquisition device described in the second aspect of the present application, where:

[0016] The color module is configured to collect a color image of a target scene;

[0017] The depth module is configured to scan the target scene to obtain point cloud data, and obtain a sparse depth image according to the point cloud data;

[0018] The acquisition device is configured to obtain a target dense depth image by using the color image and the sparse depth image.

[0019] A fourth aspect of the embodiments of the present application provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the above method when executing the computer program.

[0020] A fifth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program, where the computer program implements the steps of the above method when executed by a processor.

[0021] A sixth aspect of the embodiments of the present application provides a computer program product, which, when running on a terminal, enables the terminal to implement the steps of the method when executed.

[0022] In an embodiment of the present application, by obtaining a color image and a sparse depth image of a target scene, extracting the color features and initial features of the color image, obtaining an initial dense depth image and initial hidden features based on the color features, initial features, and sparse depth image, and performing at least one iterative optimization operation to confirm the target hidden features according to the hidden features to be confirmed obtained in each iterative optimization operation, and using the target hidden features for depth estimation to obtain the target dense depth image of the target scene. Since each iterative optimization operation needs to refer to the sparse depth image and color features to determine the hidden features to be confirmed, that is, the process of each iterative optimization operation will be guided by the information of the RGB image and the sparse depth image, the reliability of the obtained dense depth image can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0024] Figure 1 is a schematic flowchart of the implementation of a method for obtaining a depth image provided by an embodiment of the present application;

[0025] Figure 2 is a schematic flowchart of the specific implementation for determining the initial hidden features provided by an embodiment of the present application;

[0026] Figure 3 is a schematic structural diagram of a depth image model provided by an embodiment of the present application;

[0027] Figure 4 is a schematic structural diagram of a feedback module provided by an embodiment of the present application;

[0028] Figure 5 is a schematic flowchart of the specific implementation for training a depth image model provided by an embodiment of the present application;

[0029] Figure 6 is a schematic structural diagram of an apparatus for obtaining a depth image provided by an embodiment of the present application;

[0030] Figure 7 is a schematic structural diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0032] In existing depth completion technologies, generally, the scene information of a scene is guided by an RGB color image to achieve the densification of a sparse depth image. Some methods directly perform a splicing operation on the RGB and the sparse depth map, and then input the spliced image into a neural network for depth completion; some other methods input the sparse depth map into a neural network to obtain an initial dense depth map, and then fuse the dense depth map with the RGB image to obtain a depth completion result with higher accuracy.

[0033] However, these methods often do not make good use of the sparse depth image, resulting in insufficient densification of the obtained dense depth image and low reliability.

[0034] In order to illustrate the technical solution of the present application, it will be described below through specific embodiments.

[0035] Figure 1 FIG. shows a schematic implementation flowchart of a method for obtaining a depth image provided by an embodiment of the present application. This method can be applied to a terminal and is applicable to situations where the reliability of the obtained dense depth image needs to be improved.

[0036] Among them, the above-mentioned terminal can be a device capable of image processing such as a computer, a smart phone, a tablet device, etc.

[0037] Specifically, the method for obtaining the depth image may include the following steps S101 to S104.

[0038] Step S101: Obtain a color image and a sparse depth image of a target scene.

[0039] In some embodiments of the present application, the terminal can obtain a color image and point cloud data of a target scene, and then project the point cloud data onto the imaging plane of the color image to obtain a sparse depth image.

[0040] More specifically, the terminal can obtain a color image obtained by photographing a target scene with a color camera, and obtain point cloud data obtained by scanning the target scene with a depth sensor, and project the point cloud data scanned by the depth sensor onto the imaging plane of the color camera, so as to obtain a sparse depth image. Among them, the depth sensor may include, but is not limited to, lidar, direct time of flight (dTof), speckle indirect time of flight (iTof), etc.

[0041] It should be noted that the above target scene can be selected according to the actual situation, and this application does not limit it.

[0042] Step S102, extract the color features and initial features of the color image, and obtain an initial dense depth image and initial hidden features according to the color features, initial features and sparse depth image.

[0043] In an embodiment of the present application, the terminal can extract its color features and initial features through a feature extraction algorithm; among them, the feature extraction algorithms used for the color features and initial features can be the same or different. For example, the terminal can perform convolution operations on the color image through different convolution kernels to extract the color features and initial features of the color image.

[0044] Furthermore, the terminal can use the color features and initial features obtained from the color image, and the sparse depth image to determine the initial hidden features for optimizing the feedback. Among them, the initial hidden features can be used for feature optimization feedback on the initial dense depth image, so as to initially complete the depth image.

[0045] In one embodiment, as Figure 2 shown, the initial hidden features in step S102 can be obtained from steps S201 to S203.

[0046] Step S201, perform depth estimation on the initial features to obtain an initial dense depth image.

[0047] In some embodiments, the terminal can perform convolution regression on the initial features to obtain an initial dense depth image. It should be noted that the above depth estimation method can be selected according to the actual situation and is not limited here.

[0048] Step S202, fuse the initial dense depth image and the sparse depth image to obtain an initial fused feature image.

[0049] Specifically, the terminal can achieve image fusion through a concatenation (Concat) operation, and the initial fused feature image F concat (dsparse ,d dense )=Concat{d sparse ,d dense}.

[0050] Among them, d sparse is a sparse depth image, d dense is the initial dense depth image.

[0051] Step S203, using the initial fused feature image, the initial features and the color features, determine the initial hidden features.

[0052] Specifically, the terminal may perform a convolution operation on the initial fused feature image, the initial features, and the color features to determine the initial hidden features.

[0053] Step S103, using the color features, the sparse depth image and the initial hidden features, performing at least one iterative optimization operation on the initial dense depth image, and confirming the target hidden features according to the hidden features to be confirmed obtained in each iterative optimization operation.

[0054] In the implementation of the present application, after obtaining the initial hidden features, the color features, the initial hidden features and the sparse depth image are used as inputs for the first iterative optimization operation, and the hidden features to be confirmed obtained by the first iterative optimization operation are used together with the color features and the sparse depth image as inputs for the second iterative optimization operation, and so on. Each iterative optimization operation can obtain a hidden feature to be confirmed. After at least one iterative optimization operation, the target hidden features used to determine the target dense depth image can be determined from the hidden features to be confirmed obtained by each iterative optimization operation. Among them, the target dense depth image is also a depth image whose densification effect can meet the requirements.

[0055] In some embodiments, the hidden features to be confirmed output by the first iterative optimization operation in step S103 can be obtained by the following steps:

[0056] Step S204, performing depth estimation on the initial hidden features to obtain a first dense depth image;

[0057] Step S205, fusing the first dense depth image with the sparse depth image to obtain a first fused feature image;

[0058] Step S206, using the first fused feature image, the color feature and the initial hidden feature, determine the hidden feature to be confirmed output by the first iterative optimization operation.

[0059] When performing the second iterative optimization operation, the terminal can estimate the depth of the to-be-confirmed hidden features output by the first iterative optimization operation to obtain a second dense depth image. Then, the second dense depth image is fused with the sparse depth image to obtain a second fused feature image, and the to-be-confirmed hidden features output by the second iterative optimization operation are determined using the second fused feature image, the color features, and the to-be-confirmed hidden features output by the first iterative optimization operation. By analogy, after N iterations, N to-be-confirmed hidden features can be obtained, where N is a positive integer greater than or equal to 1.

[0060] It should be noted that the specific methods of steps S204 to S206 are respectively similar to those of steps S201 to S203, and will not be elaborated here.

[0061] In addition, the larger the value of N, the better the densification effect of the target hidden features finally output by the iterative optimization operation. Correspondingly, the time and computational effort required will increase. Generally, the densification effect is most obvious in the first few iterations. Therefore, the specific value of N can be set according to actual situations such as hardware conditions and densification requirements.

[0062] In some embodiments of the present application, after each time the to-be-confirmed hidden features are obtained through the iterative optimization operation, the terminal can calculate the error index between the to-be-confirmed hidden features output by the current iterative optimization operation and the to-be-confirmed hidden features output by the previous iterative optimization operation, and determine whether the error index is within a preset error threshold range. If the error index is within the preset error threshold range, the iterative optimization operation is stopped, and the to-be-confirmed hidden features obtained by the current iterative optimization operation are used as the target hidden features. If the error index is outside the preset error threshold range, the to-be-confirmed hidden features output by the current iterative optimization operation are used as the input for the next iterative optimization operation, and the next iterative optimization operation is continued.

[0063] Among them, the error threshold range can be adjusted according to actual situations, and the present application does not limit this.

[0064] That is to say, after the first iterative optimization operation, the terminal can determine whether the error index between the to-be-confirmed hidden features output by the first iterative optimization operation and the initial hidden features is within the error threshold range, so as to decide whether to perform the next iterative optimization operation. If the error index is within the error threshold range, the iterative optimization operation is stopped, and the to-be-confirmed hidden features output by the first iterative optimization operation are used as the target hidden features.

[0065] Otherwise, perform the second iterative optimization operation. After completing the second iterative optimization operation, determine whether the error metric between the to-be-confirmed hidden features output by the second iterative optimization operation and the to-be-confirmed hidden features output by the first iterative optimization operation is within the error threshold range, so as to decide whether to perform the next iterative optimization operation.

[0066] And so on, until the error metric between the to-be-confirmed hidden features output by the Nth iterative optimization operation and the to-be-confirmed hidden features output by the (N - 1)th iterative optimization operation is within the error threshold range, then stop performing the iterative optimization operation, and use the to-be-confirmed hidden features output by the Nth iterative optimization operation as the target hidden features.

[0067] In some embodiments of the present application, the error metric can also be determined based on the dense depth image corresponding to the current iterative optimization operation and the dense depth image corresponding to the previous iterative optimization operation of the current iterative optimization operation. Among them, the dense depth image corresponding to the current iterative optimization operation refers to the dense depth image obtained by performing depth estimation on the hidden features output by the current iterative optimization operation. Preferably, the terminal can subtract the dense depth image corresponding to the previous iterative optimization operation from the dense depth image obtained by the current iterative optimization operation, and then calculate the mean absolute error (MAE) value as the error metric. At this time, the number of times of performing the iterative optimization operation can balance the densification effect and efficiency.

[0068] Step S104, perform depth estimation using the target hidden features to obtain the target dense depth image of the target scene.

[0069] In an embodiment of the present application, the terminal performs depth estimation on the target hidden features confirmed in step S103, and the target dense depth image of the target scene can be obtained. This dense depth image is the image obtained after depth completion of the sparse depth image, that is, the depth image whose densification effect can meet the requirements.

[0070] In an embodiment of the present application, by acquiring the color image and the sparse depth image of the target scene, extracting the color features and initial features of the color image, obtaining the initial dense depth image and initial hidden features according to the color features, initial features and sparse depth image, and performing at least one iterative optimization operation to confirm the target hidden features according to the to-be-confirmed hidden features obtained in each iterative optimization operation, and performing depth estimation using the target hidden features to obtain the target dense depth image of the target scene. Since each iterative optimization operation needs to refer to the sparse depth image and color features to determine the to-be-confirmed hidden features, that is, the process of each iterative optimization operation will be guided by the information of the RGB image and the information of the sparse depth image, the reliability of the obtained dense depth image can be improved.

[0071] Moreover, each iterative optimization operation uses the to-be-confirmed hidden features output by the previous iterative optimization operation as a guide. Therefore, after each iterative optimization operation, the densification degree of the dense depth image will be further improved.

[0072] It should be noted that the above method for obtaining the depth image can be implemented through a network model. Figure 3 The structural schematic diagram of the depth image model is shown. The terminal can input the color image and the sparse depth image into the depth image model to obtain the target dense depth image output by the depth image model.

[0073] Among them, the depth image model may include a feature extraction module, N feedback modules, and a target depth estimation module.

[0074] The terminal can extract the color features and initial features of the color image through the feature extraction module, and obtain the initial dense depth image and the initial hidden features through the first feedback module.

[0075] Then, the terminal can sequentially perform an iterative optimization operation through the remaining feedback modules respectively, and finally, through the target depth estimation module, perform depth estimation on the target hidden features output by the last feedback module to obtain the target dense depth image of the target scene.

[0076] As Figure 4 shown, each feedback module may include an intermediate depth estimation module, a fusion module, and a sequence model module.

[0077] The steps for the terminal to perform the iterative optimization operation within a single feedback module may specifically include: through the intermediate depth estimation module of the current feedback module, performing depth estimation on the previous hidden features (i.e., the to-be-confirmed hidden features output by the previous feedback module) to obtain the dense depth image output by the current feedback module; through the fusion module of the current feedback module, fusing the dense depth image output by the current feedback module with the sparse depth image to obtain the fusion feature map of the current feedback module; through the sequence model module of the current feedback module, using the color features, the fusion feature map of the current feedback module, and the previous hidden features to determine the current to-be-confirmed hidden features (i.e., the to-be-confirmed hidden features output by the current feedback module).

[0078] The number of the above feedback modules can be set according to the actual situation. Moreover, the above depth image model can also output a parameter for characterizing the densification effect. If the parameter is greater than a pre-set threshold, the output of the current feedback module is continued to be input into the next feedback module. If the parameter is less than the threshold, the to-be-confirmed hidden features output by the current feedback module are used as the target hidden features, and through the target depth estimation module, depth estimation is performed on the target hidden features to output the target dense depth image of the target scene.

[0079] It should be noted that the specific working process of the above depth image model can be referred to the description of the method shown in the foregoing Figure 1 and Figure 2 This application will not elaborate on this.

[0080] Before using the depth image model, the terminal needs to train the depth image model. Further, during the process of training the depth image model, the number of iterations for iterative optimization operations through the feedback module is fixed. If the number of iterations is not fixed, the dense depth image obtained by iterative optimization of the feedback module will change with the adjustment of the network parameters to be trained, resulting in two variables during the training process and inaccurate training errors. During the use of the depth image model, the number of iterations of the feedback module can be not fixed, and the number of iterations can depend on the error between the dense depth image to be optimized obtained from the current iterative optimization operation and the dense depth image to be optimized obtained from the previous iterative optimization operation.

[0081] Specifically, as Figure 5 shown, the training process of the above depth image model may include steps S501 to S503.

[0082] Step S501, obtain a sample color image, a sample sparse depth image, and a corresponding reference dense depth image.

[0083] Among them, the acquisition methods of the sample color image and the sample sparse depth image can refer to the description of step S101.

[0084] The reference dense depth image is an ideal dense depth image. In some embodiments, a depth image synthesized artificially can be obtained, for example, through the unreal engine 4 (UE4), or a depth image collected by other depth sensors (such as a high-precision TOF depth camera) can also be obtained.

[0085] Step S502, input the sample color image and the sample sparse depth image into the network to be trained, obtain the sample dense depth image output by each feedback module in the network to be trained, and the sample target dense depth image output by the target depth estimation module in the network to be trained.

[0086] Among them, the model structure and working process of the network to be trained can be referred to the description of Figures 1 to 4 This application will not elaborate on this.

[0087] Step S503: Calculate the target error value based on the sample target dense depth image, each sample dense depth image, and the reference dense depth image. If the target error value is greater than the error threshold, adjust the parameters of the network to be trained to iteratively optimize the network to be trained until the target error value is less than or equal to the error threshold, and use the network to be trained as the depth image model.

[0088] Among them, the error threshold refers to the maximum value of the target error value allowed when the model is considered to converge, and can be adjusted according to the actual situation.

[0089] Specifically, in some embodiments of the present application, the terminal can calculate the initial error values between the sample target dense depth image and each sample dense depth image and the reference dense depth image respectively, and then perform weighted averaging on the initial error values to obtain the target error value, so as to ensure better densification effect of iterative optimization.

[0090] By comparing with the preset error threshold, if the target error value is greater than the error threshold, it means that the network to be trained has not converged. Therefore, it is necessary to readjust the parameters of the network to be trained and recalculate the target error value, and iterate until the target error value is less than or equal to the error threshold, indicating that the network to be trained can already output a reliable dense depth image, and this network can be used as the depth image model and put into use.

[0091] It should be noted that the number of the above sample color images, sample sparse depth images, and corresponding reference dense depth images can be multiple. In the process of each iterative training, any one or more of them can be selected for training. The training process can be implemented by the gradient descent method, and the corresponding loss function can be the L1 norm loss function, the L2 norm loss function, or other loss functions.

[0092] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences.

[0093] As Figure 6 shown in the structural schematic diagram of a depth image acquisition device 600 provided by an embodiment of the present application, the depth image acquisition device 600 is configured on a terminal.

[0094] Specifically, the depth image acquisition device 600 may include:

[0095] An image acquisition unit 601, configured to acquire a color image and a sparse depth image of a target scene;

[0096] An initial densification unit 602, configured to extract color features and initial features of the color image, and obtain an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image;

[0097] An iterative optimization unit 603, configured to perform at least one iterative optimization operation on the initial dense depth image by using the color features, the sparse depth image, and the initial hidden features, and confirm a target hidden feature according to the to-be-confirmed hidden features obtained by each iterative optimization operation;

[0098] A target densification unit 604, configured to perform depth estimation by using the target hidden features to obtain a target dense depth image of the target scene.

[0099] Wherein, the depth image acquisition device 600 may include the foregoing depth image model, please refer to Figure 3 , the initial densification unit 602 may correspond to the feature extraction module and the first feedback module of the depth image model, the iterative optimization unit 603 may correspond to other feedback modules except the first feedback module in the depth image model, and the target densification unit 604 may correspond to the target depth estimation module of the depth image model.

[0100] In some embodiments of the present application, the initial densification unit 602 may specifically be configured to: perform depth estimation on the initial features to obtain an initial dense depth image; fuse the initial dense depth image with the sparse depth image to obtain an initial fused feature image; and determine the initial hidden features by using the initial fused feature image, the initial features, and the color features.

[0101] In some embodiments of the present application, the image acquisition unit 601 may specifically be configured to: acquire a color image and point cloud data of the target scene; and project the point cloud data onto the imaging plane of the color image to obtain the sparse depth image.

[0102] In some embodiments of the present application, the iterative optimization unit 603 may specifically be configured to: perform at least one iterative optimization operation on the initial dense depth image by using the initial hidden features, calculate an error index between the to-be-confirmed hidden features output by the current iterative optimization operation and the to-be-confirmed hidden features output by the previous iterative optimization operation after each iterative optimization operation, if the error index is outside the error threshold range, continue to perform the next iterative optimization operation until the error index is within the error threshold range, stop performing the iterative optimization operation, and use the to-be-confirmed hidden features output by the current iterative optimization operation as the target hidden features.

[0103] In some embodiments of the present application, the above-mentioned acquisition device 600 of the depth image may further include a training unit, which can be used to: acquire a sample color image, a sample sparse depth image, and a corresponding reference dense depth image; input the sample color image and the sample sparse depth image into the network to be trained, and acquire the sample dense depth image output by each feedback module in the network to be trained, and the sample target dense depth image output by the target depth estimation module in the network to be trained; calculate a target error value according to the sample target dense depth image, each of the sample dense depth images, and the reference dense depth image. If the target error value is greater than the error threshold, adjust the parameters of the network to be trained to iteratively optimize the network to be trained until the target error value is less than or equal to the error threshold, and use the network to be trained as the depth image model.

[0104] In some embodiments of the present application, the above-mentioned training unit may specifically be used to: extract the color feature and the initial feature of the color image through the feature extraction module in the network to be trained; obtain an initial dense depth image and an initial hidden feature according to the color feature, the initial feature, and the sparse depth image through the first feedback module in the network to be trained; perform an iterative optimization operation respectively through other feedback modules in the network to be trained, and output the sample dense depth image obtained by each iterative optimization operation.

[0105] In some embodiments of the present application, the above-mentioned training unit may specifically be used to: calculate the initial error values between the sample target dense depth image and each of the sample dense depth images and the reference dense depth image respectively; perform weighted averaging on the initial error values to obtain the target error value.

[0106] It should be noted that for the convenience and conciseness of description, the specific working process of the above-mentioned acquisition device 600 of the depth image can refer to Figures 1 to 5 the corresponding process of the method, which will not be elaborated here.

[0107] An embodiment of the present application further provides a depth system, which specifically includes a color module, a depth module, and the aforementioned depth image acquisition device 600. Among them, the color module is used to collect a color image of a target scene; the depth module is used to scan the target scene to obtain point cloud data, and obtain a sparse depth image based on the point cloud data; the acquisition device uses the color image and the sparse depth image to obtain a target dense depth image. It should be noted that the color module includes a color camera, and the depth module includes, but is not limited to, any one of a lidar, a direct time-of-flight (dTof) camera, and an indirect time-of-flight (iTof) camera with speckles; the color module, the depth module, and the acquisition device can be an integrated device or independently set, and the data between the components can be transmitted wired or wirelessly, which is not limited here. The specific working process of the depth system can be referred to Figures 1 to 6 for description, which will not be elaborated in this application.

[0108] As Figure 7 shown, it is a schematic diagram of a terminal provided by an embodiment of the present application. The terminal 7 may include: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a depth image acquisition program. When the processor 70 executes the computer program 72, it implements the steps in the above-mentioned embodiments of the depth image acquisition method, such as Figure 1 the steps S101 to S104 shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-mentioned device embodiments, such as Figure 6 the image acquisition unit 601, the initial densification unit 602, the iterative optimization unit 603, and the target densification unit 604 shown.

[0109] The computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal.

[0110] For example, the computer program can be divided into: an image acquisition unit, an initial densification unit, an iterative optimization unit, and a target densification unit.

[0111] The specific functions of each unit are as follows:

[0112] An image acquisition unit for acquiring a color image and a sparse depth image of a target scene; an initial densification unit for extracting color features and initial features of the color image, and obtaining an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image; an iterative optimization unit for performing at least one iterative optimization operation on the initial dense depth image by using the color features, the sparse depth image, and the initial hidden features, and confirming a target hidden feature according to the to-be-confirmed hidden features obtained in each iterative optimization operation; and a target densification unit for performing depth estimation by using the target hidden feature to obtain a target dense depth image of the target scene.

[0113] The terminal may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art can understand that Figure 7 These are merely examples of the terminal and do not constitute a limitation on the terminal. It may include more or fewer components than shown in the figure, or combine certain components, or have different components. For example, the terminal may further include input / output devices, network access devices, a bus, etc.

[0114] The so-called processor 70 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0115] The memory 71 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. The memory 71 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 71 may also include both the internal storage unit and the external storage device of the terminal. The memory 71 is used to store the computer program and other programs and data required by the terminal. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0116] It should be noted that for the convenience and brevity of description, the structure of the above terminal can also refer to the specific description of the structure in the method embodiments, which will not be elaborated here.

[0117] Those skilled in the art can clearly understand that for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be elaborated here.

[0118] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0119] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0120] In the embodiments provided in this application, it should be understood that the disclosed device / terminal and method can be implemented in other ways. For example, the device / terminal embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0121] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional unit in the various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0123] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0124] The above-mentioned embodiments are only used to illustrate the technical solutions of this application, rather than to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A method for obtaining a depth image, characterized in that, Including: Obtain a color image and a sparse depth image of a target scene; Extract the color features and initial features of the color image, and obtain an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image; Utilize the color features, the sparse depth image, and the initial hidden features to perform at least one iterative optimization operation on the initial dense depth image, and confirm the target hidden features according to the to-be-confirmed hidden features obtained from each iterative optimization operation; Perform depth estimation using the target hidden features to obtain the target dense depth image of the target scene; Among them, the obtaining of the initial dense depth image and the initial hidden features according to the color features, the initial features, and the sparse depth image includes: Perform depth estimation on the initial features to obtain an initial dense depth image; Fuse the initial dense depth image and the sparse depth image to obtain an initial fused feature image; Utilize the initial fused feature image, the initial features, and the color features to determine the initial hidden features, and the initial hidden features are used for feature optimization feedback on the initial dense depth image.

2. The method for obtaining a depth image according to claim 1, wherein The obtaining of the color image and the sparse depth image of the target scene includes: Obtain the color image and point cloud data of the target scene; Project the point cloud data onto the imaging plane of the color image to obtain the sparse depth image.

3. The method for obtaining a depth image according to claim 1, wherein The utilizing of the color features, the sparse depth image, and the initial hidden features to perform at least one iterative optimization operation on the initial dense depth image, and confirm the target hidden features according to the to-be-confirmed hidden features obtained from each iterative optimization operation includes: Perform at least one iterative optimization operation on the initial dense depth image using the initial hidden features. After each iterative optimization operation, calculate the error index between the to-be-confirmed hidden feature output by the current iterative optimization operation and the to-be-confirmed hidden feature output by the previous iterative optimization operation. If the error index is outside the error threshold range, continue with the next iterative optimization operation until the error index is within the error threshold range, and stop the iterative optimization operation. Take the to-be-confirmed hidden feature output by the current iterative optimization operation as the target hidden feature.

4. The method for obtaining a depth image according to claim 1, wherein The method for obtaining the depth image is executed by a depth image model obtained through pre-training; Among them, the training process of the depth image model includes: Obtain a sample color image, a sample sparse depth image, and a corresponding reference dense depth image; Input the sample color image and the sample sparse depth image into the network to be trained, obtain the sample dense depth image output by each feedback module in the network to be trained, and the sample target dense depth image output by the target depth estimation module in the network to be trained; Calculate a target error value according to the sample target dense depth image, each of the sample dense depth images, and the reference dense depth image. If the target error value is greater than an error threshold, adjust the parameters of the network to be trained to iteratively optimize the network to be trained until the target error value is less than or equal to the error threshold, and use the network to be trained as the depth image model.

5. The method for obtaining a depth image according to claim 4, wherein The obtaining of the sample dense depth images output by each feedback module in the network to be trained includes: Extract the color features and initial features of the color image through a feature extraction module in the network to be trained; Obtain an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image through the first feedback module in the network to be trained; Perform an iterative optimization operation once through other feedback modules in the network to be trained, and output the sample dense depth images obtained from each iterative optimization operation.

6. The method for obtaining a depth image according to claim 4, wherein The calculating of the target error value according to the sample target dense depth image, each of the sample dense depth images, and the reference dense depth image includes: Calculate the initial error values between the sample target dense depth image and each of the sample dense depth images and the reference dense depth image respectively; Perform a weighted average on the initial error values to obtain the target error value.

7. An apparatus for obtaining a depth image, characterized in that, It includes: An image acquisition unit configured to acquire a color image and a sparse depth image of a target scene; An initial densification unit configured to extract the color features and initial features of the color image, and obtain an initial dense depth image and initial hidden features according to the color features, the initial features, and the sparse depth image; An iterative optimization unit configured to perform at least one iterative optimization operation on the initial dense depth image by using the color features, the sparse depth image, and the initial hidden features, and confirm target hidden features according to the hidden features to be confirmed obtained from each iterative optimization operation; A target densification unit configured to perform depth estimation by using the target hidden features to obtain a target dense depth image of the target scene; Wherein, the initial densification unit is specifically configured to: perform depth estimation on the initial features to obtain an initial dense depth image; fuse the initial dense depth image and the sparse depth image to obtain an initial fused feature image; Determine the initial hidden features by using the initial fused feature image, the initial features, and the color features, and the initial hidden features are used to perform feature optimization feedback on the initial dense depth image.

8. A depth system, characterized in that, It includes a color module, a depth module, and the obtaining device as claimed in claim 7, wherein: The color module is configured to collect a color image of a target scene; The depth module is configured to scan the target scene to obtain point cloud data, and obtain a sparse depth image according to the point cloud data; The obtaining device is configured to obtain a target dense depth image by using the color image and the sparse depth image.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the obtaining method as claimed in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the acquisition method according to any one of claims 1 to 6.

Images