Mobile equipment depth perception method and device fusing ToF and dual-vision detection

Abstract

The invention discloses a mobile device depth perception method and device fusing ToF and stereo matching, and the method comprises the steps: constructing a multi-modal fusion network model which comprises a TM module, an SM module and a depth fusion module which are connected in sequence, the TM module extracts ToF low-order features and high-order features according to a ToF depth map and a normal map, and the depth fusion module carries out the depth fusion of the ToF low-order features and the high-order features; an SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, the high-order features are used for indicating depth errors, and detection results are fused through a depth fusion module; training the model by using the final mask label and the depth error indication label; and obtaining a ToF depth map, a normal map, a cost volume and a disparity map of the detected equipment, and inputting the ToF depth map, the normal map, the cost volume and the disparity map into the trained model to obtain a depth perception detection result. According to the method, ToF and stereo matching detection can be fully fused, and the precision, efficiency and real-time performance of depth perception are improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional perception technology for mobile devices, and in particular to a depth perception method and device for mobile devices integrating ToF and dual-vision detection. Background Art

[0002] Accurate depth perception plays a vital role in a variety of 3D perception applications on mobile devices, especially depth-dependent tasks such as 3D reconstruction, 3D segmentation, 3D pose estimation, AR, and scene understanding. Without accurate depth values, it is difficult for applications to provide the best user experience. Depth perception on mobile devices usually relies on RGB-based stereo matching methods or time-of-flight (ToF) sensors, but the application of these two methods has limitations. For example, stereo matching methods cannot identify corresponding pixels in areas with weak textures or uneven lighting, and mobile ToF sensors cannot measure the depth of surfaces with specular reflections, low-reflective materials, and objects parallel to the optical axis. Therefore, the depth perception function may fail in real-world scenarios, greatly limiting its applicability.

[0003] If you want to combine the ToF detection method with the stereo matching method, the following two fusion methods are usually used:

[0004] One is a fusion method based on deep learning, which uses deep learning to learn multimodal data from a data set to estimate scene depth. However, the fusion method based on deep learning needs to rely on a large amount of data training process, and the generalization ability is limited. In particular, when encountering significantly different scene combinations, lighting changes, or surface reflectivity characteristics, the performance will be significantly reduced. In addition, in the case of a large amount of missing ToF depth, due to the ineffective ability to extract spatial structure information, it will also lead to poor depth interpolation and filling quality;

[0005] The other is a fusion method based on ToF confidence (Time of Flight Confidence). This method performs depth fusion by analyzing the quality of RGB images and ToF depth maps. ToF depth information and stereo matching depth information belong to information of different modalities. The fusion method based on ToF confidence simply uses interpolation to merge cross-modal data, which makes it difficult to fully utilize the specific characteristics of different modal data, resulting in the actual accuracy of the ToF confidence evaluation method is not high. Summary of the invention

[0006] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a depth perception method and device for a mobile device that integrates ToF and dual vision detection, which has a simple implementation method, high perception accuracy and efficiency, and strong real-time and generalization capabilities.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A depth perception method for a mobile device integrating ToF and stereo matching, comprising the following steps:

[0009] Constructing a multimodal fusion network model, the multimodal fusion network model includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to a ToF depth map and a normal map, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and the ToF high-order features are used to indicate a ToF depth error, the SM module extracts stereo matching low-order features and high-order features according to a cost volume and a disparity map, and the stereo matching high-order features are used to indicate a stereo matching depth error, and the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain a final depth detection result;

[0010] The multimodal fusion network model is trained using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates a ToF depth error value and a stereo matching depth error value;

[0011] The ToF depth map detected by the ToF sensor on the device under test is obtained, and a normal map is obtained according to the detected ToF depth map. The dual-view RGB image captured by the dual cameras on the mobile device is obtained, and the cost volume and disparity map are calculated according to the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

[0012] Further, when extracting ToF low-order features, the ToF depth map and the normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain the ToF low-order features;

[0013] When extracting low-order features for stereo matching, two layers of 3×3 convolution are used to extract features from the cost volume and the disparity map, respectively, and the features extracted from the cost volume and the disparity map are fused to obtain the low-order features for stereo matching.

[0014] Furthermore, a symmetrical U-shaped learner is used to extract ToF high-order features and / or extract stereo matching high-order features. The U-shaped learner captures local spatial information through a convolutional layer, and then globally combines and maps to output categories in a nonlinear manner through a fully connected layer, and uses skip connections to retain the spatial information of the low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

[0015] Furthermore, the loss function of the TM module during training is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, and to locate the edge of the object according to the gradient of the pixels in the ToF depth map, and to assign specified weights to the pixels at the edge of the object, so that the training process focuses on the prediction corresponding to the depth error indication label on the edge of the object.

[0016] Furthermore, the overall loss function L used by the TM module during training is ALL for:

[0017] L ALL =αL EW +βL LC

[0018]

[0019] Among them, L EW is the edge weighted TM loss, L LC It is a local consistency loss based on the normal map. and denote the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to the adjacent pixels, It represents the variance of the surface normal angle between each pixel and its neighboring pixels, and N is the total number of pixels in the dataset.

[0020] Furthermore, the depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result, including: calculating the ToF depth error and the stereo matching depth error of each pixel and comparing them, selecting the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result, if the ToF depth error is less than the stereo matching depth error, marking the pixel with a first mark T to select the ToF detection result as the fusion structure, otherwise marking the pixel with a second mark S to select the stereo matching detection result as the fusion result, and obtaining the final depth detection result from the fusion results of all pixels.

[0021] A depth sensing device for a mobile device integrating ToF and stereo matching, comprising:

[0022] A multimodal fusion network model construction component, wherein the multimodal fusion network model includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to a ToF depth map and a normal map, respectively, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and uses the ToF high-order features to indicate a ToF depth error value, the SM module extracts stereo matching low-order features and high-order features according to a cost volume and a disparity map, and uses the stereo matching high-order features to indicate a stereo matching depth error value, and the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain a final depth detection result output;

[0023] A model training component, used to train the multimodal fusion network model using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel to select a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label indicating a ToF depth error and a stereo matching depth error state;

[0024] The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor installed on the device under test, and convert the detected ToF depth map into a normal map, and obtain the dual-view RGB image collected by the dual cameras installed on the mobile device, and calculate the cost volume and disparity map based on the dual-view images, and input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain the depth perception detection result.

[0025] Furthermore, the TM module includes two feature extraction networks, a first U-shaped learner, and a first splicing unit connected in sequence. The two feature extraction networks are partially connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module.

[0026] The SM module includes two layers of 3×3 convolutions, a second U-shaped learner, and a second splicing unit connected in sequence. The two layers of 3×3 convolutions extract features from the cost volume and the disparity map, and fuse the features extracted from the cost volume and the disparity map to obtain the stereo matching low-order features, which are output to the second U-shaped learner; the second U-shaped learner extracts the stereo matching high-order features, and the second splicing unit splices the stereo matching low-order features with the stereo matching high-order features to obtain the stereo matching features and outputs them to the deep fusion module;

[0027] The first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers, and the convolutional layers are arranged alternately with the fully connected layers. Local spatial information is captured by the convolutional layers, and then globally combined and mapped to output categories in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of low-level feature maps and directly pass the spatial information to subsequent convolutional layers.

[0028] Furthermore, the depth fusion module includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. After the ToF features extracted by the TM module and the stereo matching features extracted by the high-order and SM modules pass through the convolutional layer and the fully connected layer in sequence, the fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them, and selects the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result.

[0029] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] 1. The present invention constructs an end-to-end lightweight multimodal fusion network. The TM module uses the ToF depth map and the normal map to extract low-order features and high-order features, and uses the high-order features to evaluate the ToF depth error. The SM module uses the cost volume and its disparity map to extract low-order and high-order features of stereo matching, and uses the high-order features to evaluate the depth error of stereo matching. After cross-modal fusion, the final detection result is obtained. The specific features of different modalities can be fully utilized to enable the model to have the ability to indicate the depth error of ToF and stereo matching. At the same time, the model is trained by combining two types of labels, namely the depth error indication and the final mask, so that better depth measurement values ​​can be selected from ToF and stereo matching at the pixel granularity, and inaccurate depth interpolation can be avoided, thereby improving the depth perception accuracy and real-time performance. The TM module and the SM module learn common features by combining physical priors, and learn based on the physical priors between depth errors and environmental factors, which can also provide strong model interpretability and generalization capabilities.

[0032] 2. The TM module and SM module of the present invention further form a specific loss function by analyzing the physical relationship between environmental factors and ToF depth error and stereo matching depth error, so that the model can learn the unique feature representation of each factor and map them to the depth errors in ToF and stereo matching. By combining ToF features, stereo matching and depth information for cross-modal fusion, better depth measurement results can be selected quickly and accurately from an overall perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram comparing the depth measurement and three-dimensional reconstruction effects obtained in a specific application embodiment.

[0034] Figure 2 It is a schematic diagram of the implementation flow of the depth perception method of a mobile device integrating ToF and stereo matching in this embodiment.

[0035] Figure 3 FIG. 4 is a schematic diagram of the specific structural principle of the TM module in this embodiment.

[0036] Figure 4 It is a schematic diagram of the specific structural principle of the SM module in this embodiment.

[0037] Figure 5 It is a schematic diagram of the specific structural principle of the deep fusion module in this embodiment. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0039] As shown in the disclosure of the present invention, unless the context clearly indicates an exception, the words "one", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The words "first", "second" and similar words used in the disclosure of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0040] For ease of understanding, the relevant technical background of the present invention is first introduced by way of example.

[0041] In the absence of accurate depth values, 3D reconstruction is prone to produce flawed and incomplete models. Figure 1 As shown in the figure, the first row of images corresponds to the depth map obtained from various depth measurement methods, and the second row corresponds to the 3D reconstruction effect obtained by using the obtained depth map as input. The 3D reconstruction method uses InfiniTAM. Since the dark blue part in the ToF depth map is depth loss, the corresponding effect cannot be generated in the 3D reconstruction and is therefore black. Therefore, accurate depth perception is particularly important for the 3D perception function of mobile devices.

[0042] Since ToF sensors perform well in weak texture areas, and stereo matching methods are highly robust to low object reflectivity, the complementary advantages of dual RGB-based stereo matching methods and time-of-flight sensors can be used to fuse the depth of dual RGB cameras and ToF to obtain more accurate depth. In the prior art, two methods are usually used: deep data fusion based on deep learning and deep fusion based on ToF confidence. Specifically:

[0043] 1) Deep data fusion based on deep learning

[0044] Deep learning-based fusion algorithms usually use supervised learning methods to estimate scene depth by learning multimodal data from data sets. For example, a self-attention mechanism is used to focus on more important areas in the scene based on image semantics and geometric features, or a ToF signal and RGB image containing original noise are used as input to obtain a high-quality long-distance depth map through a multi-step fusion method to effectively address the challenges posed to ToF imaging by strong ambient light and long-distance signal attenuation. A guided convolutional network structure can also be used to enhance the generalization ability of information fusion in different scenarios, or dual encoder branches can be used to extract features of RGB images and ToF depth maps respectively, and an end-edge collaborative architecture can be used to accelerate the deep fusion model.

[0045] However, since the deep data fusion method based on deep learning uses a deep neural network to predict missing or inaccurate depth by learning the relationship between the known depth area in the ToF depth map and its surrounding area in the RGB image, however, in the case of a large amount of missing ToF depth, this type of method will result in poor depth interpolation and filling quality due to its limited ability to extract spatial structural information. In addition, deep data fusion based on deep learning usually requires a large amount of training data to understand the mapping between the missing depth and RGB images. This feature-based learning method has limited generalization ability when faced with insufficient or weakly representative training data, especially when encountering significantly different scene combinations, lighting changes, or surface reflectivity characteristics, which will lead to a significant drop in performance. In addition, due to the complexity of deep learning models, if deployed on mobile devices, they will run slowly and have poor real-time performance. For example, the optimal InDepth model has a delay of 301ms on Huawei P40Pro, which cannot meet real-time requirements.

[0046] 2) Deep fusion method based on ToF confidence

[0047] ToF confidence is an indicator used to measure the quality of depth data generated by a ToF camera. It reflects the credibility of the ToF camera's measurement results (i.e., depth values), that is, the reliability or accuracy of the depth data. Confidence is usually calculated based on physical properties related to the ToF camera acquisition process, such as signal strength, amplitude, etc. Traditional depth fusion methods based on ToF confidence usually implement weighted interpolation to fuse the depth information of ToF and stereo matching based on the confidence of the ToF camera. A higher confidence means that the depth estimation of the pixel is more reliable. Therefore, in areas with higher ToF confidence, more reliance is placed on the depth output of the ToF camera, while in areas with lower confidence, reliance is placed on the depth estimation of stereo matching.

[0048] The traditional confidence-based deep fusion method can also perform deep fusion by analyzing the quality of RGB images and ToF depth maps. For example, assuming that the stereo matching acquisition system and the ToF sensor system have been calibrated, the depth is obtained in four steps: the depth information obtained by the ToF sensor is first projected onto the reference stereo camera perspective; a high-resolution depth map is calculated by the SGM stereo matching algorithm; the stereo disparity and the confidence of the ToF depth map are estimated using the CNN network; the upsampled ToF output results and the stereo disparity are fused using the improved local consistency framework. Or through a joint training method, the ToF disparity map and the binocular stereo matching disparity map are input into the neural network to learn the confidence estimation, and then the confidence estimation is used to guide the fusion of the depth data from the ToF sensor and stereo matching by deploying the fusion framework.

[0049] However, the above confidence-based depth fusion method simply uses interpolation to merge cross-modal data, which makes it difficult to fully utilize the specific features of different modal data. The depth information provided by the ToF camera is usually calculated based on the time of flight (ToF), while the stereo camera uses parallax for depth estimation. The depth information captured by the two methods comes from different sources and has different noise characteristics. ToF is very sensitive to the material and surface characteristics of the object, while stereo matching is more affected by the texture distribution and lighting conditions of the object. The traditional ToF confidence-based depth fusion method simply uses interpolation to merge cross-modal data, and does not make full use of these differences, resulting in the loss of modality-specific features, affecting the fusion effect of ToF and stereo matching depth. In addition, the accuracy of the traditional ToF confidence evaluation method is also insufficient. The traditional amplitude-based ToF confidence can only reflect the trend of the depth error distribution, but cannot accurately describe the error between the depth of each pixel and the actual depth. This is because the ToF depth error of the pixel is approximately Gaussian, and its standard deviation is proportional to the signal amplitude. The confidence cannot accurately reflect the actual depth error, which leads to inaccurate weights and depth accuracy during the fusion process.

[0050] The present invention designs an end-to-end lightweight multimodal fusion network FuseNet, which includes three different submodules: TM (ToF depth error indication network) module, SM (stereo depth error indication network) module and lightweight depth fusion module (FM). The TM module uses the ToF depth map and normal map as input. In addition to extracting low-order features, it also extracts high-level features of the main influencing factors, including object distance, angle and scene structure (object edges and planes, etc.), and evaluates the ToF depth error with high-order features to accurately evaluate the quality of ToF depth according to the main factors affecting the depth error; the SM module uses the cost volume and its disparity map as input to extract low-order and high-order features of stereo matching respectively, and evaluates the depth error of stereo matching with high-order features to reveal their relationship with the depth error of stereo matching. The FM module cross-complexes the outputs of the TM module and the SM module. Modal fusion can make full use of the specific features of different modalities, enabling the model to have the ability to indicate depth errors through ToF and stereo matching. At the same time, the model is trained in a targeted manner by combining two types of labels, depth error indication and final mask. This enables better depth measurement values ​​to be selected from ToF and stereo matching at the pixel granularity, effectively avoiding inaccurate depth interpolation, thereby improving depth perception accuracy and real-time performance. The TM module and the SM module learn common features by combining physical priors, and learn based on the physical priors between depth errors and environmental factors, which can also provide strong model interpretability and generalization capabilities.

[0051] like Figure 2As shown, the steps of the depth perception method of a mobile device integrating ToF and stereo matching in this embodiment include:

[0052] Step S01. Construct a multimodal fusion network model FuseNet, which includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, and the ToF high-order features include object distance, angle and scene structure features, and the scene structure includes object edge and / or plane features, and uses the ToF high-order features to indicate the ToF depth error, and the SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error, and the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result.

[0053] In this embodiment, the depth map and normal map are used as inputs of the TM module to extract the main factors affecting the ToF depth, including object distance, SoT angle, and scene structure (such as object edges and planes, etc.). Considering that the depth error not only increases with the increase of object distance due to path loss, but also significantly presents a sinusoidal pattern change; the angle has a significant impact on the depth error, especially in the small angle area; the edge of the object brings a higher depth error; adjacent pixels on the same plane have similar depth errors; the effect of ambient light on the mobile ToF depth error is negligible. In this embodiment, object distance, angle, and scene structure are taken as the main factors affecting the error of the mobile ToF sensor, where the scene structure includes object edges and planes, etc.

[0054] In this embodiment, the specific architecture of the TM module is as follows: Figure 3 As shown in the figure, a dual-branch structure is adopted, and each branch uses ResNet-34 (optional) to extract information from the depth map and normal map respectively, and extract the low-order features of ToF depth. Then, a symmetric U-shaped learner is used to capture the relationship between the ToF depth error and the input factors, and the high-order features of ToF depth are obtained and used as the ToF depth error indicator feature (TM-indicator) as an online indicator of the ToF depth error size.

[0055] Among them, the U-shaped learner is specifically composed of 3×3 convolution and fully connected layers, and uses operations such as upsampling, downsampling, and skip connections. The convolution layer is used to effectively capture the local spatial information of object edges, planes, and other features. This information is then globally combined in a nonlinear manner through the fully connected layer and mapped to the output category. At the same time, upsampling and downsampling operations are used to change the resolution of the feature map, enabling the module to process data of different scales. In addition, skip connections are used to retain the spatial information of the low-level feature maps and pass them directly to subsequent convolutional layers to help the module better learn the original information of the input data, thereby improving the overall accuracy. Finally, the TM module concatenates the low-order features and high-order features of the ToF depth through the concat operation to form the ToF feature, which is output to the FM module.

[0056] Specifically, when the TM module extracts ToF low-order features, the ToF depth map and the normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain ToF low-order features; a symmetrical U-shaped learner is used to extract ToF high-order features and / or extract stereo matching high-order features. The U-shaped learner captures local spatial information through a convolutional layer, and then globally combines and maps it to the output category in a nonlinear manner through a fully connected layer, and uses skip connections to retain the spatial information of the low-level feature maps, and directly passes the spatial information to subsequent convolutional layers.

[0057] The quality of the cost volume and disparity has a direct impact on the depth error in stereo matching, and the depth error can be evaluated using features in the cost volume or disparity. This embodiment uses a learning-based SM module and uses the cost volume and its related disparity map as input to reveal the relationship between the cost volume, disparity map and stereo depth error.

[0058] like Figure 4 As shown in the figure, the SM module is similar to the TM module in design, but because the relationship between stereo depth error, cost volume and disparity is relatively simple, the SM module is more lightweight and can use two layers of 3×3 convolution to extract low-order features from the cost volume and disparity map respectively. Then, the extracted low-order features are merged and input into the simplified U-shaped learner. Unlike the TM module, each convolution module of the U-shaped learner in the SM module contains only one convolution layer. The SM module can be trained using loss functions such as multi-class cross entropy, and high-order features are generated and used as stereo matching depth error indicator features SM-indicator to evaluate the error between stereo depth and actual depth. Finally, the SM module concatenates the low-order features and high-order features of the stereo matching depth through the concat operation to form stereo matching features, which are output to the FM module.

[0059] The FM module is used to combine the high-order and low-order features in the TM module and the SM module. The high-order features (generated by the TM and SM modules) guide the FM module to capture the degree of depth error of the ToF and stereo matching inputs, as well as the main factors affecting the error. On the other hand, the low-order features (the original inputs of the two error indicator network modules) can provide some basic information, such as the gradient of the depth value, the gradient of the surface normal, the divergence and curl in the disparity map, etc., which are not captured by the high-level features due to the information loss during the network propagation process.

[0060] It should be noted that even if the TM-indicator (ToF depth error indication) and SM-indicator (stereo matching depth error indication) values ​​are similar, they may correspond to different types of depth errors. Therefore, simply comparing the TM-indicator (ToF depth error indication) and the SM-indicator (stereo matching depth error indication) is not enough to determine the optimal depth. The FM module of this embodiment combines the TM-indicator (ToF depth error indication) and the SM-indicator (stereo matching depth error indication) by using a combination of high-order features and low-order features, and can give full play to the advantages of ToF detection and stereo matching detection, and effectively ensure the fusion accuracy.

[0061] like Figure 5 As shown, the FM module can be specifically composed of 3×3 convolution and full connection operations. The 3×3 convolution can capture the local structure and pattern in the fused data, learn complex features and extract higher-dimensional information, and the full connection layer can capture the complex nonlinear relationship between these features. Since the factors affecting the ToF and stereo matching depth errors not only span different fields, but also show a coupling relationship, higher accuracy can be ensured by using multiple layers of fully connected layers. Optionally, in this module, the FM module can use four layers of 3×3 convolution and two layers of fully connected operations to strike a balance between accuracy and latency. After 3×3 convolution and full connection operations, the ToF depth error and stereo matching depth error of each pixel are calculated and compared. According to the comparison result, the final depth detection result is selected from the ToF detection result and the stereo matching detection result. If the ToF depth error is less than the stereo matching depth error, the pixel is marked with the first mark T to select the ToF detection result as the fusion structure, otherwise it is marked with the second mark S to select the stereo matching detection result as the fusion result, and the final depth detection result is obtained by the fusion result of all pixels. That is, the network calculates the ToF and stereo matching depth error of each pixel. If the ToF depth error is small, the pixel is marked as "T", otherwise it is marked as "S". According to the label "S" or "T", a depth value is selected from the ToF and stereo matching depth as the fusion result.

[0062] Step S02. Use the final mask label and the depth error indication label to train the multimodal fusion network model, wherein the final mask label is a label for each pixel to select the final detection result from the ToF detection result and the stereo matching detection result, and the depth error indication label is a label indicating the ToF depth error value and the stereo matching depth error value. The depth error is the error between the depth obtained by ToF or stereo matching detection and the actual depth.

[0063] During the model training process of this embodiment, two different types of labels are used to conduct targeted training on the model, including the final mask label and the deep error indication label. The dual-label design enables the lightweight multimodal fusion network FuseNet to effectively extract discernible patterns from different input domains, thereby improving its overall accuracy. Specifically:

[0064] A. Final mask label: The goal of the lightweight multimodal fusion network FuseNet is to determine the best depth choice for each pixel, that is, using ToF sensor depth or stereo matching depth, which is a binary classification task. To do this, it is necessary to assign a numerical label "S" (stereo matching) or "T" (ToF) to the training model in the dataset as the final mask label.

[0065] B. Depth error indication label: By introducing depth error indication labels for the TM module and the SM module, the relationship between the influencing factors and the depth error can be accurately captured. The depth error indication label corresponds to the depth error value.

[0066] Considering that too many categories will lead to data sparsity, it will hinder the model from learning category-specific features, thus affecting training convergence and overall performance. On the contrary, too few categories will limit the model's grasp of details, so that its learning is limited to general patterns, which hinders accurate pattern recognition. This embodiment analyzes the distribution of the number of pixels with different depth error values ​​in the data set, and specifically classifies the depth error indication labels into 8 levels (0 to 7), ensuring that the distribution of the classified data is balanced, as shown in the table.

[0067] Table 1: Relationship between depth error indication label and depth error

[0068]

[0069] It is understandable that the number of division levels and the division of the depth error indication label can be configured in different ways according to actual needs.

[0070] Step S03. Obtain a ToF depth map detected by the ToF sensor mounted on the device under test, and convert the detected ToF depth map to obtain a normal map, and obtain a dual-view RGB image captured by the dual cameras mounted on the mobile device, and calculate the cost volume and disparity map based on the dual-view images, and input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain a depth perception detection result.

[0071] In this embodiment, the loss function of the TM module during the training process is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, and locate the edge of the object according to the gradient of the pixels in the ToF depth map, and assign specified weights to the pixels on the edge of the object, so that the training process focuses on the prediction corresponding to the depth error indication label on the edge of the object.

[0072] Optionally, the overall loss function L used by the TM module during training ALL It can be expressed as:

[0073] L ALL =αL EW +βL LC (1)

[0074] Among them, L EW is the edge weighted TM loss, L LC is a local consistency loss based on the normal map, and α and β are coefficients that measure the contribution of each component.

[0075] That is, the overall loss function for training the TM module consists of two components, which are EW The module training is guided to minimize the gap between the predicted ToF depth error indication label level and the true value. In addition, the object edge is located according to the gradient of the pixels in the ToF depth map, and then the pixels at the edge of the object are given a higher weight, so that the module training pays more attention to the prediction of the depth error indication label level on the object edge.

[0076] In the normal map, adjacent pixels on the same plane can be distinguished according to the difference in their normal angles. These pixels usually have continuous depth measurements and similar depth errors. Therefore, the change in depth error of pixels with the same texture on the same plane is consistent with the change in the normal angle of the pixel surface. In order to capture this point, this embodiment further calculates L in the following way: LC , so that adjacent pixels on the same plane but with large prediction error differences are given higher weights:

[0077]

[0078] and denotes the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to its 8 (configurable) neighboring pixels, represents the variance of the surface normal angles between each pixel and its neighboring pixels, where N is the total number of pixels in the dataset.

[0079] In this embodiment, the TM module and the SM module learn common features by combining physical priors, and learn based on the physical priors between depth errors and environmental factors. At the same time, a specific loss function is formed by analyzing the physical relationship between environmental factors and ToF depth errors and stereo matching depth errors, so that the model can learn the unique feature representation of each factor and map them to the depth errors in ToF and stereo matching, that is, the depth error indication features. Finally, the FM module combines ToF features, stereo matching and depth information for cross-modal fusion, so that better depth measurement results can be selected from an overall perspective.

[0080] In order to further improve the real-time computing efficiency of the system, a module adaptive parallel execution method can be further adopted to arrange the modules to run in parallel on suitable processors, and consider the differences in computing power of different mobile devices. When the system is first deployed on a mobile device, available heterogeneous processors (CPU, GPU, NPU, etc.) are searched, and each module is assigned to a specific processor based on minimizing the execution time. For example, the TM module and the FM fusion module can be run on the NPU, while the stereo matching and SM modules are deployed on the GPU. The TM module can also be configured to run on the CPU, while the remaining modules run on the GPU. Deploying the SM and TM modules on the same processor can eliminate the IO delay overhead caused by data replication. The execution order of each module can also be configured. The TM module and stereo matching can run in parallel on different computing units at the same time, and the FM module needs to run after the previous module completes execution.

[0081] The present invention avoids inaccurate depth interpolation by directly using better depth measurement of mobile ToF or stereo matching at pixel granularity, adopts a learning method based on physical prior between depth error and environmental factors, provides strong model interpretability and generalization ability, and can achieve low latency and seamless plug-and-play on mobile devices.

[0082] This embodiment also provides a depth perception device for a mobile device integrating ToF and stereo matching, including:

[0083] A multimodal fusion network model construction component, the multimodal fusion network model includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, and uses the ToF high-order features to indicate the ToF depth error value; the SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error value; the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result output;

[0084] A model training component, for training a multimodal fusion network model using a final mask label and a depth error indication label, wherein the final mask label is a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates the state of a ToF depth error and a stereo matching depth error;

[0085] The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor on the device under test, and convert the detected ToF depth map into a normal map, and obtain the dual-view RGB image collected by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

[0086] like Figure 3 As shown, the TM module in this embodiment includes two feature extraction networks, a first U-shaped learner and a first splicing unit connected in sequence. The two feature extraction networks are partially connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module.

[0087] like Figure 4 As shown, the SM module includes two layers of 3×3 convolutions, a second U-shaped learner and a second splicing unit connected in sequence. The two layers of 3×3 convolutions extract features from the cost volume and the disparity map, and the features extracted from the cost volume and the disparity map are fused to obtain low-order stereo matching features, which are output to the second U-shaped learner; the second U-shaped learner extracts high-order stereo matching features, and the second splicing unit splices the low-order stereo matching features with the high-order stereo matching features to obtain stereo matching features and outputs them to the deep fusion module.

[0088] Among them, the first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers. The convolutional layers are arranged alternately with the fully connected layers. The local spatial information is captured by the convolutional layers, and then globally combined and mapped to the output categories in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of the low-level feature maps and pass the spatial information directly to the subsequent convolutional layers.

[0089] like Figure 5 As shown, the depth fusion module in this embodiment includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. After the ToF features extracted by the TM module and the high-order and stereo matching features extracted by the SM module pass through the convolutional layer and the fully connected layer in sequence, the fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them, and selects the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result.

[0090] The depth perception device of a mobile device integrating ToF and stereo matching in this embodiment corresponds one to one with the depth perception method of a mobile device integrating ToF and stereo matching described above, and will not be described in detail here.

[0091] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0092] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0093] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0094] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A depth perception method for a mobile device integrating ToF and stereo matching, characterized in that the steps include: Constructing a multimodal fusion network model, the multimodal fusion network model includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to a ToF depth map and a normal map, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and the ToF high-order features are used to indicate a ToF depth error, the SM module extracts stereo matching low-order features and high-order features according to a cost volume and a disparity map, and the stereo matching high-order features are used to indicate a stereo matching depth error, and the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain a final depth detection result; The multimodal fusion network model is trained using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates a ToF depth error value and a stereo matching depth error value; The ToF depth map detected by the ToF sensor on the device under test is obtained, and a normal map is obtained according to the detected ToF depth map. The dual-view RGB image captured by the dual cameras on the mobile device is obtained, and the cost volume and disparity map are calculated according to the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

2. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: When extracting ToF low-order features, the ToF depth map and the normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain the ToF low-order features; When extracting low-order features for stereo matching, two layers of 3×3 convolution are used to extract features from the cost volume and the disparity map, respectively, and the features extracted from the cost volume and the disparity map are fused to obtain the low-order features for stereo matching.

3. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: A symmetric U-shaped learner is used to extract ToF high-order features and / or stereo matching high-order features. The U-shaped learner captures local spatial information through a convolutional layer, and then globally combines and maps to output categories in a nonlinear manner through a fully connected layer, and uses skip connections to retain the spatial information of the low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

4. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: The loss function of the TM module during the training process is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, locate the edge of the object according to the gradient of the pixels in the ToF depth map, and assign specified weights to the pixels on the edge of the object so that the training process focuses on the prediction corresponding to the depth error indication label on the edge of the object.

5. The depth perception method integrating ToF and dual vision detection according to claim 4, characterized in that: The overall loss function L used by the TM module during training ALL for: L ALL =αL EW +βL LC Among them, L EW is the edge weighted TM loss, L LC It is a local consistency loss based on the normal map. and denote the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to the adjacent pixels, It represents the variance of the surface normal angle between each pixel and its neighboring pixels, and N is the total number of pixels in the dataset.

6. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: The depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result, including: calculating the ToF depth error and the stereo matching depth error of each pixel and comparing them, selecting the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result, if the ToF depth error is less than the stereo matching depth error, marking the pixel with a first mark T to select the ToF detection result as the fusion structure, otherwise marking the pixel with a second mark S to select the stereo matching detection result as the fusion result, and obtaining the final depth detection result from the fusion results of all pixels.

7. A depth perception device for a mobile device integrating ToF and stereo matching, characterized in that: include: A multimodal fusion network model construction component, wherein the multimodal fusion network model includes a TM module, an SM module and a deep fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to a ToF depth map and a normal map, respectively, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and uses the ToF high-order features to indicate a ToF depth error value, the SM module extracts stereo matching low-order features and high-order features according to a cost volume and a disparity map, and uses the stereo matching high-order features to indicate a stereo matching depth error value, and the deep fusion module fuses the ToF detection result with the stereo matching detection result to obtain a final depth detection result output; A model training component, used to train the multimodal fusion network model using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel to select a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label indicating a ToF depth error and a stereo matching depth error state; The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor installed on the device under test, and convert the detected ToF depth map into a normal map, and obtain the dual-view RGB image collected by the dual cameras installed on the mobile device, and calculate the cost volume and disparity map based on the dual-view images, and input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain the depth perception detection result.

8. The depth perception device for mobile device integrating ToF and stereo matching according to claim 7, characterized in that: The TM module includes two feature extraction networks, a first U-shaped learner, and a first splicing unit connected in sequence. The two feature extraction networks are partially connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module; The SM module includes two layers of 3×3 convolutions, a second U-shaped learner, and a second splicing unit connected in sequence. The two layers of 3×3 convolutions extract features from the cost volume and the disparity map, and fuse the features extracted from the cost volume and the disparity map to obtain the stereo matching low-order features, which are output to the second U-shaped learner; the second U-shaped learner extracts the stereo matching high-order features, and the second splicing unit splices the stereo matching low-order features with the stereo matching high-order features to obtain the stereo matching features and outputs them to the deep fusion module; The first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers, and the convolutional layers are arranged alternately with the fully connected layers. Local spatial information is captured by the convolutional layers, and then globally combined and mapped to output categories in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of low-level feature maps and directly pass the spatial information to subsequent convolutional layers.

9. The depth perception device for mobile device integrating ToF and stereo matching according to claim 7 or 8, characterized in that: The depth fusion module includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. After the ToF features extracted by the TM module and the high-order and stereo matching features extracted by the SM module pass through the convolutional layer and the fully connected layer in sequence, the fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them. The final depth detection result is selected from the ToF detection result and the stereo matching detection result according to the comparison result.

10. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

Images