Depth perception method and device for mobile devices integrating ToF and dual-vision detection

Abstract

The present invention discloses a method and apparatus for depth perception of a mobile device that integrates ToF and stereo matching. The method comprises the following steps: constructing a multimodal fusion network model, the multimodal fusion network model comprising a TM module, an SM module, and a depth fusion module connected in sequence; the TM module extracts ToF low-order features and high-order features based on a ToF depth map and a normal map; the SM module extracts stereo matching low-order features and high-order features based on a cost volume and a disparity map, and uses the high-order features to indicate depth errors; the depth fusion module fuses the detection results; the model is trained using a final mask label and a depth error indication label; the ToF depth map, normal map, cost volume, and disparity map of the device under test are obtained and input into the trained model to obtain depth perception detection results. The present invention can fully integrate ToF and stereo matching detection to improve the accuracy, efficiency, and real-time performance of depth perception.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional perception technology for mobile devices, and in particular to a depth perception method and device for mobile devices that integrates ToF and dual-vision detection. Background Art

[0002] Accurate depth perception plays a crucial role in various 3D perception applications on mobile devices, especially depth-dependent tasks such as 3D reconstruction, 3D segmentation, 3D pose estimation, AR, and scene understanding. Without accurate depth values, applications struggle to provide an optimal user experience. Depth perception on mobile devices typically relies on RGB-based stereo matching methods or time-of-flight (ToF) sensors. However, both methods have limitations. For example, stereo matching methods cannot identify corresponding pixels in areas with weak textures or uneven lighting, while mobile ToF sensors cannot measure the depth of objects with specular reflections, low-reflective materials, and surfaces parallel to the optical axis. As a result, depth perception may fail in real-world scenarios, significantly limiting their applicability.

[0003] If you want to combine the ToF detection method with the stereo matching method, you usually use the following two fusion methods:

[0004] One is a fusion method based on deep learning, which uses deep learning to learn multimodal data from a dataset to estimate scene depth. However, this method relies on a large amount of data training and has limited generalization capabilities. Performance degrades significantly when encountering significantly different scene combinations, lighting changes, or surface reflectivity characteristics. Furthermore, when a large amount of ToF depth is missing, the quality of depth interpolation and filling is poor due to the limited ability to extract spatial structure information.

[0005] The other is a fusion method based on ToF confidence (Time of Flight Confidence). This method performs depth fusion by analyzing the quality of RGB images and ToF depth maps. ToF depth information and stereo matching depth information belong to information of different modalities. The fusion method based on ToF confidence simply uses interpolation to merge cross-modal data, which makes it difficult to fully utilize the specific characteristics of different modal data, resulting in the actual accuracy of the ToF confidence evaluation method is not high. Summary of the Invention

[0006] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a depth perception method and device for mobile devices that integrates ToF and dual-vision detection, which has a simple implementation method, high perception accuracy and efficiency, and strong real-time and generalization capabilities.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A depth perception method for a mobile device integrating ToF and stereo matching, comprising the following steps:

[0009] Constructing a multimodal fusion network model, the multimodal fusion network model includes a TM module, an SM module and a depth fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and uses the ToF high-order features to indicate the ToF depth error, the SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error, and the depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result;

[0010] The multimodal fusion network model is trained using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates a ToF depth error value and a stereo matching depth error value;

[0011] Obtain the ToF depth map detected by the ToF sensor on the device under test, and convert it into a normal map based on the detected ToF depth map. Obtain the dual-view RGB image captured by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. Input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain the depth perception detection result.

[0012] Furthermore, when extracting ToF low-order features, the ToF depth map and the normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain the ToF low-order features;

[0013] When extracting low-order features for stereo matching, two layers of 3×3 convolution are used to extract features from the cost volume and the disparity map respectively, and the features extracted from the cost volume and the disparity map are fused to obtain the low-order features for stereo matching.

[0014] Furthermore, a symmetric U-shaped learner is used to extract ToF high-order features and / or stereo matching high-order features. The U-shaped learner captures local spatial information through a convolutional layer, and then globally combines and maps it to the output category in a nonlinear manner through a fully connected layer. Skip connections are used to retain the spatial information of the low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

[0015] Furthermore, the loss function of the TM module during training is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, and locate the edge of the object according to the gradient of the pixels in the ToF depth map. The pixels at the edge of the object are assigned a specified weight so that the training process focuses on the prediction corresponding to the depth error indication label on the object edge.

[0016] Furthermore, the overall loss function L used by the TM module during training is ALL for:

[0017] L ALL =αL EW +βL LC

[0018]

[0019] Among them, L EW is the edge-weighted TM loss, L LC It is a local consistency loss based on the normal map, and denote the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to the adjacent pixels, It represents the variance of the surface normal angle between each pixel and its adjacent pixels, where N is the total number of pixels in the dataset.

[0020] Furthermore, the depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result, including: calculating the ToF depth error and the stereo matching depth error of each pixel and comparing them, selecting the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result, if the ToF depth error is less than the stereo matching depth error, marking the pixel with the first mark T to select the ToF detection result as the fusion structure, otherwise marking it with the second mark S to select the stereo matching detection result as the fusion result, and obtaining the final depth detection result from the fusion result of all pixels.

[0021] A depth perception device for a mobile device integrating ToF and stereo matching, comprising:

[0022] A multimodal fusion network model construction component, wherein the multimodal fusion network model includes a TM module, an SM module, and a depth fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, respectively. The ToF high-order features include object distance, angle, and scene structure features. The scene structure includes object edge and / or plane features, and the ToF high-order features are used to indicate the ToF depth error value. The SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error value. The depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result output;

[0023] A model training component is used to train the multimodal fusion network model using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates the status of a ToF depth error and a stereo matching depth error;

[0024] The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor on the device under test, and convert the detected ToF depth map into a normal map, and obtain the dual-view RGB image captured by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

[0025] Furthermore, the TM module includes two feature extraction networks, a first U-shaped learner, and a first splicing unit connected in sequence. The two feature extraction networks are partially connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module.

[0026] The SM module includes two layers of 3×3 convolution, a second U-shaped learner and a second splicing unit connected in sequence. The two layers of 3×3 convolution extract features from the cost volume and the disparity map, and fuse the features extracted from the cost volume and the disparity map to obtain the stereo matching low-order features, which are output to the second U-shaped learner; the second U-shaped learner extracts the stereo matching high-order features, and the second splicing unit splices the stereo matching low-order features with the stereo matching high-order features to obtain stereo matching features and outputs them to the deep fusion module;

[0027] The first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers. The convolutional layers are interleaved with the fully connected layers. Local spatial information is captured by the convolutional layers, and then globally combined and mapped to output categories in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

[0028] Furthermore, the depth fusion module includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. After the ToF features extracted by the TM module and the stereo matching features extracted by the high-order and SM modules pass through the convolutional layer and the fully connected layer in sequence, the fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them. According to the comparison result, the final depth detection result is selected from the ToF detection result and the stereo matching detection result.

[0029] An electronic device includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0030] Compared with the prior art, the advantages of the present invention are:

[0031] 1. The present invention constructs an end-to-end lightweight multimodal fusion network. The TM module uses the ToF depth map and normal map to extract low-order features and high-order features, and uses the high-order features to evaluate the ToF depth error. The SM module uses the cost volume and its disparity map to extract low-order and high-order features of stereo matching, and uses the high-order features to evaluate the depth error of stereo matching. After cross-modal fusion, the final detection result is obtained. It can make full use of the specific features of different modalities, so that the model has the ability to indicate the depth error of ToF and stereo matching. At the same time, the model is trained by combining two types of labels, depth error indication and final mask, so that better depth measurement values ​​can be selected from ToF and stereo matching at the pixel granularity, avoiding inaccurate depth interpolation, thereby improving depth perception accuracy and real-time performance. In addition, the TM module and the SM module learn common features by combining physical priors. Learning based on the physical prior between depth error and environmental factors can also provide strong model interpretability and generalization capabilities.

[0032] 2. The TM module and SM module of the present invention further form a specific loss function by analyzing the physical relationship between environmental factors and ToF depth error and stereo matching depth error, so that the model can learn the unique feature representation of each factor and map them to the depth error in ToF and stereo matching. By combining ToF features, stereo matching and depth information for cross-modal fusion, better depth measurement results can be selected quickly and accurately from a holistic perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 3D reconstruction is a schematic diagram comparing the depth measurement and 3D reconstruction effects obtained in a specific application embodiment.

[0034] Figure 2 This is a schematic diagram of the implementation process of the depth perception method for a mobile device that integrates ToF and stereo matching in this embodiment.

[0035] Figure 3 FIG. 4 is a schematic diagram of the specific structural principle of the TM module in this embodiment.

[0036] Figure 4 FIG. 4 is a schematic diagram of the specific structural principle of the SM module in this embodiment.

[0037] Figure 5 It is a schematic diagram of the specific structural principle of the deep fusion module in this embodiment. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0039] As shown in the disclosure of the present invention, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not specifically refer to the singular, but may also include the plural. The "first", "second" and similar words used in the disclosure of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect.

[0040] For ease of understanding, the relevant technical background of the present invention is first introduced by way of example.

[0041] In the absence of accurate depth values, 3D reconstruction is prone to produce flawed and incomplete models. Figure 1 As shown in the figure, the first row of images corresponds to depth maps obtained from various depth measurement methods, and the second row corresponds to the 3D reconstruction results obtained by using the obtained depth maps as input. The 3D reconstruction method used is InfiniTAM. Since the dark blue areas in the ToF depth map indicate depth loss, the corresponding effect cannot be generated in the 3D reconstruction and are therefore colored black. Therefore, accurate depth perception is particularly important for the 3D perception function of mobile devices.

[0042] Since ToF sensors perform well in areas with weak textures, and stereo matching methods are highly robust to low object reflectivity, we can leverage the complementary advantages of dual-RGB stereo matching methods and Time of Flight sensors to fuse the depth of dual-RGB cameras and ToF to obtain more accurate depth. Existing technologies typically use two methods: deep learning-based depth data fusion and ToF confidence-based depth fusion. Specifically:

[0043] 1) Deep data fusion based on deep learning

[0044] Deep learning-based fusion algorithms typically employ supervised learning methods to estimate scene depth by learning multimodal data from a dataset. For example, they employ a self-attention mechanism to focus on more important areas of a scene based on image semantics and geometric features, or use a ToF signal containing raw noise and an RGB image as input to obtain high-quality, long-range depth maps through a multi-step fusion approach. This effectively addresses challenges posed to ToF imaging by strong ambient light and long-range signal attenuation. Alternatively, guided convolutional network structures can be employed to enhance the generalization capabilities of information fusion in different scenarios, or dual encoder branches can be used to extract features from RGB images and ToF depth maps, respectively, and an edge-to-edge collaborative architecture can be employed to accelerate the deep fusion model.

[0045] However, deep learning-based deep data fusion methods use deep neural networks to predict missing or inaccurate depth by learning the relationship between known depth regions in the ToF depth map and their surrounding areas in the RGB image. However, when a large amount of ToF depth is missing, this type of approach will lead to poor depth interpolation and filling quality due to its limited ability to extract spatial structure information. Deep learning-based deep data fusion also requires a large amount of training data to understand the mapping between missing depth and RGB images. This feature-based learning method has limited generalization ability when faced with insufficient or weakly representative training data, especially when encountering significantly different scene combinations, lighting changes, or surface reflectivity characteristics, resulting in a significant performance degradation. In addition, due to the complexity of deep learning models, if deployed on mobile devices, they will run slowly and have poor real-time performance. For example, the optimal InDepth model has a latency of 301ms on the Huawei P40Pro, which cannot meet real-time requirements.

[0046] 2) Deep fusion method based on ToF confidence

[0047] ToF confidence is an indicator used to measure the quality of depth data generated by a ToF camera. It reflects the ToF camera's confidence in its measurement results (i.e., depth values), that is, the reliability or accuracy of the depth data. Confidence is usually calculated based on physical properties related to the ToF camera's acquisition process, such as signal strength and amplitude. Traditional depth fusion methods based on ToF confidence usually perform weighted interpolation to fuse the depth information of ToF and stereo matching based on the confidence of the ToF camera. A higher confidence level indicates that the depth estimation of the pixel is more reliable. Therefore, in areas with higher ToF confidence, more reliance is placed on the depth output of the ToF camera, while in areas with lower confidence, reliance is placed on the depth estimation of stereo matching.

[0048] Traditional confidence-based depth fusion can also be performed by analyzing the quality of RGB images and ToF depth maps. For example, assuming that the stereo matching acquisition system and the ToF sensor system are calibrated, the depth is obtained in four steps: the depth information obtained by the ToF sensor is first projected onto the reference stereo camera view; a high-resolution depth map is calculated by the SGM stereo matching algorithm; the stereo disparity and the confidence of the ToF depth map are estimated using a CNN network; and the upsampled ToF output and the stereo disparity are fused using a modified local consistency framework. Alternatively, through a joint training method, the ToF disparity map and the binocular stereo matching disparity map are input into a neural network to learn confidence estimation, and then the fusion framework is deployed to use the confidence estimation to guide the fusion of the depth data from the ToF sensor and stereo matching.

[0049] However, the aforementioned confidence-based depth fusion methods simply interpolate and merge cross-modal data, failing to fully utilize the specific characteristics of data from different modalities. Depth information provided by ToF cameras is typically calculated based on time of flight (ToF), while stereo cameras use parallax for depth estimation. The depth information captured by these two methods comes from different sources and has different noise characteristics. ToF is sensitive to the material and surface properties of an object, while stereo matching is more affected by the object's texture distribution and lighting conditions. Traditional ToF confidence-based depth fusion methods simply interpolate and merge cross-modal data, failing to fully utilize these differences. This results in a loss of modality-specific characteristics, compromising the fusion effect of ToF and stereo matching depth. Furthermore, traditional ToF confidence assessment methods are also inaccurate. Traditional amplitude-based ToF confidence measures can only reflect the trend of the depth error distribution but cannot accurately describe the error between each pixel's depth and the actual depth. This is because the ToF depth error of a pixel is approximately Gaussian, with a standard deviation proportional to the signal amplitude. The confidence cannot accurately reflect the actual depth error, which leads to inaccurate weights and depth accuracy during the fusion process.

[0050] The present invention designs an end-to-end lightweight multimodal fusion network FuseNet, which contains three different sub-modules: TM (ToF depth error indication network) module, SM (stereo depth error indication network) module and lightweight depth fusion module (FM). The TM module uses the ToF depth map and normal map as input. In addition to extracting low-order features, it also extracts high-level features of the main influencing factors, including object distance, angle and scene structure (object edges and planes, etc.). At the same time, it uses high-order features to evaluate the ToF depth error, so as to accurately evaluate the quality of ToF depth according to the main factors affecting the depth error; the SM module uses the cost volume and its disparity map as input, and extracts low-order and high-order features of stereo matching respectively. The high-order features are used to evaluate the depth error of stereo matching to reveal their relationship with the stereo matching depth error. The FM module cross-scaling the outputs of the TM module and the SM module. Modal fusion can make full use of the specific features of different modalities, enabling the model to have the ability to indicate depth errors through ToF and stereo matching. At the same time, the model is trained in a targeted manner by combining two types of labels: depth error indication and final mask. This enables better depth measurement values ​​to be selected from ToF and stereo matching at the pixel granularity, effectively avoiding inaccurate depth interpolation, thereby improving depth perception accuracy and real-time performance. The TM module and the SM module learn common features by combining physical priors. Learning based on the physical priors between depth errors and environmental factors can also provide strong model interpretability and generalization capabilities.

[0051] like Figure 2As shown, the steps of the depth perception method for a mobile device integrating ToF and stereo matching in this embodiment include:

[0052] Step S01. Construct a multimodal fusion network model FuseNet, which includes a TM module, an SM module and a depth fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features based on the ToF depth map and the normal map. The ToF high-order features include object distance, angle and scene structure features. The scene structure includes object edge and / or plane features, and the ToF high-order features are used to indicate the ToF depth error. The SM module extracts stereo matching low-order features and high-order features based on the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error. The depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result.

[0053] In this embodiment, a depth map and a normal map are used as inputs to the TM module to extract the main factors affecting ToF depth, including object distance, SoT angle, and scene structure (such as object edges and planes). Considering that depth error not only increases with object distance due to path loss but also exhibits a significant sinusoidal pattern, angle has a significant impact on depth error, especially in small angle regions, object edges lead to higher depth error, adjacent pixels on the same plane have similar depth errors, and the impact of ambient light on mobile ToF depth error is negligible, this embodiment considers object distance, angle, and scene structure as the main factors affecting mobile ToF sensor error, where scene structure includes object edges and planes.

[0054] In this embodiment, the specific architecture of the TM module is as follows: Figure 3 As shown in the figure, a dual-branch structure is adopted. Each branch uses ResNet-34 (optional) to extract information from the depth map and normal map respectively, extracting low-level features of ToF depth. Then, a symmetric U-shaped learner is used to capture the relationship between ToF depth error and input factors. The high-level features of ToF depth are obtained and used as the ToF depth error indicator feature (TM-indicator), which serves as an online indicator of the size of ToF depth error.

[0055] Among them, the U-shaped learner is specifically composed of 3×3 convolution and fully connected layers, and adopts operations such as upsampling, downsampling and skip connections. The convolution layer is used to effectively capture the local spatial information of object edges, planes and other features. This information is then globally combined in a nonlinear manner through the fully connected layer and mapped to the output category. At the same time, upsampling and downsampling operations are used to change the resolution of the feature map, enabling the module to process data of different scales. In addition, skip connections are used to retain the spatial information of the low-level feature map and pass it directly to the subsequent convolution layer to help the module better learn the original information of the input data, thereby improving the overall accuracy. Finally, the TM module uses the concat operation to splice the low-order features and high-order features of the ToF depth to form the ToF feature, which is output to the FM module.

[0056] Specifically, when the TM module extracts ToF low-order features, the ToF depth map and normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain ToF low-order features; a symmetrical U-shaped learner is used to extract ToF high-order features and / or extract stereo matching high-order features. The U-shaped learner captures local spatial information through the convolution layer, and then globally combines and maps it to the output category in a nonlinear manner through the fully connected layer, and uses skip connections to retain the spatial information of the low-level feature maps and pass the spatial information directly to the subsequent convolution layers.

[0057] The quality of the cost volume and disparity has a direct impact on the depth error in stereo matching. Features in the cost volume or disparity can be used to evaluate the depth error. This embodiment employs a learning-based SM module and uses the cost volume and its associated disparity map as input to reveal the relationship between the cost volume, disparity map, and stereo depth error.

[0058] like Figure 4 As shown in the figure, the SM module has a similar design to the TM module. However, due to the relatively simple relationship between stereo depth error, cost volume, and disparity, the SM module is more lightweight and can use two layers of 3×3 convolution to extract low-level features from the cost volume and disparity map, respectively. These extracted low-level features are then combined and input into a simplified U-shaped learner. Unlike the TM module, each convolutional module of the U-shaped learner in the SM module contains only one convolutional layer. The SM module can be trained using loss functions such as multi-class cross-entropy and generates high-level features that serve as stereo matching depth error indicators (SM-indicators) to assess the error between stereo depth and actual depth. Finally, the SM module concatenates the low-level and high-level features of the stereo matching depth through a concat operation to form stereo matching features, which are then output to the FM module.

[0059] The FM module is used to combine the high-order and low-order features in the TM module and the SM module. The high-order features (generated by the TM and SM modules) guide the FM module to capture the degree of depth error of the ToF and stereo matching inputs, as well as the main factors affecting the error. On the other hand, the low-order features (the original inputs of the two error indicator network modules) can provide some basic information, such as the gradient of the depth value, the gradient of the surface normal, the divergence and rotation in the disparity map, etc., which are not captured by the high-level features due to the information loss during the network propagation process.

[0060] It should be noted that even if the TM-indicator (ToF depth error indication) and SM-indicator (stereo matching depth error indication) values ​​are similar, they may correspond to different types of depth errors. Therefore, simply comparing the TM-indicator (ToF depth error indication) and the SM-indicator (stereo matching depth error indication) is not sufficient to determine the optimal depth. The FM module of this embodiment uses a combination of high-order features and low-order features, combined with the TM-indicator (ToF depth error indication) and the SM-indicator (stereo matching depth error indication), to give full play to the advantages of ToF detection and stereo matching detection, and effectively ensure fusion accuracy.

[0061] like Figure 5 As shown, the FM module can be specifically composed of 3×3 convolutions and fully connected operations. The 3×3 convolutions can capture local structures and patterns in the fused data, learn complex features, and extract higher-dimensional information. The fully connected layers can capture the complex nonlinear relationships between these features. Because the factors affecting ToF and stereo matching depth errors span different domains and exhibit coupled relationships, the use of multiple fully connected layers can ensure higher accuracy. Optionally, in this module, the FM module can use four layers of 3×3 convolutions and two layers of fully connected operations to strike a balance between accuracy and latency. After the 3×3 convolutions and fully connected operations, the ToF depth error and stereo matching depth error are calculated for each pixel and compared. Based on the comparison results, the final depth detection result is selected from the ToF detection results and stereo matching detection results. If the ToF depth error is less than the stereo matching depth error, the pixel is marked with the first tag T, selecting the ToF detection result as the fused structure. Otherwise, the pixel is marked with the second tag S, selecting the stereo matching detection result as the fused result. The final depth detection result is obtained by fusion of all pixels. That is, the network calculates the ToF and stereo matching depth error of each pixel. If the ToF depth error is small, the pixel is marked as "T", otherwise it is marked as "S". According to the label "S" or "T", a depth value is selected from the ToF and stereo matching depth as the fusion result.

[0062] Step S02: Train the multimodal fusion network model using the final mask label and depth error indicator label. The final mask label determines the label for each pixel that selects the final detection result from the ToF detection results and stereo matching detection results. The depth error indicator label indicates the ToF depth error value and the stereo matching depth error value. The depth error is the difference between the depth obtained by ToF or stereo matching detection and the actual depth.

[0063] During the model training process of this embodiment, two different types of labels are used for targeted model training, including the final mask label and the depth error indicator label. The dual-label design enables the lightweight multimodal fusion network FuseNet to effectively extract discernible patterns from different input domains, thereby improving its overall accuracy. Specifically:

[0064] A. Final Mask Label: The goal of the lightweight multimodal fusion network FuseNet is to determine the best depth choice for each pixel, that is, using the ToF sensor depth or stereo matching depth, which is a binary classification task. To do this, the training model needs to be assigned a numerical label "S" (stereo matching) or "T" (ToF) as the final mask label in the dataset.

[0065] B. Depth Error Indicator Label: By introducing depth error indicator labels for the TM module and SM module, the relationship between influencing factors and depth error is accurately captured. The depth error indicator label corresponds to the depth error value.

[0066] Considering that too many categories will lead to data sparsity, hindering the model from learning category-specific features, thereby affecting training convergence and overall performance. On the contrary, too few categories will limit the model's grasp of details, causing its learning to be limited to general patterns, hindering accurate pattern recognition. This embodiment analyzes the distribution of the number of pixels with different depth error values ​​in the dataset and specifically classifies the depth error indication labels into 8 levels (0 to 7) to ensure a balanced distribution of the classified data, as shown in the table.

[0067] Table 1: Relationship between depth error indication label and depth error

[0068]

[0069] It is understandable that the number of division levels and the division of the depth error indication label can be configured in different ways according to actual needs.

[0070] Step S03. Obtain a ToF depth map detected by the ToF sensor on the device under test, and convert the detected ToF depth map into a normal map, and obtain a dual-view RGB image captured by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. Input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain the depth perception detection result.

[0071] In this embodiment, the loss function of the TM module during the training process is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, and locate the edge of the object according to the gradient of the pixels in the ToF depth map, and assign specified weights to the pixels on the edge of the object, so that the training process focuses on the prediction corresponding to the depth error indication label on the edge of the object.

[0072] Optionally, the overall loss function L used by the TM module during training ALL It can be expressed as:

[0073] L ALL =αL EW +βL LC (1)

[0074] Among them, L EW is the edge-weighted TM loss, L LC is a local consistency loss based on the normal map, and α and β are coefficients that measure the contribution of each component.

[0075] That is, the overall loss function of training the TM module consists of two components, which are EW The module training is guided to minimize the gap between the predicted ToF depth error indication label level and the true value. In addition, the object edge is located according to the gradient of the pixels in the ToF depth map, and then the pixels at the edge of the object are given a higher weight, so that the module training focuses more on the prediction of the depth error indication label level on the object edge.

[0076] In the normal map, adjacent pixels on the same plane can be distinguished based on the difference in their normal angles. These pixels usually have continuous depth measurements and similar depth errors. Therefore, the depth error changes of pixels with the same texture on the same plane are consistent with the changes in the normal angle of the pixel surface. To capture this point, this embodiment further calculates L in the following way: LC , so that adjacent pixels on the same plane but with large prediction error differences are given higher weights:

[0077]

[0078] and denotes the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to its 8 (configurable) neighboring pixels, represents the variance of the surface normal angle between each pixel and its neighboring pixels, where N is the total number of pixels in the dataset.

[0079] In this embodiment, the TM module and the SM module learn common features by combining physical priors, and learn based on the physical priors between depth error and environmental factors. At the same time, a specific loss function is formed by analyzing the physical relationship between environmental factors and ToF depth error and stereo matching depth error, so that the model can learn the unique feature representation of each factor and map them to the depth error in ToF and stereo matching, that is, the depth error indication feature. Finally, the FM module combines ToF features, stereo matching and depth information for cross-modal fusion, so that better depth measurement results can be selected from a holistic perspective.

[0080] To further improve the system's real-time computing efficiency, a module-adaptive parallel execution approach can be employed to schedule modules to run in parallel on appropriate processors, taking into account the differences in computing power across mobile devices. When the system is initially deployed on a mobile device, available heterogeneous processors (CPU, GPU, NPU, etc.) are searched for, and each module is assigned to a specific processor based on minimizing execution time. For example, the TM module and FM fusion module can run on the NPU, while the stereo matching and SM modules can be deployed on the GPU. Alternatively, the TM module can be configured to run on the CPU, while the remaining modules run on the GPU. Deploying the SM and TM modules on the same processor eliminates I / O latency overhead caused by data duplication. The execution order of the modules can also be configured; the TM module and stereo matching can run in parallel on different computing units, while the FM module must execute only after the preceding modules have completed execution.

[0081] The present invention avoids inaccurate depth interpolation by directly using better depth measurement of mobile ToF or stereo matching at the pixel granularity, adopts a learning method based on the physical prior between depth error and environmental factors, provides strong model interpretability and generalization ability, and can achieve low latency and seamless plug-and-play on mobile devices.

[0082] This embodiment also provides a depth perception device for a mobile device that integrates ToF and stereo matching, including:

[0083] Multimodal fusion network model construction components. The multimodal fusion network model includes a TM module, an SM module, and a depth fusion module connected in sequence. The TM module extracts ToF low-order features and high-order features based on the ToF depth map and normal map, and uses the ToF high-order features to indicate the ToF depth error value. The SM module extracts stereo matching low-order features and high-order features based on the cost volume and disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error value. The depth fusion module fuses the ToF detection results with the stereo matching detection results to obtain the final depth detection result output;

[0084] A model training component is used to train the multimodal fusion network model using the final mask label and the depth error indication label, wherein the final mask label determines the label of the final detection result selected from the ToF detection result and the stereo matching detection result for each pixel, and the depth error indication label is a label indicating the ToF depth error and the stereo matching depth error status;

[0085] The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor on the device under test, and convert it into a normal map based on the detected ToF depth map, and obtain the dual-view RGB image captured by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

[0086] like Figure 3 As shown, the TM module in this embodiment includes two feature extraction networks, a first U-shaped learner and a first splicing unit connected in sequence. The two feature extraction networks are connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module.

[0087] like Figure 4 As shown, the SM module includes two layers of 3×3 convolution, a second U-shaped learner and a second splicing unit connected in sequence. The two layers of 3×3 convolution extract features from the cost volume and the disparity map, and fuse the features extracted from the cost volume and the disparity map to obtain low-order stereo matching features, which are output to the second U-shaped learner; the second U-shaped learner extracts high-order stereo matching features, and the second splicing unit splices the low-order stereo matching features with the high-order stereo matching features to obtain stereo matching features and outputs them to the deep fusion module.

[0088] Among them, the first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers. The convolutional layers and the fully connected layers are arranged in an interlaced manner. The local spatial information is captured by the convolutional layers, and then globally combined and mapped to the output category in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of the low-level feature maps and pass the spatial information directly to the subsequent convolutional layers.

[0089] like Figure 5 As shown, the depth fusion module in this embodiment includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. The ToF features extracted by the TM module and the stereo matching features extracted by the high-order and SM modules are sequentially passed through the convolutional layer and the fully connected layer. The fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them. The final depth detection result is selected from the ToF detection result and the stereo matching detection result according to the comparison result.

[0090] The depth perception device for a mobile device integrating ToF and stereo matching in this embodiment corresponds one-to-one to the depth perception method for a mobile device integrating ToF and stereo matching described above, and will not be described in detail here.

[0091] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0092] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing relevant programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0093] This embodiment further provides a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.

[0094] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0095] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A depth perception method for a mobile device integrating ToF and stereo matching, characterized in that the steps include: Constructing a multimodal fusion network model, the multimodal fusion network model includes a TM module, an SM module and a depth fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, the ToF high-order features include object distance, angle and scene structure features, the scene structure includes object edge and / or plane features, and uses the ToF high-order features to indicate the ToF depth error, the SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error, and the depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result; The multimodal fusion network model is trained using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates a ToF depth error value and a stereo matching depth error value; Obtain a ToF depth map detected by the ToF sensor on the device under test, convert the detected ToF depth map into a normal map, obtain a dual-view RGB image captured by the dual cameras on the mobile device, and calculate a cost volume and a disparity map based on the dual-view images. Input the obtained ToF depth map, normal map, cost volume, and disparity map into the trained multimodal fusion network model to obtain a depth perception detection result; The loss function of the TM module during training is constructed using the ToF depth error. This is done to minimize the gap between the predicted ToF depth error and the true value through training. Object edges are located based on the gradient of pixels in the ToF depth map, and specific weights are assigned to pixels at the object edge. This allows the training process to focus on the prediction of the depth error indicator label on the object edge. The overall loss function L used by the TM module during training ALL for: L ALL =αL EW +βL LC Among them, L EW is the edge-weighted TM loss, L LC It is a local consistency loss based on the normal map, and denote the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to the adjacent pixels, It represents the variance of the surface normal angle between each pixel and its adjacent pixels, where N is the total number of pixels in the dataset.

2. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: When extracting ToF low-order features, the ToF depth map and the normal map are respectively input into the feature extraction network for feature extraction, and the features extracted from the ToF depth map after passing through the feature extraction network are fused with the features extracted from the normal map after passing through the feature extraction network to obtain the ToF low-order features; When extracting low-order features for stereo matching, two layers of 3×3 convolution are used to extract features from the cost volume and the disparity map respectively, and the features extracted from the cost volume and the disparity map are fused to obtain the low-order features for stereo matching.

3. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: A symmetric U-shaped learner is used to extract ToF high-order features and / or stereo matching high-order features. The U-shaped learner captures local spatial information through a convolutional layer, and then globally combines and maps it to the output category in a nonlinear manner through a fully connected layer. Skip connections are used to retain the spatial information of the low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

4. The depth perception method integrating ToF and dual vision detection according to claim 1, characterized in that: The depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result, including: calculating the ToF depth error and the stereo matching depth error of each pixel and comparing them, selecting the final depth detection result from the ToF detection result and the stereo matching detection result according to the comparison result, if the ToF depth error is less than the stereo matching depth error, marking the pixel with a first mark T to select the ToF detection result as the fusion structure, otherwise marking the pixel with a second mark S to select the stereo matching detection result as the fusion result, and obtaining the final depth detection result from the fusion results of all pixels.

5. A depth perception device for a mobile device that integrates ToF and stereo matching, characterized in that: include: A multimodal fusion network model construction component, wherein the multimodal fusion network model includes a TM module, an SM module, and a depth fusion module connected in sequence, wherein the TM module extracts ToF low-order features and high-order features according to the ToF depth map and the normal map, respectively. The ToF high-order features include object distance, angle, and scene structure features. The scene structure includes object edge and / or plane features, and the ToF high-order features are used to indicate the ToF depth error value. The SM module extracts stereo matching low-order features and high-order features according to the cost volume and the disparity map, and uses the stereo matching high-order features to indicate the stereo matching depth error value. The depth fusion module fuses the ToF detection result with the stereo matching detection result to obtain the final depth detection result output; The loss function of the TM module during training is constructed using the ToF depth error to minimize the gap between the predicted ToF depth error and the true value through training, and locate the edge of the object according to the gradient of the pixel in the ToF depth map, and assign a specified weight to the pixels at the edge of the object, so that the training process focuses on the prediction corresponding to the depth error indicator label on the edge of the object; The overall loss function L used by the TM module during training ALL for: L ALL =αL EW +βL LC Among them, L EW is the edge-weighted TM loss, L LC It is a local consistency loss based on the normal map, and denote the variance of the predicted ToF depth error indication label and the ground truth depth error indication label of each pixel relative to the adjacent pixels, represents the variance of the surface normal angle between each pixel and its adjacent pixels, where N is the total number of pixels in the dataset; A model training component is used to train the multimodal fusion network model using a final mask label and a depth error indication label, wherein the final mask label determines a label for each pixel that selects a final detection result from a ToF detection result and a stereo matching detection result, and the depth error indication label is a label that indicates the status of a ToF depth error and a stereo matching depth error; The real-time detection component is used to obtain the ToF depth map detected by the ToF sensor on the device under test, and convert the detected ToF depth map into a normal map, and obtain the dual-view RGB image captured by the dual cameras on the mobile device, and calculate the cost volume and disparity map based on the dual-view images. The obtained ToF depth map, normal map, cost volume and disparity map are input into the trained multimodal fusion network model to obtain the depth perception detection result.

6. The depth perception device for mobile devices integrating ToF and stereo matching according to claim 5, characterized in that: The TM module includes two feature extraction networks, a first U-shaped learner, and a first splicing unit connected in sequence. The two feature extraction networks are partially connected to the ToF depth map and the normal map for feature extraction. The features extracted by the two feature extraction networks are fused to obtain ToF low-order features, which are output to the first U-shaped learner. The first U-shaped learner extracts ToF high-order features, and the first splicing unit splices the ToF low-order features with the ToF high-order features to obtain ToF features and outputs them to the depth fusion module. The SM module includes two layers of 3×3 convolution, a second U-shaped learner and a second splicing unit connected in sequence. The two layers of 3×3 convolution extract features from the cost volume and the disparity map, and fuse the features extracted from the cost volume and the disparity map to obtain the stereo matching low-order features, which are output to the second U-shaped learner; the second U-shaped learner extracts the stereo matching high-order features, and the second splicing unit splices the stereo matching low-order features with the stereo matching high-order features to obtain stereo matching features and outputs them to the deep fusion module; The first U-shaped learner and the second U-shaped learner include multiple convolutional layers and more than two fully connected layers. The convolutional layers are interleaved with the fully connected layers. Local spatial information is captured by the convolutional layers, and then globally combined and mapped to output categories in a nonlinear manner through the fully connected layers. Skip connections are used to retain the spatial information of low-level feature maps and pass the spatial information directly to subsequent convolutional layers.

7. The depth perception device for mobile devices integrating ToF and stereo matching according to claim 5 or 6, characterized in that: The depth fusion module includes a convolutional layer, a fully connected layer and a fusion unit connected in sequence. After the ToF features extracted by the TM module and the stereo matching features extracted by the high-order and SM modules are passed through the convolutional layer and the fully connected layer in sequence, the fusion unit calculates the ToF depth error and the stereo matching depth error of each pixel and compares them. The final depth detection result is selected from the ToF detection result and the stereo matching detection result based on the comparison result.

8. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 5.

Images