Depth map generation method and apparatus

Abstract

This specification provides a depth map generation method and apparatus. The depth map generation method includes: acquiring an original color image and a corresponding original depth map; inputting the original color image and the original depth map into a specified generation model to obtain a first depth map and feature information output by each decoding layer in the specified generation model; performing masked cross-attention calculation on the first depth map based on the feature information output by each decoding layer and a specific depth range to obtain an encoding guidance vector; using an encoder, encoding the original depth map and the first depth map based on the encoding guidance vector to obtain a first image feature, and encoding the original color image and the first depth map to obtain a second image feature; and using a decoder, decoding the first image feature and the second image feature to obtain a second depth map. This enhances the feature extraction capability of the encoder and improves the accuracy of the second depth map and the generation efficiency of the depth map.

Description

Technical Field

[0001] This specification relates to the field of image data processing technology, and in particular to a method for generating depth maps. Background Technology

[0002] With the development of data processing technology, depth maps, which can represent the spatial information of physical scenes, have important applications in spatial reconstruction and spatial perception. For example, depth maps of parking lots can be collected, and a 3D virtual model of the parking lot can be created based on the depth maps, thus spatially reconstructing the physical scene of the parking lot. Another example is the collection of depth maps of urban roads, which can be used to control the movement of autonomous vehicles.

[0003] Due to performance limitations of the acquisition unit, the acquired depth maps are mostly sparse depth maps with relatively few depth feature points, making them difficult to use directly. It is necessary to supplement these sparse depth maps with depth feature points to obtain denser depth maps with more concentrated depth feature points. Currently, the main method for depth completion is based on spatial propagation, which utilizes image features, geometric representations, and differentiable geometric propagation networks to iteratively propagate and optimize the depth features of the sparse depth map, ultimately achieving depth completion.

[0004] However, since sparse depth maps have limited depth features, obtaining dense depth maps requires iterative propagation and optimization, combined with depth analysis of image features, to achieve high accuracy. This results in insufficient depth map generation efficiency. Therefore, a depth map generation method with both high accuracy and high efficiency is urgently needed. Summary of the Invention

[0005] In view of the above, embodiments of this specification provide a method for generating depth maps. One or more embodiments of this specification also relate to a mobile device control method, a data processing method for generating depth maps, a depth map generation apparatus, a mobile device control apparatus, a data processing apparatus for generating depth maps, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a depth map generation method is provided, comprising:

[0007] Obtain the original color image and the corresponding original depth image;

[0008] The original color image and the original depth image are input into the specified generation model to obtain the first depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0009] Based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first depth map to obtain the coding guidance vector;

[0010] Using an encoder, based on the coding guidance vector, the original depth map and the first depth map are encoded to obtain the first image features, and the original color map and the first depth map are encoded to obtain the second image features;

[0011] The first image features and the second image features are decoded using a decoder to obtain the second depth map.

[0012] According to a second aspect of the embodiments of this specification, a mobile device control method is provided, wherein the mobile device is provided with an image acquisition unit and a lidar unit, including:

[0013] Acquire scene color images captured by the image acquisition unit and scene point cloud images captured by the lidar unit;

[0014] Input the scene color map and scene point cloud map into the specified generation model to obtain the first scene depth map and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth maps.

[0015] Based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the depth map of the first scene to obtain the coding guidance vector;

[0016] Using an encoder, based on the coding guidance vector, the scene point cloud map and the first scene depth map are encoded to obtain the first scene image features, and the scene color map and the first scene depth map are encoded to obtain the second scene image features.

[0017] The decoder is used to decode the features of the first scene image and the features of the second scene image to obtain the depth map of the second scene;

[0018] Motion control of the mobile device is performed based on the second scene depth map.

[0019] According to a third aspect of the embodiments of this specification, a data processing method for depth map generation is provided, applied to a cloud-side device, comprising:

[0020] Obtain a sample set, which includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images;

[0021] Extract the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images;

[0022] Input the first sample color image and the first sample depth image into the specified generation model to obtain the first training depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0023] Based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first training depth map to obtain the training coding guidance vector.

[0024] Using an encoder, based on the training encoding guidance vector, the first sample depth map and the first training depth map are encoded to obtain the first training image features, and the first sample color map and the first training depth map are encoded to obtain the second training image features.

[0025] The decoder is used to decode the features of the first training image and the features of the second training image to obtain the second training depth map;

[0026] The loss value is calculated based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image;

[0027] Based on the loss value, adjust the model parameters of the encoder and decoder, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set until the preset training termination condition is met, and obtain the model parameters of the trained encoder and decoder.

[0028] The model parameters of the encoder and decoder are sent to the end-side device.

[0029] According to a fourth aspect of the embodiments of this specification, a depth map generation apparatus is provided, comprising: a first acquisition module configured to acquire an original color image and an original depth map corresponding to the original color image; a first generation module configured to input the original color image and the original depth map into a specified generation model to obtain a first depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; a first calculation module configured to perform masked cross-attention calculation on the first depth map based on the feature information output by each decoding layer and a specific depth range to obtain an encoding guidance vector; a first encoding module configured to use an encoder to encode the original depth map and the first depth map based on the encoding guidance vector to obtain a first image feature, and to encode the original color image and the first depth map to obtain a second image feature; and a first decoding module configured to use a decoder to decode the first image feature and the second image feature to obtain a second depth map.

[0030] According to a fifth aspect of the embodiments of this specification, a mobile device control device is provided, wherein the mobile device is provided with an image acquisition unit and a lidar unit, comprising: a second acquisition module configured to acquire a scene color image acquired by the image acquisition unit and a scene point cloud image acquired by the lidar unit; a second generation module configured to input the scene color image and the scene point cloud image into a specified generation model to obtain a first scene depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding and decoding model with depth map generation function; a second calculation module configured to perform mask cross-attention calculation on the first scene depth map based on the feature information output by each decoding layer and a specific depth range to obtain an encoding guidance vector; a second encoding module configured to use an encoder to encode the scene point cloud image and the first scene depth map based on the encoding guidance vector to obtain a first scene image feature, and to encode the scene color image and the first scene depth map to obtain a second scene image feature; a second decoding module configured to use a decoder to decode the first scene image feature and the second scene image feature to obtain a second scene depth map; and a motion control module configured to perform motion control on the mobile device based on the second scene depth map.

[0031] According to a sixth aspect of the embodiments of this specification, a data processing apparatus for depth map generation is provided, applied to a cloud-side device, comprising: a third acquisition module configured to acquire a sample set, wherein the sample set includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images; a sample extraction module configured to extract a first sample color image and a first sample depth map corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images; a third generation module configured to input the first sample color image and the first sample depth map into a specified generation model to obtain a first training depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; and a third calculation module configured to perform mask cross-attention calculation on the first training depth map based on the feature information output by each decoding layer and a specific depth range to obtain a training coding guidance vector; the third An encoding module is configured to use an encoder to encode a first sample depth map and a first training depth map based on a training encoding guide vector to obtain first training image features, and to encode a first sample color image and the first training depth map to obtain second training image features; a third decoding module is configured to use a decoder to decode the first training image features and the second training image features to obtain a second training depth map; a loss calculation module is configured to calculate a loss value based on the second training depth map and a first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image; a model training module is configured to adjust the model parameters of the encoder and decoder according to the loss value, and return to the step of extracting the first sample color image and the first sample depth map corresponding to the first sample color image from the sample set until a preset training termination condition is met, thereby obtaining the trained model parameters of the encoder and decoder; a sending module is configured to send the model parameters of the encoder and decoder to the edge device.

[0032] According to a seventh aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor;

[0033] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation.

[0034] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation described above.

[0035] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation.

[0036] In one or more embodiments of this specification, an original color image and a corresponding original depth map are obtained; the original color image and the original depth map are input into a specified generation model to obtain a first depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first depth map to obtain an encoding guidance vector; using an encoder, based on the encoding guidance vector, the original depth map and the first depth map are encoded to obtain a first image feature, and the original color image and the first depth map are encoded to obtain a second image feature; using a decoder, the first image feature and the second image feature are decoded to obtain a second depth map. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second depth map, avoids depth completion through multiple iterations, and improves the generation efficiency of the depth map. Attached Figure Description

[0037] Figure 1 This is a flowchart of a depth map generation method provided in one embodiment of this specification;

[0038] Figure 2 This is a flowchart illustrating a mobile device control method according to one embodiment of this specification;

[0039] Figure 3 This is a flowchart illustrating a data processing method for depth map generation, provided in one embodiment of this specification.

[0040] Figure 4 This is a flowchart illustrating the processing steps of a depth map generation method for autonomous vehicle control, provided in one embodiment of this specification.

[0041] Figure 5 This is a schematic diagram of the processing flow of a depth map generation method provided in one embodiment of this specification;

[0042] Figure 6 This is a schematic diagram of the structure of a depth map generation device provided in one embodiment of this specification;

[0043] Figure 7 This is a schematic diagram of the structure of a mobile device control device provided in one embodiment of this specification;

[0044] Figure 8 This is a schematic diagram of a data processing apparatus for depth map generation provided in one embodiment of this specification;

[0045] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0046] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0047] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0048] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0049] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0050] Depth map: An image that uses the feature values ​​of pixels to represent the depth (distance) between the acquisition unit and objects in the physical scene.

[0051] Depth Completion: Given an input color image and its corresponding point cloud image, the neural network model extracts image features from the color image and guides the point cloud image to complete the depth pixels, resulting in a depth map with densely distributed depth pixels.

[0052] CNN (Convolutional Neural Networks) model: A multi-layer neural network model with forward propagation and backward propagation.

[0053] The Transformer model is a neural network model based on an attention mechanism. It extracts and analyzes features from data through this mechanism and can be used for both natural language and image data. One or more embodiments in this specification are used for feature extraction and analysis of image data.

[0054] ViT (Vision Transformer) model: A Transformer derivative model applied to visual image processing.

[0055] Swin Transformer model: A Transformer derivative model with multi-level sampling capability.

[0056] FFN (Feedforward Neural Network) model: A neural network model with feedforward propagation function.

[0057] Attention computation: A method that uses multiple attention heads to comprehensively integrate and compute features across different ranges, resulting in features that are more focused on the target range. It includes self-attention computation and cross-attention computation. In self-attention computation, query information, key information, and value information are determined based on the same feature information. In cross-attention computation, query information, key information, and value information are determined based on different feature information, but generally, key information and value information are determined based on the same feature information.

[0058] Masking: Masking the feature values ​​of the target range in the feature vector, so that in subsequent processing, vector processing can be more focused on the target range.

[0059] Currently, two-stage CNN models based on Corase to Fine (Curves to Fine) use image features to perform depth completion on sparse depth maps. However, since CNN models can only perform in-depth feature mining on the input image features, the accuracy of the generated dense depth maps is insufficient.

[0060] To address the aforementioned problems, this specification provides a depth map generation method. One or more embodiments of this specification also relate to a mobile device control method, a data processing method for depth map generation, a depth map generation apparatus, a mobile device control apparatus, a data processing apparatus for depth map generation, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0061] See Figure 1 , Figure 1 A flowchart of a depth map generation method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0062] Step 102: Obtain the original color image and the corresponding original depth image.

[0063] The original color image is a color space image of the physical scene, such as RGB (Red-Green-Blue) color image, HIS (Hue-Saturation-Intensity) color image, YUV (Intensity and Hue) color image, and YCbCr (Intensity and Hue) color image, etc.

[0064] The original depth map is a depth map with sparsely distributed feature points. For example, a point cloud map.

[0065] Obtaining the original color image and its corresponding original depth map can be achieved by retrieving them from an image database, which can be a local or remote image database. Alternatively, the original color image and its corresponding original depth map can be obtained from an acquisition unit, which can be a hardware unit with image acquisition capabilities, such as a camera; or a hardware unit with spatial feature point acquisition capabilities, such as an infrared sensor, acoustic sensor, or lidar unit.

[0066] It should be noted that spatial calibration can be performed on the original color map and the original depth map to ensure the accuracy of the subsequently generated first and second depth maps.

[0067] For example, an RGB color image of a community and a point cloud image of the community are obtained from a remote image database, wherein the point cloud image of the community has 20 feature points.

[0068] By acquiring the original color image and the corresponding original depth image, an image data foundation is provided for the subsequent generation of the first and second depth images.

[0069] Step 104: Input the original color image and the original depth image into the specified generation model to obtain the first depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0070] The specified generative model is an encoder-decoder model capable of generating depth maps. It features an encoder-decoder structure and an attention mechanism, such as the Transformer, ViT, and Swin Transformer models. The model is a multi-layered neural network with multiple decoding layers. Each decoding layer decodes features of different sizes to obtain feature information of the corresponding size. For example, the first decoding layer decodes 1×1 image features to obtain 4×4 features, the second decoding layer decodes 4×4 features to obtain 9×9 features, and so on. By decoding features of different sizes, the decoding layers can adaptively process features at different sizes, resulting in a first depth map with high accuracy. The specified generative model can perform feature fusion encoding on multiple input image data and then decode to obtain a depth map. Feature fusion is performed using a fusion module, which is determined based on a fusion function, such as the Concat function.

[0071] The first depth map is a depth map with a relatively dense distribution of feature points. The feature point density of the first depth map is higher than that of the original depth map but lower than that of the second depth map.

[0072] The feature information output by each decoding layer is the image feature information corresponding to the preset size of each decoding layer.

[0073] The original color image and the original depth image are input into a specified generative model to obtain a first depth image and feature information output by each decoding layer in the specified generative model. Specifically, the original color image and the original depth image are input into the specified generative model, and multiple feature points are predicted based on the image features of the original color image and the original depth image. Based on the multiple feature points and the original depth image, the first depth image and feature information output by each decoding layer in the specified generative model are obtained.

[0074] For example, the RGB color image and the point cloud image of a community are input into the Transformer model. Based on the image features of the RGB color image and the point cloud image of the community, 1000 feature points are predicted. Based on the 1000 feature points and the point cloud image of the community, a first depth map is obtained. The first depth map has 1020 feature points. The image feature information output by the 9 decoding layers in the Transformer model is obtained: Z1, Z2, Z3...Z9.

[0075] The original color image and original depth map are input into a specified generative model to obtain a first depth map and feature information output from each decoding layer in the specified generative model. The specified generative model is a pre-acquired encoding / decoding model with depth map generation capabilities. This provides a data foundation for subsequent mask cross-attention calculations.

[0076] Step 106: Based on the feature information output by each decoding layer and a specific depth range, perform mask cross-attention calculation on the first depth map to obtain the coding guidance vector.

[0077] A specific depth range refers to multiple depth ranges, or multiple distance ranges, pre-defined in the physical scene. For example, the depth range of a depth map is 0-90m, and the specific depth ranges are 0-10, 10-20, 20-30…80-90. The specific depth ranges are set according to the corresponding decoding layers in the specified generation model. For example, if the specified generation model has 9 decoding layers, corresponding to 9 specific depth ranges, or if the first decoding layer in the specified generation model decodes 2×2 size image features and the second decoding layer decodes 4×4 size image features, then the specific depth ranges are also set to depth ranges with the same depth intervals: 0-10m and 10-20m.

[0078] Masked cross-attention calculation sets a cross-attention mask for the cross-attention parameters (cross-attention weights or cross-attention matrix) of the first depth map based on a specific depth range.

[0079] The coding guide vector is a feature vector representing a specific depth range. It is obtained by setting a cross-attention mask on the cross-attention parameters based on the first depth map of that specific depth range. The coding guide vector represents the corresponding specific depth range and has encoded attention for that specific depth range. The coding guide vector serves as prior information for subsequent encoder encoding, making the subsequent encoder highly specific and enhancing its feature extraction capability within that specific depth range.

[0080] Based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first depth map to obtain the coding guidance vector. Specifically, a cross-attention mask is set for the cross-attention parameters based on the first depth map with a specific depth range.

[0081] For example, based on the image feature information (Z1, Z2, Z3...Z9) output by the 9 decoding layers and a specific depth range (0-10, 10-20, 20-30...80-90), a cross-attention mask is set on the cross-attention parameters (Attent ion_1, Attent ion_2, Attent ion_3...Attent ion_9) based on the first depth map of the specific depth range, resulting in 9 coding guidance vectors Mask (Mask_1, Mask_2, Mask_3...Mask_9).

[0082] Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first depth map to obtain the coding guidance vector. This ensures that the obtained coding guidance vector can represent the corresponding specific depth range and has coding attention for that specific depth range, thereby enabling the subsequent encoder to have high specificity and enhancing the encoder's feature extraction capability within that specific depth range.

[0083] Step 108: Using the encoder, based on the coding guide vector, encode the original depth map and the first depth map to obtain the first image features, and encode the original color map and the first depth map to obtain the second image features.

[0084] The encoder is the encoder of the depth map generation model with image feature extraction capabilities. The encoder can extract image features based on the encoded guiding vector as prior information. The depth map generation model is a pre-acquired encoding and decoding model with depth map generation capabilities. It can be the same as or different from the specified generation model. The depth map generation model can be a Transformer model, a ViT model, or a Swin Transformer model, etc.

[0085] The first image feature is a depth-level image feature. The second image feature is a pixel-level image feature. It should be noted that since both the first and second image features are obtained based on the encoding of the first depth map, both have depth features. In comparison, the first image feature is more inclined towards the depth level, while the second image feature is more inclined towards the pixel level, that is, it has a stronger representational ability for different levels.

[0086] Based on the coding guide vector, the original depth map and the first depth map are encoded to obtain the first image features. Specifically, the coding guide vector is determined as prior information for encoding, and feature fusion encoding is performed on the original depth map and the first depth map to obtain the first image features. Based on the coding guide vector, the original color image and the first depth map are encoded to obtain the second image features. Specifically, the coding guide vector is determined as prior information, and feature fusion encoding is performed on the original color image and the first depth map to obtain the second image features. The feature fusion encoding method involves using a fusion module to perform feature fusion followed by encoding. The fusion module is determined based on a fusion function, where the fusion function can be the Concat function.

[0087] For example, the coding guidance vector Mask is determined as the coding prior information, and the Transformer encoder is used to perform feature fusion encoding on the point cloud map and the first depth map of a certain community to obtain the first image feature Depth_Feature. The coding guidance vector Mask is determined as the prior information, and the Transformer encoder is used to perform feature fusion encoding on the RGB color map and the first depth map of the community to obtain the second image feature Image_Feature.

[0088] An encoder is used to encode the original depth map and the first depth map based on the coding guide vector to obtain the first image features, and the original color image and the first depth map are also encoded to obtain the second image features. Since the coding guide vector can represent a specific depth range and has encoding attention to that specific depth range, the encoder has high specificity, enhancing its feature extraction capability within that specific depth range. This ensures that the encoded image features fully represent the depth features of that specific depth range, improving the accuracy of the subsequently decoded second depth map.

[0089] Step 110: Use the decoder to decode the first image features and the second image features to obtain the second depth map.

[0090] The decoder is a depth map generation model with image feature encoding capabilities. The decoder and encoder belong to the same depth map generation model. The second depth map is a dense depth map with a high density of feature points.

[0091] The first image features and the second image features are decoded using a decoder to obtain the second depth map. Specifically, the first image features and the second image features are fused and decoded using a decoder to obtain the second depth map.

[0092] For example, using the Transformer decoder, the first image feature Depth_Feature and the second image feature Image_Feature are fused and decoded to obtain a second depth map, which has 2000 feature points.

[0093] In this embodiment, an original color image and its corresponding original depth map are obtained; the original color image and the original depth map are input into a specified generation model to obtain a first depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first depth map to obtain an encoding guidance vector; using an encoder, based on the encoding guidance vector, the original depth map and the first depth map are encoded to obtain a first image feature, and the original color image and the first depth map are encoded to obtain a second image feature; using a decoder, the first image feature and the second image feature are decoded to obtain a second depth map. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second depth map, avoids depth completion through multiple iterations, and improves the generation efficiency of the depth map.

[0094] Optionally, step 106 includes the following specific steps:

[0095] Obtain the query information corresponding to each decoding layer;

[0096] Based on the feature information output by each decoding layer, determine the key information and value information corresponding to each decoding layer;

[0097] Based on a specific depth range, the first depth map is segmented into depth ranges to obtain the first depth map corresponding to each decoding layer;

[0098] Based on the first depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information and value information corresponding to each decoding layer to obtain the encoding guidance vector.

[0099] Cross-attention processes image features using cross-attention parameters, i.e., cross-attention weights or a cross-attention matrix, to concentrate the resulting encoded guidance vector within a specific depth range. The cross-attention parameters have three parameters: query information (Q), key information (K), and value information (V). Cross-attention differs from self-attention in that the query, key, and value parameters are determined based on different feature information. In the embodiments described in this specification, the key and value information are determined based on the same feature information, i.e., the feature information output by each decoding layer. The query information corresponds to a pre-set value for each decoding layer, which can be randomly set and adjusted to represent a specific depth range. This cross-attention approach allows the resulting cross-attention parameters to represent richer feature information, i.e., setting the cross-attention mask from different dimensions, ensuring the accuracy of the encoded guidance vector.

[0100] Based on the feature information output by each decoding layer, the key information and value information corresponding to each decoding layer are determined. Specifically, the feature information output by each decoding layer is used as the key information and value information corresponding to each decoding layer.

[0101] Based on a specific depth range, the first depth map is segmented to obtain the first depth map corresponding to each decoding layer. Specifically, a depth map segmentation algorithm is generated based on the specific depth range corresponding to each decoding layer. This algorithm is then used to segment the first depth map to obtain the first depth map corresponding to each decoding layer. The depth map segmentation algorithm utilizes feature point selection criteria in the generated depth map based on the specific depth range. By selecting and retaining feature points within the specific depth range, the first depth map corresponding to each decoding layer is obtained.

[0102] Based on the first depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information and value information corresponding to each decoding layer to obtain the encoding guidance vector. Specifically, based on the depth features of the first depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information and value information corresponding to each decoding layer to obtain the encoding guidance vector.

[0103] For example, query information (Q1, Q2, Q3...Q9) corresponding to 9 decoding layers is obtained. The feature information (Z1, Z2, Z3...Z9) output by the 9 decoding layers is determined as key information (K1, K2, K3...K9) and value information (V1, V2, V3...V9) corresponding to the 9 decoding layers. Based on a specific depth range (0-10, 10-20, 20-30...80-90), the first depth map is segmented by depth range to obtain the first depth map Dc corresponding to the 9 decoding layers. Based on the depth features of the first depth maps (Dc1, Dc2, Dc3...Dc9) corresponding to the 9 decoding layers, masked cross-attention calculation is performed on the query information (Q1, Q2, Q3...Q9), key information (K1, K2, K3...K9), and value information (V1, V2, V3...V9) corresponding to the 9 decoding layers to obtain 9 coding guidance vectors Mask (Mask_1, Mask_2, Mask_3...Mask_9).

[0104] The process involves acquiring query information corresponding to each decoding layer, determining key and value information based on the feature information output by each decoding layer, segmenting the first depth map based on a specific depth range to obtain the first depth map corresponding to each decoding layer, and performing masked cross-attention calculation on the query, key, and value information corresponding to each decoding layer based on the first depth map to obtain the coding guidance vector. By determining key and value information through different feature information and combining them with the first depth map after depth feature segmentation, masked cross-attention calculation is performed, enabling the obtained coding guidance vector to more accurately represent the corresponding specific depth range and have greater coding attention to the specific depth range. This, in turn, makes the subsequent encoder more specific and further enhances the encoder's feature extraction capability within a specific depth range.

[0105] Optionally, based on the first depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to each decoding layer to obtain the encoding guidance vector, including the following specific steps:

[0106] For any decoding layer, based on the first depth map corresponding to the decoding layer, masked cross-attention calculation is performed on the query information, key information and value information corresponding to the decoding layer to obtain the encoding guidance information corresponding to the decoding layer;

[0107] Adjust the query information based on the encoding guidance information corresponding to the decoding layer and the feature information output by the decoding layer;

[0108] Based on the adjusted query information, key information, value information, and feature information output by the next decoding layer, determine the key information and value information corresponding to the next decoding layer;

[0109] Based on the encoding guidance information corresponding to each decoding layer, an encoding guidance vector is generated.

[0110] Key and value information are determined based on the feature information output by each decoding layer. Since the feature information not only represents the feature information at the depth level, but also the feature information at other levels, and the query information is adjusted during the encoding cross-attention calculation, thus representing other feature information, if the key and value information are determined for each decoding layer in isolation, it will increase the time consumption of mask cross-attention calculation, thereby affecting the generation efficiency of the depth map and making it impossible to guarantee the adjustment of the query information. The query information, key information, and value information can be passed between cascaded decoding layers to improve the generation efficiency and accuracy of the depth map.

[0111] Encoding guidance information represents feature information within a specific depth range. It is obtained by setting a cross-attention mask for the cross-attention parameters (query information, key information, and value information) based on cross-attention for any coding layer. Encoding guidance information represents the specific depth range corresponding to the current coding layer and possesses encoding attention for that specific depth range. The encoding guidance vectors generated based on the encoding guidance information of each coding layer serve as prior information for subsequent encoder encoding, making the subsequent encoder highly specific and enhancing its feature extraction capability within that specific depth range.

[0112] It should be noted that, in the embodiments of this specification, the query information corresponding to the first cascaded decoding layer can be randomly generated initial query information, along with key information and value information, which will participate in the subsequent layer-by-layer transmission.

[0113] The query information is adjusted based on the encoding guidance information corresponding to the decoding layer and the feature information output by the decoding layer. Specifically, the query information is adjusted using a depth range perception module based on the encoding guidance information corresponding to the decoding layer and the feature information output by the decoding layer. The depth range perception module is a module with feature information adjustment capabilities. In the embodiments of this specification, the depth range perception module includes an FNN model.

[0114] Based on the adjusted query information, key information, value information, and feature information output by the next decoding layer, the key information and value information corresponding to the next decoding layer are determined. Specifically, based on the adjusted query information, key information, and value information, the prior information corresponding to the next decoding layer is determined. Based on the prior information and the feature information output by the next decoding layer, the key information and value information corresponding to the next decoding layer are then determined. Furthermore, based on the adjusted query information, key information, and value information, information residuals are calculated. Based on the information residuals and the feature information output by the next decoding layer, the key information and value information corresponding to the next decoding layer are then determined.

[0115] For example, for the first-level decoding layer, the query information Q1 and the first depth map Dc1 of a specific depth range (0-10m) corresponding to the decoding layer are obtained. Based on the first depth map Dc1 corresponding to the decoding layer, a cross-attention mask is set for the query information Q1, key information K1, and value information V1 corresponding to the decoding layer. Then, the encoding guidance information Mask_1 corresponding to the first-level decoding layer is obtained. Using a preset depth range perception module, the query information Q1 is adjusted according to the encoding guidance information Mask_1 and the feature information Z1 output by the first-level decoding layer. Based on the adjusted query information Q1, key information K1, value information V1, and the feature information output by the second-level decoding layer, the key information K2 and value information V2 corresponding to the second-level decoding layer are determined. Based on the encoding guidance information Mask_1, Mask_2, Mask_3...Mask_9 corresponding to each decoding layer, the encoding guidance vector Mask(Mask_1, Mask_2, Mask_3...Mask_9) is generated.

[0116] For any decoding layer, based on the first depth map corresponding to that decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to that decoding layer to obtain the encoding guidance information corresponding to that decoding layer. Based on the encoding guidance information and the feature information output by that decoding layer, the query information is adjusted. Based on the adjusted query information, key information, value information, and the feature information output by the next decoding layer, the key information and value information corresponding to the next decoding layer are determined. Based on the encoding guidance information corresponding to each decoding layer, an encoding guidance vector is generated. This improves the efficiency and accuracy of depth map generation.

[0117] Optionally, after step 110, the following specific steps are also included:

[0118] Obtain the confidence distribution information of the first and second depth maps;

[0119] Based on the confidence distribution information of the first and second depth maps, the first and second depth maps are normalized to obtain an updated second depth map.

[0120] Since the first depth map is not generated based on a specific depth range, although its feature points are sparser compared to the second depth map, it can represent global depth features. The second depth map, on the other hand, is based on a specific depth range and undergoes more detailed segmentation (fine processing), thus representing local depth features. Combining the two and performing normalization results in an updated second depth map with higher accuracy.

[0121] The confidence distribution information of the first depth map is the pixel distribution of the confidence of each depth feature point in the first depth map, and the confidence distribution information of the second depth map is the pixel distribution of the confidence of each depth feature point in the second depth map.

[0122] In the embodiments of this specification, the normalization process is carried out by using the Softmax module, that is, normalizing the confidence information to obtain a unified confidence information within the range of (0,1), and combining the normalized confidence information with the first depth map and the second depth map.

[0123] Based on the confidence distribution information of the first and second depth maps, the first and second depth maps are normalized to obtain an updated second depth map. Specifically, based on the confidence distribution information of the first and second depth maps, the Softmax module is used to normalize the first and second depth maps to obtain an updated second depth map.

[0124] For example, the first depth map is Dc, and the second depth map is Dr. The confidence information distribution Vc of the first depth map and the confidence information distribution Vr of the second depth map are obtained. The confidence information distribution Vc of the first depth map and the confidence information distribution Vr of the second depth map are normalized using the Softmax module to obtain the updated second depth map D.

[0125] The confidence distribution information of the first and second depth maps is obtained. Based on the confidence distribution information of the first and second depth maps, the first and second depth maps are normalized to obtain an updated second depth map. By combining the first depth map, which represents global depth features, and the second depth map, which represents local depth features, the accuracy of the second depth map is improved.

[0126] Optionally, step 108 includes the following specific steps:

[0127] Based on the coding guidance vector, the attention parameters of a specified coding layer in the encoder are adjusted;

[0128] The original depth map and the first depth map are encoded using the encoder with adjusted parameters to obtain the first image features.

[0129] Since the coding guide vector represents a specific depth range and has coding attention focused on that specific depth range, the attention parameters of a specified coding layer in the encoder are adjusted based on the coding guide vector. This ensures that the coding layer focuses on encoding features within the specific depth range, resulting in image features that accurately represent that depth range. This improves the accuracy of the image features and consequently, the accuracy of the second depth map obtained through subsequent decoding.

[0130] The specified coding layer is a deep feature coding layer in the encoder, which encodes image features with depth levels. In the embodiments of this specification, the specified coding layer is a coding layer with a sparse mask structure.

[0131] Based on the coding guidance vector, the attention parameters of a specified coding layer in the encoder are adjusted. Specifically, the coding guidance vector is input into the specified coding layer as coding prior information to adjust the attention parameters of the specified coding layer in the encoder.

[0132] For example, the encoding guidance vector Mask (Mask_1, Mask_2, Mask_3...Mask_9) is input into the deep feature encoding layer as encoding prior information to adjust the attention parameter Attent ion of the deep feature encoding layer in the Swin Transformer encoder. The Swin Transformer encoder with the adjusted parameter is used to encode the point cloud map and the first depth map of a community to obtain the first image feature Depth_Feature.

[0133] Based on the coding guidance vector, the attention parameters of a specified coding layer in the encoder are adjusted. Using the parameter-adjusted encoder, the original depth map and the first depth map are encoded to obtain the first image features. This ensures that the encoder focuses on feature encoding corresponding to a specific depth range, guaranteeing that the obtained first image features can accurately represent that specific depth range, thus improving the accuracy of the first image features and consequently the accuracy of the second depth map obtained through subsequent decoding.

[0134] Optionally, prior to step 102, the following specific steps are also included:

[0135] Obtain a sample set, which includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images;

[0136] Extract the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images;

[0137] Input the first sample color image and the first sample depth image into the specified generation model to obtain the first training depth image and the feature information output by each decoding layer in the specified generation model.

[0138] Based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first training depth map to obtain the training coding guidance vector.

[0139] Using an encoder, based on the training encoding guidance vector, the first sample depth map and the first training depth map are encoded to obtain the first training image features, and the first sample color map and the first training depth map are encoded to obtain the second training image features.

[0140] The decoder is used to decode the features of the first training image and the features of the second training image to obtain the second training depth map;

[0141] The loss value is calculated based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image;

[0142] Based on the loss value, adjust the model parameters of the encoder and decoder, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set, until the preset training termination condition is met, and obtain the trained encoder and decoder.

[0143] The sample set consists of multiple pairs of sample images. Each pair includes a sample color image, a corresponding sample depth image, and a corresponding verification depth image. The sample color image is a color space image of the physical scene, such as an RGB (Red-Green-Blue) color image, a HIS (Hue-Saturation-Intensity) color image, a YUV (Intensity-Yellow) color image, or a YCbCr (Intensity-Yellow) color image. The sample depth image is a sparsely distributed depth image, such as a point cloud. The verification depth image is a densely distributed depth image. The sample set can be pre-built or obtained from an open-source sample database. In this embodiment, the open-source sample database is the KITTI sample database, and the corresponding samples are the KITTI sample set.

[0144] The preset training termination condition is a pre-defined condition for ending the iterative training. It can be a preset number of training iterations or a preset loss threshold.

[0145] It should be noted that the steps of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set to obtain the second training image features are based on the same inventive concept as steps 102 to 110 above, and will not be repeated here.

[0146] The loss value is calculated based on the second training depth map and the first validation depth map. Specifically, the loss value is calculated using the loss value calculation formula based on the second training depth map and the first validation depth map. The loss value calculation formula can be either the cross-entropy loss formula or the evaluation loss formula.

[0147] Based on the loss value, the model parameters of the encoder and decoder are adjusted. Specifically, the gradient update method is used to adjust the model parameters of the encoder and decoder according to the loss value.

[0148] For example, a KI TTI sample set is obtained, and a first sample color image and a corresponding first sample depth image are extracted from the KI TTI sample set. The first sample color image and the first sample depth image are input into a ViT model to obtain a first training depth image and feature information output by each decoding layer in the ViT model. Based on the feature information output by each decoding layer in the ViT model and a specific depth range, masked cross-attention calculation is performed on the first training depth image to obtain a training encoding guidance vector. Using the encoder of the Swin Transformer model, based on the training encoding guidance vector, the first sample depth image and the first training depth image are encoded to obtain first training image features. The first sample color image and the first training depth image are also encoded to obtain second training image features. Using the decoder of the Swin Transformer model, the first training image features and the second training image features are decoded to obtain a second training depth image. Based on the second training depth image and the first validation depth image, the loss value is calculated using the cross-loss entropy loss value calculation formula. Using the gradient update method, the Swin Transformer model is adjusted according to the loss value. The model parameters of the encoder and decoder of the Transformer model are returned, and the steps of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set are performed until the preset loss value threshold is reached, so as to obtain the encoder and decoder of the trained Swing Transformer model.

[0149] Obtain a sample set, which includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images; extract a first sample color image and a first sample depth map corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images; input the first sample color image and the first sample depth map into a specified generation model to obtain a first training depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-obtained encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, perform mask cross-attention calculation on the first training depth map to obtain a training encoding guidance vector; using the encoder, based on the training... The process involves encoding a guiding vector to encode the first sample depth map and the first training depth map, resulting in first training image features, and encoding the first sample color image and the first training depth map, resulting in second training image features. A decoder is then used to decode the first and second training image features to obtain the second training depth map. Based on the second training depth map and the first verification depth map, a loss value is calculated, where the first verification depth map is the verification depth map corresponding to the first sample color image. According to the loss value, the model parameters of the encoder and decoder are adjusted, and the process returns to extracting the first sample color image and the corresponding first sample depth map from the sample set. This process continues until a preset training termination condition is met, resulting in the trained model parameters of the encoder and decoder. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first training depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second training depth map, improves the training effect of the encoder and decoder in a single training, and improves the training efficiency of the encoder and trainer while ensuring the accuracy of the trained encoder and decoder.

[0150] See Figure 2 , Figure 2 The flowchart of a mobile device control method according to an embodiment of this specification is shown. The mobile device is equipped with an image acquisition unit and a lidar unit, and includes the following specific steps.

[0151] Step 202: Obtain the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit.

[0152] An image acquisition unit is a hardware unit with image acquisition capabilities, such as a camera.

[0153] Scene color images are color space images of physical scenes, such as RGB (Red-Green-Blue) color images, HIS (Hue-Saturation-Intensity) color images, YUV (Intensity and Hue) color images, and YCbCr (Intensity and Hue) color images.

[0154] A scene depth map is a depth map that shows the sparse distribution of feature points in a physical scene. For example, a point cloud map.

[0155] The method for acquiring the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit is as follows: receiving the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit.

[0156] Step 204: Input the scene color map and scene point cloud map into the specified generation model to obtain the first scene depth map and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth maps.

[0157] The first scene depth map is a depth map with a relatively dense distribution of feature points in the physical scene. The feature point density of the first scene depth map is higher than that of the scene depth map, but lower than that of the second scene depth map.

[0158] Step 206: Based on the feature information output by each decoding layer and a specific depth range, perform mask cross-attention calculation on the depth map of the first scene to obtain the coding guidance vector.

[0159] Step 208: Using the encoder, based on the coding guidance vector, encode the scene point cloud map and the first scene depth map to obtain the first scene image features, and encode the scene color map and the first scene depth map to obtain the second scene image features.

[0160] The first scene image features are depth-level image features of the physical scene. The second scene image features are pixel-level image features of the physical scene. It should be noted that since both the first and second scene image features are derived from the depth map encoding of the first scene, both possess depth features. However, the first scene image features are more depth-level, while the second scene image features are more pixel-level, meaning they have stronger representational capabilities across different levels.

[0161] Step 210: Use the decoder to decode the features of the first scene image and the features of the second scene image to obtain the depth map of the second scene.

[0162] The second depth map is a dense depth map for features with a high density of physical scene points.

[0163] It should be noted that steps 202 to 210 above are the same as those mentioned above. Figure 1 Steps 102 to 110 in the embodiments are based on the same inventive concept and will not be described again here.

[0164] Step 212: Perform motion control on the mobile device based on the second scene depth map.

[0165] Mobile devices are self-moving devices, such as autonomous vehicles, self-moving robots, and self-moving aircraft.

[0166] Based on the second scene depth map, motion control of the mobile device is performed. Specifically, motion control commands are generated based on the second scene depth map, and the mobile device is scheduled to move using the motion control commands.

[0167] For example, based on the second scene depth map D, a motion control command Mov_Command is generated, and the motion control command Mov_Command is used to schedule the autonomous vehicle to move.

[0168] In this embodiment, a scene color image acquired by an image acquisition unit and a scene point cloud image acquired by a lidar unit are obtained. The scene color image and scene point cloud image are input into a specified generation model to obtain a first scene depth map and feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with depth map generation function. Based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first scene depth map to obtain an encoding guidance vector. Using an encoder, based on the encoding guidance vector, the scene point cloud image and the first scene depth map are encoded to obtain first scene image features, and the scene color image and the first scene depth map are encoded to obtain second scene image features. Using a decoder, the first scene image features and the second scene image features are decoded to obtain a second scene depth map. Based on the second scene depth map, motion control is performed on the mobile device. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first scene depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded scene image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second scene depth map, improves the control accuracy of the mobile device, and improves the motion safety and operability of the mobile device. Furthermore, due to the improved generation efficiency of the depth map, the control accuracy of the mobile device is further improved, and the motion safety and operability of the mobile device are enhanced.

[0169] See Figure 3 , Figure 3 A flowchart of a data processing method for depth map generation provided in one embodiment of this specification is shown. The method is applied to a cloud-side device and includes the following specific steps.

[0170] Step 302: Obtain a sample set, which includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images;

[0171] Step 304: Extract the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images;

[0172] Step 306: Input the first sample color image and the first sample depth image into the specified generation model to obtain the first training depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0173] Step 308: Based on the feature information output by each decoding layer and a specific depth range, perform mask cross-attention calculation on the first training depth map to obtain the training coding guidance vector;

[0174] Step 310: Using the encoder, based on the training encoding guide vector, encode the first sample depth map and the first training depth map to obtain the first training image features, and encode the first sample color map and the first training depth map to obtain the second training image features.

[0175] Step 312: Using the decoder, decode the features of the first training image and the features of the second training image to obtain the second training depth map;

[0176] Step 314: Calculate the loss value based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image;

[0177] Step 316: Adjust the model parameters of the encoder and decoder according to the loss value, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set until the preset training termination condition is met, and obtain the model parameters of the trained encoder and decoder.

[0178] Step 318: Send the model parameters of the encoder and decoder to the end device.

[0179] Cloud-side devices are network cloud-based devices that provide model training capabilities; they are a type of virtual device. Edge-side devices are terminal devices that handle data storage and model execution; they can be either clients or servers of software with depth map generation capabilities.

[0180] Steps 302 to 316 above have been described above. Figure 1 The corresponding embodiments are described, and will not be repeated here.

[0181] In this embodiment, a sample set is obtained, comprising multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images; a first sample color image and a first sample depth map corresponding to the first sample color image are extracted from the sample set, wherein the first sample color image is any one of the multiple sample color images; the first sample color image and the first sample depth map are input into a specified generation model to obtain a first training depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first training depth map to obtain a training encoding guidance vector; using an encoder, based on the training encoding guidance vector... The process involves encoding a first sample depth map and a first training depth map to obtain first training image features, and encoding a first sample color image and the first training depth map to obtain second training image features. A decoder is then used to decode the first and second training image features to obtain a second training depth map. A loss value is calculated based on the second training depth map and a first verification depth map, where the first verification depth map is the verification depth map corresponding to the first sample color image. The model parameters of the encoder and decoder are adjusted according to the loss value, and the process returns to extracting the first sample color image and its corresponding first sample depth map from the sample set until a preset training termination condition is met, at which point the trained model parameters of the encoder and decoder are obtained. Finally, the model parameters of the encoder and decoder are sent to the edge device. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first training depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second training depth map, improves the training effect of the encoder and decoder in a single training, and improves the training efficiency of the encoder and trainer while ensuring the accuracy of the trained encoder and decoder.

[0182] The following is in conjunction with the appendix Figure 4 Taking the application of the depth map generation method provided in this specification in the control of autonomous vehicles as an example, the depth map generation method will be further explained. Figure 4 This specification illustrates a process flowchart of a depth map generation method for autonomous vehicle control provided in one embodiment, which specifically includes the following steps.

[0183] Step 402: Obtain the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit;

[0184] Currently, the field of autonomous vehicle control demands higher perception capabilities from autonomous vehicles. However, LiDAR units have limited perception capabilities and are costly, making it difficult to directly obtain depth maps with dense feature points. Directly using sparse point cloud maps combined with deep learning methods for depth completion, resulting in a depth map with dense feature points, can significantly reduce costs while improving perception capabilities. Therefore, achieving high-accuracy depth completion is crucial. The resulting depth map with dense feature points allows the motion control system of autonomous vehicles to further enhance perception capabilities while reducing costs, improving control accuracy, and ultimately enhancing the motion safety and handling of autonomous vehicles.

[0185] Step 404: Input the scene color map and scene point cloud map into the specified generation model to obtain the first scene depth map and the feature information output by each decoding layer in the Swing Transformer model;

[0186] Step 406: Obtain the query information corresponding to each decoding layer;

[0187] Step 408: Based on the feature information output by each decoding layer, determine the key information and value information corresponding to each decoding layer;

[0188] Step 410: Based on a specific depth range, perform depth range segmentation on the first scene depth map to obtain the first scene depth map corresponding to each decoding layer;

[0189] Step 412: Based on the first scene depth map corresponding to each decoding layer, perform masked cross-attention calculation on the query information, key information and value information corresponding to each decoding layer to obtain the encoding guidance vector;

[0190] Step 414: Based on the encoding guidance vector, adjust the attention parameters of the deep encoder in the encoder of the Swing Transformer model;

[0191] Step 416: Using the parameter-adjusted depth encoder, encode the scene point cloud map and the first scene depth map to obtain the first scene image features;

[0192] Step 418: Using the parameter-adjusted image encoder, encode the scene color image and the first scene depth image to obtain the second scene image features;

[0193] Step 420: Using the decoder of the Swing Transformer model, decode the features of the first scene image and the features of the second scene image to obtain the depth map of the second scene;

[0194] Step 422: Obtain the confidence distribution information of the first scene depth map and the second scene depth map;

[0195] Step 424: Based on the confidence distribution information of the first scene depth map and the second scene depth map, normalize the first scene depth map and the second scene depth map to obtain the updated second scene depth map.

[0196] Step 426: Perform motion control on the autonomous vehicle based on the second scene depth map.

[0197] In the embodiments of this specification, based on the first scene depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to each decoding layer to obtain the encoding guidance vector. This allows the obtained encoding guidance vector to represent the corresponding specific depth range and has encoding attention for the specific depth range, thereby enabling the subsequent encoder to have high specificity, enhancing the encoder's feature extraction capability in the specific depth range, ensuring that the encoded scene image features fully represent the depth features of the specific depth range, improving the accuracy of the decoded second scene depth map, improving the control accuracy of the autonomous vehicle, and improving the motion safety and maneuverability of the autonomous vehicle. Furthermore, due to the improved efficiency of scene depth map generation, the control accuracy of the autonomous vehicle is further improved, enhancing the motion safety and maneuverability of the autonomous vehicle.

[0198] Figure 5 This specification illustrates a schematic diagram of the processing flow of a depth map generation method according to an embodiment of this specification, as shown below. Figure 5 As shown.

[0199] The original color image and original depth image are input into a specified generative model, and feature fusion encoding is used to generate a first depth image. Based on the feature information output by each decoding layer of the specified generative model, key and value information are determined. Masked cross-attention calculation is performed on the first depth image based on query information, key information, and value information. Combined with self-attention calculation and FNN model processing, adjusted query information is obtained. The adjusted query information, key information, and value information are passed between the decoding layers of each stage to determine the encoding guidance vector. Based on the encoding guidance vector, a depth encoder is used to perform feature encoding on the first depth image and the original depth image. Based on the encoding guidance vector, an image encoder is used to perform feature encoding on the first depth image and the original color image. The image features obtained above are input into the decoder for decoding to obtain a second depth image. Based on the confidence distribution information of the first depth image and the confidence distribution information of the second depth image, the first depth image and the second depth image are normalized to obtain an updated second depth image.

[0200] It should be noted that the information and data such as color images and depth images involved in the above method embodiments are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0201] Corresponding to the above method embodiments, this specification also provides embodiments of a depth map generation apparatus. Figure 6 A schematic diagram of a depth map generation apparatus according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0202] The first acquisition module 602 is configured to acquire the original color image and the original depth image corresponding to the original color image;

[0203] The first generation module 604 is configured to input the original color image and the original depth image into a specified generation model to obtain the first depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0204] The first calculation module 606 is configured to perform mask cross-attention calculation on the first depth map based on the feature information output by each decoding layer and a specific depth range to obtain the coding guidance vector.

[0205] The first encoding module 608 is configured to use an encoder to encode the original depth map and the first depth map based on the encoding guide vector to obtain the first image features, and to encode the original color map and the first depth map to obtain the second image features.

[0206] The first decoding module 610 is configured to use a decoder to decode the first image features and the second image features to obtain a second depth map.

[0207] Optionally, the first computing module 606 is further configured as follows:

[0208] Obtain the query information corresponding to each decoding layer; determine the key information and value information corresponding to each decoding layer based on the feature information output by each decoding layer; perform depth range segmentation on the first depth map based on a specific depth range to obtain the first depth map corresponding to each decoding layer; perform masked cross-attention calculation on the query information, key information and value information corresponding to each decoding layer based on the first depth map corresponding to each decoding layer to obtain the coding guidance vector.

[0209] Optionally, the first computing module 606 is further configured as follows:

[0210] For any decoding layer, based on the first depth map corresponding to that decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to that decoding layer to obtain the encoding guidance information corresponding to that decoding layer; the query information is adjusted according to the encoding guidance information corresponding to that decoding layer and the feature information output by that decoding layer; the key information and value information corresponding to the next decoding layer are determined according to the adjusted query information, key information, value information, and the feature information output by the next decoding layer; and encoding guidance vectors are generated based on the encoding guidance information corresponding to each decoding layer.

[0211] Optionally, the device further includes:

[0212] The normalization module is configured to obtain the confidence distribution information of the first depth map and the second depth map; based on the confidence distribution information of the first depth map and the second depth map, the first depth map and the second depth map are normalized to obtain an updated second depth map.

[0213] Optionally, the first encoding module 608 is further configured as follows:

[0214] Based on the coding guidance vector, the attention parameters of the specified coding layer in the encoder are adjusted; using the encoder with adjusted parameters, the original depth map and the first depth map are encoded to obtain the first image features.

[0215] Optionally, the device further includes:

[0216] The training module is configured to acquire a sample set, which includes multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images; extract a first sample color image and a first sample depth map corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the multiple sample color images; input the first sample color image and the first sample depth map into a specified generation model to obtain a first training depth map and feature information output by each decoding layer in the specified generation model; perform masked cross-attention calculation on the first training depth map based on the feature information output by each decoding layer and a specific depth range to obtain a training coding guidance vector; and use an encoder, based on the training coding guidance vector, to perform masked cross-attention calculation on the first training depth map to obtain a training coding guidance vector. Encode the first sample depth map and the first training depth map to obtain the first training image features, and encode the first sample color image and the first training depth map to obtain the second training image features. Use a decoder to decode the first training image features and the second training image features to obtain the second training depth map. Calculate the loss value based on the second training depth map and the first verification depth map, where the first verification depth map is the verification depth map corresponding to the first sample color image. Adjust the model parameters of the encoder and decoder according to the loss value, and return to the step of extracting the first sample color image and the first sample depth map corresponding to the first sample color image from the sample set until the preset training termination condition is met to obtain the trained encoder and decoder.

[0217] In this embodiment, an original color image and its corresponding original depth map are obtained; the original color image and the original depth map are input into a specified generation model to obtain a first depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first depth map to obtain an encoding guidance vector; using an encoder, based on the encoding guidance vector, the original depth map and the first depth map are encoded to obtain a first image feature, and the original color image and the first depth map are encoded to obtain a second image feature; using a decoder, the first image feature and the second image feature are decoded to obtain a second depth map. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second depth map, avoids depth completion through multiple iterations, and improves the generation efficiency of the depth map.

[0218] The above is a schematic scheme of a depth map generation device according to this embodiment. It should be noted that the technical solution of this depth map generation device and the technical solution of the depth map generation method described above belong to the same concept. For details not described in detail in the technical solution of the depth map generation device, please refer to the description of the technical solution of the depth map generation method described above.

[0219] Corresponding to the above method embodiments, this specification also provides embodiments of mobile device control devices. Figure 7 This specification illustrates a schematic diagram of a mobile device control apparatus according to one embodiment of the present specification. The mobile device in this apparatus includes an image acquisition unit and a lidar unit, such as... Figure 7 As shown, the device includes:

[0220] The second acquisition module 702 is configured to acquire the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit;

[0221] The second generation module 704 is configured to input the scene color map and the scene point cloud map into the specified generation model to obtain the first scene depth map and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth maps.

[0222] The second calculation module 706 is configured to perform mask cross-attention calculation on the first scene depth map based on the feature information output by each decoding layer and a specific depth range, to obtain the coding guidance vector.

[0223] The second encoding module 708 is configured to use an encoder to encode the scene point cloud map and the first scene depth map based on the encoding guidance vector to obtain the first scene image features, and to encode the scene color map and the first scene depth map to obtain the second scene image features.

[0224] The second decoding module 710 is configured to use the decoder to decode the features of the first scene image and the features of the second scene image to obtain the second scene depth map;

[0225] The motion control module 712 is configured to perform motion control on the mobile device based on a second scene depth map.

[0226] In this embodiment, a scene color image acquired by an image acquisition unit and a scene point cloud image acquired by a lidar unit are obtained. The scene color image and scene point cloud image are input into a specified generation model to obtain a first scene depth map and feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with depth map generation function. Based on the feature information output by each decoding layer and a specific depth range, a masked cross-attention calculation is performed on the first scene depth map to obtain an encoding guidance vector. Using an encoder, based on the encoding guidance vector, the scene point cloud image and the first scene depth map are encoded to obtain first scene image features, and the scene color image and the first scene depth map are encoded to obtain second scene image features. Using a decoder, the first scene image features and the second scene image features are decoded to obtain a second scene depth map. Based on the second scene depth map, motion control is performed on the mobile device. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first scene depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded scene image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second scene depth map, improves the control accuracy of the mobile device, and improves the motion safety and operability of the mobile device. Furthermore, due to the improved generation efficiency of the depth map, the control accuracy of the mobile device is further improved, and the motion safety and operability of the mobile device are enhanced.

[0227] The above is a schematic scheme of a mobile device control device according to this embodiment. It should be noted that the technical solution of this mobile device control device and the technical solution of the above-described mobile device control method belong to the same concept. For details not described in detail in the technical solution of the mobile device control device, please refer to the description of the technical solution of the above-described mobile device control method.

[0228] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing apparatus applied to depth map generation. Figure 8 This specification illustrates a schematic diagram of a data processing apparatus for depth map generation according to an embodiment of this specification. This apparatus is applied to cloud-side devices, such as… Figure 8 As shown, the device includes:

[0229] The third acquisition module 802 is configured to acquire a sample set, wherein the sample set includes multiple sample color images, sample depth images corresponding to the multiple sample color images, and verification depth images corresponding to the multiple sample color images.

[0230] The sample extraction module 804 is configured to extract a first sample color image and a first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of multiple sample color images;

[0231] The third generation module 806 is configured to input the first sample color image and the first sample depth image into the specified generation model to obtain the first training depth image and the feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with the function of generating depth images.

[0232] The third calculation module 808 is configured to perform mask cross-attention calculation on the first training depth map based on the feature information output by each decoding layer and a specific depth range, to obtain the training encoding guidance vector.

[0233] The third encoding module 810 is configured to use an encoder to encode the first sample depth map and the first training depth map based on the training encoding guide vector to obtain the first training image features, and to encode the first sample color map and the first training depth map to obtain the second training image features.

[0234] The third decoding module 812 is configured to use the decoder to decode the features of the first training image and the features of the second training image to obtain the second training depth map.

[0235] The loss calculation module 814 is configured to calculate the loss value based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image;

[0236] The model training module 816 is configured to adjust the model parameters of the encoder and decoder according to the loss value, and return to perform the steps of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set until the preset training termination condition is met, so as to obtain the trained model parameters of the encoder and decoder.

[0237] The transmitting module 818 is configured to send the model parameters of the encoder and decoder to the end-side device.

[0238] In this embodiment, a sample set is obtained, comprising multiple sample color images, sample depth maps corresponding to the multiple sample color images, and verification depth maps corresponding to the multiple sample color images; a first sample color image and a first sample depth map corresponding to the first sample color image are extracted from the sample set, wherein the first sample color image is any one of the multiple sample color images; the first sample color image and the first sample depth map are input into a specified generation model to obtain a first training depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding / decoding model with depth map generation function; based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first training depth map to obtain a training encoding guidance vector; using an encoder, based on the training encoding guidance vector... The process involves encoding a first sample depth map and a first training depth map to obtain first training image features, and encoding a first sample color image and the first training depth map to obtain second training image features. A decoder is then used to decode the first and second training image features to obtain a second training depth map. A loss value is calculated based on the second training depth map and a first verification depth map, where the first verification depth map is the verification depth map corresponding to the first sample color image. The model parameters of the encoder and decoder are adjusted according to the loss value, and the process returns to extracting the first sample color image and its corresponding first sample depth map from the sample set until a preset training termination condition is met, at which point the trained model parameters of the encoder and decoder are obtained. Finally, the model parameters of the encoder and decoder are sent to the edge device. Based on the feature information output by each decoding layer and a specific depth range, masked cross-attention calculation is performed on the first training depth map, so that the resulting coding guidance vector can represent the corresponding specific depth range and has coding attention for the specific depth range. This makes the subsequent encoder highly specific, enhances the encoder's feature extraction capability in the specific depth range, ensures that the encoded image features fully represent the depth features of the specific depth range, improves the accuracy of the decoded second training depth map, improves the training effect of the encoder and decoder in a single training, and improves the training efficiency of the encoder and trainer while ensuring the accuracy of the trained encoder and decoder.

[0239] The above is an illustrative scheme of a data processing apparatus for depth map generation according to this embodiment. It should be noted that the technical solution of this data processing apparatus for depth map generation belongs to the same concept as the technical solution of the data processing method for depth map generation described above. Details not described in detail in the technical solution of the data processing apparatus for depth map generation can be found in the description of the technical solution of the data processing method for depth map generation described above.

[0240] Figure 9A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0241] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include PSTN (Public Switched Telephone Network), LAN (Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), or combinations of communication networks such as the Internet. Access device 940 may include one or more of any type of wired or wireless network interface (e.g., NIC (Network Interface Controller)), such as IEEE 802.12 WLAN (Wireless Local Area Networks) wireless interface, Wi-MAX (World Internet for Microwave Access) interface, Ethernet interface, USB (Universal Serial Bus) interface, cellular network interface, Bluetooth interface, NFC (Near Field Communication) interface, and so on.

[0242] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0243] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs (personal computers). The computing device 900 can also be a mobile or stationary server.

[0244] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the aforementioned depth map generation method, mobile device control method, or data processing method applied to depth map generation.

[0245] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the depth map generation method, the mobile device control method, and the data processing method applied to depth map generation described above. For details not described in detail in the technical solution of the computing device, please refer to the descriptions of the technical solutions of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation described above.

[0246] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation.

[0247] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the depth map generation method, the mobile device control method, and the data processing method applied to depth map generation described above. For details not described in detail in the technical solution of the storage medium, please refer to the descriptions of the technical solutions of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation described above.

[0248] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the depth map generation method, the mobile device control method, or the data processing method applied to depth map generation.

[0249] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solutions of the depth map generation method, mobile device control method, and data processing method applied to depth map generation described above. For details not described in detail in the technical solution of the computer program, please refer to the descriptions of the technical solutions of the depth map generation method, mobile device control method, or data processing method applied to depth map generation described above.

[0250] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0251] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0252] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0253] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0254] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A depth map generation method, comprising: Obtain the original color image and the corresponding original depth image; The original color image and the original depth image are input into a specified generation model to obtain a first depth image and feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with depth image generation function. Based on the feature information output by each decoding layer, the key information and value information corresponding to each decoding layer are determined; based on a specific depth range, the first depth map is segmented to obtain the first depth map corresponding to each decoding layer; based on the first depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to each decoding layer to obtain the encoding guidance vector; the specific depth range is based on multiple depth ranges set according to the decoding layers in the specified generation model; Using an encoder, based on the encoding guidance vector, the original depth map and the first depth map are encoded to obtain a first image feature, and the original color map and the first depth map are encoded to obtain a second image feature; The first image features and the second image features are decoded using a decoder to obtain a second depth map.

2. The method according to claim 1, wherein the step of performing masked cross-attention calculation on the query information, key information, and value information corresponding to each decoding layer based on the first depth map corresponding to each decoding layer to obtain the coding guidance vector includes: For any decoding layer, based on the first depth map corresponding to the decoding layer, masked cross-attention calculation is performed on the query information, key information and value information corresponding to the decoding layer to obtain the encoding guidance information corresponding to the decoding layer; Adjust the query information based on the encoding guidance information corresponding to the decoding layer and the feature information output by the decoding layer; Based on the adjusted query information, the key information, the value information, and the feature information output by the next decoding layer, determine the key information and value information corresponding to the next decoding layer; Based on the encoding guidance information corresponding to each decoding layer, an encoding guidance vector is generated.

3. The method according to claim 1, after decoding the first image features and the second image features using a decoder to obtain a second depth map, further comprising: Obtain the confidence distribution information of the first depth map and the second depth map; Based on the confidence distribution information of the first depth map and the second depth map, the first depth map and the second depth map are normalized to obtain an updated second depth map.

4. The method according to claim 1, wherein the step of encoding the original depth map and the first depth map using an encoder based on the encoding guidance vector to obtain the first image feature includes: Based on the coding guidance vector, the attention parameters of a specified coding layer in the encoder are adjusted; The encoder with adjusted parameters is used to encode the original depth map and the first depth map to obtain the first image feature.

5. The method according to any one of claims 1-4, further comprising, before obtaining the original color image and the original depth image corresponding to the original color image: Obtain a sample set, wherein the sample set includes multiple sample color images, sample depth images corresponding to the multiple sample color images, and verification depth images corresponding to the multiple sample color images; Extract a first sample color image and a first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the plurality of sample color images; Input the first sample color image and the first sample depth image into the specified generation model to obtain the first training depth image and the feature information output by each decoding layer in the specified generation model; Based on the feature information output by each decoding layer and a specific depth range, mask cross-attention calculation is performed on the first training depth map to obtain the training coding guidance vector. Using the encoder, based on the training coding guidance vector, the first sample depth map and the first training depth map are encoded to obtain the first training image features, and the first sample color map and the first training depth map are encoded to obtain the second training image features. Using a decoder, the features of the first training image and the features of the second training image are decoded to obtain a second training depth map; The loss value is calculated based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image; Based on the loss value, adjust the model parameters of the encoder and the decoder, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set, until the preset training termination condition is met, and obtain the trained encoder and decoder.

6. A mobile device control method, wherein the mobile device is equipped with an image acquisition unit and a lidar unit, comprising: The scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit are obtained; The scene color map and the scene point cloud map are input into a specified generation model to obtain a first scene depth map and feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with depth map generation function. Based on the feature information output by each decoding layer, the key information and value information corresponding to each decoding layer are determined; based on a specific depth range, the first scene depth map is segmented to obtain the first scene depth map corresponding to each decoding layer; based on the first scene depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to each decoding layer to obtain the encoding guidance vector; the specific depth range is based on multiple depth ranges set according to the decoding layers in the specified generation model; Using an encoder, based on the encoding guidance vector, the scene point cloud map and the first scene depth map are encoded to obtain first scene image features, and the scene color map and the first scene depth map are encoded to obtain second scene image features; Using a decoder, the features of the first scene image and the features of the second scene image are decoded to obtain a second scene depth map; Motion control is performed on the mobile device based on the second scene depth map.

7. A data processing method for depth map generation, applied to cloud-side devices, comprising: Obtain a sample set, wherein the sample set includes multiple sample color images, sample depth images corresponding to the multiple sample color images, and verification depth images corresponding to the multiple sample color images; Extract a first sample color image and a first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the plurality of sample color images; The first sample color image and the first sample depth image are input into a specified generation model to obtain a first training depth image and feature information output by each decoding layer in the specified generation model. The specified generation model is a pre-acquired encoding and decoding model with depth image generation function. Based on the feature information output by each decoding layer, the key information and value information corresponding to each decoding layer are determined; based on a specific depth range, the first training depth map is segmented to obtain the first training depth map corresponding to each decoding layer; based on the first training depth map corresponding to each decoding layer, masked cross-attention calculation is performed on the query information, key information, and value information corresponding to each decoding layer to obtain the training encoding guidance vector; the specific depth range is based on multiple depth ranges set according to the decoding layers in the specified generative model; Using the encoder, based on the training coding guidance vector, the first sample depth map and the first training depth map are encoded to obtain the first training image features, and the first sample color map and the first training depth map are encoded to obtain the second training image features. Using a decoder, the features of the first training image and the features of the second training image are decoded to obtain a second training depth map; The loss value is calculated based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image; Based on the loss value, adjust the model parameters of the encoder and the decoder, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set until the preset training end condition is met, and obtain the trained model parameters of the encoder and decoder. The model parameters of the encoder and decoder are sent to the end-side device.

8. A depth map generation apparatus, the apparatus comprising: The first acquisition module is configured to acquire the original color image and the original depth image corresponding to the original color image; The first generation module is configured to input the original color image and the original depth image into a specified generation model to obtain a first depth image and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding and decoding model with depth image generation function; The first calculation module is configured to determine the key information and value information corresponding to each decoding layer based on the feature information output by each decoding layer; to perform depth range segmentation on the first depth map based on a specific depth range to obtain the first depth map corresponding to each decoding layer; and to perform masked cross-attention calculation on the query information, key information, and value information corresponding to each decoding layer based on the first depth map corresponding to each decoding layer to obtain the encoding guidance vector; the specific depth range is determined according to multiple depth ranges set for the decoding layers in the specified generation model. The first encoding module is configured to use an encoder to encode the original depth map and the first depth map based on the encoding guidance vector to obtain a first image feature, and to encode the original color map and the first depth map to obtain a second image feature; The first decoding module is configured to use a decoder to decode the first image features and the second image features to obtain a second depth map.

9. A mobile device control device, wherein the mobile device is provided with an image acquisition unit and a lidar unit, the device comprising: The second acquisition module is configured to acquire the scene color image acquired by the image acquisition unit and the scene point cloud image acquired by the lidar unit; The second generation module is configured to input the scene color image and the scene point cloud image into a specified generation model to obtain a first scene depth map and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding and decoding model with depth map generation function. The second calculation module is configured to determine the key information and value information corresponding to each decoding layer based on the feature information output by each decoding layer; to perform depth range segmentation on the first scene depth map based on a specific depth range to obtain the first scene depth map corresponding to each decoding layer; and to perform masked cross-attention calculation on the query information, key information, and value information corresponding to each decoding layer based on the first scene depth map corresponding to each decoding layer to obtain the encoding guidance vector; the specific depth range is determined according to multiple depth ranges set for the decoding layers in the specified generation model. The second encoding module is configured to use an encoder to encode the scene point cloud map and the first scene depth map based on the encoding guidance vector to obtain first scene image features, and to encode the scene color map and the first scene depth map to obtain second scene image features. The second decoding module is configured to use a decoder to decode the features of the first scene image and the features of the second scene image to obtain a second scene depth map. The motion control module is configured to perform motion control on the mobile device based on the second scene depth map.

10. A data processing apparatus for depth map generation, applied to cloud-side equipment, the apparatus comprising: The third acquisition module is configured to acquire a sample set, wherein the sample set includes multiple sample color images, sample depth images corresponding to the multiple sample color images, and verification depth images corresponding to the multiple sample color images. The sample extraction module is configured to extract a first sample color image and a first sample depth image corresponding to the first sample color image from the sample set, wherein the first sample color image is any one of the plurality of sample color images; The third generation module is configured to input the first sample color image and the first sample depth image into a specified generation model to obtain a first training depth image and feature information output by each decoding layer in the specified generation model, wherein the specified generation model is a pre-acquired encoding and decoding model with depth image generation function. The third calculation module is configured to determine the key information and value information corresponding to each decoding layer based on the feature information output by each decoding layer; to perform depth range segmentation on the first training depth map based on a specific depth range to obtain the first training depth map corresponding to each decoding layer; and to perform masked cross-attention calculation on the query information, key information, and value information corresponding to each decoding layer based on the first training depth map corresponding to each decoding layer to obtain the training encoding guidance vector; the specific depth range is determined according to multiple depth ranges corresponding to the decoding layers in the specified generative model. The third encoding module is configured to use an encoder to encode the first sample depth map and the first training depth map based on the training encoding guidance vector to obtain the first training image features, and to encode the first sample color map and the first training depth map to obtain the second training image features. The third decoding module is configured to use a decoder to decode the features of the first training image and the features of the second training image to obtain a second training depth map. The loss calculation module is configured to calculate the loss value based on the second training depth map and the first verification depth map, wherein the first verification depth map is the verification depth map corresponding to the first sample color image; The model training module is configured to adjust the model parameters of the encoder and the decoder according to the loss value, and return to the step of extracting the first sample color image and the first sample depth image corresponding to the first sample color image from the sample set until the preset training end condition is met, so as to obtain the trained model parameters of the encoder and decoder. The sending module is configured to send the model parameters of the encoder and decoder to the end-side device.

11. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the depth map generation method according to any one of claims 1 to 5, the mobile device control method according to claim 6, or the data processing method applied to depth map generation according to claim 7.

12. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the depth map generation method of any one of claims 1 to 5, the mobile device control method of claim 6, or the data processing method applied to depth map generation of claim 7.

Images