Visible light image guided lidar depth completion in-memory indexing method and apparatus

Abstract

The application provides a visible light image guided laser radar depth completion memory index method and device, and relates to the technical field of digital image processing. The method comprises the following steps: extracting and splicing RGB features and depth features; obtaining depth neighborhood feature values corresponding to multiple spatial neighborhood directions for a depth center point in the spliced feature map, and classifying the multiple spatial neighborhood directions according to spatial orientations; generating quantized input indexes corresponding to the spatial neighborhood directions according to the depth center feature value, the RGB center feature value and the depth neighborhood feature values; according to the classification results and the quantized input indexes, querying a pre-constructed lookup table in parallel to obtain intermediate fusion results of the spatial neighborhood directions; summing the intermediate fusion results of all the spatial neighborhood directions to obtain a fusion output feature; wherein, the fusion output of all the indexes is pre-calculated by using a trained RGBD fusion network to construct the pre-constructed lookup table. In this way, the calculation amount and inference delay are greatly reduced.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, and in particular to a method and apparatus for in-memory indexing of depth completion for lidar guided by visible light images. Background Technology

[0002] The depth completion task of LiDAR aims to reconstruct high-quality, dense depth maps based on sparse point clouds and corresponding high-resolution color (RGB) images. High-quality, dense depth information is crucial for numerous downstream applications, such as 3D reconstruction, autonomous driving environmental perception, robot navigation and obstacle avoidance, and scene understanding and interaction in augmented reality or virtual reality. In depth completion neural networks, the visible light-depth (RGB-D) fusion method is one of the core components determining overall performance. This RGBD fusion method is responsible for extracting semantic and texture features from RGB images and depth information from the depth map, and then synergistically integrating the information from these two modalities through a specific fusion strategy. This guides the network to accurately fill in missing regions in the depth map and restore fine depth details.

[0003] Currently, existing deep completion methods have made significant progress in improving completion accuracy, and can be mainly classified into methods based on spatial propagation networks, methods based on attention mechanisms, and methods based on multi-scale feature interactions. These methods typically design complex RGBD fusion modules to capture the correlation of cross-modal features, such as using iterative spatial propagation networks to repeatedly update neighborhood features, or introducing global attention mechanisms to enhance long-distance dependencies. However, these high-performance fusion strategies generally suffer from excessively high computational complexity. For example, the fusion module in multi-scale feature interaction-based methods often requires billions of Giga Multiply-Accumulate Operations (GMACs). This huge computational overhead is manageable on workstations equipped with high-performance graphics processing units (GPUs), but on edge devices with limited computing resources, such as embedded autonomous driving computing platforms and mobile robot controllers, it can cause severe inference latency. Summary of the Invention

[0004] The purpose of this invention is to provide a visible light image-guided lidar depth completion in-memory indexing method and apparatus to solve the inference delay problem caused by the large amount of computation in the prior art.

[0005] To address the aforementioned technical problems, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention provides a visible light image-guided lidar depth completion in-memory indexing method, comprising:

[0007] Feature extraction is performed on the RGB image and LiDAR depth image to be completed, and the extracted RGB features and the extracted depth features are concatenated along the channel dimension to obtain the concatenated feature map;

[0008] For the depth center point in the stitched feature map, obtain the depth neighborhood feature values ​​of multiple spatial neighborhood directions in the stitched feature map, and classify the multiple spatial neighborhood directions according to their spatial orientation to obtain the classification results;

[0009] Based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point, a quantized input index corresponding to each spatial neighborhood direction is generated.

[0010] Based on the classification results and the corresponding quantization input index, a pre-built lookup table matching the categories in the classification results is queried in parallel to obtain intermediate fusion results for each spatial neighborhood direction.

[0011] The intermediate fusion results of all spatial neighborhood directions are summed to obtain the fusion output feature of the depth center point. The fusion output feature is used to generate the final completed depth map.

[0012] The pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the deep center features, RGB center features and deep neighborhood features are quantized into integers with a preset bit width and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table.

[0013] A second aspect of the present invention provides a visible light image-guided lidar depth completion memory indexing device, comprising:

[0014] The extraction and stitching module is used to extract features from the RGB image and the LiDAR depth image to be completed, and to stitch the extracted RGB features and the extracted depth features along the channel dimension to obtain a stitched feature map.

[0015] The classification module is used to obtain the corresponding depth neighborhood feature values ​​of multiple spatial neighborhood directions in the stitched feature map for the depth center point, and classify the multiple spatial neighborhood directions according to the spatial orientation to obtain the classification result.

[0016] The generation module is used to generate quantized input indices corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point.

[0017] The parallel query module is used to query the pre-built lookup table that matches the category in the classification result based on the classification result and the corresponding quantized input index, so as to obtain the intermediate fusion result in each spatial neighborhood direction. The pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the deep center feature, RGB center feature and deep neighborhood feature are quantized into integers with a preset bit width and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table.

[0018] The summation module is used to sum the intermediate fusion results in all spatial neighborhood directions to obtain the fusion output features of the depth center point. The fusion output features are used to generate the final completed depth map.

[0019] Compared to existing technologies, the visible light image-guided lidar depth completion in-memory indexing method and apparatus provided by this invention extracts features from the RGB image to be completed and the lidar depth image, and concatenates the extracted RGB features and the extracted depth features along the channel dimension to obtain a concatenated feature map. For the depth center point in the concatenated feature map, the depth neighborhood feature values ​​corresponding to multiple spatial neighborhood directions in the concatenated feature map are obtained, and the multiple spatial neighborhood directions are classified according to their spatial orientation to obtain classification results. Based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point, a database indexing method and apparatus are generated for each spatial neighborhood feature point. The quantized input index corresponds to the neighborhood direction; based on the classification result and the corresponding quantized input index, a pre-built lookup table matching the category in the classification result is queried in parallel to obtain the intermediate fusion result for each spatial neighborhood direction; the intermediate fusion results for all spatial neighborhood directions are summed to obtain the fusion output feature of the depth center point; the pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the depth center feature, RGB center feature and depth neighborhood feature are quantized into integers of a preset bit width and concatenated into an index; the fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table. In this way, based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point, a quantized input index corresponding to each spatial neighborhood direction is generated, which compresses the input index. Using the trained RGBD fusion network, the depth center feature, RGB center feature, and depth neighborhood feature are quantized into integers of a preset bit width and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct a pre-built lookup table. This pre-calculated and stored lookup table is extremely small and can be directly embedded into the on-chip cache. Parallel memory access operations can be performed based on the quantized input index. This not only completely eliminates the multiplication and addition overhead of convolution operations, but also greatly reduces data transfer and off-chip memory access latency, significantly reducing computational load and inference latency. Attached Figure Description

[0020] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, with the same or corresponding reference numerals denoteing the same or corresponding parts, wherein:

[0021] Figure 1 A flowchart illustrating a visible light image-guided lidar depth completion in-memory indexing method is shown.

[0022] Figure 2 A schematic diagram illustrating a decoupling method for spatial propagation networks is shown.

[0023] Figure 3 A schematic diagram illustrating depth-visible feature map rearrangement and neighborhood types is shown.

[0024] Figure 4 The diagram illustrates the process of quantitative perception training and conversion into a look-up table (LUT).

[0025] Figure 5 A schematic diagram illustrating the parallel table lookup process is shown below.

[0026] Figure 6 A schematic diagram of the affinity sensing module structure is shown.

[0027] Figure 7 A schematic diagram of the propagation calculation module structure is shown.

[0028] Figure 8 A schematic diagram of the overall structure of the completion method is shown.

[0029] Figure 9 A visual comparison diagram illustrating the differences between the method of the present invention and the prior art is shown schematically.

[0030] Figure 10 A schematic diagram of a visible light image-guided lidar depth completion memory indexing device is shown. Detailed Implementation

[0031] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0032] It should be noted that, unless otherwise stated, the technical or scientific terms used in this invention should have the ordinary meaning as understood by one of ordinary skill in the art.

[0033] The methods described in the embodiments of the present invention will be explained in detail below.

[0034] Figure 1 A flowchart illustrating the in-memory indexing method for visible light image-guided lidar depth completion in an embodiment of the present invention is shown. See [link to relevant documentation]. Figure 1 As shown, the visible light image-guided lidar depth completion in-memory indexing method may include:

[0035] S101. Extract features from the RGB image and the LiDAR depth image to be completed, and then concatenate the extracted RGB features and the extracted depth features along the channel dimension to obtain a concatenated feature map.

[0036] Specifically, the extracted RGB features and the extracted depth features are concatenated along the channel dimension to obtain a concatenated feature map. This includes: using an alternating arrangement strategy to alternately arrange the RGB features and depth features along the channel dimension to obtain the concatenated feature map.

[0037] Figure 2 A schematic diagram illustrating a decoupling method for spatial propagation networks, employing an interval arrangement strategy, is shown below. Figure 2 As shown in the left image, RGB features and depth features are arranged alternately. This alternating arrangement is suitable for the grouped convolution operation of the subsequent affinity sensing module, ensuring that the data processed by each group of convolution kernels conforms to the visible light-depth data structure.

[0038] S102. For the depth center point in the stitched feature map, obtain the depth neighborhood feature values ​​of multiple spatial neighborhood directions in the stitched feature map, and classify the multiple spatial neighborhood directions according to their spatial orientation to obtain the classification results.

[0039] in, Figure 3 A schematic diagram illustrating depth-visible feature map rearrangement and neighborhood types is shown below. Figure 3 As shown, the classification results include S-type neighborhood types and C-type neighborhood types. The S-type neighborhood type corresponds to the type of deep neighborhood feature values ​​in the four orthogonal directions, while the C-type neighborhood type corresponds to the type of deep neighborhood feature values ​​in the four diagonal directions.

[0040] Step S102 can be referred to as a decoupling method for spatial propagation networks.

[0041] For details, see Figure 2As shown in the right-hand figure, for the depth center point in the stitched feature map, the depth neighborhood feature values ​​corresponding to multiple spatial neighborhood directions in the stitched feature map are obtained, and the multiple spatial neighborhood directions are classified according to their spatial orientation to obtain the classification results, including:

[0042] Step A1: Divide the multiple spatial neighborhood directions of the depth center point into four orthogonal directions and four diagonal directions according to their spatial orientation.

[0043] The four orthogonal directions are up, down, left, and right, and the four diagonal directions are up-left, up-right, down-left, and down-right.

[0044] Step A2: Based on the preset translation and stitching operations, obtain the depth neighborhood feature values ​​corresponding to the four orthogonal directions and the four diagonal directions respectively.

[0045] Specifically, step A2 includes:

[0046] Step A21: Perform a filling operation on the depth features in the stitched feature map to obtain a filled feature map.

[0047] Step A22: Translate the filled feature map according to the first set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four orthogonal directions coincide with the coordinates of the depth center feature, and obtain the depth neighborhood feature values ​​corresponding to the four orthogonal directions.

[0048] The first group of translation vectors are translation vectors of the S-class neighborhood type. The first group of translation vectors are: upward translation (1,0), right translation (0,1), left translation (0,-1), and downward translation (-1,0).

[0049] By combining the filling operation with the first set of translation vectors, the feature map is translated so that the coordinates of the S-type deep neighborhood feature pixels in the four directions of up, down, left, and right coincide with the coordinates of the depth center feature, and the corresponding deep neighborhood feature values ​​in the four directions of up, down, left, and right are obtained.

[0050] Step A23: Translate the filled feature map according to the second set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four diagonal directions coincide with the coordinates of the depth center feature, and obtain the depth neighborhood feature values ​​corresponding to the four diagonal directions.

[0051] The second group of translation vectors are translation vectors of the C-class neighborhood type. The second group of translation vectors are left-top translation (1,1), right-top translation (1,-1), left-bottom translation (-1,1) and right-bottom translation (-1,-1).

[0052] By combining the filling operation with the second set of translation vectors, the filled feature map is translated so that the coordinates of the C-type deep neighborhood feature pixels in the four directions of top left, top right, bottom left, and bottom right coincide with the coordinates of the depth center feature, and the corresponding deep neighborhood feature values ​​in the four directions of top left, top right, bottom left, and bottom right are obtained.

[0053] Step A3: The depth neighborhood feature values ​​corresponding to the four orthogonal directions are spliced ​​along the first new dimension to form an S-type neighborhood feature cube.

[0054] The four feature maps corresponding to the depth neighborhood feature values ​​in the four directions of up, down, left, and right after the movement are stitched together along the first new dimension to form an S-type neighborhood feature cube.

[0055] Step A4: The depth neighborhood feature values ​​corresponding to the four diagonal directions are spliced ​​along the second new dimension to form a C-type neighborhood feature cube.

[0056] Among them, the classification results for multiple spatial neighborhood directions are composed of S-class neighborhood feature cubes and C-class neighborhood feature cubes.

[0057] The four feature maps corresponding to the depth neighborhood feature values ​​in the four directions of top left, top right, bottom left, and bottom right after the movement are spliced ​​along the second new dimension to form a C-type neighborhood feature cube.

[0058] S103. Generate a quantized input index corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point.

[0059] For example, based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point, eight sets of quantized input indices corresponding to each spatial neighborhood direction are generated.

[0060] Specifically, based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point, a quantized input index corresponding to each spatial neighborhood direction is generated, including:

[0061] Step B1: Quantize the depth center feature value into a first integer value with a first preset bit width.

[0062] The first preset bit width is 4 bits, and the first integer value with the first preset bit width is the first integer value with 4 bits.

[0063] The Quantization Aware Training (QAT) method is used to perform 4-bit quantization on the deep center feature values.

[0064] Step B2: Quantize the RGB center feature value into a second integer value with a first preset bit width.

[0065] The second integer value, having a first preset bit width, is a 4-bit second integer value. The RGB center feature value is quantized using the QAT method with 4 bits.

[0066] Step B3: Quantize the depth neighborhood feature values ​​corresponding to each spatial neighborhood direction into a third integer value with a first preset bit width.

[0067] The third integer value with a first preset bit width is a 4-bit third integer value. The QAT method is used to perform 4-bit quantization processing on the depth neighborhood feature values ​​corresponding to each spatial neighborhood direction.

[0068] Step B4: Concatenate the first integer value, the second integer value, and the third integer value to generate a quantized input index corresponding to each spatial neighborhood direction.

[0069] S104. Based on the classification results and the corresponding quantization input index, perform parallel queries on the pre-built lookup table that matches the category in the classification results to obtain intermediate fusion results for each spatial neighborhood direction.

[0070] The pre-built lookup table is constructed as follows: using the trained RGBD fusion network, the deep center features, RGB center features, and deep neighborhood features are quantized into integers of a preset bit width and concatenated into indices. The fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table. The pre-built lookup table includes S-type and C-type pre-built lookup tables.

[0071] Figure 4 The diagram illustrates the process of quantization training and conversion into a look-up table (LUT). See [link / reference]. Figure 4 As shown, specifically, the preset bit width is 4 bits. Using the trained RGBD fusion network, the deep center features, RGB center features, and deep neighborhood features are quantized into 4-bit integers and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct two types of LUTs: S-class pre-built lookup tables and C-class pre-built lookup tables. The two types of LUTs correspond to C-class neighborhood types and S-class neighborhood types, respectively.

[0072] See Figure 4As shown, the QAT method is used to perform 4-bit quantization on the deep center features, RGB center features, and deep neighborhood features, quantizing them into int4 integers. During training, pseudo-quantization nodes are inserted to simulate the quantization effect. In the dequantization process, a straight-through estimator (STE) is used to maintain gradient flow. The fused output is also quantized using the same method.

[0073] In both S-class and C-class neighborhood types, only one of the deep center feature, RGB center feature, and deep neighborhood feature is selected. After quantization, the deep center feature, RGB center feature, and deep neighborhood feature are all 4-bit values. The values ​​of these three features are concatenated and iterated to form 4096 indices. The trained RGBD fusion network is used to pre-calculate the fusion output corresponding to all input features. Finally, two types of LUTs are constructed, each containing 4096 groups of 16-channel 4-bit fusion outputs.

[0074] For example, based on the classification results and the corresponding 8 sets of quantized input indices, 8 table lookups are performed in parallel to obtain intermediate fusion results for each spatial neighborhood direction.

[0075] Specifically, Figure 5 A schematic diagram illustrating the parallel table lookup process is shown below. Figure 5 As shown, based on the classification results and the corresponding quantized input index, a pre-built lookup table matching the categories in the classification results is queried in parallel to obtain intermediate fusion results for each spatial neighborhood direction, including:

[0076] Step C1: Based on the S-type or C-type neighborhood type, synchronously send the corresponding quantized input index to the corresponding S-type or C-type pre-built lookup table for parallel retrieval.

[0077] Step C2: Read the intermediate fusion results for each spatial neighborhood direction from the S-class pre-built lookup table or the C-class pre-built lookup table.

[0078] Specifically, the training process of the completed RGBD fusion network includes:

[0079] Step D1: Calculate the affinity matrices corresponding to the S-type and C-type neighborhood types respectively using the affinity sensing module.

[0080] in, Figure 6 A schematic diagram of the affinity sensing module structure is shown below. Figure 6 As shown, the affinity sensing module includes a rotation integration operation module, a mask depth convolution module, and a point convolution module connected in sequence.

[0081] For example, the affinity sensing modules are divided into two categories based on neighborhood type. These two types of affinity sensing modules are used to calculate the affinity matrix in four directions. For different C-type or S-type neighborhood types in different directions, the affinity sensing modules are reused through rotation ensemble operations. Each affinity sensing module consists of a custom depthwise separable convolution and a point convolution concatenated together, outputting an affinity matrix.

[0082] Specifically, step D1 includes:

[0083] Step D11: Input the feature data of S-type and C-type neighborhood types into the rotation integration operation module respectively, so as to rotate the feature data of the same type of neighborhood to the preset reference direction, calculate using the shared network parameters, and rotate the calculation result back to the original orientation, and output the intermediate features after orientation correction.

[0084] For example, the preset reference direction can be the direction corresponding to 0°, 90°, 180°, and 270°. Feature data of S-type and C-type neighborhoods are input into the rotation integration module, allowing neighborhoods of the same type in different directions to share the same affinity sensing module parameters. During implementation, the feature data of neighborhoods of the same type in the stitched feature map are rotated by 0°, 90°, 180°, and 270° respectively. Pixel padding is performed on the rotated feature data to ensure that the same position of the pre-sized convolution kernel in the subsequent mask depth convolution module can read all pixels of the stitched feature map at different rotation angles. After subsequent network calculations, the calculation results are rotated back to the original direction. This allows four C-type neighborhoods and four S-type neighborhoods to each share one affinity sensing module.

[0085] Step D12: Input the orientation-corrected intermediate features into the mask depth convolution module, so as to use the convolution kernel of preset size and the position mask to perform convolution operation on the orientation-corrected intermediate features and output local convolution features based on three feature pixels.

[0086] The location mask is configured to enable the convolution operation to take effect at the pixel positions corresponding to the depth center feature, the RGB center feature, and the depth neighborhood feature.

[0087] The preset size of the convolution kernel is the kernel size of... The convolution kernel is [value]. The orientation-corrected intermediate features are input into the mask depthwise convolution module, whose kernel size is [value]. Simultaneously, a position mask is set, with the position mask of the pixel corresponding to the depth center feature, RGB center feature, and depth neighborhood feature set to 1, and the position mask of the rest set to 0. This position mask is then multiplied by the convolution kernel to ensure that the effective input features participating in the convolution operation only include the pixels corresponding to the aforementioned depth center feature, RGB center feature, and depth neighborhood feature.

[0088] During implementation, the splicing features of the splicing feature map are first received, using a size of [size missing]. The convolution kernel performs convolution operations, and the position mask preserves the pixels corresponding to the depth center features, the RGB center features, and the depth neighborhood features. Based on the S-type and C-type neighborhood types, the position mask preserves the S-type and C-type neighborhoods respectively.

[0089] The convolution operation only applies to the three retained pixels. Assume that a certain size in the concatenated feature is... The feature map in the window is The calculation formula is:

[0090] ;

[0091] in, This represents the convolution operation. Indicates multiplication. These are the three-dimensional position coordinates within the convolution kernel window. For the first The width direction of the feature map, the first The feature map height direction and the first feature map height direction and the Network weight values ​​for each channel dimension For the first The width direction of the feature map, the first The feature map height direction and the first feature map height direction and the Location mask for each channel dimension This represents the output feature map of the convolution operation.

[0092] The above mapping relationship is denoted as:

[0093] ;

[0094] in, Indicates a certain size Feature map in window Output feature map of convolution operation The mapping relationship.

[0095] Preferably, for neighborhood type C, the convolution kernel... The location is the pixel corresponding to the deep neighborhood feature. The location is the pixel corresponding to the depth center feature. The position is the pixel corresponding to the center feature of RGB; for S-class neighborhood patterns, the convolution kernel... The location is the pixel corresponding to the deep neighborhood feature. The location is the pixel corresponding to the depth center feature. The position is the pixel corresponding to the center feature of the RGB.

[0096] Step D13: Input the local convolutional features into the point convolution module to perform nonlinear transformation on the local convolutional features through multiple consecutive point-by-point convolutional layers and inter-layer nonlinear activation functions, generating an affinity matrix corresponding to the current neighborhood direction.

[0097] Multiple consecutive pointwise convolutional layers can be configured as four consecutive pointwise convolutional layers, with ReLU activation function used as the inter-layer nonlinear activation function. Affinity matrices are generated by nonlinearly transforming the local convolutional features. In each of the four consecutive pointwise convolutional layers, except for the last pointwise convolution, ReLU activation is applied after each pointwise convolution for nonlinear activation.

[0098] In implementation, four consecutive pointwise convolutional layers are used. For local convolutional features, the first convolutional layer expands the intermediate features into channel expansion coefficients. ( (where the integer is any integer), the second and third convolutional layers preserve the channel expansion coefficients. The fourth convolutional layer outputs one channel. Except for the fourth convolutional layer, each subsequent convolutional layer undergoes ReLU activation after output. The point convolution module ensures that the output depends only on three original input pixels in the same group: the pixel corresponding to the depth center feature, the pixel corresponding to the RGB center feature, and the pixel corresponding to the depth neighborhood feature. Complex mapping relationships are constructed through multiple layers of non-linear activation.

[0099] ;

[0100] in, For the first A linear transformation, Indicates the index of the linear transformation. For ReLU function, The output feature map of the convolution operation is an index along the channel dimension. include , For the first The output feature map of the convolution operation corresponding to each channel dimension. For the first The affinity matrix corresponding to the channel dimension, the Affinity matrix corresponding to each channel dimension Belongs to the affinity matrix After combining the calculations from each channel, the above mapping relationship is denoted as:

[0101] ;

[0102] in, Represents the output feature map of the convolution operation With affinity matrix The mapping relationship.

[0103] because With a certain size Feature map in window There is a mapping relationship between them, so the above formula can also be written as:

[0104] ;

[0105] in, yes and The cascaded connections. The optimal channel spread factor for the intermediate layer of the network. .

[0106] Step D2: Using the affinity matrix, the deep neighborhood features and deep center features are weighted and fused through the propagation calculation module to complete the training of the RGBD fusion network.

[0107] Specifically, Figure 7 A schematic diagram of the propagation computation module structure is shown below. Figure 7 As shown, step D2 includes:

[0108] Step D21: Multiply the affinity matrix by the corresponding deep neighborhood feature values ​​to obtain the affinity neighborhood features.

[0109] Step D22: Using learnable weight parameters, the affinity neighborhood features and the deep center features are weighted and added together to obtain the preliminary fused features.

[0110] The C-type neighborhood type and the S-type neighborhood type each share a weight parameter.

[0111] In implementation, the affinity matrix is ​​first multiplied by the corresponding deep neighborhood feature values ​​to obtain the affinity neighborhood features. Then, using learnable weight parameters, the affinity neighborhood features are weighted and added to the deep center features to obtain the preliminary fusion features. The calculation formula is:

[0112] ;

[0113] in, Represents the pixel corresponding to the depth center feature. This represents the pixel corresponding to the deep neighborhood feature. As learnable weight parameters, each of the C-class and S-class neighborhood types has an independent weight parameter. .

[0114] In the above mapping relationship, the pixels corresponding to the depth center feature Pixels corresponding to deep neighborhood features The result after a simple positional offset can be directly represented as a certain size. Feature map in window Therefore, the initial fusion of the feature and affinity matrix with the original input, i.e., a certain size, is performed. Feature map in window Related, denoted as:

[0115] ;

[0116] in, This is another representation of the initial fusion characteristics. This is the mapping function from the original input to the preliminary fused features. Because... It can also be represented as a certain size. Feature map in window The mapping is such that the above mapping relationship can also be written as:

[0117] ;

[0118] Another representation of the initial fusion characteristics include , For the first Preliminary fusion features corresponding to each channel dimension This is the final form of the direct mapping function from the original input to the initial fused features.

[0119] Step D23: Perform propagation computation on the preliminary fused features, and perform cross-channel information fusion through multiple consecutive pointwise convolutional layers and inter-layer nonlinear activation functions to output unidirectional fused features.

[0120] Figure 8 A schematic diagram illustrating the overall structure of the completion method is shown below. Figure 8 As shown, in implementation, four consecutive pointwise convolutional layers are used. The first convolutional layer expands the input into channel expansion coefficients. ( (where the integer is any integer), the second and third convolutional layers preserve the channel expansion coefficients. The fourth convolutional layer outputs 16 channels, with a corresponding channel expansion factor of 16. Except for the fourth convolutional layer, each of the remaining convolutional layers undergoes ReLU activation after its output, as expressed by the formula:

[0121] ;

[0122] in, For the first A linear transformation, Indicates the index of the linear transformation. For ReLU function, For channel-dimensional indexing, For the first The unidirectional fusion feature corresponding to each channel dimension.

[0123] because It can also be represented as raw input. The mapping, therefore The mapping relationship can also be written as:

[0124] ;

[0125] Among them, the One-way fusion features corresponding to each channel dimension belong , It is a unidirectional fusion feature. For a certain size Feature map in window Fusion features in one direction A complete and direct mapping function.

[0126] Considering the channel expansion coefficient When the value is large, it increases the network complexity and makes training more difficult. Therefore, it is advisable to optimize the channel spread coefficient of the intermediate layers of the network. .

[0127] In conclusion, the final The output can be represented as the original input, i.e., a certain value of . Feature map in window For the direct mapping and overall operation process, please refer to [link / reference]. Figure 6 As shown.

[0128] S105. Summing the intermediate fusion results for all spatial neighborhood directions yields the fusion output features of the depth center point.

[0129] The fused output features are used to generate the final completed depth map.

[0130] This invention addresses the computational bottleneck of the RGBD fusion module in depth completion tasks by proposing an in-memory indexing method for LiDAR depth completion guided by visible light images. By converting complex convolutional operations in the fusion process into in-memory access operations, the computational load and inference latency are significantly reduced. During model training, a convolutional neural network architecture is used to build the RGBD fusion network: first, feature cubes of C-class and S-class neighborhoods are constructed; then, an affinity matrix is ​​generated through an affinity perception module; and finally, the fusion output is obtained through a propagation calculation module. During quantization, the RGB center features, depth center features, and depth neighborhood features are quantized into 4-bit integers and concatenated to form the input features. The input features are then processed by the RGBD fusion network to produce a 16-channel 4-bit fusion output. All input features are traversed, and the fusion outputs corresponding to all input features are used to construct a LUT (Local Ultimate Test), with a single LUT size of only 32KB. After the LUT is deployed on the hardware device, eight lookup operations are performed simultaneously for the eight deep neighbor features of the deep center feature. The lookup results are added together by the accumulator to obtain the fused output. This allows the process of generating the fused output from the input features to avoid a large number of multiply-accumulate operations in the convolution calculation process. The inference latency is reduced by 90% compared with the existing technology, which is suitable for the real-time depth completion requirements of edge devices.

[0131] This invention first extracts RGB and depth features. After concatenating these two types of features along the channel dimensions, for specific locations of the depth center and RGB center features, the depth neighborhood features are divided into S-type neighborhoods (i.e., top, bottom, left, right) and C-type neighborhoods (i.e., top left, bottom left, top right, bottom right). An affinity matrix for these two types of depth neighborhood features is generated using an affinity sensing module. Then, the depth neighborhood features and depth center features are weighted and added according to the affinity matrix. Finally, a propagation calculation module performs interactive propagation of channel information to achieve the fusion of depth and RGB features. Compared with existing technologies, the traditional multiply-accumulate operations in the entire RGBD fusion process of this invention can be replaced by efficient memory access operations, significantly reducing the computational load and inference latency of the fusion process. This invention concatenates the two types of center features and depth neighborhood features and performs 4-bit quantization, using this as an index to retrieve the pre-built LUT, directly obtaining the 16-channel 4-bit fused output. Since the input features involve only 3 pixels (i.e., the pixel corresponding to the depth center feature, the pixel corresponding to the RGB center feature, and the pixel corresponding to a single depth neighborhood feature), the LUT size is only 32KB, which can be directly embedded into the on-chip cache to achieve in-memory computing. This invention reduces the inference latency to 10ms while maintaining completion accuracy, and improves the inference speed by approximately 10 times compared to existing technologies, thereby enabling real-time visible light-guided depth completion on edge devices.

[0132] The effectiveness of the method of this invention can be further illustrated by the following experimental data. To verify the effectiveness of the method of this invention, a comparative experiment was conducted on the NYU Depth v2 dataset. In the data preprocessing stage, the image size was first resized to 320×240, and then center-cropped to obtain an image with a resolution of 304×228. Subsequently, 500 points were randomly sampled as sparse depth input. The hardware computing platform used in the experiment was an NVIDIA GeForce RTX3090 GPU and an AMD EPYC 7542 CPU. To ensure the reliability of the data, all experimental indicators are the average of 5 independent experiments. The visible light image-guided LiDAR depth completion in-memory indexing method proposed in this invention, and the existing completion methods, both collect indicators based on 4-bit quantization. The specific comparison results are shown in Table 1.

[0133] Table 1. Comparison of results between the present invention and traditional methods

[0134]

[0135] As shown in Table 1, the in-memory indexing method for visible light image-guided LiDAR depth completion proposed in this invention was comprehensively compared with existing mainstream depth completion algorithms, namely Convolutional Spatial Propagation Network (CSPN), Geometry-Aware Embedding Network (GAENet), and ImprovingDC (Deep Completion Improvement through Depth Feature Upsampling). To ensure that the evaluation results closely reflect the actual deployment environment of edge devices, all methods involved in the comparison, including the method of this invention, underwent 4-bit quantization processing before metric collection and statistics. As shown in the experimental data in Table 1, existing depth completion algorithms generally have high resource consumption. CSPN consumes as much as 8.7MB of storage, while GAENet and ImprovingDC, although optimized in some metrics, still have a computational load of tens of GMACs, making it difficult to meet the requirements of extreme lightweight design. In contrast, this invention, by constructing an in-memory indexing mechanism, successfully transforms traditional convolution multiplication-addition operations into efficient table lookup operations, achieving a significant performance breakthrough. Specifically, the storage consumption of the method in this invention is only 0.54MB, which is about 16 times lower than CSPN and about 3 times lower than the best-performing ImprovingDC. In terms of computation, this invention requires only 0.629 GMACs, or 0.629 MACs(G), which is about 70 times lower than ImprovingDC. In terms of the most critical inference speed, the method in this invention requires only 10.21ms, which is about 5 times faster than GAENet and about 10 times faster than the most accurate ImprovingDC. Meanwhile, in terms of completion accuracy, the root mean square error (RMSE) of the method in this invention is 107.3mm, which is better than CSPN's 117.2mm and GAENet's 114.6mm, and is at the same level as ImprovingDC's 105.2mm, with a difference of about 1%. This shows that while significantly reducing computing resources and inference latency, the method in this invention still maintains high-quality deep completion capabilities and can adapt to the real-time application requirements of edge devices.

[0136] Figure 9 The diagram schematically illustrates a visual comparison between the method of the present invention and the prior art. See attached image. Figure 9As shown in Table 1, a comparison of the visualization depth completion results of the method of this invention and the comparative method under 4-bit quantization is presented. From the quantitative indicators in Table 1, the root mean square error of the method of this invention is slightly higher than that of the computationally more complex ImprovingDC method. However, at the actual visual perception level, the method of this invention still maintains extremely high reconstruction quality, and the two do not show a significant difference in visual effect. Through observation... Figure 9 The magnified local detail area reveals that, compared to CSPN and the blurring and artifacts that appear at object edges, the depth map generated by the method of this invention is highly consistent with the ground truth in terms of overall structure. Particularly in areas with rich geometric structures, such as window edges and corners, the method of this invention generates sharp, clear edges without losing crucial geometric details despite using in-memory indexing and low-bit quantization. This demonstrates that the method of this invention effectively balances completion accuracy and visual effect while significantly reducing computational overhead, accurately recovering the geometric structural information in the scene.

[0137] In practical implementation, the aforementioned visible light image-guided lidar depth completion in-memory indexing method can be implemented through software, hardware, or a combination of both. Those skilled in the art should understand that the described embodiments are only for clearly illustrating the core ideas of the invention and are not intended to limit the scope of protection. For example, the feature quantization bit depth is not limited to 4 bits and can be adjusted to 2 bits, 6 bits, or other precisions depending on hardware resources; the neighborhood direction partitioning method can be extended to hexagonal, rhomboid, or other topological structures; the combination order of the depth center value, RGB center value, and neighboring point values ​​in the LUT index construction can be dynamically adjusted, and the quantization range can be adapted to different sensor characteristics for nonlinear mapping.

[0138] It should be noted that the number of convolutional layers and channel spread coefficients (such as...) in the weight generator... or The number of directions can be increased or decreased according to the model complexity requirements; the activation function in the point convolution module can be replaced with a leaky linear rectified function (LeakyReLU), a self-gated activation function (Swish), or other equivalent nonlinear units; the angle partitioning method of the rotation multiplexing module can also be adjusted to a bidirectional multiplexing mechanism with 180° intervals. In addition, multiple directions refer to a set of two or more directions, including but not limited to 2-direction, 4-direction, and other extended implementations.

[0139] The fusion method provided by this invention is applicable to various depth completion architectures, and its LUT deployment mechanism can be embedded in edge computing platforms such as mobile robot vision systems, augmented reality / virtual reality (AR / VR) headsets, and autonomous driving perception modules. Any scheme that uses a combination of depth center value, RGB center value, and neighborhood depth value as an index, replaces real-time convolution operations with a pre-computed LUT, and performs multi-directional parallel lookup fusion, regardless of how its neighborhood construction form, weight generation network structure, or quantization strategy is adjusted, falls within the scope of equivalent substitution of the claims of this invention.

[0140] This invention first extracts RGB and depth features and performs cross-modal concatenation. Then, it constructs feature cubes for S-class and C-class neighborhoods respectively. Next, it generates two types of orientation-aware weights through grouped convolution, using two independent weight parameters to perform data propagation for each type of neighborhood. Utilizing in-memory processing, the input index and corresponding output value are combined to form a LUT, avoiding numerous multiplication and addition operations in traditional convolutions and achieving efficient RGBD feature fusion. By converting traditional convolution calculations to LUTs, the computational cost and inference latency of RGBD fusion are significantly reduced. The method quantizes the RGB center feature, depth center feature, and depth neighborhood features into 4-bit integers, concatenating them to form the input features. Each input feature corresponds to a 16-channel 4-bit fusion output, resulting in a LUT size of only 32KB. Thanks to the in-memory processing mechanism, the inference latency of this invention is reduced by 90% compared to existing technologies, adapting to the real-time depth completion requirements of edge devices.

[0141] Based on the above Figure 1As can be seen from the implementation method, the embodiments of the present invention extract features from the RGB image and the LiDAR depth image to be completed, and concatenate the extracted RGB features and the extracted depth features along the channel dimension to obtain a concatenated feature map; for the depth center point in the concatenated feature map, obtain the depth neighborhood feature values ​​corresponding to multiple spatial neighborhood directions in the concatenated feature map, and classify the multiple spatial neighborhood directions according to the spatial orientation to obtain the classification result; generate a quantized input index corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point; query the pre-built lookup table that matches the category in the classification result in parallel according to the classification result and the corresponding quantized input index to obtain the intermediate fusion result of each spatial neighborhood direction; sum the intermediate fusion results of all spatial neighborhood directions to obtain the fusion output feature of the depth center point; wherein, the pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the depth center feature, RGB center feature and depth neighborhood feature are quantized into integers with a preset bit width and concatenated as an index, and the fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table. In this way, based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point, a quantized input index corresponding to each spatial neighborhood direction is generated, thus compressing the input index. Using the trained RGBD fusion network, the depth center feature, RGB center feature, and depth neighborhood feature are quantized into integers of a preset bit width and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct a pre-built lookup table. This pre-calculated and stored lookup table is extremely small and can be directly embedded into the on-chip cache. Parallel memory access operations can be performed based on the quantized input index. This not only completely eliminates the multiplication and addition overhead of convolution operations but also greatly reduces data transfer and off-chip memory access latency, resulting in a significant reduction in computational load and inference latency.

[0142] Based on the same inventive concept, as an implementation of the above-mentioned visible light image-guided lidar depth completion memory indexing method, this embodiment of the invention also provides a visible light image-guided lidar depth completion memory indexing device. Figure 10 This is a structural diagram of the device in an embodiment of the present invention. See also: Figure 10 As shown, the device may include:

[0143] The extraction and stitching module 1001 is used to extract features from the RGB image and the LiDAR depth image to be completed, and to stitch the extracted RGB features and the extracted depth features along the channel dimension to obtain a stitched feature map.

[0144] The classification module 1002 is used to obtain the corresponding depth neighborhood feature values ​​of multiple spatial neighborhood directions in the stitched feature map for the depth center point in the stitched feature map, and classify the multiple spatial neighborhood directions according to the spatial orientation to obtain the classification result.

[0145] The generation module 1003 is used to generate a quantized input index corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point;

[0146] The parallel query module 1004 is used to query the pre-built lookup table that matches the category in the classification result based on the classification result and the corresponding quantized input index in parallel, so as to obtain the intermediate fusion result in each spatial neighborhood direction. The pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the deep center feature, RGB center feature and deep neighborhood feature are quantized into integers with a preset bit width and concatenated into an index. The fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table.

[0147] The summation module 1005 is used to sum the intermediate fusion results of all spatial neighborhood directions to obtain the fusion output features of the depth center point. The fusion output features are used to generate the final completed depth map.

[0148] The extraction and stitching module 1001 is specifically used to employ an interval arrangement strategy to alternately arrange RGB features and LiDAR depth features in the channel dimension to obtain a stitched feature map.

[0149] The classification module 1002 is specifically used to divide the multiple spatial neighborhood directions of the depth center point into four orthogonal directions and four diagonal directions according to their spatial orientation; based on preset translation and splicing operations, it obtains the depth neighborhood feature values ​​corresponding to the four orthogonal directions and the four diagonal directions respectively; it splices the depth neighborhood feature values ​​corresponding to the four orthogonal directions along a first new dimension to form an S-type neighborhood feature cube; it splices the depth neighborhood feature values ​​corresponding to the four diagonal directions along a second new dimension to form a C-type neighborhood feature cube; wherein, the S-type neighborhood feature cube and the C-type neighborhood feature cube constitute the classification result for multiple spatial neighborhood directions.

[0150] In classification module 1002, the depth neighborhood feature values ​​corresponding to the four orthogonal directions and the four diagonal directions are obtained respectively, including: filling the depth features in the spliced ​​feature map to obtain a filled feature map; translating the filled feature map according to the first set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four orthogonal directions coincide with the coordinates of the depth center feature, thereby obtaining the depth neighborhood feature values ​​corresponding to the four orthogonal directions; and translating the filled feature map according to the second set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four diagonal directions coincide with the coordinates of the depth center feature, thereby obtaining the depth neighborhood feature values ​​corresponding to the four diagonal directions.

[0151] The generation module 1003 is specifically used to quantize the depth center feature value into a first integer value with a first preset bit width; quantize the RGB center feature value into a second integer value with a first preset bit width; quantize the depth neighborhood feature value corresponding to each spatial neighborhood direction into a third integer value with a first preset bit width; and concatenate the first integer value, the second integer value, and the third integer value to generate a quantized input index corresponding to each spatial neighborhood direction.

[0152] The parallel query module 1004 is specifically used to synchronously send the corresponding quantized input index to the corresponding S-class or C-class pre-built lookup table for parallel retrieval based on the S-class or C-class neighborhood type; read the intermediate fusion results of each spatial neighborhood direction from the S-class or C-class pre-built lookup table; the classification results include S-class neighborhood type and C-class neighborhood type, where the S-class neighborhood type is the type corresponding to the depth neighborhood feature values ​​in the four orthogonal directions, and the C-class neighborhood type is the type corresponding to the depth neighborhood feature values ​​in the four diagonal directions. The pre-built lookup table includes the S-class pre-built lookup table and the C-class pre-built lookup table.

[0153] It should be noted that the above description of the visible light image-guided LiDAR depth completion in-memory indexing device embodiment is similar to the description of the visible light image-guided LiDAR depth completion in-memory indexing method embodiment, and has similar beneficial effects as the visible light image-guided LiDAR depth completion in-memory indexing method embodiment. For technical details not disclosed in the embodiments of the visible light image-guided LiDAR depth completion in-memory indexing device of this invention, please refer to the description of the method embodiment of this invention for understanding.

[0154] The above embodiments are merely typical implementations of the present invention. Those skilled in the art, after reading this specification, can make reasonable modifications or parameter adjustments to elements such as feature quantization strategies, LUT index construction rules, and neighborhood direction partitioning methods without departing from the concept of the present invention. Such deductions, modifications, or substitutions based on the core innovations of the present invention should all be considered to fall within the protection scope of this patent.

[0155] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A visible light image-guided lidar depth completion in-memory indexing method, characterized in that, include: Feature extraction is performed on the RGB image and LiDAR depth image to be completed, and the extracted RGB features and the extracted depth features are concatenated along the channel dimension to obtain the concatenated feature map; For the depth center point in the stitched feature map, obtain the depth neighborhood feature values ​​corresponding to multiple spatial neighborhood directions in the stitched feature map, and classify the multiple spatial neighborhood directions according to their spatial orientation to obtain the classification result; Based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point, a quantized input index corresponding to each spatial neighborhood direction is generated. Based on the classification result and the corresponding quantized input index, a pre-built lookup table matching the category in the classification result is queried in parallel to obtain the intermediate fusion result for each spatial neighborhood direction; wherein, the pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the deep center feature, RGB center feature and deep neighborhood feature are quantized into integers of a preset bit width and concatenated into an index, and the fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table; The intermediate fusion results of all spatial neighborhood directions are summed to obtain the fusion output feature of the depth center point, which is used to generate the final completed depth map.

2. The method according to claim 1, characterized in that, The step of concatenating the extracted RGB features and the extracted depth features along the channel dimension to obtain a concatenated feature map includes: An alternating arrangement strategy is adopted to alternately arrange the RGB features and the depth features along the channel dimension to obtain the stitched feature map.

3. The method according to claim 1, characterized in that, For the depth center point in the stitched feature map, multiple spatial neighborhood directions are obtained, corresponding to the depth neighborhood feature values ​​in the stitched feature map. These multiple spatial neighborhood directions are then classified according to their spatial orientation to obtain a classification result, including: The spatial neighborhood directions of the depth center point are divided into four orthogonal directions and four diagonal directions according to the spatial orientation. Based on preset translation and splicing operations, the depth neighborhood feature values ​​corresponding to the four orthogonal directions and the four diagonal directions are obtained respectively; The depth neighborhood feature values ​​corresponding to the four orthogonal directions are spliced ​​along the first new dimension to form an S-type neighborhood feature cube; The depth neighborhood feature values ​​corresponding to the four diagonal directions are spliced ​​along the second new dimension to form a C-type neighborhood feature cube; wherein, the S-type neighborhood feature cube and the C-type neighborhood feature cube constitute the classification result for the multiple spatial neighborhood directions.

4. The method according to claim 3, characterized in that, The step of obtaining the depth neighborhood feature values ​​corresponding to the four orthogonal directions and the four diagonal directions based on preset translation and stitching operations includes: The depth features in the stitched feature map are filled to obtain a filled feature map. The filled feature map is translated according to the first set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four orthogonal directions coincide with the coordinates of the depth center feature, and the depth neighborhood feature values ​​corresponding to the four orthogonal directions are obtained. The filled feature map is translated according to the second set of translation vectors so that the coordinates of the depth neighborhood feature pixels in the four diagonal directions coincide with the coordinates of the depth center feature, thereby obtaining the depth neighborhood feature values ​​corresponding to the four diagonal directions.

5. The method according to claim 1, characterized in that, The step of generating a quantized input index corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value, and depth neighborhood feature value of the depth center point includes: The depth center feature value is quantized into a first integer value with a first preset bit width; The RGB center feature value is quantized into a second integer value having the first preset bit width; The depth neighborhood feature values ​​corresponding to each spatial neighborhood direction are quantized into a third integer value with the first preset bit width. The first integer value, the second integer value, and the third integer value are concatenated to generate a quantized input index corresponding to each spatial neighborhood direction.

6. The method according to claim 3, characterized in that, The classification results include S-type neighborhood types and C-type neighborhood types. The S-type neighborhood types are the types corresponding to the depth neighborhood feature values ​​in the four orthogonal directions, and the C-type neighborhood types are the types corresponding to the depth neighborhood feature values ​​in the four diagonal directions. The pre-built lookup table includes an S-type pre-built lookup table and a C-type pre-built lookup table. The step of querying a pre-built lookup table that matches the category in the classification result and the corresponding quantized input index in parallel to obtain intermediate fusion results for each spatial neighborhood direction includes: Based on the S-type neighborhood type or the C-type neighborhood type, the corresponding quantized input index is synchronously sent to the corresponding S-type pre-built lookup table or C-type pre-built lookup table for parallel retrieval. Read the intermediate fusion results for each spatial neighborhood direction from the S-type pre-built lookup table or the C-type pre-built lookup table.

7. The method according to claim 6, characterized in that, The training process of the completed RGBD fusion network includes: The affinity matrix corresponding to the S-type neighborhood type and the C-type neighborhood type is calculated using the affinity sensing module. Using the affinity matrix, the deep neighborhood features and the deep center features are weighted and fused through the propagation calculation module to complete the training of the RGBD fusion network.

8. The method according to claim 7, characterized in that, The affinity sensing module includes a rotation integration operation module, a mask depthwise convolution module, and a point convolution module connected in sequence; the step of calculating the affinity matrices corresponding to the S-type and C-type neighborhood types through the affinity sensing module includes: The feature data of the S-type neighborhood type and the C-type neighborhood type are respectively input into the rotation integration operation module to rotate the feature data of the same type of neighborhood to a preset reference direction, and calculate using shared network parameters. The calculation result is then rotated back to the original orientation, and the intermediate feature after direction correction is output. The orientation-corrected intermediate features are input into the mask depth convolution module to perform convolution operations on the orientation-corrected intermediate features using a convolution kernel of a preset size and a position mask, and output local convolution features based on three feature pixels. The position mask is configured to make the convolution operation effective at the pixel positions corresponding to the depth center feature, the RGB center feature and the depth neighborhood feature. The local convolutional features are input into the point convolution module to perform nonlinear transformation on the local convolutional features through multiple consecutive point-by-point convolutional layers and inter-layer nonlinear activation functions, thereby generating an affinity matrix corresponding to the current neighborhood direction.

9. The method according to claim 7, characterized in that, The step of using the affinity matrix to perform weighted fusion of the deep neighborhood features and the deep center features through a propagation calculation module includes: Multiply the affinity matrix by the corresponding deep neighborhood feature value to obtain the affinity neighborhood feature; By using learnable weight parameters, the affinity neighborhood features and the deep center features are weighted and added together to obtain preliminary fused features; The preliminary fusion features are subjected to propagation computation operations, and cross-channel information is fused through multiple consecutive pointwise convolutional layers and inter-layer nonlinear activation functions to output unidirectional fusion features.

10. A visible light image-guided lidar depth completion memory indexing device, characterized in that, include: The extraction and stitching module is used to extract features from the RGB image and the LiDAR depth image to be completed, and to stitch the extracted RGB features and the extracted depth features along the channel dimension to obtain a stitched feature map. The classification module is used to obtain the depth neighborhood feature values ​​of multiple spatial neighborhood directions corresponding to the depth center point in the stitched feature map, and classify the multiple spatial neighborhood directions according to their spatial orientation to obtain the classification result. The generation module is used to generate a quantized input index corresponding to each spatial neighborhood direction based on the depth center feature value, RGB center feature value and depth neighborhood feature value of the depth center point; The parallel query module is used to query a pre-built lookup table that matches the category in the classification result based on the classification result and the corresponding quantized input index, so as to obtain the intermediate fusion result of each spatial neighborhood direction; wherein, the pre-built lookup table is pre-built in the following way: using the trained RGBD fusion network, the deep center feature, RGB center feature and deep neighborhood feature are quantized into integers of a preset bit width and concatenated into an index, and the fusion output corresponding to all indices is pre-calculated to construct the pre-built lookup table; The summation module is used to sum the intermediate fusion results of all spatial neighborhood directions to obtain the fusion output feature of the depth center point. The fusion output feature is used to generate the final completed depth map.

Images