Multi-sensor fusion target detection method and system based on RGB-D structure perception and robot positioning error evaluation

Abstract

This invention relates to a multi-sensor fusion target detection method and system based on RGB-D structure perception and robot localization error evaluation, belonging to the field of robot multi-sensor fusion measurement and performance testing technology. This invention addresses the problems of traditional methods failing to effectively decouple and suppress inter-modal conflicts, lacking adaptability in fusion methods, and exhibiting poor detection performance in dynamic and noisy scenarios. Through a differential attention mechanism, this invention actively separates shared structural information and modal-specific information between RGB and depth modalities, obtaining purer and more reliable multi-modal feature representations, fundamentally suppressing inter-modal conflicts. Furthermore, the invention's dynamic routing mechanism based on hybrid experts enables the router to automatically allocate weights according to the real-time content of the input image, dynamically selecting experts that emphasize appearance, geometry, or a deep fusion of both, greatly enhancing the model's adaptability and robustness in variable and complex scenarios. This method can be applied to RGB-D multi-modal fusion target detection.

Description

Technical Field

[0001] This invention belongs to the field of robot multi-sensor fusion measurement and performance testing technology, specifically involving a multi-sensor fusion target detection method and system based on RGB-D structural perception and robot positioning error evaluation. Background Technology

[0002] In recent years, target detection technology that utilizes complementary fusion of RGB images (which provide rich texture and color information) and depth images (which provide accurate geometric and spatial information) has become a core technology in fields such as robot navigation, augmented reality, and intelligent monitoring. Compared to single-modal fusion, multimodal fusion can effectively address challenges in complex scenes, such as drastic changes in lighting, target occlusion, and texture blurring.

[0003] However, existing RGB-D object detection methods still have significant shortcomings in their fusion strategies:

[0004] 1. Insufficient Structural Modeling and Modal Conflicts: Most methods employ simple "static" fusion methods such as feature stitching, element-wise addition, or cascading, lacking explicit modeling of the inherent structural dependencies between RGB and depth modalities. RGB images excel at capturing appearance and semantics, while depth images reflect geometric structure and spatial layout. When the information from the two is inconsistent (e.g., depth sensor noise distorts object boundaries, or RGB images lose texture due to reflection), this simple fusion method couples conflicting information, introducing noise and actually reducing model performance. Existing attention-based fusion methods (such as cross-attention and Transformer) attempt to model intermodal interactions, but they often couple "shared structures" and "modal-specific structures" in the same attention calculation process, failing to effectively decouple and suppress intermodal conflicts caused by sensor noise or scene degradation.

[0005] 2. Rigid Fusion Strategies and Lack of Scene Adaptability: Existing methods typically use fixed parameters for their fusion networks, with fusion rules determined after training. However, real-world scenes are highly dynamic. For example, depth information is relatively more reliable in dimly lit environments; RGB information is more discriminative in textured, well-lit environments; and when objects severely occlude each other, inference requires combining complementary cues from both. Fixed fusion strategies cannot dynamically adjust the confidence level for different modalities based on the real-time contextual characteristics of the input samples, resulting in limited generalization ability and robustness in complex and variable scenes.

[0006] 3. Sensitive to dynamic scenes and noise: Indoor service robots, drones and other application scenarios have a large number of dynamic objects and complex occlusions. Existing methods have significantly reduced detection performance when dealing with these situations, mainly because their fusion mechanism cannot effectively distinguish which are reliable shared structural information and which are noise or conflict information that should be suppressed.

[0007] Therefore, there is an urgent need for an RGB-D target detection method that can explicitly decouple cross-modal structures, adaptively fuse data, and improve robustness in dynamic and noisy scenarios.

[0008] In the process of developing robot inspection standards and evaluating robot performance, it is necessary to quantify and analyze the robot's motion accuracy using multi-sensor fusion measurement technology, with target localization error being one of the most critical evaluation indicators. Existing visual inspection methods primarily focus on detection accuracy and lack a direct correlation with robot performance testing indicators, making it difficult to meet the requirements of spatial measurement consistency and error repeatability in standardized testing. Therefore, it is necessary to research visual perception methods that can provide stable spatial measurement results and support localization error calculation. Summary of the Invention

[0009] This invention addresses the problems of existing technologies failing to effectively decouple and suppress intermodal conflicts, lacking adaptability in fusion methods, and exhibiting poor detection performance in dynamic and noisy scenarios. It proposes a multi-sensor fusion target detection method and system based on RGB-D structural perception and robot positioning error evaluation. Furthermore, the calculation of robot target positioning error can support the development of robot motion accuracy testing and detection standards.

[0010] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0011] According to one aspect of the present invention, a multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation specifically includes the following steps:

[0012] Step 1: Simultaneously acquire RGB and depth images of the scene to be detected using an RGB-D sensor, and preprocess the acquired depth image.

[0013] Step 2: Use the acquired RGB image as the input to the first backbone network branch, and use the preprocessed depth image as the input to the second backbone network branch;

[0014] Dynamic structure-aware fusion is performed on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The features extracted from corresponding positions of the first and second backbone network branches are added together, and the sum is used as the input of the neck network. The output of the neck network is then used as the input of the detection head. The detection head outputs the target detection results, that is, predicts the bounding box coordinates, category and confidence of each target.

[0015] Furthermore, the preprocessing of the acquired depth image specifically includes:

[0016] The acquired depth images are sequentially processed by hole filling, bilateral filtering for noise reduction, and scale normalization.

[0017] Furthermore, the sequence composed of multi-scale RGB feature maps extracted from the first backbone network branch is as follows: ,in:

[0018] This represents the RGB feature map output by the third convolutional layer in the first backbone network branch;

[0019] This represents the RGB feature map output by the fourth convolutional layer in the first backbone network branch;

[0020] This represents the RGB feature map output by the fifth convolutional layer in the first backbone network branch.

[0021] Furthermore, the sequence composed of multi-scale deep feature maps extracted from the second backbone network branch is as follows: ,in:

[0022] This represents the depth feature map output by the third convolutional layer in the second backbone network branch;

[0023] This represents the depth feature map output by the fourth convolutional layer in the second backbone network branch;

[0024] This represents the depth feature map output by the fifth convolutional layer in the second backbone network branch.

[0025] Furthermore, on and Dynamic structure-aware fusion is performed, specifically as follows:

[0026] Step 1: Process the RGB feature map Perform intramodal self-attention calculation and record the result as... For depth feature maps Perform intramodal self-attention calculation and record the result as... ,in, This indicates a bullish self-attention strategy;

[0027] Again and Perform residual join to obtain the Enhanced features ;right and Perform residual join to obtain the Enhanced features ;

[0028] Step 2: Enhance features and enhanced features Cross-modal cross and differential attention calculations are performed to obtain the cross-attention map. and ;

[0029] Step 3, according to , The differential attention mechanism is used to decouple shared and unique components in RGB and depth images, specifically including RGB unique components. RGB shared components Deeply specific components and deep shared ingredients ;

[0030] Step 4, for , , and Perform dynamic route fusion to obtain Corresponding structure-aware enhancement features as well as Corresponding structure-aware enhancement features ;

[0031] Step 5, according to get Features output by the dynamic structure perception fusion module, based on get Features output by the dynamic structure-aware fusion module:

[0032]

[0033]

[0034] in, express Features output by the dynamic structure perception fusion module express Features output by the dynamic structure perception fusion module.

[0035] Furthermore, the specific process of step 2 is as follows:

[0036] Step 21: Enhance features Projection as query vector Key vector Sum value vector This will enhance the features Projection as query vector Key vector Sum value vector ;

[0037] Step 22: Calculate the cross-attention map based on the results of Step 21. and :

[0038]

[0039]

[0040] in, This represents matrix multiplication, with the superscript T indicating the transpose of the matrix. Representing feature dimension, This represents the Softmax activation function. and This represents a cross-attention map.

[0041] Furthermore, the specific process of step 3 is as follows:

[0042] Introduce a difference parameter Using difference parameters Perform a difference operation to decompose the cross-modal information into four components:

[0043]

[0044]

[0045]

[0046]

[0047] in:

[0048] Indicates RGB specific components, Indicates RGB shared components, Indicates deep-specific components, Indicates deep shared components, Value vector Sum value vector The splicing result.

[0049] Furthermore, the specific process of step 4 is as follows:

[0050] Step 41, , , and By splicing, the spliced ​​result is obtained. Then splice the results As input to the router, the router is a multilayer perceptron. The three-dimensional vector output by the router is then normalized using the Softmax function to obtain a weight vector composed of three routing weights. ,and ;

[0051] Step 42: Construct the first expert network, the second expert network, and the third expert network;

[0052] Will and Using these as inputs to the first expert network, we obtain the output of the first expert network. and ;Will and splicing result As input to the second expert network, the output of the second expert network is obtained. ;Will , , and splicing result As input to the third expert network, the output of the third expert network is obtained. ;

[0053] Step 43, for , and By performing weighted fusion, we obtain Corresponding structure-aware enhancement features ;

[0054] Step 44, for , and By performing weighted fusion, we obtain Corresponding structure-aware enhancement features .

[0055] Furthermore, each expert network is a multi-head self-attention layer.

[0056] Furthermore, the aforementioned Corresponding structure-aware enhancement features for:

[0057]

[0058] in, express After passing through the first expert network, the output of the first expert network is passed; express After passing through the second expert network, the output of the second expert network is shown. express After passing through the third expert network, the output of the third expert network is shown.

[0059] The Corresponding structure-aware enhancement features for:

[0060]

[0061] in, express After passing through the first expert network, the output of the first expert network is passed; express The corresponding structure-aware enhancement features.

[0062] Furthermore, the parameters of the first backbone network, the second backbone network, the neck network, and the detection head are obtained through joint training, and the iteration stopping condition for joint training is:

[0063] The robot reconstructs its 3D spatial coordinates based on the 2D target position and depth information output by the detection head on the validation set. The target positioning error (TPE) is then calculated based on the reconstructed 3D spatial coordinates and the target reference coordinates. Training stops when the positioning error on the validation set converges. The target positioning error also characterizes the robot's motion accuracy and consistency with spatial measurements.

[0064] According to another aspect of the present invention, a multi-sensor fusion target detection system based on RGB-D structure perception and robot localization error evaluation, the system includes an image acquisition module, an image preprocessing module, a first backbone network branch, a second backbone network branch, a perception fusion module, a neck network, and a detection head, wherein:

[0065] The image acquisition module is used to acquire RGB images and depth images of the scene to be detected;

[0066] The image preprocessing module is used to preprocess the depth image;

[0067] The input to the first backbone network branch is an RGB image, and the input to the second backbone network branch is a preprocessed depth image;

[0068] The perception fusion module is used to perform dynamic structure perception fusion on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The fusion result continues to be passed down along the first backbone network branch and the second backbone network branch.

[0069] The input to the neck network is the sum of the features extracted from the first backbone network branch and the second backbone network branch at corresponding positions;

[0070] The input to the detection head is the output of the neck network, and the target detection result is output through the detection head.

[0071] The beneficial effects of this invention are:

[0072] 1. Explicit structural decoupling effectively suppresses conflicts: This invention can actively separate the shared structural information and modal-specific information between RGB and depth modal through differential attention mechanism, and mathematically weaken the interference of inconsistent information (noise, conflict), thereby obtaining a purer and more reliable multimodal feature representation, fundamentally suppressing intermodal conflicts and improving fusion quality.

[0073] 2. Scene-adaptive fusion and dynamic strategy adjustment: The dynamic routing mechanism based on hybrid experts introduced in this invention makes the network no longer a fixed "one-size-fits-all" fusion. The router can automatically allocate weights according to the real-time content of the input image (such as lighting conditions, degree of occlusion, and texture clarity), and dynamically select experts that rely on appearance, geometry, or a deep fusion of both, which greatly enhances the model's adaptability and robustness in changing and complex scenes.

[0074] 3. Significantly Improved Detection Performance: Experiments on publicly available benchmark datasets demonstrate that the method of this invention significantly outperforms existing advanced RGB-D detection methods and conventional fusion baselines in terms of mean average accuracy (mAP). The performance advantage is even more pronounced in challenging scenarios with severe occlusion, motion blur, and sensor noise.

[0075] 4. Modular design and strong compatibility: The dynamic structure perception fusion module of this invention is ingeniously designed and can be plugged into any existing dual-stream RGB-D detection framework as a feature enhancement module to improve the detection performance of the model. It has good universality and scalability. Attached Figure Description

[0076] Figure 1 This is an overall flowchart of the method of the present invention;

[0077] Figure 2 This is a flowchart of the dataset processing of the present invention;

[0078] Figure 3 This is a flowchart illustrating the modification of the single-modal YOLO extended to multimodal mode according to the present invention.

[0079] Figure 4 This is a flowchart illustrating the implementation of the DSAFM module of the present invention;

[0080] Figure 5 This is a diagram showing the detection results of this invention on the SUN RGB-D dataset;

[0081] Figure 6 This is a diagram showing the detection results of this invention on the THUD_Robotic dataset;

[0082] Figure 7 This is a network structure diagram of the traditional YOLO11 model. Detailed Implementation

[0083] The object detection method of this invention is based on the YOLO11 model. YOLO11 is the latest version of the object detection algorithm proposed by the Ultralytics team in 2024. It continues the design philosophy of the YOLO series of "single-stage, end-to-end, and high real-time performance," while making targeted improvements in feature extraction, multi-scale fusion, and loss function. Compared with the previous YOLOv8 network, YOLO11 introduces a more efficient C3k2 module in the backbone network and optimizes the connection method of feature pyramid and path aggregation in the neck structure, thereby significantly improving detection accuracy and inference speed while maintaining lightweight design. Figure 7 The diagram shows the structure of the YOLO11 model, which specifically includes a backbone network, a neck network, and a head.

[0084] Within the backbone network, the input image sequentially passes through the first convolutional layer, the second convolutional layer, the first C3k2 module, the third convolutional layer, the second C3k2 module, the fourth convolutional layer, the third C3k2 module, the fifth convolutional layer, the fourth C3k2 module, the first SPPF module, and the first C2PSA module.

[0085] Within the neck network, the output of the first C2PSA module is used as the input of the first upsampling unit. The output of the first upsampling unit is then concatenated with the output of the third C3k2 module, and the concatenated result is used as the input of the fifth C3k2 module. The output of the fifth C3k2 module is used as the input of the second upsampling unit, and the output of the second upsampling unit is then concatenated with the output of the second C3k2 module, and the concatenated result is used as the input of the sixth C3k2 module. The output of the sixth C3k2 module is used as the input of the sixth convolutional layer, and the output of the sixth convolutional layer is then concatenated with the output of the fifth C3k2 module, and the concatenated result is used as the input of the seventh C3k2 module. The output of the seventh C3k2 module is used as the input of the seventh convolutional layer, and the output of the seventh convolutional layer is concatenated with the output of the first C2PSA module, and the concatenated result is used as the input of the eighth C3k2 module.

[0086] Inside the detection head, the outputs of the sixth, seventh, and eighth C3k2 modules are used as inputs to the detection head, and the detection results are output through the detection head.

[0087] This invention improves upon the YOLO11 model by designing two parallel and identical backbone network branches. The outputs of corresponding positions of the two backbone network branches are added together, and the sum is used as the input to the neck network. Finally, the output of the neck network is used as the input to the detection head. Overall, the improved YOLO11 model architecture adopted in this invention includes two backbone network branches, one neck network, and one detection head.

[0088] Specifically, the designed backbone network has made the following improvements based on the traditional backbone network:

[0089] A DSAFM fusion module was added between the third convolutional layer and the second C3k2 module, a DSAFM fusion module was added between the fourth convolutional layer and the third C3k2 module, and a DSAFM fusion module was added between the fifth convolutional layer and the fourth C3k2 module.

[0090] In this invention, the second C3k2 module in the backbone network is referred to as the P3 layer, the third C3k2 module in the backbone network is referred to as the P4 layer, and the C2SPA module in the backbone network is referred to as the P5 layer.

[0091] Specific implementation method one: Combining Figure 1 This embodiment describes a multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation. The method specifically includes the following steps:

[0092] Step 1: Simultaneously acquire RGB and depth images of the scene to be detected using an RGB-D sensor (such as Kinect, RealSense, LiDAR, or camera), and preprocess the acquired depth image.

[0093] Specifically, the acquired depth images are sequentially subjected to hole filling, bilateral filtering for noise reduction, and scale normalization to eliminate sensor noise and invalid values.

[0094] Step 2: Use the acquired RGB image as the input to the first backbone network branch, and use the preprocessed depth image as the input to the second backbone network branch;

[0095] Dynamic structure-aware fusion is performed on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The features extracted from the corresponding positions of the first and second backbone network branches are added together, and the result is used as the input of the neck network. The output of the neck network (generating a multi-scale feature pyramid) is then used as the input of the detection head. The detection head outputs the target detection results, that is, predicts the bounding box coordinates, category and confidence of each target.

[0096] Specifically, the sequence of multi-scale RGB feature maps extracted from the first backbone network branch is denoted as... The sequence of multi-scale deep feature maps extracted from the second backbone network branch is denoted as... ;

[0097] in:

[0098] This represents the RGB feature map output by the third convolutional layer in the first backbone network branch. This represents the RGB feature map output by the fourth convolutional layer in the first backbone network branch. This represents the RGB feature map output by the fifth convolutional layer in the first backbone network branch;

[0099] This represents the depth feature map output by the third convolutional layer in the second backbone network branch. This represents the depth feature map output by the fourth convolutional layer in the second backbone network branch. This represents the depth feature map output by the fifth convolutional layer in the second backbone network branch;

[0100] Will and As input to the dynamic structure-aware fusion module, such as Figure 4 As shown, the specific process of dynamic structure-aware fusion is as follows:

[0101] Step 1: Process the RGB feature map Perform intramodal self-attention calculation and record the result as... For depth feature maps Perform intramodal self-attention calculation and record the result as... ,in, This indicates a bullish self-attention strategy;

[0102] By calculating intramodal self-attention, long-range dependencies within each modality can be established, enhancing feature consistency and structural stability within each modality and providing a more robust foundation for cross-modal interaction. Taking RGB feature maps as an example, they are first projected onto the map to flatten it into a one-dimensional sequence. The flattened result is then input into the multi-head self-attention layer to calculate the correlation weights between all positions within the one-dimensional sequence.

[0103] Again and Perform residual join to obtain the Enhanced features ;right and Perform residual join to obtain the Enhanced features ;

[0104] Step 2: Enhance features and enhanced features Cross-modal cross and differential attention calculations are performed to obtain the cross-attention map. and ;Specifically:

[0105] Step 21: Enhance features Projection as query vector Key vector Sum value vector This will enhance the features Projection as query vector Key vector Sum value vector ;

[0106] Step 22: Calculate the cross-attention map based on the results of Step 21. and :

[0107]

[0108]

[0109] in, This represents matrix multiplication, with the superscript T indicating the transpose of the matrix. Representing feature dimension, This represents the Softmax activation function. and Represents a cross-attention map;

[0110] Step 3, according to , The differential attention mechanism is used to decouple shared and specific components in RGB and depth images; specifically:

[0111] Introduce a learnable difference parameter Its initial value can be determined based on the network depth and optimized during training through backpropagation, utilizing the difference parameters. By performing a differential operation, the cross-modal information is decomposed into four components with clear physical meaning:

[0112]

[0113]

[0114]

[0115]

[0116] in:

[0117] Indicates RGB specific components, Indicates RGB shared components, Indicates deep-specific components, Indicates deep shared components, Value vector Sum value vector The splicing result;

[0118] The unique structural components of the RGB and depth modes actively weaken the interference information of the other mode through differential operations, thereby highlighting the unique structural features of the mode and effectively suppressing mode conflict.

[0119] Step 4: Based on the hybrid expert system... , , and Perform dynamic route fusion to obtain Corresponding structure-aware enhancement features as well as Corresponding structure-aware enhancement features ;Specifically:

[0120] Step 41, , , and By splicing, the spliced ​​result is obtained. Then splice the results As input to the router, the router is a multilayer perceptron (MLP). The three-dimensional vector output by the router is then normalized using the Softmax function to obtain a weight vector composed of three routing weights. ,and Each weight reflects the degree of reliance on different "experts" in the current scenario;

[0121] Step 42: Construct the first expert network, the second expert network, and the third expert network. Each expert network is a multi-head self-attention layer; among them, the shared feature expert is used for specialized processing. and The shared structural information represents the extraction of high-level semantics common to both modalities; unique feature experts specialize in handling RGB-specific components. and deep specific components Each modality's unique geometric or textural cues are explored in depth; fusion experts are responsible for high-level interaction and fusion of shared and specific information to achieve deep fusion of the two types of information.

[0122] Will and Using these as inputs to the first expert network, we obtain the output of the first expert network. and ;Will and splicing result As input to the second expert network, the output of the second expert network is obtained. ;Will , , and splicing result As input to the third expert network, the output of the third expert network is obtained. ;

[0123] Step 43, for , and Perform weighted fusion:

[0124]

[0125] in, express After passing through the first expert network, the output of the first expert network is passed; express After passing through the second expert network, the output of the second expert network is shown. express After passing through the third expert network, the output of the third expert network is shown. express The corresponding structure-aware enhancement features;

[0126] Step 44, for , and Perform weighted fusion:

[0127]

[0128] in, express After passing through the first expert network, the output of the first expert network is passed; express The corresponding structure-aware enhancement features;

[0129] Step 5, according to get Features output by the dynamic structure perception fusion module, based on get Features output by the dynamic structure-aware fusion module:

[0130]

[0131]

[0132] in, express Features output by the dynamic structure perception fusion module express The features output by the dynamic structure perception fusion module are stably transferred through residual connections.

[0133] Using the above method to and After fusion processing, the following is obtained and .Will Continuing to serve as the input to the second C3k2 module within the first backbone network, Continuing as the input to the second C3k2 module within the second backbone network, the output of the second C3k2 module within the first backbone network is added element-wise with the output of the second C3k2 module within the second backbone network. The result of this addition is then used as the input to the neck network (i.e., the result of the addition is concatenated with the output of the second upsampling unit within the neck network). This operation directly merges the RGB features and depth features refined by the Dynamic Structure Aware Fusion Module (DSAFM) at the corresponding scale, achieving deep modal complementarity.

[0134] Continue to propagate forward within the primary backbone network. Continue propagating forward within the second backbone network, then use the RGB feature map output from the fourth convolutional layer in the first backbone network branch. The depth feature map output by the fourth convolutional layer in the second backbone network branch After dynamic structure perception fusion module The features output by the dynamic structure perception fusion module continue to propagate forward within the first backbone network. The features output by the dynamic structure perception fusion module continue to propagate forward in the second backbone network. The outputs of the third C3k2 module in the first backbone network and the third C3k2 module in the second backbone network are added element by element. The sum is used as the input of the neck network (that is, the sum is concatenated with the output of the first upsampling unit in the neck network).

[0135] Then take the output of the fifth convolutional layer in the first backbone network branch. The output of the fifth convolutional layer in the second backbone network branch After dynamic structure perception fusion module The features output by the dynamic structure perception fusion module continue to propagate forward within the first backbone network. The features output by the dynamic structure-aware fusion module continue to propagate forward in the second backbone network. The outputs of the C2PSA modules in the first backbone network and the C2PSA modules in the second backbone network are added element by element, and the sum is used as the input to the neck network (that is, the sum is used as the input to the first upsampling unit of the neck network).

[0136] like Figure 7 As shown, the feature maps after multimodal fusion are then fed into the neck network of YOLO11. The neck network constructs a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN) through a series of upsampling and concat operations. The FPN aims to address the problem of multi-scale targets in object detection by transferring high-level semantic information to low-level features, thus enhancing the detection capability for small targets. The PAN further adds bottom-up paths to the FPN, promoting the flow of low-level positional information to higher levels. In this invention, the neck network utilizes these mature mechanisms to achieve effective interaction and enhancement of multi-scale features, ensuring the model's ability to detect targets of different sizes.

[0137] Finally, the multi-scale features processed by the neck network are fed into the detection head. The YOLO11 network's detection head employs a decoupled design, processing classification and regression tasks separately. This design stems from an understanding of the differences between classification and localization tasks in object detection, avoiding conflicts between the two tasks in optimizing targets, thereby further improving object detection accuracy. The detection head ultimately outputs bounding box coordinates, target confidence, and class probability in parallel, completing the entire detection process. In multi-sensor fusion measurement applications, the three-dimensional spatial coordinates of the target can be recovered based on the detected two-dimensional target position and depth information, and compared with standard reference coordinates to calculate the target positioning error (TPE), which characterizes the consistency between robot motion accuracy and spatial measurement.

[0138] The specific process of training the improved YOLO11 model in this invention is as follows:

[0139] S1. Dataset Selection and Preprocessing:

[0140] In terms of dataset selection, this invention uses two publicly available indoor RGB-D target detection datasets: SUN RGB-D and THUD_Robotic. The dataset processing procedure is as follows: Figure 2 As shown. SUN RGB-D is an indoor scene dataset proposed by Princeton University in 2015. The dataset contains approximately 10,000 images covering ten common indoor object categories, including beds, tables, sofas, chairs, toilets, desks, dressing tables, bedside tables, bookshelves, and bathtubs. Its data was collected by multiple depth sensors, exhibiting rich scene diversity. THUD_Robotic is a dataset designed by Tsinghua University in 2024 specifically for robot grasping tasks. It contains over 12,000 images and 42 object categories, focusing on small objects on desktops, and features significant occlusion and complex backgrounds. In the data preprocessing stage, the system first ensures that RGB images and depth images within the same scene are accurately aligned and then uniformly scales all images to 640×640 pixels. For the depth map, the present invention also performs special preprocessing, including using morphological closing operation and neighborhood interpolation to fill holes in invalid values ​​in the depth map, applying bilateral filtering to smooth noise while preserving the sharpness of object edges, and normalizing the depth values ​​to the 0 to 1 range and applying histogram equalization to enhance the contrast of depth features.

[0141] S2. Training Strategy and Hyperparameter Settings

[0142] Regarding training strategies and hyperparameter settings, this invention directly adopts the overall training strategy and default data augmentation pipeline of YOLO11 to ensure comparability with official benchmarks. Furthermore, it extends this approach for RGB-D dual-stream input, such as... Figure 3As shown.

[0143] The specific training parameters are set as follows: the optimizer is stochastic gradient descent (SGD) with momentum, the initial learning rate (lr0) is set to 0.01, the final learning rate (lrf) is 0.0001, the momentum is 0.937, and the weight decay is 0.0005. Regarding the loss function, the classification task uses Variational Loss to balance the classification difficulty of positive and negative samples and easy and difficult samples; the regression task uses CIoU Loss, which, in addition to considering the overlapping area, incorporates center point distance and aspect ratio constraints to make the bounding box regression more accurate; and together with DFL Loss, they form the total loss function to optimize the probability distribution of the target location. For data augmentation, Mosaic augmentation with a probability of 1.0, horizontal flipping with a probability of 0.5, and HSV color gamut perturbation (H coefficient of 0.015, S coefficient of 0.7, and V coefficient of 0.4) were employed. Random scaling (scale: 0.5) and translation (translate: 0.1) were also used to improve the model's robustness to changes in viewpoint, lighting, and background. The entire training process was performed on a single NVIDIA RTX 3090 GPU with a batch size of 4 for 200 epochs to ensure stable convergence of the training results.

[0144] This training strategy is consistent with the release implementation of YOLO11, and combines the structural adjustments of this invention on dual-stream input and DSAFM modules, enabling the model to fully utilize the complementary information of RGB and depth modalities while maintaining the original real-time performance, thereby improving detection performance in complex indoor scenes.

[0145] In terms of model structure, the system first inputs the preprocessed image pairs into two parallel feature extraction paths. The RGB stream and the depth stream maintain complete symmetry in structure, both using standard modules such as the C3k2 module, SPPF module, and C2PSA module for multi-scale feature extraction up to the P5 layer, thus ensuring that the outputs of the two streams strictly correspond in semantic hierarchy and spatial resolution.

[0146] The detection results of the method of the present invention on the two datasets are as follows: Figure 5 and Figure 6 As shown. After training, the model of this invention not only improves the robustness of RGB-D target detection, but also enables stable calculation of robot target localization error, providing a unified spatial measurement index expression method for multi-sensor fusion measurement systems, and can serve as an important technical support for the formulation of robot detection and performance evaluation standards.

[0147] Specific Implementation Method Two: A multi-sensor fusion target detection system based on RGB-D structured perception and robot localization error evaluation. The system includes an image acquisition module, an image preprocessing module, a first backbone network branch, a second backbone network branch, a perception fusion module, a neck network, and a detection head, wherein:

[0148] The image acquisition module is used to acquire RGB images and depth images of the scene to be detected;

[0149] The image preprocessing module is used to preprocess the depth image;

[0150] The input to the first backbone network branch is an RGB image, and the input to the second backbone network branch is a preprocessed depth image;

[0151] The perception fusion module is used to perform dynamic structure perception fusion on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The fusion result continues to be passed down along the first backbone network branch and the second backbone network branch.

[0152] The input to the neck network is the sum of the features extracted from the first backbone network branch and the second backbone network branch at corresponding positions;

[0153] The input to the detection head is the output of the neck network, and the target detection result is output through the detection head.

[0154] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multi-sensor fusion object detection method based on RGB-D structure perception and robot positioning error evaluation, characterized in that, The method specifically includes the following steps: Step 1: Simultaneously acquire RGB and depth images of the scene to be detected using an RGB-D sensor, and preprocess the acquired depth image. Step 2: Use the acquired RGB image as the input to the first backbone network branch, and use the preprocessed depth image as the input to the second backbone network branch; Dynamic structure-aware fusion is performed on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The features extracted from corresponding positions of the first and second backbone network branches are added together, and the sum is used as the input of the neck network. The output of the neck network is then used as the input of the detection head. The detection head outputs the target detection results, that is, predicts the bounding box coordinates, category and confidence of each target.

2. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 1, characterized in that, The sequence composed of the multi-scale RGB feature maps extracted by the first backbone network branch is wherein: RGB feature map representing the output of a third convolutional layer in the first backbone network branch; RGB feature map representing the output of the fourth convolutional layer in the first backbone network branch; RGB feature map representing the output of the fifth convolutional layer in the first backbone network branch; The sequence composed of multi-scale depth feature maps extracted from the second backbone network branch is ,in: This represents the depth feature map output by the third convolutional layer in the second backbone network branch; This represents the depth feature map output by the fourth convolutional layer in the second backbone network branch; This represents the depth feature map output by the fifth convolutional layer in the second backbone network branch.

3. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 2, characterized in that, right and Dynamic structure-aware fusion is performed, specifically as follows: Step 1: Process the RGB feature map Perform intramodal self-attention calculation and record the result as... For depth feature maps Perform intramodal self-attention calculation and record the result as... ,in, This indicates a bullish self-attention strategy; Again and Perform residual join to obtain the Enhanced features ;right and Perform residual join to obtain the Enhanced features ; Step 2: Enhance features and enhanced features Cross-modal cross and differential attention calculations are performed to obtain the cross-attention map. and ; Step 3, according to , The differential attention mechanism is used to decouple shared and specific components in RGB and depth images, specifically including RGB specific components. RGB shared components Deeply specific components and deep shared ingredients ; Step 4, for , , and Perform dynamic route fusion to obtain Corresponding structure-aware enhancement features as well as Corresponding structure-aware enhancement features ; Step 5, according to get Features output by the dynamic structure perception fusion module, based on get Features output by the dynamic structure-aware fusion module: in, express Features output by the dynamic structure perception fusion module express Features output by the dynamic structure perception fusion module.

4. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 3, characterized in that, The specific process of step 2 is as follows: Step 21: Enhance features Projection as query vector Key vector Sum value vector This will enhance the features Projection as query vector Key vector Sum value vector ; Step 22: Calculate the cross-attention map based on the results of Step 21. and : in, This represents matrix multiplication, with the superscript T indicating the transpose of the matrix. Representing feature dimension, This represents the Softmax activation function. and This represents a cross-attention map.

5. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 4, characterized in that, The specific process of step 3 is as follows: Introduce a difference parameter Using difference parameters Perform a difference operation to decompose the cross-modal information into four components: in: Indicates RGB specific components, Indicates RGB shared components, Indicates deep-specific components, Indicates deep shared components, Value vector Sum value vector The splicing result.

6. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 5, characterized in that, The specific process of step 4 is as follows: Step 41, , , and By splicing, the spliced ​​result is obtained. ; Then splice the results As input to the router, the router is a multilayer perceptron. The three-dimensional vector output by the router is then normalized using the Softmax function to obtain a weight vector composed of three routing weights. ,and ; Step 42: Construct the first expert network, the second expert network, and the third expert network; Will and Using these as inputs to the first expert network, we obtain the output of the first expert network. and ;Will and splicing result As input to the second expert network, the output of the second expert network is obtained. ;Will , , and splicing result As input to the third expert network, the output of the third expert network is obtained. ; Step 43, for , and By performing weighted fusion, we obtain Corresponding structure-aware enhancement features ; Step 44, for , and By performing weighted fusion, we obtain Corresponding structure-aware enhancement features .

7. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 6, characterized in that, Each expert network is a multi-head self-attention layer.

8. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 7, characterized in that, The Corresponding structure-aware enhancement features for: in, express After passing through the first expert network, the output of the first expert network is passed; express After passing through the second expert network, the output of the second expert network is shown. express After passing through the third expert network, the output of the third expert network is shown. The Corresponding structure-aware enhancement features for: in, express After passing through the first expert network, the output of the first expert network is passed; express The corresponding structure-aware enhancement features.

9. The multi-sensor fusion target detection method based on RGB-D structure perception and robot localization error evaluation according to claim 8, characterized in that, The parameters of the first backbone network, the second backbone network, the neck network, and the detection head are obtained through joint training. The iteration stopping condition for joint training is: The target's three-dimensional spatial coordinates are recovered based on the two-dimensional target position and depth information output by the detector on the validation set. The target positioning error is calculated based on the recovered target three-dimensional spatial coordinates and the target reference coordinates. Training stops when the positioning error on the validation set converges.

10. A multi-sensor fusion target detection system based on RGB-D structure perception and robot localization error evaluation, characterized in that, The system includes an image acquisition module, an image preprocessing module, a first backbone network branch, a second backbone network branch, a perception fusion module, a neck network, and a detection head, wherein: The image acquisition module is used to acquire RGB images and depth images of the scene to be detected; The image preprocessing module is used to preprocess the depth image; The input to the first backbone network branch is an RGB image, and the input to the second backbone network branch is a preprocessed depth image; The perception fusion module is used to perform dynamic structure perception fusion on the multi-scale RGB feature map extracted from the first backbone network branch and the multi-scale depth feature map extracted from the second backbone network branch. The fusion result continues to be passed down along the first backbone network branch and the second backbone network branch. The input to the neck network is the sum of the features extracted from the first backbone network branch and the second backbone network branch at corresponding positions; The input to the detection head is the output of the neck network, and the target detection result is output through the detection head.

Images