Instance-level fusion three-dimensional object detection method based on double-path attention mechanism

Abstract

The invention discloses an instance-level fusion three-dimensional object detection method and device based on a double-path attention mechanism, and relates to the technical field of three-dimensional object detection. The method comprises the following steps: according to RGB image data and three-dimensional point cloud data, performing indoor three-dimensional object recognition through an instance-level double-path attention 3D detection network to obtain an instance prediction frame and a prediction category probability; carrying out loss function calculation according to the three-dimensional point cloud data, the instance prediction frame and the prediction category probability to obtain target detection loss; performing parameter optimization on the instance-level double-path attention 3D detection network according to the target detection loss to obtain an optimized instance-level double-path attention 3D detection network; and according to the current RGB image data and the current three-dimensional point cloud data, performing indoor three-dimensional object recognition by using an optimization instance-level double-path attention 3D detection network. The method is a three-dimensional object detection method which is based on instance-level feature fusion of a double-path attention mechanism and is high in efficiency and robustness.

Description

An instance-level fusion 3D object detection method based on dual-path attention mechanism Technical Field

[0001] This invention relates to the field of 3D target detection technology, and in particular to an instance-level fusion 3D object detection method and apparatus based on a dual-path attention mechanism. Background Technology

[0002] 3D object detection is a technique for locating and identifying objects in 3D space, typically outputting object categories and their 3D bounding boxes. This technology is widely used in autonomous driving, robotics, warehousing and logistics, AR / VR perception, and digital twins. In a typical workflow, multi-source sensor data is calibrated and preprocessed, followed by feature extraction and fusion using spatial representations such as points, voxels, and bird's-eye views. This generates candidate objects and regresses 3D bounding box parameters and confidence scores. The development of high-quality spatial representation, multi-scale feature aggregation, and multi-sensor fusion is crucial for 3D detection performance. 3D detection faces various sources of uncertainty, including long-range sparse point clouds, occlusion and truncation, sensor noise and calibration errors, long-tailed categories, and domain offsets. Effective uncertainty modeling helps assess the reliability of detection results and supports downstream planning and decision-making.

[0003] Multimodal 3D object detection aims to jointly utilize heterogeneous sensor data such as point clouds, RGB images, and depth data to complete target localization and recognition in 3D space, outputting category, 3D orientation bounding box parameters, and confidence scores. Compared to unimodal approaches, multimodal approaches can complement the high-resolution semantics and texture discrimination of images with the geometric metrics and scale accuracy of point clouds, improving robustness in long-range sparse, strongly occluded, and appearance-similar classes. A typical workflow requires feature extraction from different modalities and a unified spatial representation, followed by cross-modal feature fusion to generate candidate boxes for downstream planning and decision-making. The core challenge of multimodal fusion in 3D perception lies in cross-modal alignment and full fusion: under the premise of consistent spatiality and visibility, it is necessary to efficiently and robustly fuse the high-resolution semantics of images with the precise geometry of point clouds at multiple levels, avoiding information loss and performance degradation caused by alignment errors and misleading context.

[0004] To encode scene context information, early point cloud-driven methods often transformed irregular point cloud voxels into regular meshes and processed them using dense 3D convolutions. However, these methods incurred significant computational and memory costs. Further research has shifted towards more efficient representations and encodings: one type is point-based methods that directly model points, exemplified by VoteNet, which employs a voting mechanism inspired by Hough transforms for center proposal and combines semantic priors with hybrid geometric primitives to improve candidate generation and robustness; the other type is convolutional encoding centered on sparse convolutions, represented by Generative Shape Proposal Networks (GSPN), which significantly improves network depth and efficiency through sparsification. In recent years, Transformer has been further used to extract global context. Although early methods still relied on post-processing of nonmaximum suppression, with the introduction of decoders and advanced matching strategies, 3D Detection Transformer (3DETR) has achieved end-to-end prediction. Subsequent methods have improved performance and scalability through designs such as voxel-image similarity, hybrid query point aggregation, vertex-guided attention, and state-space models.

[0005] Image-driven 3D object detection methods have gained increasing attention in recent years due to their low data acquisition cost and convenient data collection process. Benefiting from continuous advancements in geometric perception mechanisms, implicit and explicit spatial representations, and large-scale model-driven semantic transfer, image-based 3D object detection has achieved rapid development. Represented by Image to Voxels Network (ImVoxelNet), multi-view RGB features are projected and fused into a 3D voxel space, achieving 3D object detection using only images. Inspired by Neural Radiance Fields, NeRF-Det (NeRF-for-Detection) integrates a NeRF branch that predicts the transparency of sampling points, utilizing self-supervision to improve consistency across multiple views. In recent years, the rapid development of 3D Gaussian has been represented by Gaussian-Det (Learning Closed-Surface Gaussians for 3D Object Detection), which uses a continuous density field representation based on Gaussian points to achieve fine-grained geometric reconstruction and efficient inference, and expands the spatial representation capabilities based on RGB methods.

[0006] Point cloud-driven and image-driven 3D object detection methods offer complementary advantages. Point cloud-driven methods excel at capturing precise geometric structures and object localization, but have limited capabilities in high-level semantic understanding, struggling to directly obtain rich semantic relationships. Image-driven methods benefit from low-cost data acquisition and rich semantic information, but their direct geometric reasoning capabilities in 3D space are limited, resulting in relatively weaker performance. Multimodal fusion of point clouds and images can jointly utilize geometric structure and appearance semantics to achieve more accurate and robust 3D scene understanding. Most existing methods integrating 2D image information, such as ImVoteNet and TR3D, primarily rely on point-to-pixel feature stitching, which is insufficient to capture holistic object semantics. These methods' utilization of 2D information remains superficial. They depend on sparse 2D-to-3D point projection, feature stitching, or basic cross-modal alignment.

[0007] In the existing technology, there is a lack of a highly efficient and robust 3D object detection method based on instance-level feature fusion using a dual-path attention mechanism. Summary of the Invention

[0008] To address the shortcomings of existing technologies, such as poor semantic understanding, unstable detection of objects with extreme aspect ratios, susceptibility to multi-view consistency errors and projection biases, and low accuracy in 3D localization and scale inference, this invention provides an instance-level fusion 3D object detection method and apparatus based on a dual-path attention mechanism. The technical solution is as follows:

[0009] On the one hand, an instance-level fusion 3D object detection method based on a dual-path attention mechanism is provided. This method is implemented by an instance-level fusion 3D object detection device and includes: capturing an indoor target scene with an RGB-D camera to obtain RGB image data and a depth map; back-projecting the depth map based on preset camera intrinsic and extrinsic parameters to obtain 3D point cloud data; performing indoor 3D object recognition using an instance-level dual-path attention 3D detection network based on the RGB image data and the 3D point cloud data to obtain instance prediction boxes and predicted class probabilities; calculating a loss function based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities to obtain an object detection loss; optimizing the parameters of the instance-level dual-path attention 3D detection network based on the object detection loss to obtain an optimized instance-level dual-path attention 3D detection network; capturing the current indoor scene with an RGB-D camera to obtain current RGB image data and a current depth map; back-projecting the current depth map based on preset camera intrinsic and extrinsic parameters to obtain current 3D point cloud data; and using the optimized instance-level dual-path attention 3D detection network to perform indoor 3D object recognition based on the current RGB image data and the current 3D point cloud data to obtain current instance prediction boxes and current predicted class probabilities.

[0010] On the other hand, an instance-level fusion 3D object detection device based on a dual-channel attention mechanism is provided. This device is applied to an instance-level fusion 3D object detection method based on a dual-channel attention mechanism. The device includes: a data acquisition module, used to capture indoor target scenes using an RGB-D camera to obtain RGB image data and a depth map; based on preset camera intrinsic and extrinsic parameters, back-projecting the depth map to obtain 3D point cloud data; a 3D detection module for the training phase, used to perform indoor 3D object recognition through an instance-level dual-channel attention 3D detection network based on the RGB image data and 3D point cloud data, obtaining instance prediction boxes and predicted class probabilities; and a network parameter optimization module, used to optimize the network parameters based on the 3D point cloud data. The loss function is calculated using the instance prediction bounding box and the predicted class probability to obtain the object detection loss. Based on the object detection loss, the parameters of the instance-level dual-channel attention 3D detection network are optimized to obtain the optimized instance-level dual-channel attention 3D detection network. In the inference stage, the 3D detection module is used to capture the current indoor scene using an RGB-D camera to obtain the current RGB image data and the current depth map. Based on the preset camera intrinsic and extrinsic parameters, back projection is performed on the current depth map to obtain the current 3D point cloud data. Based on the current RGB image data and the current 3D point cloud data, the optimized instance-level dual-channel attention 3D detection network is used to perform indoor 3D object recognition to obtain the current instance prediction bounding box and the current predicted class probability.

[0011] On the other hand, an instance-level fusion 3D object detection device is provided, the instance-level fusion 3D object detection device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the above-described instance-level fusion 3D object detection methods based on a dual-path attention mechanism.

[0012] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described instance-level fusion 3D object detection methods based on a dual-path attention mechanism.

[0013] The beneficial effects of the technical solutions provided by the embodiments of this invention include at least the following: This invention proposes an instance-level fusion 3D object detection method based on a dual-path attention mechanism. While maintaining geometric localization accuracy and scalability, it introduces instance-level semantics from images to resolve category ambiguities of geometrically sparse targets. Through dynamic trade-offs and fusion at the instance level using dual-path attention, it obtains a unified, robust, and scale-sensitive cross-modal instance representation, reducing redundant computation and background noise, thus improving inference efficiency while enhancing detection accuracy. This invention also proposes a unified spatial instance-level feature encoding scheme, mapping 2D features to a unified 3D space through intrinsic and extrinsic parameter calculations, but no longer performing point cloud point-to-pixel mapping. This invention employs a one-to-one mapping of elements; utilizing candidate boxes generated by a coarse instance candidate generator, it aggregates and aligns 2D semantic features within the same instance at the object level, forming a unified instance representation; the aggregation of candidate boxes achieves consistent 2D-3D semantic transfer, reducing the computational overhead and noise pollution of point / pixel-level alignment; instance-level aggregation reduces sensitivity to point-to-point mapping, improves the identification and stability of difficult-to-class objects such as long, thin, and clinging objects, and significantly simplifies the inference process and engineering implementation; this invention combines rich 2D texture information with 3D geometric data to enhance the discrimination ability of these long and thin objects; the geometry-guided fusion module explicitly aligns complementary cues between 3D geometric features and 2D semantic texture, enhancing the distinguishability of 2D texture information for target objects; the semantic-guided fusion module utilizes 3D geometric information to enhance the spatial layout and shape cues of 2D features; the dual-path design enriches the geometric and texture content of instance features. This invention is a highly efficient and robust 3D object detection method based on a dual-path attention mechanism for instance-level feature fusion. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 is a flowchart of an instance-level fusion 3D object detection method based on a dual-path attention mechanism provided by an embodiment of the present invention; Figure 2 is a block diagram of an instance-level fusion 3D object detection device based on a dual-path attention mechanism provided by an embodiment of the present invention; Figure 3 is a structural schematic diagram of an instance-level fusion 3D object detection device provided by an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0017] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0018] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0019] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] This invention provides an instance-level fusion 3D object detection method based on a dual-path attention mechanism. This method can be implemented by an instance-level fusion 3D object detection device, which can be a terminal or a server. As shown in Figure 1, the flowchart of the instance-level fusion 3D object detection method based on the dual-path attention mechanism includes the following steps:

[0022] S1. Capture an indoor target scene using an RGB-D camera to obtain RGB image data and a depth map; based on preset camera intrinsic and extrinsic parameters, perform back projection according to the depth map to obtain three-dimensional point cloud data; in one feasible implementation, the present invention uses an RGB-D camera to collect information about an indoor scene, and the data obtained using the RGB-D camera is in the form of a two-dimensional RGB image and a corresponding depth map.

[0023] Using camera intrinsic and extrinsic parameters, the depth map is converted into 3D point cloud data. The converted 3D point cloud, image, and their intrinsic and extrinsic parameters are then input into the network model for target detection.

[0024] S2. Based on RGB image data and 3D point cloud data, indoor 3D object recognition is performed using an instance-level dual-channel attention 3D detection network to obtain instance prediction boxes and predicted category probabilities. The instance-level dual-channel attention 3D detection network includes a feature extraction network, a coarse-grained instance proposal generator, an instance feature aggregation module, an instance feature fusion module, and a prediction head. The feature extraction network includes a 2D feature extraction network and a 3D feature extraction network. The coarse-grained instance proposal generator adopts a feedforward neural network architecture. The instance feature aggregation module includes a gated neural network and a spatial feature aggregation module. The instance feature fusion module includes a multilayer perceptron and a multi-head cross-attention module. The prediction head includes a regression head and a classification head.

[0025] In one feasible implementation, the present invention proposes an instance-level dual-path attention 3D detection network framework, the structure of which includes a feature extraction network, a coarse-grained instance proposal generator, an instance feature aggregation module, an instance feature fusion module, and a prediction head.

[0026] Optionally, based on RGB image data and 3D point cloud data, an instance-level dual-attention 3D detection network is used to recognize indoor 3D objects, obtaining instance prediction boxes and predicted class probabilities. This includes: in a preset unified 3D space, using a feature extraction network to perform feature alignment encoding based on RGB image data and 3D point cloud data to obtain pixel mapping features and point cloud mapping features; using a coarse-grained instance proposal generator to encode based on the point cloud mapping features to obtain point cloud proposal features and corresponding coarse prediction boxes; based on the coarse prediction boxes, using an instance feature aggregation module to perform dynamic feature integration based on pixel mapping features, point cloud mapping features, and point cloud proposal features to obtain 2D instance features and 3D instance features; based on a dual-attention mechanism, using an instance feature fusion module to perform symmetrical cross-fusion based on the coarse prediction boxes, 2D instance features, and 3D instance features to obtain fused features; and using a prediction head to perform recognition based on the fused features to obtain instance prediction boxes and predicted class probabilities.

[0027] In one feasible implementation, this invention proposes a candidate instance selection method based on a coarse-grained instance proposal generator (CPG). The CPG generates high-recall instance-level candidates in 3D space with low computational overhead, covering multi-scale and geometrically sparse targets, providing a high-quality starting point for subsequent instance-level encoding. The model structure used by the coarse-grained instance proposal generator is a feedforward neural network.

[0028] The 3D point cloud is input into a sparse 3D backbone network, voxelized and its features are encoded to obtain the point cloud mapping feature F_3d_point; F_3d_point is fed into a coarse-grained instance proposal generator, and a set of coarse-grained instance candidates are output, including coarse estimates of center, size, and orientation, confidence, and proposal features F_3d_proposal, which are used for instance-level encoding.

[0029] Unlike existing methods that fuse instances, this approach proposes a coarse-grained instance proposal using 2D images. By using point clouds, it can directly reflect the 3D geometric structure, maintain the true geometric relationship of objects in 3D space, and make the proposed coarse-grained instances more accurate.

[0030] Optionally, in a preset unified three-dimensional space, feature alignment encoding is performed using a feature extraction network based on RGB image data and three-dimensional point cloud data to obtain pixel mapping features and point cloud mapping features. This includes: extracting features using a two-dimensional feature extraction network based on RGB image data to obtain two-dimensional pixel features; performing linear mapping based on the two-dimensional pixel features in a preset unified three-dimensional space according to preset camera intrinsic and extrinsic parameters to obtain pixel mapping features; extracting features using a three-dimensional feature extraction network based on three-dimensional point cloud data to obtain three-dimensional point cloud features; and performing linear mapping based on the three-dimensional point cloud data features in a preset unified three-dimensional space to obtain point cloud mapping features.

[0031] In one feasible implementation, unlike existing technologies that project 3D features onto the BEV space and fuse them with 2D features or project 2D features one by one onto the point cloud, the former does not make full use of the geometric information in three-dimensional space for positioning, and the latter's feature fusion is too rudimentary and destroys the original object features.

[0032] This invention proposes to unify the mapping of 2D and 3D features into a three-dimensional space, fully utilizing the geometric information of 3D features for localization and the texture information of 2D features for auxiliary classification. By spatially aggregating features in a unified space, efficient collaboration between 2D and 3D features at the object level can be achieved, reducing inconsistencies and alignment errors between modalities and improving the consistency and interpretability of cross-modal representations. Instance-level feature extraction is divided into 3D instance feature aggregation and 2D instance feature aggregation.

[0033] Optionally, based on the coarse prediction box, and according to pixel mapping features, point cloud mapping features, and point cloud proposal features, an instance feature aggregation module is used to perform dynamic feature integration to obtain two-dimensional instance features and three-dimensional instance features. This includes: based on the coarse prediction box, and according to pixel mapping features, using a spatial aggregation feature module to perform instance-level dynamic feature aggregation to obtain two-dimensional instance features; based on the coarse prediction box, performing feature extraction on the point cloud mapping features to obtain voxel features within the coarse prediction box; based on the voxel features, using a spatial aggregation feature module to perform pooling processing to obtain local point cloud features; and based on the coarse prediction box, and according to point cloud proposal features and local point cloud features, using a gated neural network to perform instance-level dynamic feature fusion to obtain three-dimensional instance features.

[0034] In one feasible implementation, the point cloud proposal features generated by the coarse instance proposal generator typically fail to capture a comprehensive object-level representation. To facilitate instance identification, this invention further encodes instance-level features based on the bounding box of each proposal.

[0035] To maintain spatial consistency and geometric consistency of instances, this invention uses a spatial aggregation feature module to aggregate 2D features within instances to achieve this. 2D features are used to assist in instance classification; therefore, they do not contain instance proposal features. This process is illustrated in equations (1) and (2):

[0036] F_2d_point = Proj(F_2d_pixel) (1); F_2d_ins = Agg(F_2d_point, P) (2); where F_2d_point is the pixel mapping feature output by the 2D backbone, and P is a coarse-grained instance candidate. Proj() is the function that maps pixels to a uniform space. Agg() is the spatial aggregation feature module, which is... and It consists of two layers of 3D sparse convolutions. F_2d_ins is the aggregated 2D instance feature.

[0037] For each proposed feature F_3d_proposal, voxels included within the predicted bounding box are selected. Unlike the max pooling or linear pooling of existing technologies, this invention employs a spatial aggregation feature module as the RoI-Conv pooling module, which is a two-layer sparse abstraction block that progressively aggregates voxel features associated with each 3D proposal from the backbone network.

[0038] The contributions of global bounding box information F_3d_proposal and point cloud local features F_3d_roi to different objects are not the same. Simply adding them together cannot distinguish when to rely on global cues and when to rely on local details, which may lead to information redundancy or conflicts. This invention proposes a gating mechanism that can dynamically allocate weights based on contextual features, thereby more flexibly emphasizing useful information and producing more consistent object representations.

[0039] For each candidate box P_i, the adaptive gating weights are calculated as a channel-dimensional vector α_i ∈ [0,1]C, and F_3d_proposal and F_3d_roi are further aggregated through a small gating network. This process is shown in equations (3) and (4):

[0040] F_3d_roi = Agg(F_3d_point, P) (3); F_3d_ins = Gate(F_3d_proposal, F_3d_roi) (4); where F_3d_point is the point cloud mapping feature output by the 3D backbone, P is the coarse-grained instance candidate, and F_3d_proposal is the point cloud proposal feature. Agg() is the spatial aggregation feature module. Gate() is the gated network. F_3d_ins is the aggregated 3D instance feature.

[0041] Optionally, based on the dual-path attention mechanism, symmetrical cross-fusion is performed using an instance feature fusion module based on the coarse prediction box, two-dimensional instance features, and three-dimensional instance features to obtain fused features. This includes: learning position encoding using a multilayer perceptron based on the basic attributes of the coarse prediction box to obtain instance position encoding; performing geometry-guided feature fusion using a multi-head cross-attention module based on the three-dimensional instance features, instance position encoding, and two-dimensional instance features to obtain a first fused feature; performing texture-guided feature fusion using a multi-head cross-attention module based on the two-dimensional instance features, instance position, and three-dimensional instance features to obtain a second fused feature; and concatenating features based on the first and second fused features to obtain the fused features.

[0042] In one feasible implementation, the dual-path attention instance-level fusion module proposed in this invention performs equivalent cross-modal fusion of 2D and 3D features in a unified space, which differs from traditional point-by-point / pixel alignment and asymmetric cross-modal attention. It achieves efficient and consistent transfer of cross-modal semantic and geometric information at the instance level through object-level aggregation (based on coarse instance candidate boxes).

[0043] For each instance, a dedicated positional encoding is learned to cover the instance's center location and size, thereby improving the network's sensitivity to shape. Unlike existing technologies that use the instance's center point for encoding, this invention uses the instance's bounding box and volume as the instance's basic attributes, and further uses an MLP to learn the positional encoding, as shown in equation (5):

[0044] Position_i = MLP(b_i, volume(b_i))(5; F_3d_ins as q, F_2d_ins as k, v, and shape-guided instance position encoding are fed into the multi-head cross-attention module to obtain the first fusion feature F_ggf dominated by 3D geometric features. The process is as follows (6): (6); where v_q and v_k are function mappings of the MLP, SoftMax is the normalization function, and Position_i is the instance position encoding.

[0045] F_2d_ins as q, F_3d_ins as k, v, and the shape-guided instance position encoding are fed into the multi-head cross-attention module to obtain the second fusion feature F_sgf dominated by 2D texture features. The process is as follows (7): (7); The dual-path attention mechanism is completely symmetrical, and alignment and fusion are performed at the instance level. The 2D channel and the 3D channel achieve symmetrical information updates and cross-modal interactions at the same stage and level, ensuring the equivalence of the two paths of information in weight allocation and propagation direction, and reducing unidirectional bias. In order to obtain a unified instance representation, F_ggf and F_sgf are fused along the feature dimension to generate symmetrical and complementary fused features F_ins, which is shown in the following equation (8):

[0046] F_ins = F_ggf + F_sgf (8); S3. Calculate the target detection loss based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities; optimize the parameters of the instance-level dual-channel attention 3D detection network based on the target detection loss to obtain an optimized instance-level dual-channel attention 3D detection network; Optionally, calculate the target detection loss based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities, including: calculating the classification loss based on the cross-entropy loss function, according to the class labels of the 3D point cloud data and the predicted class probabilities; calculating the regression loss based on the intersection-union ratio calculation method, according to the ground truth boxes of the 3D point cloud data and the instance prediction boxes; and calculating the target detection loss by weighting the classification loss and the regression loss.

[0047] In one feasible implementation, the present invention calculates a loss function based on the category labels and ground truth annotations in the 3D point cloud data, combined with the predicted strength prediction box and the predicted category probability.

[0048] The classification loss is calculated using the cross-entropy loss function, and the calculation process is as follows (9): L_cls = CE(p, y) (9); where CE is the cross-entropy loss function, p is the predicted class probability of the network output, and y is the class label.

[0049] The regression loss is calculated using the intersection-union ratio (IoU) method. The calculation process is as follows: (10) and (11): IoU = area(b_p ∩ b_gt) / area(b_p ∪ b_gt) (10); L_reg = 1–IoU (11); where IoU represents the intersection-union ratio between the instance prediction box b_p and the ground truth label box b_gt. Since the larger the ratio, the more similar the objects are, the upper bound of 1 minus IoU is used as the loss function. area() represents the area function to obtain the area of ​​the target region.

[0050] S4. Capture the current indoor scene using an RGB-D camera to obtain the current RGB image data and current depth map; based on preset camera intrinsic and extrinsic parameters, back-project the current depth map to obtain the current 3D point cloud data; based on the current RGB image data and the current 3D point cloud data, use an optimized instance-level dual-path attention 3D detection network to perform indoor 3D object recognition, and obtain the current instance prediction box and the current prediction category probability.

[0051] In one feasible implementation, the present invention takes into account both geometric localization and semantic discrimination while maintaining inference efficiency. It can dynamically balance the importance of 3D geometry and 2D semantics for geometrically sparse and extreme aspect ratio targets, robustly generate high-quality 3D candidates and complete object-level feature fusion, reduce irrelevant background processing and post-processing dependencies, improve the detection accuracy of difficult classes and reduce system complexity.

[0052] This invention proposes an instance-level fusion 3D object detection method based on a dual-path attention mechanism. While maintaining geometric localization accuracy and scalability, it introduces instance-level semantics from images to resolve category ambiguity in geometrically sparse targets. Through dynamic trade-offs and fusion at the instance level using dual-path attention, it obtains a unified, robust, and scale-sensitive cross-modal instance representation, reducing redundant computation and background noise, thus improving both detection accuracy and inference efficiency. This invention also proposes a unified spatial instance-level feature encoding scheme, mapping 2D features to a unified 3D space through intrinsic and extrinsic parameter calculations, but without performing point-to-pixel mapping of point cloud points to pixels. It utilizes a coarse-grained real-time feature encoding method. For example, the candidate boxes generated by the candidate generator aggregate and align 2D semantic features within the same instance at the object level to form a unified instance representation. The aggregation of candidate boxes achieves consistent 2D-3D semantic transfer, reducing the computational overhead and noise pollution of point / pixel-level alignment. Instance-level aggregation reduces sensitivity to point-by-point mapping, improves the identification and stability of difficult-to-class objects such as long, thin, and clinging objects, and significantly simplifies the inference process and engineering implementation. This invention combines rich 2D texture information with 3D geometric data to enhance the discrimination ability of these long, thin objects. The geometry-guided fusion module explicitly aligns complementary cues between 3D geometric features and 2D semantic textures, enhancing the distinguishability of 2D texture information for target objects. The semantic-guided fusion module utilizes 3D geometric information to enhance the spatial layout and shape cues of 2D features. The dual-path design enriches the geometric and texture content of instance features. This invention is a highly efficient and robust 3D object detection method based on a dual-path attention mechanism for instance-level feature fusion.

[0053] Figure 2 is a block diagram of an instance-level fusion 3D object detection device based on a dual-path attention mechanism provided in an embodiment of the present invention. This device is used in an instance-level fusion 3D object detection method based on a dual-path attention mechanism. Referring to Figure 2, the device includes a data acquisition module 210, a training phase 3D detection module 220, a network parameter optimization module 230, and an inference phase 3D detection module 240. Wherein:

[0054] The data acquisition module 210 is used to capture indoor target scenes using an RGB-D camera, obtaining RGB image data and depth maps; based on preset camera intrinsic and extrinsic parameters, it performs back projection based on the depth maps to obtain 3D point cloud data; the 3D detection module 220 in the training phase is used to perform indoor 3D object recognition using an instance-level dual-path attention 3D detection network based on the RGB image data and 3D point cloud data, obtaining instance prediction boxes and predicted class probabilities; the network parameter optimization module 230 is used to calculate the loss function based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities to obtain the object detection loss; Based on the target detection loss, the parameters of the instance-level dual-channel attention 3D detection network are optimized to obtain an optimized instance-level dual-channel attention 3D detection network. During the inference phase, the 3D detection module 240 is used to capture the current indoor scene using an RGB-D camera, obtaining current RGB image data and a current depth map. Based on preset camera intrinsic and extrinsic parameters, back-projection is performed on the current depth map to obtain current 3D point cloud data. Based on the current RGB image data and the current 3D point cloud data, the optimized instance-level dual-channel attention 3D detection network is used to perform indoor 3D object recognition, obtaining the current instance prediction bounding box and the current prediction category probability.

[0055] The instance-level dual-channel attention 3D detection network includes a feature extraction network, a coarse-grained instance proposal generator, an instance feature aggregation module, an instance feature fusion module, and a prediction head. The feature extraction network includes a two-dimensional feature extraction network and a three-dimensional feature extraction network. The coarse-grained instance proposal generator adopts a feedforward neural network architecture. The instance feature aggregation module includes a gated neural network and a spatial feature aggregation module. The instance feature fusion module includes a multilayer perceptron and a multi-head cross-attention module. The prediction head includes a regression head and a classification head.

[0056] Optionally, the 3D detection module 220 during the training phase is further configured to: in a preset unified three-dimensional space, use a feature extraction network to perform feature alignment encoding based on RGB image data and three-dimensional point cloud data to obtain pixel mapping features and point cloud mapping features; based on the point cloud mapping features, use a coarse-grained instance proposal generator for encoding processing to obtain point cloud proposal features and corresponding coarse prediction boxes; based on the coarse prediction boxes, use an instance feature aggregation module to perform dynamic feature integration based on pixel mapping features, point cloud mapping features, and point cloud proposal features to obtain two-dimensional instance features and three-dimensional instance features; based on a dual-path attention mechanism, use an instance feature fusion module to perform symmetrical cross-fusion based on the coarse prediction boxes, two-dimensional instance features, and three-dimensional instance features to obtain fused features; and based on the fused features, use a prediction head for recognition to obtain instance prediction boxes and predicted class probabilities.

[0057] Optionally, the 3D detection module 220 during the training phase is further configured to: extract features from RGB image data using a two-dimensional feature extraction network to obtain two-dimensional pixel features; perform linear mapping based on preset camera intrinsic and extrinsic parameters in a preset unified three-dimensional space to obtain pixel mapping features; extract features from three-dimensional point cloud data using a three-dimensional feature extraction network to obtain three-dimensional point cloud features; and perform linear mapping based on the three-dimensional point cloud data features in a preset unified three-dimensional space to obtain point cloud mapping features.

[0058] Optionally, the 3D detection module 220 during the training phase is further configured to: based on the coarse prediction box, perform instance-level dynamic feature aggregation using a spatial aggregation feature module according to the pixel mapping features to obtain two-dimensional instance features; based on the coarse prediction box, perform feature extraction on the point cloud mapping features to obtain voxel features within the coarse prediction box; perform pooling processing using a spatial aggregation feature module according to the voxel features to obtain local point cloud features; and based on the coarse prediction box, perform instance-level dynamic feature fusion using a gated neural network according to the point cloud proposal features and local point cloud features to obtain three-dimensional instance features.

[0059] Optionally, during the training phase, the 3D detection module 220 is further configured to: learn position codes using a multilayer perceptron based on the basic attributes of the coarse prediction box to obtain instance position codes; perform geometry-guided feature fusion using a multi-head cross-attention module based on the 3D instance features, the instance position codes, and the 2D instance features to obtain a first fused feature; perform texture-guided feature fusion using a multi-head cross-attention module based on the 2D instance features, the instance positions, and the 3D instance features to obtain a second fused feature; and perform feature concatenation based on the first fused feature and the second fused feature to obtain a fused feature.

[0060] Optionally, the network parameter optimization module 230 is further configured to: calculate the classification loss based on the cross-entropy loss function, according to the category labels and predicted category probabilities of the 3D point cloud data; calculate the regression loss based on the intersection-union ratio (IU) calculation method, according to the ground truth bounding boxes and instance prediction boxes of the 3D point cloud data; and calculate the target detection loss by weighting the classification loss and the regression loss.

[0061] This invention proposes an instance-level fusion 3D object detection method based on a dual-path attention mechanism. While maintaining geometric localization accuracy and scalability, it introduces instance-level semantics from images to resolve category ambiguity in geometrically sparse targets. Through dynamic trade-offs and fusion at the instance level using dual-path attention, it obtains a unified, robust, and scale-sensitive cross-modal instance representation, reducing redundant computation and background noise, thus improving both detection accuracy and inference efficiency. This invention also proposes a unified spatial instance-level feature encoding scheme, mapping 2D features to a unified 3D space through intrinsic and extrinsic parameter calculations, but without performing point-to-pixel mapping of point cloud points to pixels. It utilizes a coarse-grained real-time feature encoding method. For example, the candidate boxes generated by the candidate generator aggregate and align 2D semantic features within the same instance at the object level to form a unified instance representation. The aggregation of candidate boxes achieves consistent 2D-3D semantic transfer, reducing the computational overhead and noise pollution of point / pixel-level alignment. Instance-level aggregation reduces sensitivity to point-by-point mapping, improves the identification and stability of difficult-to-class objects such as long, thin, and clinging objects, and significantly simplifies the inference process and engineering implementation. This invention combines rich 2D texture information with 3D geometric data to enhance the discrimination ability of these long, thin objects. The geometry-guided fusion module explicitly aligns complementary cues between 3D geometric features and 2D semantic textures, enhancing the distinguishability of 2D texture information for target objects. The semantic-guided fusion module utilizes 3D geometric information to enhance the spatial layout and shape cues of 2D features. The dual-path design enriches the geometric and texture content of instance features. This invention is a highly efficient and robust 3D object detection method based on a dual-path attention mechanism for instance-level feature fusion.

[0062] Figure 3 is a schematic diagram of an instance-level fusion 3D object detection device provided in an embodiment of the present invention. As shown in Figure 3, the instance-level fusion 3D object detection device may include the instance-level fusion 3D object detection device based on the dual-path attention mechanism shown in Figure 2. Optionally, the instance-level fusion 3D object detection device 310 may include a first processor 2001.

[0063] Optionally, the instance-level fused 3D object detection device 310 may also include a memory 2002 and a transceiver 2003.

[0064] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0065] The following describes the various components of the instance-level fused 3D object detection device 310 in detail with reference to Figure 3: The first processor 2001 is the control center of the instance-level fused 3D object detection device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0066] Optionally, the first processor 2001 can execute various functions of the instance-level fused 3D object detection device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0067] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG3.

[0068] In a specific implementation, as one example, the instance-level fusion 3D object detection device 310 may also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 3. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0069] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0070] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be coupled to the first processor 2001 through the interface circuit (not shown in Figure 3) of the instance-level fused 3D object detection device 310. This embodiment of the invention does not specifically limit this.

[0071] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0072] Optionally, transceiver 2003 may include a receiver and a transmitter (not shown separately in Figure 3). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0073] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and is coupled to the first processor 2001 through the interface circuit (not shown in Figure 3) of the instance-level fusion 3D object detection device 310. This embodiment of the invention does not specifically limit this.

[0074] It should be noted that the structure of the instance-level fused 3D object detection device 310 shown in Figure 3 does not constitute a limitation on the router. The actual instance-level fused 3D object detection device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] Furthermore, the technical effect of the instance-level fusion 3D object detection device 310 can be referred to the technical effect of the instance-level fusion 3D object detection method based on the dual-path attention mechanism described in the above method embodiments, and will not be repeated here.

[0076] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0077] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0078] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0079] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0080] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0081] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An instance-level fusion 3D object detection method based on a dual-path attention mechanism, characterized in that, The method includes: capturing an indoor target scene using an RGB-D camera to obtain RGB image data and a depth map; performing back projection based on the depth map using preset camera intrinsic and extrinsic parameters to obtain 3D point cloud data; performing indoor 3D object recognition using an instance-level dual-channel attention 3D detection network based on the RGB image data and the 3D point cloud data to obtain instance prediction boxes and predicted class probabilities; calculating a loss function based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities to obtain an object detection loss; optimizing the parameters of the instance-level dual-channel attention 3D detection network based on the object detection loss to obtain an optimized instance-level dual-channel attention 3D detection network; capturing the current indoor scene using an RGB-D camera to obtain current RGB image data and a current depth map; performing back projection based on the current depth map using preset camera intrinsic and extrinsic parameters to obtain current 3D point cloud data; and performing indoor 3D object recognition using the optimized instance-level dual-channel attention 3D detection network based on the current RGB image data and the current 3D point cloud data to obtain current instance prediction boxes and current predicted class probabilities.

2. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 1, characterized in that, The instance-level dual-path attention 3D detection network includes a feature extraction network, a coarse-grained instance proposal generator, an instance feature aggregation module, an instance feature fusion module, and a prediction head; the feature extraction network includes a two-dimensional feature extraction network and a three-dimensional feature extraction network; the coarse-grained instance proposal generator adopts a feedforward neural network architecture; the instance feature aggregation module includes a gated neural network and a spatial feature aggregation module; The instance feature fusion module includes a multilayer perceptron and a multi-head cross-attention module; the prediction head includes a regression head and a classification head.

3. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 2, characterized in that, The method of identifying indoor 3D objects using an instance-level dual-channel attention 3D detection network based on RGB image data and 3D point cloud data to obtain instance prediction boxes and predicted class probabilities includes: in a preset unified 3D space, using a feature extraction network to perform feature alignment encoding based on RGB image data and 3D point cloud data to obtain pixel mapping features and point cloud mapping features; using a coarse-grained instance proposal generator to encode based on the point cloud mapping features to obtain point cloud proposal features and corresponding coarse prediction boxes; using an instance feature aggregation module to dynamically integrate based on the coarse prediction boxes, pixel mapping features, point cloud mapping features, and point cloud proposal features to obtain 2D instance features and 3D instance features; using an instance feature fusion module to perform symmetrical cross-fusion based on the coarse prediction boxes, 2D instance features, and 3D instance features using a dual-channel attention mechanism to obtain fused features; and using a prediction head to identify based on the fused features to obtain instance prediction boxes and predicted class probabilities.

4. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 3, characterized in that, The step of obtaining pixel mapping features and point cloud mapping features by using a feature extraction network to perform feature alignment encoding based on RGB image data and 3D point cloud data in a preset unified 3D space includes: extracting features from RGB image data using a 2D feature extraction network to obtain 2D pixel features; performing linear mapping based on the 2D pixel features in a preset unified 3D space according to preset camera intrinsic and extrinsic parameters to obtain pixel mapping features; extracting features from 3D point cloud data using a 3D feature extraction network to obtain 3D point cloud features; and performing linear mapping based on the 3D point cloud data features in a preset unified 3D space to obtain point cloud mapping features.

5. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 3, characterized in that, The process of obtaining two-dimensional and three-dimensional instance features by dynamically integrating pixel mapping features, point cloud mapping features, and point cloud proposal features using an instance feature aggregation module, based on a coarse prediction box, includes: performing instance-level dynamic feature aggregation using a spatial aggregation feature module based on the pixel mapping features to obtain two-dimensional instance features; performing feature extraction on the point cloud mapping features based on the coarse prediction box to obtain voxel features within the coarse prediction box; performing pooling processing using the spatial aggregation feature module based on the voxel features to obtain local point cloud features; and performing instance-level dynamic feature fusion using a gated neural network based on the coarse prediction box, point cloud proposal features, and local point cloud features to obtain three-dimensional instance features.

6. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 3, characterized in that, The dual-path attention mechanism involves using an instance feature fusion module to perform symmetrical cross-fusion based on coarse prediction boxes, 2D instance features, and 3D instance features to obtain fused features. This includes: learning positional encoding using a multilayer perceptron based on the basic attributes of the coarse prediction boxes to obtain instance positional encoding; using a multi-head cross-attention module to perform geometry-guided feature fusion based on 3D instance features, instance positional encoding, and 2D instance features to obtain a first fused feature; using a multi-head cross-attention module to perform texture-guided feature fusion based on 2D instance features, instance position, and 3D instance features to obtain a second fused feature; and concatenating the first and second fused features to obtain the final fused feature.

7. The instance-level fusion 3D object detection method based on dual-path attention mechanism according to claim 1, characterized in that, The step of calculating the target detection loss based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities includes: calculating the classification loss based on the cross-entropy loss function, using the class labels of the 3D point cloud data and the predicted class probabilities; calculating the regression loss based on the intersection-union ratio (IUU) calculation method, using the ground truth bounding boxes and instance prediction boxes of the 3D point cloud data; and performing a weighted calculation based on the classification loss and regression loss to obtain the target detection loss.

8. An instance-level fusion 3D object detection device based on a dual-path attention mechanism, wherein the instance-level fusion 3D object detection device based on the dual-path attention mechanism is used to implement the instance-level fusion 3D object detection method based on the dual-path attention mechanism as described in any one of claims 1-7, characterized in that, The device includes: a data acquisition module for capturing an indoor target scene using an RGB-D camera to obtain RGB image data and a depth map; and back-projecting the depth map based on preset camera intrinsic and extrinsic parameters to obtain 3D point cloud data; a training-phase 3D detection module for recognizing indoor 3D objects using an instance-level dual-channel attention 3D detection network based on the RGB image data and 3D point cloud data to obtain instance prediction boxes and predicted class probabilities; a network parameter optimization module for calculating a loss function based on the 3D point cloud data, instance prediction boxes, and predicted class probabilities to obtain an object detection loss; and optimizing the parameters of the instance-level dual-channel attention 3D detection network based on the object detection loss to obtain an optimized instance-level dual-channel attention 3D detection network; and an inference-phase 3D detection module for capturing the current indoor scene using an RGB-D camera to obtain current RGB image data and a current depth map; back-projecting the current depth map based on preset camera intrinsic and extrinsic parameters to obtain current 3D point cloud data; and using the optimized instance-level dual-channel attention 3D detection network based on the current RGB image data and current 3D point cloud data to recognize indoor 3D objects to obtain current instance prediction boxes and current predicted class probabilities.

9. An instance-level fusion 3D object detection device, characterized in that, The instance-level fusion 3D object detection device includes: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images