Weak supervision three-dimensional target detection method based on multi-signal decoupling

Abstract

The invention belongs to the field of three-dimensional scene understanding, and provides a weak supervision three-dimensional target detection method based on multi-signal decoupling. According to the method, a multi-signal decoupling learning framework is constructed, and the position, the size and the orientation of a three-dimensional bounding box are respectively supervised by utilizing two-dimensional target frame constraint, category size priori generated by a language model and point cloud geometric information. The method specifically comprises a centrality-enhanced projection constraint module used for optimizing target positioning, a semantic prior anchoring module used for guiding size regression, a rotation consistency regularization module used for improving orientation discrimination capability, and an adversarial geometric alignment module used for dynamically adjusting a three-dimensional frame boundary. All the modules are trained cooperatively, a high-quality pseudo label can be generated for detector training without three-dimensional labeling, and finally the high-precision weak supervision three-dimensional target detection method is realized.

Description

Technical Field

[0001] This invention belongs to the field of 3D scene understanding, and more specifically, it relates to a weakly supervised 3D target detection method based on multi-signal decoupling. Background Technology

[0002] Weakly supervised 3D object detection methods based on 2D annotation are driving a shift in 3D detection from reliance on expensive manual annotation to a lightweight learning paradigm. This technology enables autonomous driving systems, robotic platforms, and embedded devices to overcome the bottleneck of acquiring 3D annotations. By dynamically acquiring image data during operation, simple 2D annotations are sufficient to continuously improve model detection accuracy. These methods typically generate pseudo-3D bounding boxes based on 2D annotations by designing geometric constraints or projection rules to aid in the training of 3D detection models. For example, Yi Wei et al.'s "FGR: Frustum-Aware GeometricReasoning for Weakly Supervised 3D Vehicle Detection" utilizes point cloud clustering to enhance spatial constraints, while Liang Peng et al.'s "WeakM3D: Towards Weakly Supervised Monocular 3D Object Detection" predicts 3D bounding boxes based on geometric priors. However, these methods are usually based on a single modal input or signal and jointly optimize the three-dimensional parameters (position, size, rotation angle) through a unified loss function. They ignore the physical semantic differences between parameters, making it difficult to achieve independent modeling and step-by-step reasoning of position, size and orientation. This leads to problems such as scale depth ambiguity and rotation angle ambiguity during the projection process.

[0003] To address these issues, existing research has attempted to incorporate rich supervisory information to improve performance. For example, Guowen Zhang et al.'s "General Geometry-aware Weakly Supervised 3D Object Detection" uses prior sizes generated by a large language model to assist in 3D bounding box prediction, improving detection results through semantic constraints. However, this method relies on manually generated rules for prior matching, limiting its flexibility, and the optimization process still suffers from parameter coupling problems, making it difficult to fully decouple the various parameters of the 3D bounding box. Furthermore, existing methods still have significant room for improvement in the synergistic effect of multimodal information and hierarchical signal optimization. These limitations affect the performance and generalization ability of weakly supervised 3D detection in multi-class detection tasks. Summary of the Invention

[0004] This invention proposes a weakly supervised 3D object detection method based on multi-signal decoupling, which enables 3D object detection under 2D supervision signals, effectively alleviating the dependence on 3D annotations. In this method, 2D annotations are used to supervise the projection of the 3D prediction box onto the 2D plane, helping the model acquire basic perceptual abilities for different categories. However, an inherent information gap exists between the 2D plane and 3D space, directly leading to serious projection ambiguity problems. Specifically, this manifests as: 1) scale-depth ambiguity, where 3D objects of different sizes and depths may produce identical 2D projection boxes; 2) orientation ambiguity, where objects with different orientations may project the same 2D box. Furthermore, existing methods typically use a unified loss function to coupledly optimize all 3D parameters (position, size, rotation angle), ignoring the heterogeneous physical semantics of each parameter, leading to conflicts in the optimization process and inaccurate prediction results.

[0005] To address the aforementioned problems, this invention proposes an innovative multi-signal decoupled learning and detection framework called DiMuS. The core idea of ​​this framework is to design different supervisory signals and decouple them from specific 3D parameters for targeted optimization. Specifically, this includes: enhancing position localization through enhanced 2D bounding box projection supervision; performing self-supervised learning to resolve object orientation through rotation-aware consistency regularization; and fine-tuning boundaries using the bidirectional interaction between point cloud and predicted bounding box through adversarial geometric alignment. This decoupling design effectively overcomes the two major challenges of projection ambiguity and parameter optimization conflicts, achieving high-precision weakly supervised 3D detection.

[0006] The technical solution adopted in this invention is as follows:

[0007] A weakly supervised 3D target detection method based on multi-signal decoupling learning, comprising the following steps:

[0008] Step 1: Obtain the prior dimensions of the object;

[0009] (1.1) Design the problem “Please give the average three-dimensional size of the object category to be studied” as a question to the multimodal large language model GPT4o, where the object category to be studied includes three-dimensional object categories;

[0010] (1.2) The multimodal large language model GPT4o is used to answer the design question in step (1.1) to obtain the prior dimensions of the object category to be studied, including length, width and height;

[0011] Step 2: Point cloud rotation enhancement;

[0012] (2.1) Use lidar to scan the scene to be detected to obtain scene point cloud, use a camera to take a picture of the scene to be detected to obtain color image, and record the camera intrinsic and extrinsic parameters corresponding to the color image;

[0013] (2.2) The color image obtained in step (2.1) is labeled, and the two-dimensional bounding boxes of all objects to be studied in the color image are labeled. Based on the two-dimensional bounding boxes and the intrinsic and extrinsic parameters of the camera, the view frustum cloud corresponding to each object to be studied is extracted from the scene point cloud.

[0014] (2.3) For the view frustum cloud corresponding to each object to be studied obtained in step (2.2), sort them according to the average depth of the view frustum cloud, and execute the region growing algorithm from shallow to deep to obtain the object point cloud that constitutes all the original objects.

[0015] (2.4) Perform rotation enhancement on the object point cloud of the original object extracted in step (2.3) to obtain the object point cloud of the rotated object, generate the object pair of the original object and the rotated object, and record the rotation enhancement angle.

[0016] (2.5) For the scene point cloud obtained in step (2.1), the RANSAC algorithm is used to calculate the ground plane height of the scene point cloud. A large number of candidate points are randomly selected on the ground plane. The candidate points are used to perform collision detection with all other objects to be studied in the scene according to the pre-set distance threshold. All non-collision candidate points are saved and used to embed the rotating object to generate an enhanced scene point cloud containing the original object and the rotating object.

[0017] Step 3: Train a weakly supervised 3D object detection model;

[0018] (3.1) A weakly supervised 3D target detection model is constructed based on the CenterPoint model framework, including a 3D backbone network and a target detection head. The 3D backbone network includes a voxelization layer, an encoder layer and a feature pyramid layer. The target detection head receives the output features of the feature pyramid layer to predict the 3D bounding box.

[0019] (3.2) The enhanced scene point cloud obtained in step (2.5) is passed through the 3D backbone network and the target detection head of the weakly supervised 3D target detection model in sequence, and the output includes the position, scale factor and rotation angle of the predicted 3D bounding box.

[0020] (3.3) The prior size obtained in step (1.2) is combined with the scale factor of the predicted 3D bounding box obtained in step (3.2) to obtain the actual predicted size, as shown in the following formula:

[0021]

[0022] in, For the actual predicted size, The prior size provided for the multimodal large language model GPT4o Hyperparameters for controlling the magnitude of change, To predict the scale factor of the 3D bounding box;

[0023] For the prediction output of a weakly supervised 3D object detection model, three supervision algorithms are designed to decouple and supervise the parameters in the output:

[0024] (1) For each object to be studied, multiple 3D boxes are predicted, a set of predicted boxes is constructed, and the average 3D box of the predicted box set is calculated. The reciprocal of the distance between the position parameters of the predicted 3D box and the average 3D box is used as the weight of each predicted 3D box. All predicted 3D boxes are projected onto a 2D plane, and the L1 loss function is calculated with the 2D bounding box labeled in step (2.2). The calculated weights of the predicted 3D boxes are used to weight the loss and sum the losses as the weighted 2D projection supervision loss.

[0025] (2) For the object pair of the original object and the rotating object generated in step (2.4), the predicted rotation angle and predicted size of the weakly supervised 3D target detection model for the original object and the rotating object are obtained respectively. The difference between the predicted rotation angle and the difference between the predicted size are calculated. The difference between the predicted rotation angle and the rotation enhancement angle recorded in step (2.4) are used to calculate the rotation error. The difference between the predicted rotation angle and the predicted size are added to the difference between the predicted size as the rotation consistency regularization loss.

[0026] (3) From the perspective of BEV, the point clouds of the original object and the rotated object obtained in step 2 are divided into internal points and external points according to the boundary of the predicted 3D box. The distance between all internal points and external points and the nearest edge of the predicted 3D box is calculated. The sum of the distances between internal points and the boundary is used as the attractive force part in the adversarial geometric alignment loss, and the sum of the distances between external points and the boundary is used as the repulsive force part in the adversarial geometric alignment loss. When the proportion of internal points is too small, the repulsive force dominates to expand the boundary. When there are enough internal points, the attractive force and the repulsive force work together to achieve accurate fitting of the boundary.

[0027] The beneficial effects of this invention are:

[0028] 1. The present invention matches the position, size and rotation angle parameters of the 3D bounding box with decoupled supervision signals such as weighted 2D projection supervision, large model prior size, rotation consistency regularization and 3D geometric alignment, respectively. This effectively avoids the coupling optimization conflict between parameters, thereby significantly improving the accuracy and robustness of weakly supervised 3D target detection.

[0029] 2. This invention overcomes the problem of inaccurate positioning of large objects in existing methods by introducing weighted two-dimensional projection supervision, effectively improving the prediction accuracy of the object's center position.

[0030] 2. This invention utilizes the prior dimensions provided by a multimodal large language model as the benchmark anchor point for predicted dimensions, effectively solving the scale-depth ambiguity problem that exists when relying solely on two-dimensional projection supervision. This makes the predicted object dimensions more stable and consistent with common sense in the physical world, significantly improving the accuracy and rationality of dimension prediction.

[0031] 3. This invention proposes a rotation consistency regularization method, which obtains original object-rotating object pairs through data augmentation and learns rotation invariant features in a self-supervised manner. This method effectively enhances the model's sensitivity to object orientation, overcomes the orientation ambiguity caused by uneven angle distribution in the dataset and two-dimensional projection, and significantly improves the accuracy of rotation angle prediction.

[0032] 4. This invention designs an adversarial geometric alignment module, which establishes a dynamic bidirectional force of attraction and repulsion between the object point cloud and the 3D bounding box boundary, achieving precise optimization of the predicted bounding box boundary. This mechanism can adaptively adjust the boundary according to the spatial distribution relationship of the point cloud, thereby improving the fidelity of the predicted bounding box boundary.

[0033] 5. Compared with weakly supervised 3D target detection methods such as GGA, DiMuS avoids problems such as insensitivity to rotation angle perception and large size prediction errors, achieving high-precision detection results and having broad potential for technology promotion. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0036] This invention is a 3D target detection method based on 2D supervised signals, proposing a technical solution called DiMuS. DiMuS designs a weakly supervised 3D target detection training method with multi-signal decoupled supervision, thoroughly analyzes and addresses the projection ambiguity problem of 2D bounding boxes, and introduces weighted 2D projection supervision, large model prior size, rotational consistent regularization, and adversarial geometric alignment algorithm to independently optimize different parameters of the predicted bounding box in a directional manner, thereby improving detection accuracy.

[0037] To improve the accuracy of predicted size, this method relies on the open-world knowledge of the multimodal large language model GPT4o to obtain the prior size of the category under study. Based on the prior size information, a scale factor prediction method is introduced to replace the traditional absolute size regression method, so that the size of the predicted box can be obtained by adaptively combining the prior information and the scale factor.

[0038] To enhance the model's perception of rotation angles, this method proposes a point cloud rotation enhancement strategy. Specifically, for the input scene point cloud, firstly, the view frustum point cloud is extracted based on the 2D bounding box, and then the object point cloud is extracted using a region growing algorithm. Next, the extracted object point cloud is rotated and enhanced. Then, the ground plane height of the scene point cloud is calculated, and candidate points are searched on the ground to embed the rotated object, thus obtaining the enhanced scene point cloud.

[0039] To improve the prediction quality of different parameters of the 3D bounding box, this method proposes a decoupled training approach for weakly supervised 3D target models. First, the model takes a rotated enhanced scene point cloud as input and outputs the position, scale factor, and rotation angle of the predicted 3D bounding box. Then, the scale factor of the predicted 3D bounding box is combined with the prior size to obtain the actual predicted size. Weighted 2D projection supervision loss and rotation consistency regularization loss are used to specifically supervise the position and rotation angle parameters of the predicted 3D bounding box, respectively. Furthermore, adversarial geometric alignment loss is used to help achieve accurate fitting of the predicted 3D bounding box boundary. Figure 1 As shown, a weakly supervised 3D target detection method based on multi-signal decoupling has the following steps:

[0040] Step 1: Obtain the prior dimensions of the object;

[0041] (1.1) Design the problem “Please give the average three-dimensional size of the object category to be studied” as a question to the multimodal large language model GPT4o, where the object category to be studied includes three-dimensional object categories;

[0042] (1.2) The multimodal large language model GPT4o is used to answer the design question in step (1.1) to obtain the prior dimensions of the object category to be studied, including length, width and height;

[0043] Step 2: Point cloud rotation enhancement;

[0044] (2.1) Use lidar to scan the scene to be detected to obtain scene point cloud, use a camera to take a picture of the scene to be detected to obtain color image, and record the camera intrinsic and extrinsic parameters corresponding to the color image;

[0045] (2.2) The color image obtained in step (2.1) is labeled, and the two-dimensional bounding boxes of all objects to be studied in the color image are labeled. Based on the two-dimensional bounding boxes and the intrinsic and extrinsic parameters of the camera, the view frustum cloud corresponding to each object to be studied is extracted from the scene point cloud.

[0046] (2.3) For the view frustum cloud corresponding to each object to be studied obtained in step (2.2), sort them according to the average depth of the view frustum cloud, and execute the region growing algorithm from shallow to deep to obtain the object point cloud that constitutes all the original objects.

[0047] (2.4) Perform rotation enhancement on the object point cloud of the original object extracted in step (2.3) to obtain the object point cloud of the rotated object, generate the object pair of the original object and the rotated object, and record the rotation enhancement angle.

[0048] (2.5) For the scene point cloud obtained in step (2.1), the RANSAC algorithm is used to calculate the ground plane height of the scene point cloud. A large number of candidate points are randomly selected on the ground plane. The candidate points are used to perform collision detection with all other objects to be studied in the scene according to the pre-set distance threshold. All non-collision candidate points are saved and used to embed the rotating object to generate an enhanced scene point cloud containing the original object and the rotating object.

[0049] Step 3: Train a weakly supervised 3D object detection model;

[0050] (3.1) A weakly supervised 3D target detection model is constructed based on the CenterPoint model framework, including a 3D backbone network and a target detection head. The 3D backbone network includes a voxelization layer, an encoder layer and a feature pyramid layer. The target detection head receives the output features of the feature pyramid layer to predict the 3D bounding box.

[0051] (3.2) The enhanced scene point cloud obtained in step (2.5) is passed through the 3D backbone network and the target detection head of the weakly supervised 3D target detection model in sequence, and the output includes the position, scale factor and rotation angle of the predicted 3D bounding box.

[0052] (3.3) The prior size obtained in step (1.2) is combined with the scale factor of the predicted 3D bounding box obtained in step (3.2) to obtain the actual predicted size, as shown in the following formula:

[0053]

[0054] in, For the actual predicted size, The prior size provided for the multimodal large language model GPT4o Hyperparameters for controlling the magnitude of change, To predict the scale factor of the 3D bounding box;

[0055] For the prediction output of a weakly supervised 3D object detection model, three supervision algorithms are designed to decouple and supervise the parameters in the output:

[0056] (1) For each object to be studied, multiple 3D boxes are predicted, a set of predicted boxes is constructed, and the average 3D box of the predicted box set is calculated. The reciprocal of the distance between the position parameters of the predicted 3D box and the average 3D box is used as the weight of each predicted 3D box. All predicted 3D boxes are projected onto a 2D plane, and the L1 loss function is calculated with the 2D bounding box labeled in step (2.2). The calculated weights of the predicted 3D boxes are used to weight the loss and sum the losses as the weighted 2D projection supervision loss.

[0057] (2) For the object pair of the original object and the rotating object generated in step (2.4), the predicted rotation angle and predicted size of the weakly supervised 3D target detection model for the original object and the rotating object are obtained respectively. The difference between the predicted rotation angle and the difference between the predicted size are calculated. The difference between the predicted rotation angle and the rotation enhancement angle recorded in step (2.4) are used to calculate the rotation error. The difference between the predicted rotation angle and the predicted size are added to the difference between the predicted size as the rotation consistency regularization loss.

[0058] (3) From the perspective of BEV, the point clouds of the original object and the rotated object obtained in step 2 are divided into internal points and external points according to the boundary of the predicted 3D box. The distance between all internal points and external points and the nearest edge of the predicted 3D box is calculated. The sum of the distances between internal points and the boundary is used as the attractive force part in the adversarial geometric alignment loss, and the sum of the distances between external points and the boundary is used as the repulsive force part in the adversarial geometric alignment loss. When the proportion of internal points is too small, the repulsive force dominates to expand the boundary. When there are enough internal points, the attractive force and the repulsive force work together to achieve accurate fitting of the boundary.

[0059] Example:

[0060] In this embodiment, the framework of the present invention is built based on the CenterPoint model, using the KITTI dataset for training and validation, employing the AdamW optimizer with a learning rate of 0.0015 and weight decay of 0.01. After 120 training epochs based on 2D bounding boxes as supervised information, 3D bounding boxes are generated as pseudo-labels for the dataset, and the accuracy is validated on public datasets and the model. All experiments are performed on a 4-bit machine. Performed on an NVIDIA A800 Tensor Core GPU.

[0061] This invention significantly improves accuracy across all detection tasks on the KITTI dataset. Compared to the "General Geometry-aware Weakly Supervised 3D Object Detection" method, it achieves higher detection accuracy of 18.82, 8.41, and 19.51 percentage points for vehicles, pedestrians, and cyclists, respectively. This demonstrates the invention's superior learning ability with 2D annotations and ensures cross-category generalization, exhibiting robust detection performance across multiple research categories. Furthermore, this invention does not rely on expensive 3D annotations; it can train a high-precision 3D detection model using only readily available 2D annotations, proving its applicability to real-world autonomous driving and robotics applications.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weakly supervised 3D target detection method based on multi-signal decoupling learning, characterized in that, The steps are as follows: Step 1: Obtain the prior dimensions of the object; (1.1) Design problem "Please give the average three-dimensional size of the object category to be studied", as a question to be asked to the multimodal large language model GPT4o, where the object category to be studied includes three-dimensional object categories; (1.2) The multimodal large language model GPT4o is used to answer the design question in step (1.1) to obtain the prior dimensions of the object category to be studied, including length, width and height; Step 2: Point cloud rotation enhancement; (2.1) Use lidar to scan the scene to be detected to obtain scene point cloud, use a camera to take a picture of the scene to be detected to obtain color image, and record the camera intrinsic and extrinsic parameters corresponding to the color image; (2.2) The color image obtained in step (2.1) is labeled, and the two-dimensional bounding boxes of all objects to be studied in the color image are labeled. Based on the two-dimensional bounding boxes and the intrinsic and extrinsic parameters of the camera, the view frustum cloud corresponding to each object to be studied is extracted from the scene point cloud. (2.3) For the view frustum cloud corresponding to each object to be studied obtained in step (2.2), sort them according to the average depth of the view frustum cloud, and execute the region growing algorithm from shallow to deep to obtain the object point cloud that constitutes all the original objects. (2.4) Perform rotation enhancement on the object point cloud of the original object extracted in step (2.3) to obtain the object point cloud of the rotated object, generate the object pair of the original object and the rotated object, and record the rotation enhancement angle. (2.5) For the scene point cloud obtained in step (2.1), the RANSAC algorithm is used to calculate the ground plane height of the scene point cloud. A large number of candidate points are randomly selected on the ground plane. The candidate points are used to perform collision detection with all other objects to be studied in the scene according to the pre-set distance threshold. All non-collision candidate points are saved and used to embed the rotating object to generate an enhanced scene point cloud containing the original object and the rotating object. Step 3: Train a weakly supervised 3D object detection model; (3.1) A weakly supervised 3D target detection model is constructed based on the CenterPoint model framework, including a 3D backbone network and a target detection head. The 3D backbone network includes a voxelization layer, an encoder layer and a feature pyramid layer. The target detection head receives the output features of the feature pyramid layer to predict the 3D bounding box. (3.2) The enhanced scene point cloud obtained in step (2.5) is passed through the 3D backbone network and the target detection head of the weakly supervised 3D target detection model in sequence, and the output includes the position, scale factor and rotation angle of the predicted 3D bounding box. (3.3) The prior size obtained in step (1.2) is combined with the scale factor of the predicted 3D bounding box obtained in step (3.2) to obtain the actual predicted size, as shown in the following formula: ， in, For the actual predicted size, The prior size provided for the multimodal large language model GPT4o Hyperparameters for controlling the magnitude of change, To predict the scale factor of the 3D bounding box; For the prediction output of a weakly supervised 3D object detection model, three supervision algorithms are designed to decouple and supervise the parameters in the output: (1) For each object to be studied, multiple 3D boxes are predicted, a set of predicted boxes is constructed, and the average 3D box of the predicted box set is calculated. The reciprocal of the distance between the position parameters of the predicted 3D box and the average 3D box is used as the weight of each predicted 3D box. All predicted 3D boxes are projected onto a 2D plane, and the L1 loss function is calculated with the 2D bounding box labeled in step (2.2). The calculated weights of the predicted 3D boxes are used to weight the loss and sum the losses as the weighted 2D projection supervision loss. (2) For the object pair of the original object and the rotating object generated in step (2.4), the predicted rotation angle and predicted size of the weakly supervised 3D target detection model for the original object and the rotating object are obtained respectively. The difference between the predicted rotation angle and the difference between the predicted size are calculated. The difference between the predicted rotation angle and the rotation enhancement angle recorded in step (2.4) are used to calculate the rotation error. The difference between the predicted rotation angle and the predicted size are added to the difference between the predicted size as the rotation consistency regularization loss. (3) From the perspective of BEV, the point clouds of the original object and the rotated object obtained in step 2 are divided into internal points and external points according to the boundary of the predicted 3D box. The distance between all internal points and external points and the nearest edge of the predicted 3D box is calculated. The sum of the distances between internal points and the boundary is used as the attractive force part in the adversarial geometric alignment loss, and the sum of the distances between external points and the boundary is used as the repulsive force part in the adversarial geometric alignment loss. When the proportion of internal points is too small, the repulsive force dominates to expand the boundary. When there are enough internal points, the attractive force and the repulsive force work together to achieve accurate fitting of the boundary.

Images