Three-dimensional target detection refinement method based on multi-expert model

Abstract

The invention relates to the field of computer vision and artificial intelligence, in particular to a three-dimensional target detection refining method based on a multi-expert model, and the method comprises the steps: carrying out the feature extraction of original point cloud data, and obtaining the point cloud features of the original point cloud data; generating an initial three-dimensional candidate frame based on the point cloud features; based on a sparseness perception gating mechanism and the number of original point clouds in the initial three-dimensional candidate frame, calculating to obtain a sparseness score vector; inputting the initial three-dimensional candidate box into a hierarchical expert network to output prediction results of all sparseness expert branches; based on the sparseness score vector, performing weighted fusion on prediction results of all sparseness expert branches to generate a final three-dimensional bounding box; in the model training stage, the loss of each sparseness expert branch is weighted based on the sparseness score vector, and network parameters are updated through back propagation. According to the method, the accuracy and robustness of detection and positioning can be improved while the calculation efficiency is maintained.

Description

Technical Field

[0001] The embodiments of this application relate to the fields of computer vision and artificial intelligence, and in particular to a refined method for 3D object detection based on a multi-expert model. Background Technology

[0002] With the development of autonomous driving, unmanned systems, and intelligent transportation, LiDAR-based 3D target detection technology has gradually become a core component of environmental perception. 3D target detection can accurately identify and locate targets such as vehicles, pedestrians, and bicycles in complex scenes, providing a reliable basis for path planning and decision control. Among these methods, the two-stage detection approach is widely used due to its high accuracy: the first stage generates candidate bounding boxes, and the second stage refines these boxes to achieve even higher detection accuracy.

[0003] However, existing two-stage refinement methods still have significant shortcomings. On the one hand, multiple attributes of the detection boxes (such as center position, size, and orientation) depend on different geometric feature information. If a single network regresses them simultaneously, feature conflicts are likely to occur, leading to negative transfer in multi-task learning and affecting overall accuracy. On the other hand, the sparsity of point clouds within candidate boxes varies greatly. Some candidate boxes contain fewer points or are severely occluded. A uniform refinement strategy is difficult to adapt to such data differences, resulting in unstable detection results.

[0004] Existing technologies primarily improve performance by enhancing feature extraction or refining network structure, but often overlook two key issues: multi-attribute learning conflicts and sample sparsity differences. This makes it difficult for the refinement stage to play its due role, limiting detection accuracy and robustness.

[0005] Therefore, there is an urgent need for a new refined method for 3D target detection that can provide independent optimization strategies for different attribute learning conflicts and sample sparsity problems, thereby improving the accuracy and robustness of detection and localization while maintaining computational efficiency. Summary of the Invention

[0006] To address the aforementioned technical issues, embodiments of this application propose a refined method for 3D target detection based on a multi-expert model. This method provides independent optimization strategies for different attribute learning conflicts and sample sparsity problems, thereby improving the accuracy and robustness of detection and localization while maintaining computational efficiency.

[0007] To achieve the above objectives, embodiments of this application propose a refined method for 3D target detection based on a multi-expert model, the method comprising: Feature extraction is performed on the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data; Initial 3D candidate boxes are generated based on point cloud features; wherein, the initial 3D candidate boxes are 3D bounding boxes used to represent the position, size and orientation of potential objects; A sparsity-aware gating mechanism is constructed, and a sparsity score vector is calculated based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes; wherein, the sparsity score vector is used to represent the suitability of the initial 3D candidate boxes with each sparsity expert branch; Construct a hierarchical expert network containing multiple sparse expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches; Based on the sparsity score vector, the prediction results of all sparse expert branches are weighted and fused to generate the final 3D bounding box. During the model training phase, the loss of each sparse expert branch is weighted based on the sparse score vector to obtain the total loss, and the network parameters are updated through backpropagation.

[0008] To achieve the above objectives, embodiments of this application also propose a three-dimensional target detection refinement device based on a multi-expert model, the device comprising: The extraction module is used to extract features from the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data. The generation module is used to generate initial 3D candidate boxes based on point cloud features; wherein, the initial 3D candidate boxes are 3D bounding boxes used to represent the position, size and orientation of potential objects; The computation module is used to construct a sparsity-aware gating mechanism and calculate a sparsity score vector based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes; wherein, the sparsity score vector is used to represent the suitability of the initial 3D candidate boxes with each sparsity expert branch; The computation module is also used to construct a hierarchical expert network containing multiple sparse expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches; The weighting module is used to weight and fuse the prediction results of all sparse expert branches based on the sparse score vector to generate the final 3D bounding box. The training module is used during the model training phase to weight the loss of each sparse expert branch based on the sparse score vector to obtain the total loss, and then updates the network parameters through backpropagation.

[0009] To achieve the above objectives, embodiments of this application also propose an electronic device, including: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a multi-expert model-based method for refining 3D target detection as described above.

[0010] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of a multi-expert model-based method for refining 3D target detection as described above.

[0011] This application proposes a refined 3D object detection method based on a multi-expert model. First, raw point cloud data is acquired, and features are extracted from the raw point cloud data to obtain point cloud features. Then, initial 3D candidate boxes are generated based on the point cloud features. A sparsity score vector is calculated based on a sparsity-aware gating mechanism and the number of raw points within the initial 3D candidate boxes. A hierarchical expert network containing multiple sparsity expert branches is constructed, and the initial 3D candidate boxes are input into the hierarchical expert network to output the prediction results of all sparsity expert branches. Based on the sparsity score vector, the prediction results of all sparsity expert branches are weighted and fused to generate the final 3D bounding box. During the model training phase, the loss of each sparsity expert branch is weighted based on the sparsity score vector, and the network parameters are updated through backpropagation. This provides independent optimization strategies for learning conflicts of different attributes and sample sparsity issues, thereby improving the accuracy and stability of the detection results while maintaining computational efficiency.

[0012] Optionally, the original point cloud data is denoted as , , This represents the total number of points in a single frame of the point cloud. Representing a three-dimensional real space, the step of extracting features from the acquired raw point cloud data to obtain point cloud features of the raw point cloud data includes: For raw point cloud data Voxelization is performed to divide the point cloud space, and a 3D sparse convolutional network is used to process the non-empty voxels to extract the initial geometric features of the original point cloud data. Multiple key points are sampled from the original point cloud data, and the voxel set abstraction module is used to aggregate the voxel features around each key point to obtain point cloud features.

[0013] Optionally, generating initial 3D candidate boxes based on point cloud features includes: inputting the point cloud features into a candidate region generation network, so that the candidate region generation network classifies and regresses the point cloud features to obtain initial 3D candidate boxes; wherein, the initial 3D candidate boxes... Specifically, it is expressed as follows: ;in, This represents the total number of candidate boxes generated. Representing the A 3D candidate box, Candidate boxes The three-dimensional coordinates of the center point These are candidate boxes Length, width, and height, Candidate boxes The yaw angle is defined as its orientation angle in a top view.

[0014] Optionally, a sparsity-aware gating mechanism is constructed, and based on the sparsity-aware gating mechanism and the number of original point clouds within the initial 3D candidate boxes, a sparsity score vector is calculated, including: Calculate the initial 3D candidate box Each candidate box Number of raw point clouds contained inside , as a metric for measuring the sparsity of point clouds, Specifically, it is expressed as follows: ; set up A sparsity expert branch Each sparsity expert branch is an independent neural network module used to specialize the processing of candidate boxes within its corresponding sparsity range. Gaussian kernel-based scoring function The original point cloud quantity Mapped to a Sparsity score vector of dimension , Candidate boxes With the A sparsity expert branch The relevance score; Specifically, this can be expressed by the following formula: ; in, It is the first The point range mainly overseen by each expert. It is the lower bound of the interval. It is the upper bound of the interval. Center of the interval Controlling the rate of score decay, the number of points in the point range specifically refers to the number of original LiDAR points contained within the candidate box. This value is the core metric for measuring the sparsity of candidate boxes.

[0015] Optionally, a hierarchical expert network containing multiple sparse expert branches is constructed, and the initial 3D candidate boxes are input into the hierarchical expert network to output the prediction results of all sparse expert branches, including: Each sparsity expert branch It is divided into three independent attribute expert sub-branches, including: the center point optimization branch. Size optimization branch and Orientation Optimization Branch ; For candidate boxes By utilizing a shared region of interest feature extractor, the corresponding features are extracted from the backbone network feature map.

[0016] In each sparsity expert branch Internally, features Input into the center point optimization branch respectively Size optimization branch and Orientation Optimization Branch In this process, each optimization branch learns the optimization of a specific attribute through its internal multilayer perceptron, thereby outputting the prediction results of all sparse expert branches. The prediction results include classification confidence and central residuals, specifically including: Center point optimization branches Output classification confidence score and central residual ; Size optimization branch Output classification confidence score and dimensional residuals ; Towards Optimized Branches Output classification confidence score and central residual .

[0017] Optionally, based on the sparsity score vector, the prediction results of all sparse expert branches are weighted and fused to generate the final 3D bounding box, including: For each sparsity expert First, the prediction results of its three attribute expert sub-branches are aggregated. Specifically, it includes: ; ; in, Indicates the final classification confidence level. This represents the final regression vector; thus, each sparsity expert is obtained. For candidate boxes Prediction results ; Using sparsity score vectors Normalization is performed to obtain the fusion weights. Specifically, it includes: ; in, This represents a small constant used to maintain numerical stability; Based on fusion weight The predictions from all sparsity expert branches are weighted to obtain the final 3D bounding box. Specifically, it includes: .

[0018] Optionally, during the model training phase, the losses of each sparse expert branch are weighted based on the score vector to obtain the total loss, and the network parameters are updated through backpropagation, including: Define the standard optimization loss function The optimized loss function includes classification loss and regression loss to measure the performance of the first class. The sparsity expert branch for the first Prediction results of candidate boxes With real labels The differences between them; During model training, sparsity scores are utilized. As weights, the losses of each sparse expert branch are weighted to obtain the total loss of each sparse expert branch; where, the th The total loss of each sparsity expert branch The calculation method specifically includes: ; The total loss is determined by summing the losses of all sparsity expert branches. Specifically, it includes: ; Total loss The learnable parameters in all expert branches and the backbone network are updated using the backpropagation algorithm. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0020] Figure 1 This is a flowchart of a three-dimensional target detection refinement method based on a multi-expert model provided in one embodiment of this application; Figure 2 This is a flowchart illustrating the implementation of a multi-expert model-based method for refining 3D target detection, provided in one embodiment of this application. Figure 3 This is a schematic diagram of an overall network architecture provided in one embodiment of this application; Figure 4 This is a schematic diagram of a sparsity-aware gating mechanism provided in one embodiment of this application; Figure 5 This is an example of an attribute expert branch decoupling structure diagram provided in one embodiment of this application; Figure 6 This is an example diagram of a detection failure on a sparse target provided in one embodiment of this application; Figure 7 This is an example image of successful detection on sparse targets provided in one embodiment of this application; Figure 8 This is a schematic diagram of the structure of a three-dimensional target detection refinement device based on a multi-expert model provided in another embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0022] With the development of autonomous driving, unmanned systems, and intelligent transportation, LiDAR-based 3D target detection technology has gradually become a core component of environmental perception. 3D target detection can accurately identify and locate targets such as vehicles, pedestrians, and bicycles in complex scenes, providing a reliable basis for path planning and decision control. Among these methods, the two-stage detection approach is widely used due to its high accuracy: the first stage generates candidate bounding boxes, and the second stage refines these boxes to achieve even higher detection accuracy.

[0023] However, existing two-stage refinement methods still have significant shortcomings. On the one hand, multiple attributes of the detection boxes (such as center position, size, and orientation) depend on different geometric feature information. If a single network regresses them simultaneously, feature conflicts are likely to occur, leading to negative transfer in multi-task learning and affecting overall accuracy. On the other hand, the sparsity of point clouds within candidate boxes varies greatly. Some candidate boxes contain fewer points or are severely occluded. A uniform refinement strategy is difficult to adapt to such data differences, resulting in unstable detection results.

[0024] Existing technologies primarily improve performance by enhancing feature extraction or refining network structure, but often overlook two key issues: multi-attribute learning conflicts and sample sparsity differences. This makes it difficult for the refinement stage to play its due role, limiting detection accuracy and robustness.

[0025] Therefore, there is an urgent need for a new refined method for 3D target detection that can provide independent optimization strategies for different attribute learning conflicts and sample sparsity problems, thereby improving the accuracy and stability of detection results while maintaining computational efficiency.

[0026] In view of this, embodiments of this application propose a refined method for 3D object detection based on a multi-expert model. The technical problem this application aims to solve is that existing two-stage 3D LiDAR object detectors typically employ a single, holistic optimization head network during the second-stage optimization process. This network needs to simultaneously regress heterogeneous attributes such as center position, size, and orientation, leading to feature conflicts (i.e., "inter-attribute conflicts") between different attribute learning tasks, thus causing negative transfer problems. Furthermore, this single model lacks generalization ability for samples with significant differences in point cloud density within the candidate bounding box (i.e., "inter-sample conflicts"), making it difficult to efficiently handle both sparse and dense point clouds simultaneously, ultimately limiting the detection accuracy.

[0027] To achieve the above objectives, the solution adopted in this application is to propose a refined 3D object detection method based on a multi-expert model, specifically a 3D object optimization method based on a hierarchical mixture-of-experts (HMI) model. This method reconstructs the 3D object optimization problem into a hierarchical decision-making process. First, a sparse gating network is used to "soft-route" candidate bounding boxes to different sparse perception expert networks based on the point cloud density within the bounding boxes. Then, within each sparse expert network, the optimization task is further decomposed into three sub-tasks: center, size, and orientation, which are processed by dedicated attribute expert branches. This method systematically resolves the dual conflicts between attributes and between samples through a divide-and-conquer strategy, thereby improving the final accuracy of 3D object detection.

[0028] For a detailed description of a refined method for 3D target detection based on a multi-expert model, please refer to the following embodiments.

[0029] One embodiment of this application proposes a refined 3D target detection method based on a multi-expert model, applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for illustration. The implementation details of the refined 3D target detection method based on a multi-expert model proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0030] The specific process of the refined 3D target detection method based on a multi-expert model proposed in this embodiment can be described as follows: Figure 1 As shown, it includes the following steps 101 to 106.

[0031] Step 101: Extract features from the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data; For example, the acquired raw point cloud data can be voxelized, a process that divides the continuous 3D space containing the point cloud into discrete voxel grids and encodes their features. Subsequently, a 3D sparse convolution-based backbone network and a voxel set abstraction module are used to efficiently extract keypoint features. These keypoint features are a small set of representative point feature vectors sampled and encoded from the original point cloud. Each point feature vector carries rich geometric and contextual information from its neighborhood, providing a high-quality feature foundation for subsequent object detection and optimization.

[0032] In one possible embodiment, the original point cloud data is denoted as... , , This represents the total number of points in a single frame of the point cloud. It represents the three-dimensional real number space.

[0033] The step of extracting features from the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data includes: For raw point cloud data Voxelization is performed to divide the point cloud space, and a 3D sparse convolutional network is used to process the non-empty voxels to extract the initial geometric features of the original point cloud data. Multiple key points are sampled from the original point cloud data, and the voxel set abstraction module is used to aggregate the voxel features around each key point to obtain information-rich and robust point cloud features for each key point.

[0034] Step 102: Generate initial 3D candidate boxes based on point cloud features.

[0035] The initial 3D candidate box is used to represent the 3D bounding box of the potential object's position, size, and orientation.

[0036] For example, using the point cloud features extracted in the above embodiments, i.e., keypoint features, a candidate region generation network (RPN) is used to perform foreground and background classification and initial bounding box regression on each keypoint feature, thereby generating a series of coarse 3D candidate boxes (i.e., initial 3D candidate boxes) covering potential objects in the scene. The 3D candidate boxes are 3D bounding boxes used to initially indicate the position, size, and orientation of potential objects.

[0037] In one possible embodiment, generating an initial 3D candidate bounding box based on point cloud features includes: Point cloud features are input into a candidate region generation network, which then classifies and regresses the point cloud features to obtain initial 3D candidate boxes. Specifically, it is expressed as follows: ; in, This represents the total number of candidate boxes generated. Representing the A 3D candidate box, Candidate boxes The three-dimensional coordinates of the center point These are candidate boxes Length, width, and height, Candidate boxes The yaw angle is defined as its orientation angle in a top view.

[0038] Step 103: Construct a sparsity-aware gating mechanism, and calculate the sparsity score vector based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes.

[0039] The sparsity score vector is used to represent the suitability of the initial 3D candidate box with each sparsity expert branch.

[0040] For example, a sparsity-aware gating mechanism is designed based on the number of original point clouds contained within each candidate box. This mechanism uses the number of original point cloud points falling within the geometric volume of each 3D candidate box as a direct indicator of the sparsity of the targets within the box. Subsequently, a score vector is calculated for each candidate box, where each dimension of the vector corresponds to a sparse expert network, and its value represents the appropriateness of the current candidate box for processing by that expert. This vector will be used for subsequent dynamic and adaptive "soft routing" processing of the candidate boxes, that is, assigning processing weights to each expert network based on the score vector, realizing a weighted and parallel processing method, rather than forcibly assigning them to a single expert.

[0041] In one possible embodiment, a sparsity-aware gating mechanism is constructed, and based on the sparsity-aware gating mechanism and the number of original point clouds within the initial 3D candidate bounding boxes, a sparsity score vector is calculated, including: Calculate the initial 3D candidate box Each candidate box Number of raw point clouds contained inside , as a metric for measuring the sparsity of point clouds, Specifically, it is expressed as follows: ; set up A sparsity expert branch Each sparsity expert branch is an independent neural network module used to specialize the processing of candidate boxes within its corresponding sparsity range. Gaussian kernel-based scoring function The original point cloud quantity Mapped to a Sparsity score vector of dimension , Candidate boxes With the A sparsity expert branch The relevance score; Specifically, this can be expressed by the following formula: ; in, It is the first The point range mainly overseen by each expert. It is the lower bound of the interval. It is the upper bound of the interval. Center of the interval Controlling the rate of score decay, the number of points in the point range specifically refers to the number of original LiDAR points contained within the candidate box. This value is the core metric for measuring the sparsity of candidate boxes.

[0042] Step 104: Construct a hierarchical expert network containing multiple sparsity expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparsity expert branches.

[0043] For example, the initial 3D candidate boxes are fed into a hierarchical expert network. Here, the candidate box features refer to the feature vectors corresponding to the spatial location of the candidate box, extracted from the feature map generated by the backbone network through methods such as region-of-interest pooling. This network first consists of multiple sparsity expert branches, and within each sparsity expert, it is further divided into three attribute expert sub-branches specifically responsible for optimizing the center, size, and orientation. This achieves hierarchical and refined adjustment of the candidate boxes in both the sample and task dimensions. The core purpose of this adjustment is to systematically address the two core challenges faced in the refinement stage of 3D object detection through structured division of labor. In the sample dimension, by establishing different sparsity expert branches, specialized processing is achieved for samples with different point cloud densities. For example, candidate boxes containing a small number of point clouds (such as distant or occluded targets) differ greatly in data characteristics from candidate boxes containing a large number of point clouds (such as nearby targets). Processing them separately by specialized expert networks can significantly improve their respective optimization accuracy. In the task dimension, within each sparsity expert, attribute expert sub-branches (center, size, orientation) are used to resolve feature conflicts between different regression tasks. Since the geometric features required to optimize position, size, and orientation are not identical, decoupling them allows each sub-branch to focus on learning the features most relevant to its own task, avoiding performance compromises in multi-task learning by a single network. The final output of this hierarchical expert network is a set of finely tuned parameters, specifically including classification confidence and bounding box regression residuals. Its internal processing flow is as follows: the input candidate box features are fed in parallel into all sparse expert branches. Within each sparse expert branch, the features are further fed in parallel into three attribute expert sub-branches: center, size, and orientation. Each sub-branch consists of an independent multilayer perceptron and independently predicts the regression residuals and classification confidence it is responsible for.

[0044] In one possible embodiment, a hierarchical expert network containing multiple sparse expert branches is constructed, and the initial 3D candidate boxes are input into the hierarchical expert network to output the prediction results of all sparse expert branches, including: Each sparsity expert branch It is divided into three independent attribute expert sub-branches, including: the center point optimization branch. Size optimization branch and Orientation Optimization Branch ; For candidate boxes By utilizing shared region of interest feature extractors (e.g., RoI-grid pooling), the corresponding features are extracted from the backbone network feature map.

[0045] In each sparsity expert branch Internally, features Input into the center point optimization branch respectively Size optimization branch and Orientation Optimization Branch In this process, each optimization branch learns the optimization of a specific attribute through its internal multilayer perceptron, thereby outputting the prediction results of all sparse expert branches. The prediction results include classification confidence and central residuals, specifically including: Center point optimization branches Output classification confidence score and central residual ; Size optimization branch Output classification confidence score and dimensional residuals ; Towards Optimized Branches Output classification confidence score and central residual .

[0046] Step 105: Based on the sparsity score vector, the prediction results of all sparsity expert branches are weighted and fused to generate the final 3D bounding box.

[0047] For example, the prediction results of each attribute branch within each sparsity expert are first aggregated. Then, the sparsity score calculated in S3 is used as the weight to perform weighted fusion of the optimization results of all sparsity experts. By combining the opinions of all sparsity experts, a single, high-precision final 3D bounding box is generated.

[0048] In one possible implementation, the prediction results of all sparse expert branches are weighted and fused based on the sparse score vector to generate the final 3D bounding box, including: For each sparsity expert First, the prediction results of its three attribute expert sub-branches are aggregated. The final classification confidence score is obtained by averaging, and the final regression vector is obtained by concatenation, specifically including: ; ; in, Indicates the final classification confidence level. This represents the final regression vector; thus, each sparsity expert is obtained. For candidate boxes Prediction results ; Using sparsity score vectors Normalization is performed to obtain the fusion weights. Specifically, it includes: ; in, This represents a small constant used to maintain numerical stability; Based on fusion weight The predictions from all sparsity expert branches are weighted to obtain the final 3D bounding box. Specifically, it includes: .

[0049] Step 106: During the model training phase, the loss of each sparsity expert branch is weighted based on the score vector to obtain the total loss, and the network parameters are updated through backpropagation.

[0050] For example, during the model training phase, the loss function of each expert branch is weighted using sparsity scores. This weighted supervision signal can guide different experts to focus on learning data within their corresponding sparsity range, and update the parameters of the entire network end-to-end through the standard gradient backpropagation algorithm, thereby achieving expert-specific learning.

[0051] In one possible implementation, during the model training phase, the loss of each sparse expert branch is weighted based on the score vector, and the network parameters are updated through backpropagation, including: Define the standard optimization loss function The optimized loss function includes classification loss (e.g., Focal Loss) and regression loss (e.g., Smooth L1 Loss) to measure the... The sparsity expert branch for the first Prediction results of candidate boxes With real labels The differences between them; During model training, sparsity scores are utilized. As weights, the losses of each sparse expert branch are weighted to obtain the total loss of each sparse expert branch; where, the th The total loss of each sparsity expert branch The calculation method specifically includes: ; The total loss is determined by summing the losses of all sparsity expert branches. Specifically, it includes: ; Total loss The learnable parameters in all expert branches and the backbone network are updated using the backpropagation algorithm.

[0052] This application proposes a refined 3D object detection method based on a multi-expert model. First, raw point cloud data is acquired and its features are extracted to obtain point cloud features. Then, initial 3D candidate boxes are generated based on these features. A sparsity score vector is calculated based on a sparsity-aware gating mechanism and the number of raw points within the initial 3D candidate boxes. A hierarchical expert network containing multiple sparsity expert branches is constructed, and the initial 3D candidate boxes are input into the hierarchical expert network to output the prediction results of all sparsity expert branches. Based on the sparsity score vector, the prediction results of all sparsity expert branches are weighted and fused to generate the final 3D bounding box. During model training, the loss of each sparsity expert branch is weighted based on the sparsity score vector, and the network parameters are updated through backpropagation. This provides independent optimization strategies for learning conflicts of different attributes and sample sparsity issues, thereby improving the accuracy and stability of the detection results while maintaining computational efficiency.

[0053] For example, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating the implementation of a multi-expert model-based method for refining 3D target detection, as provided in one embodiment of this application. Figure 2 The process includes inputting point clouds and extracting features; generating initial 3D candidate boxes; refining and optimizing the candidate boxes through a hierarchical hybrid expert network; and finally outputting high-precision 3D detection results.

[0054] The following section uses vehicle detection in an autonomous driving scenario (based on the KITTI dataset) to illustrate the refined 3D object detection method based on a multi-expert model provided in this application.

[0055] First, the input point cloud is voxelized and features are extracted. For example... Figure 3 As shown, the system receives a frame of raw point cloud data obtained by vehicle-mounted LiDAR scanning. The point cloud data is first voxelized and then fed into a 3D sparse convolution-based backbone network for processing to extract deep geometric and semantic features of the point cloud.

[0056] Second, a candidate region generation network (RPN) is used to generate candidate boxes. The first-stage RPN uses features extracted by the backbone network to generate a series of coarse 3D candidate boxes. For example, for a car that is far away in the scene, the RPN may generate a preliminary bounding box, but the position, size and orientation of the box may have significant errors, and due to the distance, the box contains only a small number of point cloud points.

[0057] Third, sparsity-aware gating is used to calculate coefficient scores. For the candidate boxes mentioned above, the sparsity-aware gating mechanism is first activated, such as... Figure 4 As shown, the system calculates the number of point clouds contained within the candidate bounding box, assuming it to be 45 points. This number of points is used as an input signal and fed into a scoring function. Assume we have two sparsity experts: Expert 1 handles candidate boxes with fewer points (e.g., 0-100), and Expert 2 handles candidate boxes with more points (>100). In this case, the scoring function will generate a high sparsity score (e.g., 0.95) for Expert 1 and a low score (e.g., 0.05) for Expert 2. This score vector will be used for subsequent weighted fusion.

[0058] Fourth, the candidate box features are fed into a hierarchical expert network. In this network, the candidate box features are fed in parallel into all sparse expert branches (expert 1 and expert 2 in this example) for processing. Within each sparse expert branch, such as... Figure 5 As shown, the optimization task is further decoupled into three independent attribute expert sub-branches: center point optimization branch, size optimization branch, and orientation optimization branch. Each sub-branch consists of an independent network layer, which performs residual prediction on the center coordinates, length, width, height, and yaw angle of the candidate box, respectively.

[0059] Fifth, the prediction results of each expert branch are aggregated. During the inference phase, the prediction results of each sparse expert's three attribute sub-branches are first integrated to obtain a complete optimization suggestion for the candidate box. Then, the system uses the sparsity scores calculated in step (3) (0.95 for expert 1 and 0.05 for expert 2) as weights to perform a weighted average of the optimization suggestions from the two sparse experts. Since the weight of expert 1 is much higher than that of expert 2, its optimization result dominates the final decision, thus achieving precise adjustment of the sparse point cloud candidate box.

[0060] Sixth, during training, the loss of each expert is weighted by sparsity scores, and parameters are updated end-to-end through backpropagation, allowing experts to focus on data within a specific sparsity range and achieving specialized learning. The final output is an optimized 3D bounding box. Comparison shows that the method in this application can significantly improve detection accuracy. Figure 6The results of the traditional single-head optimization method for detecting this sparse target (dark box) show significant deviations between its predicted size and location and the ground truth (light box). Figure 7 The results after optimization by the method of this invention are shown. The final output bounding box (dark box) is highly consistent with the ground truth (light box), which proves that this application effectively solves the challenges brought about by data heterogeneity through expert specialization and achieves more robust and accurate detection.

[0061] Overall, compared with existing two-stage 3D detection methods that use a single optimization head, this invention has the following advantages: First, a hierarchical expert specialization mechanism: This paper proposes a hierarchical hybrid expert (MoE) architecture for the first time, which decomposes the 3D object optimization problem into two orthogonal dimensions: "sample" and "task". The method first uses sparsity experts to deal with the differences in point cloud density between samples, and then uses attribute experts within each sparsity expert to resolve feature learning conflicts between tasks, forming a new "divide and conquer" refined optimization paradigm.

[0062] Second, adaptive handling of data heterogeneity: Through a sparsity-aware gating mechanism, this invention can adaptively assign points within candidate boxes to the expert network best suited for processing that type of data, based on the density of the point cloud within the candidate box. This significantly improves the model's detection accuracy for distant, small, or occluded targets (sparse point cloud) and enhances the model's robustness in complex scenarios.

[0063] Third, negative transfer suppression in multi-task learning: By establishing optimization branches for center, size, and orientation within the expert network, this invention effectively isolates the inherent conflicts between different regression tasks. Each attribute branch can learn a feature representation optimized for its specific task, avoiding the performance compromises that occur when a single network learns heterogeneous targets, thereby significantly improving the regression accuracy of each attribute of the bounding box.

[0064] Fourth, efficient modular integration: This invention is designed as a standalone, plug-and-play optimized head module that can be easily integrated into any existing two-stage 3D detection framework to improve its performance. By sharing the main feature extraction network among expert branches and separating only the final lightweight fully connected layer, this method achieves significant accuracy gains with almost no increase in inference time cost, making it highly practical.

[0065] Fifth, robust soft routing and ensemble prediction: This invention employs a "soft" routing mechanism, allowing all experts to learn from all samples, albeit with different focuses. This approach learns more generalized features than rigidly partitioning the data. During inference, by weighted fusion of the predictions from all experts, a more stable and accurate final prediction result can be obtained than any single model, effectively utilizing the collective wisdom of the experts.

[0066] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0067] Another embodiment of this application proposes a refined 3D target detection device based on a multi-expert model. The details of this refined 3D target detection device based on a multi-expert model are described below. The following content is for ease of understanding and is not essential for implementing this example. Figure 8 This is a schematic diagram of the structure of a 3D target detection refinement device based on a multi-expert model proposed in this embodiment, including: The extraction module 810 is used to extract features from the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data. The generation module 820 is used to generate initial 3D candidate boxes based on point cloud features; wherein, the initial 3D candidate boxes are used to represent the 3D bounding boxes of the potential object's position, size and orientation; The computation module 830 is used to construct a sparsity-aware gating mechanism and calculate a sparsity score vector based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes; wherein, the sparsity score vector is used to represent the suitability of the initial 3D candidate boxes with each sparsity expert branch; The computation module 830 is also used to construct a hierarchical expert network containing multiple sparse expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches; The weighting module 840 is used to weight and fuse the prediction results of all sparse expert branches based on the sparse score vector to generate the final 3D bounding box. Training module 850 is used during the model training phase to weight the loss of each sparse expert branch based on the sparse score vector to obtain the total loss, and to update the network parameters through backpropagation.

[0068] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0069] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0070] Another embodiment of this application provides an electronic device, such as Figure 9 As shown, it includes a processor 91 and a memory 92. The memory 92 stores instructions that the processor 91 can execute. When the processor 91 is configured to execute the instructions, the electronic device can implement a multi-expert model-based method for refining 3D target detection as described in the above method embodiment.

[0071] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0072] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0073] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a multi-expert model-based method for refining 3D target detection as described in the above method embodiments.

[0074] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0075] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A refined method for 3D target detection based on a multi-expert model, characterized in that, The method includes: Feature extraction is performed on the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data; Initial 3D candidate boxes are generated based on point cloud features; wherein, the initial 3D candidate boxes are 3D bounding boxes used to represent the position, size and orientation of potential objects; A sparsity-aware gating mechanism is constructed, and a sparsity score vector is calculated based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes; wherein, the sparsity score vector is used to represent the suitability of the initial 3D candidate boxes with each sparsity expert branch; Construct a hierarchical expert network containing multiple sparse expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches; Based on the sparsity score vector, the prediction results of all sparse expert branches are weighted and fused to generate the final 3D bounding box. During the model training phase, the loss of each sparse expert branch is weighted based on the sparse score vector to obtain the total loss, and the network parameters are updated through backpropagation.

2. The method according to claim 1, characterized in that, The original point cloud data is denoted as , , This represents the total number of points in a single frame of the point cloud. Representing a three-dimensional real space, the step of extracting features from the acquired raw point cloud data to obtain point cloud features of the raw point cloud data includes: For raw point cloud data Voxelization is performed to divide the point cloud space, and a 3D sparse convolutional network is used to process the non-empty voxels to extract the initial geometric features of the original point cloud data. Multiple key points are sampled from the original point cloud data, and the voxel set abstraction module is used to aggregate the voxel features around each key point to obtain point cloud features.

3. The method according to claim 2, characterized in that, The generation of initial 3D candidate boxes based on point cloud features includes: Point cloud features are input into a candidate region generation network, which then classifies and regresses the point cloud features to obtain initial 3D candidate boxes. Specifically, it is expressed as follows: ； in, This represents the total number of candidate boxes generated. Representing the A 3D candidate box, Candidate boxes The three-dimensional coordinates of the center point These are candidate boxes Length, width, and height, Candidate boxes The yaw angle is defined as its orientation angle in a top view.

4. The method according to claim 3, characterized in that, The construction of a sparsity-aware gating mechanism, and the calculation of a sparsity score vector based on the sparsity-aware gating mechanism and the number of original point clouds within the initial 3D candidate boxes, includes: Calculate the initial 3D candidate box Each candidate box Number of raw point clouds contained inside , as a metric for measuring the sparsity of point clouds, Specifically, it is expressed as follows: ; set up A sparsity expert branch Each sparsity expert branch is an independent neural network module used to specialize the processing of candidate boxes within its corresponding sparsity range. Gaussian kernel-based scoring function The original point cloud quantity Mapped to a Sparsity score vector of dimension , Candidate boxes With the A sparsity expert branch The relevance score; Specifically, this can be expressed by the following formula: ； in, It is the first The point range mainly overseen by each expert. It is the lower bound of the interval. It is the upper bound of the interval. Center of the interval Controlling the rate of score decay, the number of points in the point range specifically refers to the number of original LiDAR points contained within the candidate box. This value is the core metric for measuring the sparsity of candidate boxes.

5. The method according to claim 4, characterized in that, The process involves constructing a hierarchical expert network containing multiple sparse expert branches, and inputting the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches, including: Each sparsity expert branch It is divided into three independent attribute expert sub-branches, including: the center point optimization branch. Size optimization branch and Orientation Optimization Branch ; For candidate boxes By utilizing a shared region of interest feature extractor, the corresponding features are extracted from the backbone network feature map. In each sparsity expert branch Internally, features Input into the center point optimization branch respectively Size optimization branch and Orientation Optimization Branch In this process, each optimization branch learns the optimization of a specific attribute through its internal multilayer perceptron, thereby outputting the prediction results of all sparse expert branches. The prediction results include classification confidence and central residuals, specifically including: Center point optimization branches Output classification confidence score and central residual ; Size optimization branch Output classification confidence score and dimensional residuals ; Towards Optimized Branches Output classification confidence score and orientation residual .

6. The method according to claim 5, characterized in that, The step of weighted fusion of prediction results from all sparse expert branches based on the sparse score vector to generate the final 3D bounding box includes: For each sparsity expert First, the prediction results of its three attribute expert sub-branches are aggregated. Specifically, it includes: ； ； in, Indicates the final classification confidence level. This represents the final regression vector; thus, each sparsity expert is obtained. For candidate boxes Prediction results ; Using sparsity score vectors Normalization is performed to obtain the fusion weights. Specifically, it includes: ； in, This represents a small constant used to maintain numerical stability; Based on fusion weight The predictions from all sparsity expert branches are weighted to obtain the final 3D bounding box. Specifically, it includes: 。 7. The method according to claim 6, characterized in that, During the model training phase, the losses of each sparse expert branch are weighted based on the score vector to obtain the total loss, and the network parameters are updated through backpropagation, including: Define the standard optimization loss function The optimized loss function includes classification loss and regression loss to measure the performance of the first class. The sparsity expert branch for the first Prediction results of candidate boxes With real labels The differences between them; During model training, sparsity scores are utilized. As weights, the losses of each sparse expert branch are weighted to obtain the total loss of each sparse expert branch; where, the th The total loss of each sparsity expert branch The calculation method specifically includes: ； The total loss is determined by summing the losses of all sparsity expert branches. Specifically, it includes: ； Total loss The learnable parameters in all expert branches and the backbone network are updated using the backpropagation algorithm.

8. A refined 3D target detection device based on a multi-expert model, characterized in that, The device includes: The extraction module is used to extract features from the acquired raw point cloud data to obtain the point cloud features of the raw point cloud data. The generation module is used to generate initial 3D candidate boxes based on point cloud features; wherein, the initial 3D candidate boxes are 3D bounding boxes used to represent the position, size and orientation of potential objects; The computation module is used to construct a sparsity-aware gating mechanism and calculate a sparsity score vector based on the sparsity-aware gating mechanism and the number of original point clouds in the initial 3D candidate boxes; wherein, the sparsity score vector is used to represent the suitability of the initial 3D candidate boxes with each sparsity expert branch; The computation module is also used to construct a hierarchical expert network containing multiple sparse expert branches, and input the initial 3D candidate boxes into the hierarchical expert network to output the prediction results of all sparse expert branches; The weighting module is used to weight and fuse the prediction results of all sparse expert branches based on the sparse score vector to generate the final 3D bounding box. The training module is used during the model training phase to weight the loss of each sparse expert branch based on the sparse score vector to obtain the total loss, and then updates the network parameters through backpropagation.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a multi-expert model-based method for refining 3D target detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a refined method for three-dimensional target detection based on a multi-expert model as described in any one of claims 1 to 7.

Images