Adaptive feature fusion method, network and detection method in point cloud target detection

Abstract

The invention discloses an adaptive feature fusion method, a network and a detection method in point cloud target detection. The fusion method comprises the following steps: acquiring N1 feature maps of different levels, and performing channel alignment processing on the feature map of each level; self-adaptively generating weights corresponding to the feature maps after the channels of the second level to the N1th level are aligned, and obtaining N1-1 weights; based on N1-1 weights, sequentially fusing the feature map of the N1th hierarchy after channel alignment with the feature map of the previous hierarchy after channel alignment until fusion with the feature map of the first hierarchy after channel alignment is completed, and obtaining a fused feature map; the feature fusion can be realized more accurately, so that the robustness and the detection performance in a complex scene are improved. The invention further provides a loss function adopting gradient guide correction, and the problem that the learning effect of the model is affected due to unbalanced distribution of various target samples and unbalanced distribution of difficult samples in a data set can be solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional target detection, and particularly relates to a self-adaptive feature fusion method, a network and a detection method in point cloud target detection. BACKGROUND

[0002] Three-dimensional target detection is a core technology of automatic driving perception system, which can provide accurate environmental perception information for path planning, motion prediction and collision avoidance tasks. Since the DARPA challenge, automatic driving technology has been continuously developing, especially in the target recognition ability in complex dynamic scenes, which has made important breakthroughs, which has promoted the development of efficient, real-time and robust perception system.

[0003] Automatic driving relies on multi-sensor fusion, among which passive sensors (such as cameras) provide rich color and texture information, but are greatly affected by light; active sensors (such as lidar) provide high-precision depth information through point cloud data, which are not limited by light conditions and are more suitable for complex environments. Three-dimensional target detection based on lidar can accurately classify and locate surrounding objects, but still faces challenges such as sparsity and irregularity of point cloud data, limited feature expression ability, and target scale variation and class distribution imbalance. Current research mainly improves the adaptability of point cloud data through voxelization (such as VoxelNet) or point cloud feature extraction network (such as PointNet), and adopts feature fusion strategy to enhance the perception ability of multi-scale targets. In addition, to solve the problem of class imbalance, optimizing sampling strategy, designing adaptive loss function and other methods can effectively improve the detection accuracy of tail classes.

[0004] Overall, three-dimensional target detection technology plays a key role in improving the safety and reliability of automatic driving vehicles, especially in low light and bad weather conditions. With the continuous development of technology, it will lay a solid foundation for realizing L5 level unmanned driving.

[0005] In recent years, researchers have proposed a large number of innovative methods for 3D object detection. These methods can be categorized into three main categories: image-based methods, point cloud-based methods, and multimodal fusion-based methods. Image-based methods analyze 2D image data acquired from a camera and leverage techniques such as deep learning to detect and localize objects. However, image data often lacks depth information, which limits the performance of methods that rely solely on images for 3D object detection. To address this deficiency, point cloud-based methods have begun to gain widespread application. 3D point cloud data generated by LiDAR, in particular, provides accurate depth information, enabling more precise object detection and localization. Building on this, multimodal fusion-based methods have emerged. By combining multiple data sources, such as images and point clouds, they leverage the strengths of various sensors to improve detection accuracy and robustness. However, current feature fusion methods often exhibit poor fusion performance and fail to achieve precise feature fusion, which in turn impacts subsequent detection accuracy and robustness. Summary of the Invention

[0006] In order to solve the above-mentioned problems existing in the prior art, the present invention provides an adaptive feature fusion method, network and detection method in point cloud target detection, which can not only realize the precise fusion of features, but also alleviate the problem that the learning effect of the model is affected by the unbalanced distribution of various target samples and the unbalanced distribution of difficult and easy samples in the data set.

[0007] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] The present invention provides an adaptive feature fusion method for point cloud target detection, comprising:

[0009] Obtain N1 feature maps of different levels to be fused; N1 is a positive integer greater than or equal to 2;

[0010] Perform channel alignment on the feature maps of each level to obtain N1 channel-aligned feature maps of different levels;

[0011] Adaptively generate the weights corresponding to the feature maps after channel alignment at the 2nd to N1th levels, and obtain N1-1 weights;

[0012] Based on the N1-1 weights, starting from the channel-aligned feature map of the N1th level, the feature maps are sequentially fused with the channel-aligned feature maps of the previous level until the fusion with the channel-aligned feature map of the 1st level is completed, thereby obtaining a fused feature map.

[0013] The application further provides a self-adaptive feature fusion network in point cloud target detection, which is used for executing the steps of the self-adaptive feature fusion method in point cloud target detection.

[0014] The application further provides a point cloud target detection method, comprising:

[0015] obtaining to-be-detected point cloud data;

[0016] converting the to-be-detected point cloud data into a pseudo two-dimensional image form to obtain a to-be-detected pseudo two-dimensional image;

[0017] extracting feature maps of different levels of the to-be-detected pseudo two-dimensional image by using a backbone network;

[0018] performing fusion on the feature maps of different levels by using a self-adaptive feature fusion network through the self-adaptive feature fusion method in point cloud target detection to obtain a fused feature map;

[0019] performing target detection according to the fused feature map by using a three-dimensional target detector.

[0020] Compared with the prior art, the application has the following beneficial effects:

[0021] The self-adaptive feature fusion method and the self-adaptive feature fusion network in point cloud target detection can not only adaptively generate weights, but also perform deep interaction and information fusion between multiple layers of features, ensure that the detailed information of low-level features and the semantic information of high-level features are effectively combined, and thus can more accurately realize feature fusion. Through this adaptive fusion method, the model can fully exert the complementary advantages of multiple layers of features, thereby improving the robustness and detection performance in complex scenes.

[0022] The point cloud target detection method provided by the application uses a three-dimensional target detector trained by using a gradient-guided correction loss function. The loss function reduces the weights of simple negative samples and extremely difficult samples in the training process, while increasing the weight of normal difficult samples, so that the model can focus more on learning samples with higher value, thereby improving the detection performance of the model. In addition, the loss function makes rare and difficult samples contribute more than common difficult samples in loss calculation, enhances the learning ability of the model for a small number of sample categories, improves the robustness and generalization performance of target detection, and thus alleviates the problem that the learning effect of the model is affected by the unbalanced distribution of various target samples and the unbalanced distribution of difficult and easy samples in the data set.

[0023] The application will be described in further detail below with reference to the drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flowchart of an adaptive feature fusion method in point cloud target detection provided by an embodiment of the application;

[0025] Figure 2 is a structural diagram of an adaptive feature fusion network in point cloud target detection provided by an embodiment of the application;

[0026] Figures 3A-3C is a visual comparison diagram of detection results of LS-LiDAR before and after three groups of models adopt the adaptive feature fusion structure provided by the application;

[0027] Figures 4A-4C is a visual comparison diagram of detection of LS-LiDAR before and after a class balancing module provided by the application is adopted. DETAILED DESCRIPTION

[0028] The application will be described in further detail below with reference to the drawings and specific embodiments, but the embodiments of the application are not limited thereto.

[0029] Feature layers at different scales have their own unique characteristics in capturing information. Shallow feature maps primarily capture detailed image information, such as edges and textures, while deep feature maps are capable of learning more abstract, high-level semantic information, such as object category and location. However, when using these different-scale feature layers individually for object detection, their feature information does not fully overlap, and the expressive power of each layer varies, resulting in poor multi-scale object detection performance. To overcome this problem, a feature fusion structure is often added after the backbone network. The goal of feature fusion is to effectively combine feature maps of different scales to achieve a more comprehensive feature representation, thereby improving detection accuracy. Through feature fusion, the model can fully leverage the complementarity of shallow and deep features to better capture detailed and semantic information about the object. FPN is a method widely used in feature fusion modules. Its core concept is to construct a top-down feature pyramid structure, allowing features of different scales to be fused layer by layer, thereby enhancing the model's multi-scale detection capabilities. Specifically, FPN performs upsampling on high-level feature maps and then fuses them with lower-level feature maps. This layer-by-layer fusion approach gradually enriches the feature information at each layer, enabling the model to better handle objects of varying sizes. However, FPN also has certain limitations in practical applications. Because feature fusion is performed top-down, noise and inaccurate information in the top feature layers are propagated layer by layer to the lower feature maps, causing feature contamination. Especially when performing object detection in lower-level feature maps, noise in the top feature layers can interfere with detailed information in the lower layers, resulting in a decrease in detection accuracy. Therefore, how to effectively fuse features from different layers while avoiding the negative impact of noise has become a key research direction in current feature fusion technology. In this context, many new feature fusion structures have emerged, such as feature fusion methods combined with attention mechanisms. Channel attention mechanisms and spatial attention mechanisms can suppress useless features through weighted methods while enhancing important features, preventing excessive noise propagation and further improving object detection performance. The channel attention mechanism (such as SENet) enhances important features by generating weights for each channel. Its weight generation process mainly relies on the modeling of global information such as global average pooling (GAP). However, this mechanism ignores the local relationship between features in spatial positions and has difficulty capturing fine-grained spatial features. The spatial attention mechanism (such as CBAM) enhances important spatial regions in the feature map by generating weights for each spatial position. It usually uses GAP and maximum pooling operations to extract global and local information of the feature map in the spatial dimension, and learns the dependencies in the spatial dimension through convolutional layers. This mechanism can effectively highlight the key areas where the target is located, such as the edges or salient areas of the object, thereby improving the model's ability to model spatial context and complementing the channel attention mechanism.However, the calculation process of spatial attention may introduce additional computational overhead, especially when the feature map size is large, thus posing certain challenges for real-time applications. Whether it is a channel attention mechanism or a spatial attention mechanism, it is usually limited to a single network layer, lacking modeling of interactions between different scale feature maps, thus having certain limitations in multi-scale information fusion and processing complex scenes. Therefore, in the three-dimensional target detection task in the automatic driving scene, the problem of difficult extraction of data features caused by unsatisfactory feature fusion effect is faced. In view of this problem, the present application proposes a kind of adaptive fusion method and network based on multi-layer feature perception to extract multi-scale features, so as to realize accurate feature fusion.

[0030] In the three-dimensional target detection task in the automatic driving scene, in addition to the above-mentioned problem of difficult extraction of data characteristics, there are also problems of class imbalance, difficult sample imbalance and positive and negative sample imbalance in the training process of the detection model. Specifically, (1) target class imbalance. In the target detection task, the frequency of different classes of targets is quite different. This imbalance will cause the head class to dominate in the training data, causing the model to learn more towards these common classes during the learning process. Although the model can accurately detect the head class, the detection accuracy of the tail class is insufficient, which may result in a high rate of missed detection or false detection. Especially in automatic driving, security monitoring and other scenarios that require the recognition of multiple targets, the missed detection of tail class targets may have serious consequences. In addition, due to the scarcity of tail class samples in the training data, the model cannot access enough tail class samples during training, which makes it difficult for the model to learn effective features from them. Even if the model has a good optimization goal for the head class, the training of the tail class is still a bottleneck, making it difficult to achieve global optimization. In the long-term training process, the model may overfit the features of the head class and fall into a local optimum.(2) Difficulty sample imbalance. In target detection, samples can be divided into simple samples and difficult samples. Simple samples usually refer to negative samples that are easy for the model to identify, or targets with typical and easily distinguishable shapes, sizes, and features in the foreground. These simple samples have features that match the model's existing weights, and the model can easily make correct predictions. Difficult samples are those with complex backgrounds, fuzzy edges, occlusions, deformations, scale changes or extreme poses, or a small number of classes that are sparsely distributed in the data. These difficult samples differ greatly from the features learned by the model and often result in large classification errors or regression errors, but they help the model learn more discriminative features. Some difficult samples may become extreme difficult samples or outliers due to noise, label errors or other abnormal factors. Overemphasis on these extreme difficult samples or excessive adjustment of model parameters will cause the model to deviate from the normal learning path, and may even lead to unstable convergence of the model in the later training period. During network training, there are usually more simple samples than difficult samples, which causes most of the gradient updates of the model to come from these simple samples, making it difficult for the model to learn effective difficult sample features and thus reducing its adaptability to complex scenarios.(3) Positive and negative sample imbalance. In single-stage target detection algorithms, the number of negative samples is much larger than that of positive samples, and most negative samples are simple samples. Simple negative samples are usually easily predicted by the classifier, and their corresponding gradients are small, contributing relatively little to the model's learning.However, due to the large number of these simple negative samples, their gradients will dominate the total gradient calculation, causing the model to pay more attention to these simple negative samples during training, ignoring positive samples or more valuable negative samples, thereby affecting the learning effect of the model. In order to solve these problems, the present application also proposes a Gradient-Guided Correction Loss (GGCL) to solve these problems.

[0031] Figure 1 is a flowchart of an adaptive feature fusion method in point cloud target detection provided by an embodiment of the present application, as shown in Figure 1 , the method comprises:

[0032] S101, obtaining N1 feature maps of different levels to be fused.

[0033] S102, respectively performing channel alignment processing on the feature maps of each level to obtain N1 channel-aligned feature maps of different levels.

[0034] S103, adaptively generating weights corresponding to the channel-aligned feature maps of the 2nd to N1th levels respectively to obtain N1-1 weights.

[0035] S104, based on the N1-1 weights, starting from the channel-aligned feature map of the N1th level, sequentially fusing with the channel-aligned feature map of the previous level until the fusion with the channel-aligned feature map of the 1st level is completed, to obtain a fused feature map.

[0036] The adaptive fusion method proposed by the present application can dynamically perceive the relative importance of different feature layers and adaptively adjust the fusion weights between feature maps of each layer according to task requirements, thereby improving the discrimination ability of feature expression. On the one hand, this structure adaptively adjusts the fusion weights of background and foreground features, strengthens the suppression ability of background points and the attention degree of foreground points, thereby reducing the false detection problem caused by noise propagation. On the other hand, with the fusion of multi-scale features, the structure further enhances the ability to capture sparse and incomplete features. This enables the network to better handle targets in low resolution and occlusion situations, significantly reducing the risk of missing weak feature targets and occluded targets, and effectively improving the robustness and detection accuracy of the model in complex environments.

[0037] Specifically, the above S103 is implemented through steps S1031-S1034:

[0038] S1031, for the channel-aligned feature map of each level in the 2nd to N1th levels, extracting local features of the channel-aligned feature map of the level and capturing spatial information to obtain initial features.

[0039] S1032: Standardize and activate the initial features to obtain activated features.

[0040] S1033. Perform spatial dimension reduction, channel adjustment, and channel relationship capture on the activated features to obtain processed features.

[0041] S1034. Compress the values ​​of the processed features to the range of [0, 1] to obtain the weights corresponding to the feature maps after channel alignment at this level.

[0042] Specifically, the above S104 is implemented through steps S1041 to S1042:

[0043] S1041, feature map after aligning the channels of the N1th level As the fusion feature of the N1th level

[0044] S1042, feature map after channel alignment according to the N1th level Corresponding weight Fusion features of the N1th level The feature map after weighted adjustment and alignment with the channel of the N1-1th level Fuse and then integrate the fused features to obtain the N1-1th level fusion features After that, the feature map after channel alignment at level N1-1 is Corresponding weight Fusion features of the N1-1th level The feature map after weighted adjustment and alignment with the channel of the N1-2th level Fuse and then integrate the fused features to obtain the N1-2th level fusion features The fusion is iterated in this way until the fusion feature F1 of the first level is obtained, and the fusion feature F1 of the first level is used as the fusion feature map.

[0045] For example, the fusion feature F of the jth level j Expressed as:

[0046]

[0047] The present invention also provides an adaptive feature fusion network (i.e., an adaptive feature fusion structure) for point cloud target detection, which is used to execute the above-mentioned adaptive feature fusion method.

[0048] In some embodiments, the adaptive feature fusion network includes: N1 channel alignment modules, 1 adaptive weight module, N1-1 weight adjustment modules, N1-1 fusion modules and N1-1 integration modules. Exemplarily, each channel alignment module is a 1×1 convolution layer, and each integration module is a 3×3 convolution layer. Exemplarily, the adaptive weight module includes: a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, a global average pooling layer, a 1×1 convolution layer and a Sigmoid normalization function; wherein, the inputs and outputs of the 3×3 convolution layer, the normalization layer, the ReLU activation function, the global average pooling layer, the 1×1 convolution layer and the Sigmoid normalization function are connected in sequence, and the input of the 3×3 convolution layer serves as the input of the adaptive weight module, and the output of the Sigmoid normalization function serves as the output of the adaptive weight module.

[0049] For example, Figure 2 A structural diagram of the adaptive feature fusion network in point cloud object detection. Figure 2 In the figure, N1=3, the yellow, purple and green cubes represent the feature maps of three different levels, and the yellow, purple and green cubes represent the feature maps of the first level, the second level and the third level, namely the L1, L2 and L3 levels respectively. The purple and green lines respectively represent the input of the feature maps of the L2 and L3 levels into the adaptive weight module for processing and generating the corresponding weights. Figure 2As shown in Figure 1, the feature fusion process can be divided into four main steps: channel alignment, adaptive weight generation, feature fusion, and feature integration. (1) Channel alignment. Since feature maps at different levels may contain different numbers of channels, direct fusion may lead to information mismatch or loss. Therefore, the input feature map is first aligned through 1×1 convolution. This not only ensures information alignment in subsequent fusion operations, but also improves the effectiveness and expressiveness of feature fusion without adding additional computational burden. (2) Adaptive weight generation. Except for the lowest L1 level feature map, all other feature maps are processed by the adaptive weight module to generate weight coefficients adapted to the current feature layer. The adaptive weight module consists of multiple layers, including 3×3 convolution layer, batch normalization (BN) layer, ReLU activation function, global average pooling layer, 1×1 convolution layer, and Sigmoid normalization function. First, the 3×3 convolution layer is used to extract local features, capture spatial information, and provide rich feature representation for subsequent weighting operations. Then, the BN layer and ReLU activation function further process the convolution results. The BN layer reduces the internal covariate shift by standardizing the input of each layer, thereby accelerating the training process and improving the stability of the network. The ReLU activation function introduces nonlinear characteristics, allowing the network to better capture the complex patterns in the input data. The global average pooling layer performs spatial dimensionality reduction on the feature map, thereby extracting the global information of each channel and retaining the contextual information between channels. Subsequently, the 1×1 convolution layer is used to adjust the number of feature channels and capture the relationship between different channels by learning the weights in the convolution kernel. Finally, the Sigmoid function compresses the output value to the range of [0, 1] and generates a channel weighting coefficient so that the contribution of different feature layers can be flexibly adjusted in the subsequent feature weighting process. (3) Feature fusion. First, the feature map of the previous layer is weighted and adjusted according to the weight coefficient generated by the adaptive weight module, so that the features of specific channels or spatial regions are highlighted, thereby enhancing the network's attention to key features. Then, the weighted feature map is resized using the nearest interpolation method to make it consistent with the size of the feature map to be fused in the next layer. Finally, the feature map that has completed weighting and resizing is fused with the feature map of the next layer. The present invention adopts a pixel-by-pixel addition method. The purpose of this step is to fuse feature information of different scales and make full use of the diversity and complementarity of feature maps at each layer, thereby improving the network's feature expression capabilities at different levels. The feature maps processed by the feature fusion module will provide richer contextual information for subsequent tasks, enhancing the performance and robustness of the model. (4) Feature integration. After the feature fusion step, the feature maps obtained have combined information from different levels, but the fusion effect of this information still needs to be further optimized.To enhance the expression ability of the fused feature map and extract more effective features, a 3x3 convolution is used to further integrate the fused feature map. This process not only helps to improve the expression ability of the feature map, but also enables more accurate fusion of information at different levels while maintaining computational efficiency, so that the fused feature map can better adapt to the needs of downstream tasks and improve the overall performance of the model. Figure 2 As shown in FIG. 1, F3 is the third level of fused features, F2 is the second level of fused features, F1 is the first level of fused features, and F1 is the final fused features. As an example, as shown in FIG. 2, the scales of the feature maps at L1, L2 and L3 levels can be Figure 2 As shown in FIG. 1, F3 is the third level of fused features, F2 is the second level of fused features, F1 is the first level of fused features, and F1 is the final fused features. As an example, as shown in FIG. 2, the scales of the feature maps at L1, L2 and L3 levels can be As shown in FIG. 1, F3 is the third level of fused features, F2 is the second level of fused features, F1 is the first level of fused features, and F1 is the final fused features. As an example, as shown in FIG. 2, the scales of the feature maps at L1, L2 and L3 levels can be

[0050] The adaptive fusion method and network based on multi-level feature perception proposed by the present application can effectively solve the challenges in the current feature fusion process. Compared with the traditional feature fusion method based on fixed weight or simple attention mechanism, the adaptive fusion method and network of the present application can more finely adjust the weight relationship between different levels of features by introducing a multi-level feature perception mechanism. This mechanism dynamically learns the expression importance of different levels of features, enabling the network to automatically selectively strengthen meaningful features according to the level difference and context information of each layer of feature map, thereby achieving more accurate feature fusion. In addition, the feature fusion process of FPN is fixed and cannot adaptively adjust the weight of feature fusion according to the specific task or data input, which makes it difficult to fully capture the fine-grained spatial information and scale difference of each layer of feature in complex scenes. The present application not only adaptively generates weights, but also enables deep interaction and information fusion between multiple levels of features, ensuring that the detailed information of low-level features and the semantic information of high-level features are effectively combined. Through this adaptive fusion method, the model can fully utilize the complementary advantages of multi-level features, improving the robustness and detection performance in complex scenes.

[0051] The present application also provides a point cloud object detection method, comprising:

[0052] S201, obtaining point cloud data to be detected.

[0053] S202, converting the point cloud data to be detected into a pseudo two-dimensional image form to obtain a pseudo two-dimensional image to be detected.

[0054] As an example, the point cloud data can be converted into a pseudo two-dimensional image by performing point cloud voxelization processing on the point cloud data to be detected.

[0055] S203, extracting feature maps of different levels of the pseudo two-dimensional image to be detected using a backbone network.

[0056] Exemplarily, the backbone network can be ResNet-50, and the feature maps output by the first three stages of ResNet-50 are taken as the feature maps of different levels which need to be fused subsequently. Exemplarily, the scales of the feature maps output by the first three stages extracted by ResNet-50 are (64x248x216), (128x124x108) and (256x62x54) respectively.

[0057] S204, the adaptive feature fusion network is adopted to fuse the feature maps of different levels by performing the adaptive feature fusion method to obtain a fused feature map.

[0058] S205, a three-dimensional target detector is adopted to perform target detection according to the fused feature map.

[0059] Exemplarily, the three-dimensional target detector is a single-stage three-dimensional target detector, that is, a classifier. Exemplarily, the single-stage three-dimensional target detector is trained by using a gradient-guided correction loss function, wherein the gradient-guided correction loss function is obtained by introducing a gradient coordination mechanism into a cross-entropy loss.

[0060] Exemplarily, the expression of the gradient-guided correction loss function is as follows:

[0061]

[0062] wherein N2 is the total number of training samples, G i is the gradient norm of the i-th training sample, D(G) is the gradient density function of the training sample, a is a parameter for balancing positive and negative samples, p is the prediction probability of the candidate box of the i-th training sample by the model, and a is a focus factor representing the basic behavior of the classifier in the sample balance scene, and n is a hyperparameter.

[0063] The design principle of the gradient-guided correction loss function proposed in the application is described as follows.

[0064] In designing the gradient-guided correction loss function, the cross-entropy (CE) loss in a binary classification task is taken as the basis, and the expression of the CE loss is as follows: wherein p [0, 1] is the prediction probability of the candidate box by the model, and y [0, 1] is the true value label of a certain class. In addition, the Focal Loss is a loss function for solving the sample imbalance problem, two modulation parameters are introduced into the cross-entropy loss, which balances the positive and negative samples and the difficult and easy samples to a certain extent. The expression of the Focal Loss is as follows: FL L γ = FL (p) = -a (1-p) log (p), wherein a is a parameter for balancing positive and negative samples, and (1-p)γ The contribution of simple samples to the loss is reduced by predicting the score p and the focusing parameter γ, and attention is paid to the learning of difficult samples. A large number of negative samples are easy to classify, while positive samples are usually difficult to classify. The imbalance between positive and negative samples can be considered as the imbalance between difficult and easy samples. The focusing parameter γ determines the impact of FocalLoss. A larger γ will greatly reduce the loss contribution of most negative samples. Therefore, the higher the imbalance between positive and negative samples, the greater the expected value of γ. Since different degrees of positive and negative sample imbalance require different γ values, the present invention designs a class weight factor (Class Weight Factor, CWF) based on Focal Loss, and adaptively sets the γ value according to the positive and negative imbalance. The classification loss of the kth category is expressed as: Among them, γ k Represents the focus parameter of the kth class, which is similar to the γ in Focal Loss. Focusing factor γ k is decoupled into two components, namely the category-independent parameter γ a and category-related parameters That is, there is a formula: Among them, the parameter γ a Represents the focusing factor that controls the basic behavior of the classifier in the sample balance scenario, parameter It is a variable parameter related to the imbalance degree of the k-th class sample, which is responsible for regulating the concentration of the classifier learning on the class imbalance problem.

[0065] The present invention adopts gradient guidance parameter Let x be the output of the model, then p = softmax(x), then the gradient of the loss function with respect to the model output is: The norm of the gradient is defined as The value of G reflects the difficulty of a sample, and also the imbalance between positive and negative samples, indicating the impact of the sample on the global gradient. This paper uses the gradient norm G to guide the parameters The choice is: Among them, the hyperparameter n is the factor that determines L CWF Scaling factor for the upper bound of the mid-gradient guidance.

[0066] In a single-stage detection model, the distribution of G is dominated by simple negative samples, which have a significant impact on the global gradient. Furthermore, the convergence model cannot handle extremely difficult samples, which can be considered outliers. To address the problem of inconsistency in the distribution of G in a single-stage detector, this paper introduces a method for harmonizing gradient density. The gradient density function of the training sample is shown below: Among them, N2 is the total number of training samples, G i is the gradient norm of the i-th training sample, and G represents the gradient norm of all training samples.ε and l ε The calculation formulas of and l are respectively: The gradient density of G represents the number of samples in the area centered on G with a length of ε, and is used to describe the gate function, is the center of the gate, is the gate width. By normalizing the effective length of the area, the gradient density correction parameter (GDCP) is defined as: The gradient density correction parameter is embedded into the classification loss L CWF The final gradient-guided modified loss function is:

[0067] The effectiveness of the present application is further illustrated by experiments below.

[0068] Experiment 1

[0069] In the target detection task, it is difficult to meet the needs of complex autonomous driving scenes by relying only on the feature maps extracted by the backbone network for target representation. In order to make full use of features of different scales, this time, research and experiments are conducted on how to effectively fuse multi-scale features. The traditional multi-scale feature fusion method has significant defects in practical application. The top-down feature fusion method is easy to introduce noise information in high-level features and propagate downward in the feature fusion process. This noise transmission can weaken the representation ability of bottom-level features, thereby negatively affecting target detection. In view of this problem, the present application proposes an adaptive fusion structure based on multi-layer feature perception to achieve more efficient feature fusion processing. This method gives adaptive weights to features of different levels, dynamically adjusts the fusion process of features, effectively suppresses the problem of high-level noise transmission to the bottom layer, and thus improves the expression ability and detection performance of multi-scale features.

[0070] Figures 3A-3C The visualization comparison chart of the detection results of the LS-LiDAR of the three groups of models before and after using the adaptive feature fusion structure proposed by the present application is shown. Figures 3A-3C In the a group, b group and c group shown in the figure, the left figure of each group of figures represents the detection result figure without introducing the adaptive feature fusion module, and the right figure represents the detection result figure after introducing the adaptive feature fusion module. In order to facilitate analysis, different colors are used to render the points corresponding to different target categories, wherein green represents a small car, blue represents a non-motor vehicle, yellow represents a large vehicle, pink represents a pedestrian, orange represents an obstacle, and black represents the background. From Figure 3A and Figure 3BAs can be observed from the a group and b group comparison diagrams shown, in the detection result diagram on the left side without introducing the adaptive feature fusion module, there is a phenomenon of false detection of pedestrians in the upper right corner. This shows that the traditional top-down feature fusion method cannot effectively suppress the noise accumulation between different feature layers when processing multi-scale information, resulting in background points being wrongly predicted as pedestrians, which exposes its limitations in noise robustness. Moreover, as can be observed from the a group, b group and c group comparison diagrams shown in Figure 3A 、 Figure 3B and Figure 3C As can be observed from the a group, b group and c group comparison diagrams shown, in the detection result diagram on the left side without using the adaptive feature fusion module, there are various missing detection phenomena, including sparse targets and targets with incomplete features due to occlusion. This shows that the traditional method has the problem of insufficient ability to capture key information when dealing with sparse point cloud distribution and occlusion interference. In contrast, the adaptive fusion structure based on multi-layer feature perception proposed by the present application can dynamically perceive the relative importance of different feature layers and adaptively adjust the fusion weights between feature maps of each layer according to task requirements, thereby improving the discrimination ability of feature expression. On the one hand, the structure strengthens the suppression ability of background points and the attention degree of foreground points by adaptively adjusting the fusion weights of background and foreground features, thereby reducing the false detection problem caused by noise propagation. On the other hand, with the fusion of multi-scale features, the structure further enhances the ability to capture sparse and incomplete features. This enables the network to better handle targets in low resolution and occlusion conditions, significantly reducing the missing detection risk of weak feature targets and occluded targets, and effectively improving the robustness and detection accuracy of the model in complex environments.

[0071] In addition, the adaptive fusion structure based on multi-layer feature perception proposed by the present application can help the network learn the features of various targets more efficiently, thereby improving the detection accuracy of multiple targets. Table 1 compares the detection accuracy of the model on LS-LiDAR under different feature fusion methods.

[0072] Table 1 Detection accuracy comparison of LS-LiDAR under different feature fusion methods

[0073] Algorithm Sedan Non-motor vehicle Large vehicle Pedestrian Obstacle Average accuracy Baseline 76.09 62.73 68.81 26.98 30.84 53.09 Baseline+SENet 79.12 62.29 74.92 29.05 33.12 55.70 Baseline+CBAM 80.67 63.87 74.83 30.93 36.97 57.45 Baseline+Ours 81.02 65.91 75.21 36.02 41.46 59.92

[0074] It can be observed that the overall detection accuracy is improved when introducing the attention mechanism in the feature layer. This is because the attention mechanism optimizes the feature by weighting the channel dimension or spatial dimension of the feature map, highlighting the more critical feature area in the target detection task, thereby enhancing the feature expression ability and improving the attention of the network to important information. However, the traditional attention mechanism has certain limitations, which mainly focus on extracting information in a single feature map, and pays insufficient attention to the fusion problem between feature layers. To solve this problem, the adaptive fusion structure based on multi-layer perception proposed in the present application can realize effective information flow between multi-scale feature layers through adaptive weight adjustment, thereby realizing the sharing of key features from top to bottom. This method not only can significantly reduce noise interference, but also can fully exploit the features of the target, further improving the detection accuracy. Especially in the detection task of small and weak targets at a long distance, the fusion structure proposed in the present application shows obvious advantages. Because the point cloud features of such targets are sparse and distributed discretely, it is often difficult for traditional methods to accurately capture their feature information, resulting in missed detection. The feature fusion structure proposed in the present application dynamically focuses on the key feature information of small and weak targets, and preferentially learns and optimizes the feature area with greater detection value. This not only effectively improves the detection ability of the network for small and weak targets, but also significantly reduces the missed detection rate, ultimately achieving overall improvement in the detection performance of the model.

[0075] Table 2 shows the detection accuracy of different algorithms for LS-LiDAR. The experiments used are classical algorithms in the field of three-dimensional target detection based on point cloud. According to the specific direction classification, it includes Second, VoxelNet, PointPillars, CenterPoint, Part-A2, TANet, SE-SSD, BtcDet based on voxel; PointRCNN, 3DSSD based on points; PV-RCNN, BADet based on the combination of points and voxels.

[0076] Table 2 shows the detection accuracy of different algorithms for LS-LiDAR. The experiments used are classical algorithms in the field of three-dimensional target detection based on point cloud. According to the specific direction classification, it includes Second, VoxelNet, PointPillars, CenterPoint, Part-A2, TANet, SE-SSD, BtcDet based on voxel; PointRCNN, 3DSSD based on points; PV-RCNN, BADet based on the combination of points and voxels.

[0077] Algorithm Sedan Non-motor vehicle Large vehicle Pedestrian Obstacle Average accuracy Second 69.85 51.90 62.36 30.87 32.66 49.53 VoxelNet 68.64 49.85 63.78 22.94 29.53 46.95 PointPillars 70.17 56.32 63.92 30.76 31.63 50.56 PointRCNN 70.65 53.67 66.26 29.48 32.59 50.53 3DSSD 78.23 56.92 70.73 30.81 36.15 54.57 Part-A 2 ]]> 78.27 59.18 72.48 29.74 37.39 55.41 PV-RCNN 79.73 61.92 72.35 31.93 38.94 56.97 CenterPoint 73.68 58.34 68.92 28.67 36.04 53.13 TANet 76.62 63.88 69.26 34.83 42.35 57.39 SE-SSD 81.45 63.52 72.87 32.76 38.45 57.81 BADet 80.52 63.19 73.42 32.58 39.84 57.91 BtcDet 81.96 64.83 73.92 33.21 39.68 58.72 Ours 81.02 65.91 75.21 36.02 41.46 59.92

[0078] It can be found that BtcDet has the highest precision in the small car detection task, considering that BtcDet introduces an occupancy probability prediction mechanism, which can effectively identify and process the occlusion and signal loss area in the point cloud. The target distribution in the autonomous driving scene is usually dense, so BtcDet performs well on LS-LiDAR. TANet has the highest precision in the pedestrian detection task, which benefits from the triple attention module introduced in the network, which can effectively enhance the key feature points while suppressing unstable points. In dealing with the relatively scarce point cloud features of pedestrian targets, TANet can significantly improve the detection ability of the model. However, the algorithm proposed in the present application (“Ours” in Table 2) has the highest average detection precision on LS-LiDAR, which fully demonstrates that reasonable feature channel expansion during point cloud voxelization and effective multi-scale feature fusion can improve the detection effect of the model in complex scenes, especially when dealing with multi-scale targets, the method of the present application shows stronger adaptability and superiority.

[0079] As shown in Table 3, the detection precision of different algorithms on the public dataset KITTI is also compared.

[0080] Table 3 Detection precision comparison of different algorithms on KITTI

[0081] Algorithm Car Non-motor vehicle Pedestrian Average accuracy Second 74.16 54.85 42.56 57.19 VoxelNet 64.11 49.16 33.29 48.85 PointPillars 75.31 59.67 42.52 59.17 PointRCNN 75.14 57.62 40.17 57.64 3DSSD 79.27 63.81 44.84 62.64 Part-A 2 ]]> 77.27 63.12 43.58 61.32 PV-RCNN 81.05 62.37 44.01 62.48 CenterPoint 73.67 57.05 41.26 57.33 TANet 76.13 59.84 45.03 60.33 SE-SSD 82.37 63.58 44.21 63.39 BADet 81.39 64.25 44.31 63.32 BtcDet 82.74 65.31 44.63 64.23 Ours 82.53 67.04 45.81 65.13

[0082] Consistent with the experimental results on LS-LiDAR, BtcDet still obtains the highest detection precision in the car detection task. However, the algorithm proposed in the present application obtains the highest average detection precision in the multi-target detection task, which fully proves that the detection algorithm designed based on feature expansion and multi-layer adaptive feature fusion in the present application has superior detection performance, can effectively identify targets in different datasets, and the algorithm performs well in the autonomous driving scene.

[0083] In summary, combined with the voxelization processing of feature expansion and the adaptive multi-scale feature fusion mechanism, the model proposed in the present application exhibits superior performance in the point cloud target detection task, especially in dealing with multi-scale targets, it shows better detection precision and robustness. The feature fusion module in the model effectively overcomes the noise propagation problem existing in traditional methods, and significantly improves the adaptability of the model in complex scenes through strengthening target feature learning. Experimental results show that whether on the self-made dataset LS-LiDAR or the public dataset KITTI, the algorithm of the present application can improve the performance in multiple tasks, proving its good generalization in the autonomous driving scene.

[0084] Experiment two

[0085] To evaluate the effect of the gradient-guided modified loss function on improving the learning ability of the model and the detection accuracy, and the rationality of the loss function design, the experimental results will be described in detail from the aspects of comparative experiments and ablation experiments.

[0086] To make a comprehensive comparison, the gradient-guided modified loss function proposed in this chapter and other five classical classification loss functions are used for constraint training based on the Second, PointPillars, and 3DSSD benchmark detectors, including cross-entropy loss (CE), Focal Loss, Equalized Focal Loss, Balanced Softmax, and GHM. Among them, the cross-entropy loss is used to measure the difference between the predicted probability distribution and the true distribution of the model, and is known for its lightweight and efficiency. It has always been the most commonly used loss function for classification models in the field of object detection. Focal Loss solves the problem of imbalance between positive and negative samples in single-stage detectors to some extent, and is also a commonly used classification loss function in the field of computer vision. Therefore, these loss constraint methods are used for training on LS-LiDAR, and testing and evaluation are performed. Table 4 shows the quantitative evaluation results of the detection accuracy of the models trained under the gradient-guided modified loss function and other loss functions.

[0087] Table 4 Influence of different classification loss functions on LS-LiDAR detection

[0088]

[0089]

[0090] It can be seen that the model trained using the loss function proposed in the present application achieves the highest average precision on the test set, and the detection accuracy of a small number of sample categories is significantly improved. Specifically, compared with the Focal Loss classification loss function used in the Second and PointPillars models, the use of the loss function proposed in the present application can improve the average detection accuracy by 5.2% and 4.12%, respectively, among which the detection accuracy of pedestrians and obstacles is improved most obviously, by 12.73% and 8.23%, and 8.34% and 5.14%, respectively. Compared with the CE classification loss function used in the 3DSSD model, the use of the loss function proposed in the present application can improve the average detection accuracy by 4.49%, among which the detection accuracy of pedestrians and obstacles is improved by 6.1% and 5.94%, respectively.

[0091] The key control parameters of the gradient-guided modified loss function include the class weight factor (CWF) and the gradient density correction parameter (GDCP). In the corresponding experiments in Table 5, the effectiveness of the two control parameters and the rationality of the loss function design are analyzed through the design of ablation experiments.

[0092] Table 5: Influence of regulation parameters on LS-LiDAR detection

[0093]

[0094]

[0095] Influence of class weight factor. In the ablation experiment, it can be observed that the class weight factor has a significant influence on the performance of the model. By introducing the class weight factor, i.e., using a class-related adaptive adjustment factor for classes with different foreground and background imbalance situations, the loss contribution of each class of samples can be independently rebalanced. The problem of class imbalance is solved on the basis of solving the problem of foreground and background imbalance. Experimental results show that the application of the class weight factor significantly improves the performance of the model on long-tail datasets, especially in tasks with extremely unbalanced class distribution, the accuracy and recall of the model are significantly improved.

[0096] Influence of gradient density correction parameter. The introduction of the gradient density correction parameter has a positive influence on the stability and final performance of the model training. By introducing the gradient density correction parameter, the negative influence of simple negative samples and extremely difficult samples on model training can be reduced, while the contribution of normal difficult samples is enhanced. This mechanism effectively solves the problem of gradient imbalance in single-stage object detection algorithms, making the model more focused on learning samples that are more critical to improving detection accuracy. Ablation experiment results show that the application of the gradient density correction parameter significantly improves the convergence behavior of the model, reduces fluctuations in the training process, and improves performance on multiple tasks.

[0097] Figures 4A-4C The visualization comparison chart of LS-LiDAR detection before and after using the class balancing module proposed in the present application is shown, and the class balancing module refers to the gradient-guided correction loss function proposed in the present application. Figures 4A-4C Similar to the detection visualization chart of Experiment 1, Figure 4A 、 Figure 4B 、 Figure 4C In the a group, b group and c group shown in the figure, the left side of each group of figures represents the detection result chart without introducing the class balancing module, and the right side represents the detection result chart after introducing the class balancing module. Different colors are used to render the points corresponding to different target classes, wherein green represents a small car, blue represents a non-motor vehicle, yellow represents a large vehicle, pink represents a pedestrian, orange represents an obstacle, and black represents the background. From the figure, it can be observed that the use of the class balancing module can effectively reduce the occurrence of missed detection and false detection of tail class targets such as pedestrians, obstacles and non-motor vehicles, and the detection accuracy of each class of targets is also improved to a certain extent.

[0098] The present application deeply explores a series of challenges in the point cloud target detection task under the automatic driving scene, including the sparse and irregular distribution of point clouds, certain information loss after data processing, difficulty for the network to effectively extract target features, and a large number of simple negative samples interfering with the gradient update of the network and the like. In view of these problems, the present application proposes an innovative and applicable solution from the perspective of network structure design and training strategy optimization, so as to construct a point cloud target detector with better detection effect, and improve the performance and reliability of the automatic driving system in the complex traffic environment.

[0099] It should be noted that the terms "first", "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0100] In the description of the present application, the description referring to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present specification.

[0101] In the specification, the word "comprising" does not exclude other components or steps, and "one" or "a" does not exclude a plurality. Some measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0102] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be regarded as falling within the scope of protection of the present application.

Claims

1. An adaptive feature fusion method for point cloud object detection, characterized in that: include: Obtain N1 feature maps of different levels to be fused; N1 is a positive integer greater than or equal to 2; Perform channel alignment on the feature maps of each level to obtain N1 channel-aligned feature maps of different levels; Adaptively generate the weights corresponding to the feature maps after channel alignment at the 2nd to N1th levels, and obtain N1-1 weights; Based on the N1-1 weights, starting from the channel-aligned feature map of the N1th level, the feature maps are sequentially fused with the channel-aligned feature maps of the previous level until the fusion with the channel-aligned feature map of the 1st level is completed, thereby obtaining a fused feature map.

2. The method according to claim 1, characterized in that The adaptive generation of weights corresponding to the feature maps after channel alignment at the 2nd to N1th levels includes: For the channel-aligned feature maps of each level in the 2nd to N1th levels, extract the local features of the channel-aligned feature maps of the level and capture the spatial information to obtain the initial features; Normalizing and activating the initial features to obtain activated features; Performing spatial dimension reduction, channel adjustment, and channel relationship capture on the activated features to obtain processed features; The values ​​of the processed features are compressed to the range of [0, 1] to obtain the weights corresponding to the feature maps after channel alignment at this level.

3. The method according to claim 1, characterized in that The method of combining the channel-aligned feature map of the N1-th level with the channel-aligned feature map of the previous level based on the N1-1 weights, and fusing the feature map with the channel-aligned feature map of the first level until the fusion with the channel-aligned feature map of the first level is completed, thereby obtaining a fused feature map, including: The feature map after channel alignment at the N1th level is used as the fusion feature at the N1th level; According to the weight corresponding to the channel-aligned feature map of the N1-th level, the fused features of the N1-th level are weighted adjusted and then fused with the channel-aligned feature map of the N1-1-th level, and the fused features are integrated to obtain the fused features of the N1-1-th level. Afterwards, according to the weight corresponding to the channel-aligned feature map of the N1-1-th level, the fused features of the N1-1-th level are weighted adjusted and then fused with the channel-aligned feature map of the N1-2-th level, and the fused features are integrated to obtain the fused features of the N1-2-th level. The fusion is iterated in this way until the fused features of the first level are obtained, and the fused features of the first level are used as the fused feature map.

4. An adaptive feature fusion network for point cloud object detection, characterized in that: Used to execute the method described in any one of claims 1 to 3 above, the adaptive feature fusion network includes: N1 channel alignment modules, 1 adaptive weight module, N1-1 weighted adjustment modules, N1-1 fusion modules and N1-1 integration modules.

5. The adaptive feature fusion network according to claim 4, characterized in that The adaptive weight module includes: a 3×3 convolution layer, a batch normalization layer, a ReLU activation function, a global average pooling layer, a 1×1 convolution layer and a Sigmoid normalization function; wherein, the input and output of the 3×3 convolution layer, the normalization layer, the ReLU activation function, the global average pooling layer, the 1×1 convolution layer and the Sigmoid normalization function are connected in sequence, and the input of the 3×3 convolution layer serves as the input of the adaptive weight module, and the output of the Sigmoid normalization function serves as the output of the adaptive weight module.

6. The adaptive feature fusion network according to claim 4, characterized in that Each channel alignment module is a 1×1 convolutional layer, and each integration module is a 3×3 convolutional layer.

7. A point cloud target detection method, characterized in that: include: Obtain the point cloud data to be detected; Converting the point cloud data to be detected into a pseudo two-dimensional image to obtain a pseudo two-dimensional image to be detected; Using a backbone network to extract feature maps of different levels of the pseudo two-dimensional image to be detected; Adopting an adaptive feature fusion network, and fusing the feature maps of different levels by executing the method described in any one of claims 1 to 3 above, to obtain a fused feature map; A three-dimensional target detector is used to perform target detection based on the fused feature map.

8. The method according to claim 7, characterized in that The three-dimensional object detector is a single-stage three-dimensional object detector.

9. The method according to claim 8, characterized in that The single-stage three-dimensional object detector is trained using a gradient-guided corrected loss function, wherein the gradient-guided corrected loss function is obtained by introducing a gradient coordination mechanism in the cross-entropy loss.

10. The method according to claim 9, characterized in that The expression of the gradient-guided corrected loss function is as follows: Among them, N2 is the total number of training samples, G i is the gradient norm of the i-th training sample, D(G) is the gradient density function of the training sample, α is the parameter for balancing positive and negative samples, p is the model's predicted probability for the candidate box of the i-th training sample, γ a represents the focusing factor that controls the basic behavior of the classifier in the sample-balanced scenario, and n is a hyperparameter.

Images