Monocular three-dimensional target detection method based on adaptive convolution and intelligent label distribution

By introducing adaptive convolution and intelligent label allocation methods in three-dimensional object detection technology, combining multiple data enhancement strategies and label allocation algorithms, the problem of insufficient sample diversity, computational complexity and generalization capabilities in three-dimensional object detection is solved. , realizes efficient and accurate three-dimensional object detection.

CN118982806APending Publication Date: 2024-11-19四川吉利学院

Patent Information

Application Number
CN202411063370.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the fields of autonomous driving, three-dimensional object detection technology faces problems such as insufficient sample diversity, high computing complexity, large resource requirements, difficulty in multi-scale detection, sensitive environmental changes, low labeling accuracy and insufficient generalization capabilities, which limits its wide application.

Method used

Using a monocular three-dimensional object detection method based on adaptive convolution and intelligent label allocation, data augmentation is carried out through multi-layer adaptive convolution units and dynamic feature fusion technology, and using Mosaic, RandomAffine, Mixup and RandomHSV transformations, combined with Gaussian hybrid model and the expected maximization algorithm for label allocation, dynamically adjusting the positive sample number to suit the targets of different scales and shapes.

Benefits of technology

It significantly improves the training effect and generalization ability of the three-dimensional object detection model, realizes efficient feature extraction, precise classification and regression, improves the robustness of the model for environmental changes and the accuracy of label allocation, and adapts to target detection of different perspectives and scales. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982806A_ABST
    Figure CN118982806A_ABST
Patent Text Reader

Abstract

The invention provides a monocular three-dimensional target detection method based on adaptive convolution and intelligent label distribution, and relates to the technical field of target detection, and the method comprises the following steps: carrying out the data enhancement of 3D data; dividing the enhanced data into a training set and a test set; training a target detection neural network model through the training set, and verifying the target detection neural network model by using the test set; target detection of an actual scene is carried out through the trained target detection neural network model; the architecture of the target detection neural network model comprises a backbone network part and a detection head part; the structure of the detection head part is as follows: two parallel self-adaptive convolution layers, a softmax layer and a classification layer are connected behind a first self-adaptive convolution unit, and a regression layer is connected behind a second self-adaptive convolution unit; the classification layer and the regression layer are connected with a label distribution module. The network architecture of the invention realizes efficient feature extraction and accurate classification and regression ability by introducing the adaptive convolution kernel and dynamic feature fusion technology, thereby achieving high precision on a 3D target detection task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a monocular three-dimensional target detection method. Background Art

[0002] In the field of autonomous driving, data enhancement technology plays a vital role. It can not only effectively increase the diversity of data and prevent model overfitting, but also simulate difficult samples in actual driving scenarios to ensure the stability and safety of the system. Alexnet uses random cropping, flipping and rotation to increase the training data and thus improve the overall performance of the network. ResNet uses random scaling, flipping, cropping, and finally color enhancement. These basic data enhancement methods can be divided into the following categories: scaling, cropping, flipping, affine transformation, and HSV transformation.

[0003] In recent years, data augmentation technology has also continued to develop. Yolov4 first proposed the Mosaic data augmentation method, which increases data diversity by splicing four different training images into a new image. This method effectively improves the generalization ability and performance of the target detection model. Mixup enhances training data by mixing the pixel matrices and corresponding annotation information of two pictures. This linear interpolation method not only increases the diversity of the image, but also helps the model learn to deal with label noise. Yolox further combines Mosaic and Mixup for data augmentation, and strategically turns off data augmentation in the final stage of training, significantly improving model performance.

[0004] At present, 3D object detection technology faces many challenges in practical applications. First, the diversity of training data samples is insufficient, resulting in poor model generalization ability. Second, the high computational complexity and large resource requirements make the training and inference time long, making it difficult to apply on devices with limited resources. In addition, multi-scale object detection is difficult, and the model performs poorly when dealing with small and large objects. Environmental changes and data noise pose a threat to the robustness and stability of the model, and real-time and latency issues also limit its application in scenarios that require rapid response. The annotation process of 3D data is complex and costly, and the annotation accuracy problem further affects the model training effect. Finally, the domain adaptability and generalization ability of existing models are insufficient, and cross-domain applications are difficult. These problems jointly limit the widespread application of 3D object detection technology in fields such as autonomous driving, robotics, and intelligent monitoring, and require in-depth research and improvement in data diversity, computational efficiency, feature extraction, environmental adaptability, real-time processing, annotation accuracy, and model generalization ability. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a monocular three-dimensional target detection method based on adaptive convolution and intelligent label allocation. The technical solution of the present invention is as follows:

[0006] A monocular 3D target detection method based on adaptive convolution and intelligent label assignment comprises the following steps:

[0007] Perform data enhancement on 3D data;

[0008] Divide the augmented data into training and testing sets;

[0009] The target detection neural network model is trained using the training set, and the target detection neural network model is verified using the test set;

[0010] Perform target detection in actual scenes through the trained target detection neural network model;

[0011] The architecture of the target detection neural network model includes a backbone network part, a feature fusion network part and a detection head part;

[0012] The backbone network consists of multiple layers of adaptive convolutional units and weight generators. The specific structure of the backbone network is as follows:

[0013] The multi-layer adaptive convolution unit consists of multiple convolutional layers, each of which dynamically adjusts the effective range of the convolution kernel weights through a trainable envelope function;

[0014] The weight generator is used to generate a dynamic feature fusion weight matrix. Its input is the feature map obtained by the top convolution layer of the backbone network. After passing through two fully connected layers and an activation function, the output is a one-dimensional vector data of the weights of each layer of the feature fusion network.

[0015] The feature fusion network fuses the feature maps of each layer in the multi-layer adaptive convolutional unit, and integrates multi-scale features through adaptive dynamic weight fusion;

[0016] The feature fusion map is the final output of the feature fusion network and is passed to the detection head.

[0017] The detection head part includes two parallel adaptive convolution layers, namely the first adaptive convolution unit and the second adaptive convolution unit; the specific structure is as follows:

[0018] The first adaptive convolution unit uses a residual structure, which consists of a convolution layer and a skip connection. The convolution kernel is an adaptive convolution kernel with a size of 9x9. The output feature map is normalized by a softmax layer, followed by a classification layer for target classification.

[0019] The second adaptive convolution unit also uses a residual structure, and the convolution kernel is also an adaptive convolution kernel with a size of 9x9. The output feature map is directly input into the regression layer to predict the position information of the target bounding box.

[0020] The results of the classification layer and the regression layer are input into the label assignment module to obtain the target detection results.

[0021] Specifically, the data enhancement includes Mosaic data enhancement, RandomAffine transformation, Mixup image fusion and RandomHSV adjustment.

[0022] Specifically, the steps of Mosaic data enhancement are as follows:

[0023] Randomly select 4 images from the training batch;

[0024] Initialize a blank canvas whose length and width are both twice the size of the input image;

[0025] Stitch 4 randomly selected images onto the canvas, with each image occupying 1 / 4 of the canvas. If the stitched image exceeds the size of the canvas, crop the excess to ensure that the size of the final image matches the model input size.

[0026] Adjust the annotation boxes in each image accordingly to ensure that they correctly reflect the location and size of the object in the new image;

[0027] The steps of the RandomAffine transformation are as follows:

[0028] Perform RandomAffine transformation on the stitched images, including but not limited to flipping, scaling, and translation operations; and perform corresponding operations on the real frame;

[0029] The Mixup image fusion operation is as follows:

[0030] Randomly select another image and perform random scaling;

[0031] Crop it to the same size as the image obtained by the previous Mosaic data augmentation step;

[0032] Generate the final training image by changing the weights of the two images and fusing them together;

[0033] The steps of RandomHSV adjustment are as follows:

[0034] Use RandomHSV technology to convert the image from RGB color space to HSV color space and randomly adjust the hue, saturation and brightness.

[0035] Specifically, the specific connections of the backbone network are as follows:

[0036] The input image first passes through the first convolutional layer, and the feature map output by the first convolutional layer is passed to the second convolutional layer; the feature map output by the second convolutional layer is sequentially passed to the third convolutional layer, and so on, until it is passed to the last convolutional layer. The output feature map of the last convolutional layer is then processed by a weight generator to generate the dynamic weights of each modulated convolutional layer.

[0037] The feature fusion network includes multiple feature fusion layers, which are the i-th feature fusion layer from top to bottom, where i ranges from 1 to N, and the value of N corresponds to the number of convolutional layers of the backbone network; both the first feature fusion layer and the N-th feature fusion layer include two adaptively connected convolutional layers in sequence, and the remaining feature fusion layers include three adaptively connected convolutional layers in sequence and a residual structure; the output of the first adaptively connected convolutional layer of the first feature fusion layer is connected to the input of the second adaptively connected convolutional layer of the second adaptively connected convolutional layer by upsampling for an addition operation; starting from the second feature fusion layer, the output of the second adaptively connected convolutional layer of the i-th feature fusion layer is connected to the input of the second adaptively connected convolutional layer of the i+1-th feature fusion layer for an addition operation, and i < N - 2; the output of the first adaptively connected convolutional layer of the N-1-th feature fusion layer is connected to the input of the second adaptively connected convolutional layer of the N-1-th adaptively connected convolutional layer by upsampling for an addition operation; starting from the N-th feature fusion layer, the output of the third adaptively connected convolutional layer of each layer is added to the input of the third adaptively connected convolutional layer of the previous feature fusion layer.

[0038] Specifically, the algorithm in the label assignment module is as follows:

[0039] To evaluate the quality of each sample point, a quality evaluation value index function is constructed, which combines the predicted class probability of each sample point and the matching degree of the bounding box.

[0040] Based on the evaluation value of each sample point, the Gaussian mixture model and the expectation maximization algorithm are applied to cluster the sample points, and finally two Gaussian distributions are obtained. The sample points with evaluation values lower than the smaller mean of the two Gaussian distributions represent the positive sample candidate regions.

[0041] For the points within each candidate region, a cost matrix between the true bounding box and each predicted point is constructed by combining the classification loss and the regression loss.

[0042] According to the principle of minimizing the calculated loss, k positive samples with the lowest cost are assigned to each true bounding box from the positive sample candidate regions, and the other samples are classified as negative samples, where k is the dynamic number of positive sample assignments.

[0043] Specifically, based on the evaluation value of each sample point, the Gaussian mixture model and the expectation maximization algorithm are applied to cluster the sample points, and finally two Gaussian distributions are obtained. The sample points with evaluation values ​​lower than the smaller mean of the two Gaussian distributions indicate that the positive sample candidate regions are specifically:

[0044] Construct the quality evaluation value indicator function, the formula is

[0045] c i = FL(p i , C)+(1-IoU(b i , B))

[0046] Among them, c i represents the quality evaluation value of the i-th sample point, p i and b i They represent the probability and bounding box of the predicted category of the point, C and B represent the true category and true bounding box of the point, and FL represents Focal Loss.

[0047] Specifically, for each point in the candidate region, the cost matrix between the true bounding box and each predicted point is constructed by combining the classification loss and the regression loss:

[0048] After determining a set of positive sample candidate points, the remaining points in the true box are classified as negative samples. For these negative sample points, the cost between them and the true box is set to a constant value. Specifically, the cost value is defined as C ij , represents the cost value of the i-th true box and the j-th predicted point, which is the weighted sum of the classification loss and the regression loss:

[0049]

[0050] Among them, Loss cls and Loss reg Respectively represent the classification loss function and regression loss function; the classification loss function is Focal Loss, and the regression loss function is the smoothL1 loss function; and They represent the prediction scores of the predicted point j under the hyperparameter of θ, and They represent the true classification and bounding box attributes of the i-th true box, respectively. The bounding box attributes include position, size, and rotation angle. α represents the weight ratio of classification and regression. Focal Loss is

[0051] Loss cls (pt) = -(1-p t ) γ log(p t )

[0052] Among them, -(1-p t ) γ represents the modulation factor, and γ represents the adjustable parameter.

[0053] Specifically, according to the calculated loss minimization principle, k positive samples with the lowest cost are allocated to each true bounding box from the positive sample candidate area, and other samples are classified as negative samples as follows:

[0054] By calculating the sum of the IoU values ​​of the predicted boxes and rounding them, the number of positive samples k that should be assigned to each true box is dynamically determined. The calculation formula is:

[0055]

[0056] Among them, K j represents the number of dynamic positive samples corresponding to the jth gt box, q is a predefined constant used to select the first q predicted boxes with the largest intersection-over-union ratio with each true box; represents the iou value of the i-th maximum iou prediction box in the j-th gt box, G j represents the jth gt box, P i represents the i-th predicted box.

[0057] After adopting the above scheme, the beneficial effects of the present invention are:

[0058] (1) The comprehensive data enhancement strategy adopted in this paper aims to significantly improve the training effect and generalization ability of the 3D object detection model.

[0059] (2) The network architecture of the present invention realizes efficient feature extraction and accurate classification and regression capabilities by introducing adaptive convolution kernels and dynamic feature fusion technology, thereby achieving high precision in 3D object detection tasks.

[0060] (3) A dynamic positive sample number determination method based on the OTA concept is adopted, so that the label assignment process can more flexibly and accurately reflect the matching degree between each true box and the predicted box. Compared with the traditional label assignment method, this strategy does not simply assign a fixed number of positive samples to each true box or select only a single best match, but dynamically adjusts the number of positive samples according to the actual IoU value distribution. The significant advantage of this method is that it can adaptively adjust the number of positive samples to adapt to true boxes of different sizes and shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The schematic diagram of the backbone network of the target detection neural network model of the present invention;

[0062] Figure 2 This is a diagram of the architecture of the detection head portion of the target detection neural network model of the present invention;

[0063] Figure 3 A schematic diagram of an algorithm in a label allocation module of the present invention;

[0064] Figure 4 It is a schematic diagram of the Mosaic data enhancement process of the present invention;

[0065] Figure 5 It is a schematic diagram of the Mixup image fusion result of the present invention. DETAILED DESCRIPTION

[0066] A monocular 3D target detection method based on adaptive convolution and intelligent label assignment comprises the following steps:

[0067] S1. Perform data enhancement on 3D data;

[0068] The data enhancement includes Mosaic data enhancement, RandomAffine transformation, Mixup image fusion and RandomHSV adjustment. These transformations not only increase the visual diversity of the image, but also, since the target detection task involves 3D real boxes, the labels of these real boxes also need to be geometrically transformed accordingly to maintain the accuracy of the annotation information. This step is crucial to improving the model's adaptability to different perspectives and scale changes.

[0069] The steps of Mosaic data enhancement are as follows:

[0070] like Figure 4 As shown, 4 images are randomly selected from the training batch;

[0071] Initialize a blank canvas whose length and width are both twice the size of the input image;

[0072] Stitch 4 randomly selected images onto the canvas, with each image occupying 1 / 4 of the canvas; if the stitched image exceeds the size of the canvas, crop the excess part to ensure that the size of the final image matches the model input size;

[0073] Adjust the annotation boxes in each image accordingly to ensure that they correctly reflect the location and size of the object in the new image;

[0074] The steps of the RandomAffine transformation are as follows:

[0075] Perform RandomAffine transformation on the stitched images, including but not limited to flipping, scaling, and translation operations; and perform corresponding operations on the real frame;

[0076] like Figure 5 As shown, the Mixup image fusion operation is as follows:

[0077] Randomly select another image and perform random scaling;

[0078] Crop it to the same size as the image obtained by the previous Mosaic data augmentation step;

[0079] Generate the final training image by changing the weights of the two images and fusing them together;

[0080] The steps of RandomHSV adjustment are as follows:

[0081] Use RandomHSV technology to convert the image from RGB color space to HSV color space and randomly adjust the hue, saturation and brightness. This step can effectively increase the diversity of the image in the color dimension and help the model learn to accurately identify targets under different lighting and color conditions. RandomHSV adjustment This step can effectively increase the diversity of the image in the color dimension and help the model learn to accurately identify targets under different lighting and color conditions.

[0082] The comprehensive data enhancement strategy adopted by the present invention is intended to significantly improve the training effect and generalization ability of the 3D target detection model. Through the Mosaic data enhancement technology, the present invention can integrate multiple scenes and targets in a single training image, greatly enriching the visual input of the model, thereby effectively improving the model's adaptability to different target configurations, occlusion conditions, and scene complexity. RandomAffine transformation and Mixup image fusion further enhance the model's robustness in the face of scale, position, posture changes and different lighting conditions, ensuring that the model can learn more generalized feature expressions. Finally, through RandomHSV adjustment, the diversity of the model in color space has been significantly enhanced, allowing the model to more effectively handle common lighting changes and color deviation problems in practical applications. Overall, this series of data enhancement methods work together to provide the model with a more comprehensive and diverse training environment, significantly improving the accuracy and robustness of the algorithm, and laying a solid foundation for 3D target detection tasks in complex environments.

[0083] S2. Divide the enhanced data into training set and test set;

[0084] S3. Train the target detection neural network model using the training set, and verify the target detection neural network model using the test set;

[0085] S4. Perform target detection in actual scenes using the trained target detection neural network model;

[0086] The architecture of the target detection neural network model includes a backbone network part, a feature fusion network part and a detection head part; the structure of the backbone network part is:

[0087] The multi-layer adaptive convolution unit consists of multiple convolution layers. Each convolution layer dynamically adjusts the effective range of the convolution kernel weights through a trainable envelope function. Based on the network input features, a modulation matrix based on the envelope function can be generated. The learnable modulation matrix performs an element-wise product with the convolution kernel weights to adaptively adjust the convolution kernel size.

[0088] The weight generator is used to generate a dynamic feature fusion weight matrix. Its input is the feature map obtained from the top convolution layer of the backbone network. After passing through two fully connected layers and an activation function, the output is a one-dimensional vector data of the weights of each layer of the feature fusion network.

[0089] The feature fusion network fuses the feature maps of each layer in the multi-layer adaptive convolution unit, synthesizes multi-scale features through adaptive dynamic weight fusion, and improves the feature expression ability.

[0090] The specific connection of the backbone network is as follows:

[0091] The input image first passes through the first convolution layer, and the feature map output by the first convolution layer is passed to the second convolution layer. The feature map output by the second convolution layer is sequentially passed to the third convolution layer, and so on until it is passed to the last convolution layer. The output feature map of the last convolution layer is then processed by a weight generator to generate the dynamic weights of each modulated convolution layer.

[0092] The feature fusion network includes multiple feature fusion layers, which are the i-th feature fusion layer from top to bottom, where i ranges from 1 to N, and the value of N corresponds to the number of convolution layers in the backbone network. The first feature fusion layer and the N-th feature fusion layer both include two sequentially connected adaptive convolution layers, and the remaining feature fusion layers include three sequentially connected adaptive convolution layers and a residual structure. The output of the first adaptive convolution layer of the first feature fusion layer is connected to the input of the second adaptive convolution layer of the second adaptive convolution layer through upsampling for an addition operation. Starting from the second feature fusion layer, the output of the second adaptive convolution layer of the i-th feature fusion layer is connected to the input of the second adaptive convolution layer of the i + 1-th feature fusion layer for an addition operation, and i < N - 2. The output of the first adaptive convolution layer of the N - 1-th feature fusion layer is connected to the input of the second adaptive convolution layer of the N - 1-th adaptive convolution layer through upsampling for an addition operation. Starting from the N-th feature fusion layer, the output of the third adaptive convolution layer of each layer is added to the input of the third adaptive convolution layer of the previous feature fusion layer.

[0093] As shown in the figure, the feature fusion network receives the output feature maps of each convolution layer in the backbone network. These feature maps come from different levels and contain information from low-level features to high-level features. The feature fusion network uses a bidirectional up- and down-sampling path to process the feature maps through multiple layers of feature fusion layers. The arrows in the architecture diagram represent the flow of data, and each point represents an adaptive convolution layer. The curved arrows represent skip connections (residual connections), the downward arrows represent upsampling, which expands the size of the feature map, and the upward arrows represent downsampling, which reduces the size of the feature map. It is worth noting that Figure 1 Only 5 layers are drawn in the figure, and there are only 5 weights behind them, which does not correspond to the number of convolutional layers of the backbone network. This is because the figure is just a simplified diagram. In fact, the number of layers in the feature fusion network is consistent with the number of convolutional layers in the backbone network.

[0094] The upsampling path starts from the high-level feature map, as shown by the downward arrow in the figure. The low-resolution feature map is upsampled to a higher resolution through an upsampling operation (such as nearest neighbor interpolation or bilinear interpolation). These upsampled feature maps are fused with the low-level feature maps of the same resolution.

[0095] The downsampling path starts from the low-level feature map, as shown by the upward arrow in the figure. The high-resolution feature map is downsampled to a lower resolution through a downsampling operation (such as maximum pooling or average pooling). These downsampled feature maps are fused with the high-level feature maps of the same resolution.

[0096] Skip connections are used to achieve effective information transfer at the same level. Through skip connections, the input feature map can be directly passed to the subsequent layer and added to the output of the convolutional layer, thereby alleviating the gradient vanishing problem and improving the training efficiency of deep networks. In addition, residual connections also promote the effective learning of features, prevent information loss in deep networks, and enhance the representation ability of the network.

[0097] In the last layer of dynamic feature fusion, the feature fusion network uses weighted sum to fuse high-level and low-level feature maps. The weights of each feature map (w1, w2, w3, w4, w5) are dynamically learned during the training process by the top-level weight generator of the backbone network to ensure the optimal feature fusion effect.

[0098] The feature fusion map is the final output of the feature fusion network and is passed to the detection head.

[0099] The detection head part includes two parallel adaptive convolution layers, namely the first adaptive convolution unit and the second adaptive convolution unit; the specific structure is as follows:

[0100] The first adaptive convolution unit uses a residual structure, which consists of a convolution layer and a skip connection. The convolution kernel is an adaptive convolution kernel with a size of 9x9. The output feature map is normalized by a softmax layer, followed by a classification layer for target classification.

[0101] The second adaptive convolution unit also uses a residual structure, and the convolution kernel is also an adaptive convolution kernel with a size of 9x9. The output feature map is directly input into the regression layer to predict the position information of the target bounding box.

[0102] The results of the classification layer and the regression layer are input into the label assignment module to obtain the target detection results. The label assignment module is used to more efficiently assign positive and negative samples during training, affecting the classification loss and regression loss, thereby achieving better training results during the training process.

[0103] The feature fusion map is the final output of the backbone network and is passed to the detection head.

[0104] The detection head part includes two parallel adaptive convolution layers, namely the first adaptive convolution unit and the second adaptive convolution unit. The specific structure is as follows:

[0105] The first adaptive convolution unit uses a residual structure, which consists of a convolution layer and a skip connection. The convolution kernel is an adaptive convolution kernel with a size of 9x9. The output feature map is normalized by a softmax layer, followed by a classification layer for target classification.

[0106] The second adaptive convolution unit also uses a residual structure, and the convolution kernel is also an adaptive convolution kernel with a size of 9x9. The output feature map is directly input into the regression layer to predict the position information of the target bounding box.

[0107] The core mechanism of adaptive convolution kernel is realized through a trainable envelope function. The envelope function is responsible for dynamically adjusting the effective range of convolution kernel weights so that it can adapt to image features of different scales. Specifically, this process involves the element-wise product between the convolution kernel weights and a dynamically generated modulation matrix based on the envelope function. The generation of this modulation matrix is ​​done dynamically according to the characteristics of the network input, thereby realizing the adaptive adjustment of the convolution kernel size. The design of the envelope function is based on a parameterized mathematical model that can automatically adjust its parameters according to the feature distribution in the training data to optimally control the shape and size of the convolution kernel.

[0108]

[0109] This formula explains the role of the envelope function, where U is the envelope function, W is the weight coefficient of the convolution kernel, X is the input feature map, and O is the output feature map. is the element-wise product. The envelope function U can be defined as A smoothly differentiable function of

[0110] U:((k,l),θ)→u k,l ∈R

[0111] where{(k,l)|k,l∈{1,2,..,n}}andθ={θ,θ2,..θ p}

[0112] The envelope function controls the effective size and shape of the convolution kernel, and its error derivative can be calculated using the chain rule in the back-propagation behavior, and the weights can be updated along with the network:

[0113]

[0114] Among them, η represents the learning rate, w′ k,l and θ′ p represents the updated convolution kernel weights and envelope function parameters, and E represents the error.

[0115] The Gaussian envelope function is the first choice because of its continuous differentiability:

[0116]

[0117] in Exists in two-dimensional Euclidean space, the central variable μ=(μ x , μ y ) can control the center of the convolution kernel. Here, for the convenience of training, the center variable is simply set to a fixed convolution kernel center. This controls the distribution of the convolution kernel.

[0118] The adaptive convolution layer first defines a basic grid, the size of which determines the maximum possible size of the convolution kernel. Then, by adjusting the parameters of the Gaussian envelope, the network can form an effective convolution region of dynamic size within this basic grid. This process involves learning the parameters of the Gaussian envelope, which are optimized by the gradient descent algorithm during the back-propagation phase of the network. This adaptive mechanism allows the convolution layer to not only adapt to image features of different scales, but also to mitigate the risk of overfitting to a certain extent, because it achieves more efficient feature learning by reducing the number of parameters and intelligently adjusting the size of the convolution kernel.

[0119] The standard deviation σ of the Gaussian envelope function is a learnable parameter that controls the effective coverage of the convolution kernel, allowing the network to autonomously learn the convolution kernel size that best suits the task during training. This design not only increases the adaptability of the network, but also improves the model's sensitivity and generalization ability to changes in input data.

[0120] The output of the backbone network is a feature fusion map, which is input into the detection head.

[0121] like Figure 2 As shown in the figure, the structure of the detection head is:

[0122] There are two parallel adaptive convolutional layers, namely the first adaptive convolutional unit and the second adaptive convolutional unit. The residual structure is used in the first adaptive convolutional unit and the second adaptive convolutional unit. The convolution kernel size is 9*9. The first adaptive convolutional unit is connected to the softmax layer and the classification layer, and the second adaptive convolutional unit is connected to the regression layer. The classification layer and the regression layer are connected to the label assignment module.

[0123] The network architecture of the present invention achieves efficient feature extraction and accurate classification and regression capabilities by introducing adaptive convolution kernels and dynamic feature fusion technology, thereby achieving high accuracy in 3D object detection tasks. The adaptive convolution kernel allows the network to dynamically adjust the convolution kernel size according to the input image, enhancing the adaptability to features of different scales. At the same time, the dynamic feature fusion mechanism further optimizes the integration of features at different levels, enabling the model to process complex visual information more flexibly. This innovation and optimization of the architecture significantly improves the ability to handle complex 3D detection problems while ensuring high efficiency.

[0124] like Figure 3 As shown, the algorithm of the label allocation module is as follows:

[0125] In order to evaluate the quality of each sample point, a quality evaluation value indicator function is constructed, which combines the predicted category probability and the matching degree of the bounding box of each sample point;

[0126] Based on the evaluation value of each sample point, the Gaussian mixture model and expectation maximization algorithm are used to cluster the sample points, and finally two Gaussian distributions are obtained. The sample points with evaluation values ​​lower than the smaller mean of the two Gaussian distributions represent the positive sample candidate areas.

[0127] In order to more accurately assign positive and negative samples, for each point in the candidate region, the classification loss and regression loss are combined to construct a cost matrix between the true bounding box and each predicted point;

[0128] The dynamic number of positive sample allocations k is introduced. According to the calculated loss minimization principle, k lowest-cost positive samples are allocated to each true bounding box from the positive sample candidate area, and other samples are classified as negative samples. k is the dynamic number of positive sample allocations. This dynamic allocation mechanism allows the model to adjust the learning focus more flexibly and optimize for different training situations, thereby effectively improving the detection performance and generalization ability of the model.

[0129] The method mixes two Gaussian distributions of prediction scores and takes the smaller mean of the two as the threshold. Anchor points below this threshold are considered as positive sample candidates:

[0130] Construct the quality evaluation value indicator function, the formula is

[0131] c i = FL(p i , C)+(1-IoU(b i , B))

[0132] Among them, c i represents the quality evaluation value of the i-th sample point, p i and b i Respectively represent the probability and bounding box of the predicted category of the point, C and B represent the true category and true bounding box of the point, respectively. FL represents Focal Loss, which is a classification loss function mainly used to deal with the problem of category imbalance. Based on the evaluation value of each point, the Gaussian mixture model and the expectation maximization algorithm are applied to cluster the sample points to obtain two Gaussian distributions. In these two distributions, the sample points with evaluation values ​​lower than the smaller mean u1 are judged as positive sample candidate areas. The core advantages of this step are:

[0133] Probability-based sample quality assessment: By combining the predicted probability of the sample point and the IoU with the true bounding box, this method can comprehensively evaluate the quality of the sample from multiple dimensions.

[0134] Application of Gaussian mixture model and expectation maximization algorithm: GMM and EM algorithms are used to perform efficient clustering analysis on samples, which significantly improves the accuracy and efficiency of positive and negative sample judgment.

[0135] Automated determination of positive sample candidate regions: The positive sample candidate regions are automatically identified through cluster analysis, which avoids the limitation of relying on manual threshold setting in traditional methods and improves the versatility and adaptability of the method.

[0136] Flexibility and scalability: The probability-based model design makes this method flexible and applicable to different data types and task requirements, and has excellent scalability. At the same time, the model parameters can be adjusted according to actual conditions to optimize performance.

[0137] Comprehensive considerations: This method comprehensively reflects the quality of samples by considering multiple factors such as predicted category probability and bounding box matching, thereby improving the accuracy and robustness of label assignment.

[0138] For each point in the candidate area, the cost matrix between the true bounding box and each predicted point is constructed by combining the classification loss and the regression loss:

[0139] After determining a set of positive sample candidate points, the remaining points in the true box are classified as negative samples. For these negative sample points, the cost between them and the true box is set to a constant value. Specifically, the cost value is defined as C ij , represents the cost value of the i-th true box and the j-th predicted point, which is the weighted sum of the classification loss and the regression loss:

[0140]

[0141] Among them, Loss cls and Loss reg Respectively represent the classification loss function and regression loss function; the classification loss function is Focal Loss, and the regression loss function is the smoothL1 loss function; and They represent the prediction scores of the predicted point j under the hyperparameter of θ, and They represent the true classification and bounding box attributes of the i-th true box, respectively. The bounding box attributes include position, size, and rotation angle. α represents the weight ratio of classification and regression. Focal Loss is

[0142] Loss cls (p t )=-(1-p t ) γ log(p t )

[0143] Among them, -(1-p t ) γ represents the modulation factor, and γ represents the adjustable parameter.

[0144] In the algorithm of the present invention, constructing a cost matrix between the true bounding box (gt box) and each predicted point is a key step, which aims to comprehensively evaluate the degree of match of each predicted point relative to each true bounding box. This process is achieved by calculating the cost value of each predicted point corresponding to each true box, thereby forming a dense cost relationship matrix to provide a basis for subsequent label assignment. Unlike most label assignment methods that only independently assign positive and negative samples and ignore contextual relationships, the method of the present invention constructs a cost matrix from a global perspective to establish dense mutual relationships between each true box and all predicted points. This global consideration helps to optimize the label assignment process. In particular, when a single predicted point may be associated with multiple true boxes, each true box will be assigned to the partial prediction box with the lowest cost through screening by the cost matrix, which will effectively reduce erroneous predictions and label assignments.

[0145] The dynamic number of positive sample allocations k is introduced, and according to the principle of minimizing the calculated performance loss, k positive samples with the lowest cost are allocated to each true bounding box, and other samples are classified as negative samples as follows:

[0146] By calculating the sum of the IoU values ​​of the predicted boxes and rounding them, the number of positive samples k that should be assigned to each true box is dynamically determined. The calculation formula is:

[0147]

[0148] Among them, q is a predefined constant used to select the first q predicted boxes with the largest intersection-over-union ratio with each true box; represents the iou value of the i-th maximum iou prediction box in the j-th gt box, G j represents the jth gt box, P i Represents the i-th predicted box. The core of this step is to dynamically adjust the number of positive samples according to the relationship between the predicted box and the true box, thereby effectively improving the label allocation efficiency and detection performance of the model. The dynamic positive sample number determination method based on the OTA idea is adopted, so that the label allocation process can more flexibly and accurately reflect the degree of match between each true box and the predicted box. Compared with the traditional label allocation method, this strategy does not simply assign a fixed number of positive samples to each true box or select only a single best match, but dynamically adjusts the number of positive samples according to the actual IoU value distribution. The significant advantage of this method is that it can adaptively adjust the number of positive samples to adapt to real boxes of different sizes and shapes. For example, for a real box with a larger area, more positive samples may be required to fully cover its range, while for a smaller real box, fewer positive samples are sufficient. Such dynamic adjustment not only improves the detection accuracy of the model for objects of different sizes, but also enhances the generalization ability and overall performance of the model.

Claims

1. A monocular 3D object detection method based on adaptive convolution and intelligent label assignment, characterized in that: The steps include: Perform data enhancement on 3D data; Divide the augmented data into training and testing sets; The target detection neural network model is trained using the training set, and the target detection neural network model is verified using the test set; Perform target detection in actual scenes through the trained target detection neural network model; The architecture of the target detection neural network model includes a backbone network part, a feature fusion network part and a detection head part; The backbone network consists of multiple layers of adaptive convolutional units and weight generators. The specific structure of the backbone network is as follows: The multi-layer adaptive convolution unit consists of multiple convolutional layers, each of which dynamically adjusts the effective range of the convolution kernel weights through a trainable envelope function; The weight generator is used to generate a dynamic feature fusion weight matrix. Its input is the feature map obtained by the top convolution layer of the backbone network. After passing through two fully connected layers and an activation function, the output is a one-dimensional vector data of the weights of each layer of the feature fusion network. The feature fusion network fuses the feature maps of each layer in the multi-layer adaptive convolutional unit, and integrates multi-scale features through adaptive dynamic weight fusion; The feature fusion map is the final output of the feature fusion network and is passed to the detection head. The detection head part includes two parallel adaptive convolution layers, namely the first adaptive convolution unit and the second adaptive convolution unit; the specific structure is as follows: The first adaptive convolution unit uses a residual structure, which consists of a convolution layer and a skip connection. The convolution kernel is an adaptive convolution kernel with a size of 9x9. The output feature map is normalized by a softmax layer, followed by a classification layer for target classification. The second adaptive convolution unit also uses a residual structure, and the convolution kernel is also an adaptive convolution kernel with a size of 9x9. The output feature map is directly input into the regression layer to predict the position information of the target bounding box. The results of the classification layer and the regression layer are input into the label assignment module to obtain the target detection results.

2. According to claim 1, a monocular three-dimensional target detection method based on adaptive convolution and intelligent label allocation is characterized in that: The data enhancement includes Mosaic data enhancement, RandomAffine transformation, Mixup image fusion and RandomHSV adjustment.

3. The monocular 3D target detection method based on adaptive convolution and intelligent label allocation according to claim 3, characterized in that: The steps of Mosaic data enhancement are as follows: Randomly select 4 images from the training batch; Initialize a blank canvas whose length and width are both twice the size of the input image; Stitch 4 randomly selected images onto the canvas, with each image occupying 1 / 4 of the canvas. If the stitched image exceeds the size of the canvas, crop the excess to ensure that the size of the final image matches the model input size. Adjust the annotation boxes in each image accordingly to ensure that they correctly reflect the location and size of the object in the new image; The steps of the RandomAffine transformation are as follows: Perform RandomAffine transformation on the stitched images, including but not limited to flipping, scaling, and translation operations; and perform corresponding operations on the real frame; The Mixup image fusion operation is as follows: Randomly select another image and perform random scaling; Crop it to the same size as the image obtained by the previous Mosaic data augmentation step; Generate the final training image by changing the weights of two images and fusing them together; The steps of the RandomHSV adjustment are as follows: Use the RandomHSV technique to convert the picture from the RGB color space to the HSV color space and randomly adjust the hue, saturation, and brightness.

4. The monocular 3D target detection method based on adaptive convolution and intelligent label assignment according to claim 1, characterized in that: The specific connections of the backbone network are as follows: The input image first passes through the first convolutional layer, and the feature map output by the first convolutional layer is passed to the second convolutional layer; the feature map output by the second convolutional layer is successively passed to the third convolutional layer, and so on until it is passed to the last convolutional layer. The output feature map of the last convolutional layer is then processed by a weight generator to generate the dynamic weights of each convolutional layer after modulation; The feature fusion network includes multiple feature fusion layers, which are the i-th feature fusion layer from top to bottom, where i ranges from 1 to N, and the value of N corresponds to the number of convolutional layers of the backbone network; Both the first feature fusion layer and the N-th feature fusion layer include two sequentially connected adaptive convolutional layers, and the remaining feature fusion layers include three sequentially connected adaptive convolutional layers and a residual structure; The output of the first adaptive convolutional layer of the first feature fusion layer is connected to the input of the second adaptive convolutional layer of the second adaptive convolutional layer through upsampling for an addition operation; starting from the second feature fusion layer, the output of the second adaptive convolutional layer of the i-th feature fusion layer is connected to the input of the second adaptive convolutional layer of the i + 1-th feature fusion layer for an addition operation, and i < N - 2; the output of the first adaptive convolutional layer of the N - 1-th feature fusion layer is connected to the input of the second adaptive convolutional layer of the N - 1-th adaptive convolutional layer through upsampling for an addition operation; starting from the N-th feature fusion layer, the output of the third adaptive convolutional layer of each layer is added to the input of the third adaptive convolutional layer of the previous feature fusion layer.

5. The monocular 3D target detection method based on adaptive convolution and intelligent label assignment according to claim 1, characterized in that: The algorithm in the label assignment module is as follows: To evaluate the quality of each sample point, construct a quality evaluation index function that combines the predicted class probability of each sample point and the matching degree of the bounding box; Based on the evaluation value of each sample point, apply the Gaussian mixture model and the expectation maximization algorithm to cluster the sample points, and finally obtain two Gaussian distributions. The sample points with evaluation values lower than the smaller mean of the two Gaussian distributions represent the positive sample candidate regions; For the points within each candidate region, combine the classification loss and the regression loss to construct a cost matrix between the true bounding box and each predicted point; According to the principle of minimizing the calculated loss, assign k positive samples with the lowest cost to each true bounding box from the positive sample candidate regions, and divide the other samples into negative samples, where k is the dynamic number of positive sample assignments.

6. The monocular 3D target detection method based on adaptive convolution and intelligent label assignment according to claim 5, characterized in that: The application of the Gaussian mixture model and the expectation maximization algorithm to cluster the sample points based on the evaluation value of each sample point, and finally obtain two Gaussian distributions. The sample points with evaluation values lower than the smaller mean of the two Gaussian distributions represent the positive sample candidate regions specifically as: Construct a quality evaluation index function, and the formula is c i =FL(p i ,C)+(1-IoU(b i ,B)) Among them, c i represents the quality evaluation value of the i-th sample point, p i and b i They represent the probability and bounding box of the predicted category of the point, C and B represent the true category and true bounding box of the point, and FL represents FocalLoss.

7. The monocular 3D target detection method based on adaptive convolution and intelligent label assignment according to claim 5, characterized in that: For each point in the candidate region, the cost matrix between the true bounding box and each predicted point is constructed by combining the classification loss and the regression loss: After determining a set of positive sample candidate points, the remaining points in the true box are classified as negative samples. For these negative sample points, the cost between them and the true box is set to a constant value. Specifically, the cost value is defined as C ij , represents the cost value of the i-th true box and the j-th predicted point, which is the weighted sum of the classification loss and the regression loss: Among them, Loss cls and Loss reg Respectively represent the classification loss function and regression loss function; the classification loss function is FocalLoss, and the regression loss function is the smoothL1 loss function; and They represent the prediction scores of the predicted point j under the hyperparameter of θ, and They represent the true classification and bounding box attributes of the i-th true box, respectively. The bounding box attributes include position, size, and rotation angle. α represents the weight ratio of classification and regression. Focal Loss is Loss cls (p t )=-(1-p t ) γ log(p t ) Among them, -(1-p t ) γ represents the modulation factor, and γ represents the adjustable parameter.

8. The monocular 3D target detection method based on adaptive convolution and intelligent label assignment according to claim 1, characterized in that: According to the calculated loss minimization principle, k lowest-cost positive samples are assigned to each true bounding box from the positive sample candidate area, and other samples are classified as negative samples as follows: By calculating the sum of the IoU values ​​of the predicted boxes and rounding them, the number of positive samples k that should be assigned to each true box is dynamically determined. The calculation formula is: Among them, K j represents the number of dynamic positive samples corresponding to the jth gt box, q is a predefined constant used to select the first q predicted boxes with the largest intersection-over-union ratio with each true box; represents the iou value of the i-th maximum iou prediction box in the j-th gt box, G j represents the jth gt box, P i represents the i-th predicted box.

Citation Information

Patent Citations

  • Image target detection method based on convolutional neural network

    CN114511710A

  • Laser point cloud three-dimensional target detection method based on unmanned system

    CN116758497A

  • Multi-source target detection method based on prototype network dynamic balance optimization strategy

    CN118015342A

Cited By

  • Neural network simplification optimization method and system for unmanned aerial vehicle target identification

    CN120544007A

  • A neural network simplification optimization method and system for UAV target recognition

    CN120544007B