Target detection method, device and equipment and readable storage medium
By performing feature enhancement processing on LiDAR data and combining two-dimensional convolution kernels and attention mechanisms, the low efficiency problem caused by multi-sensor fusion in intelligent driving is solved, achieving high-precision and efficient target detection.
Patent Information
- Application Number
- CN202411544642.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-18
AI Technical Summary
In current technologies for intelligent driving, the fusion of multi-sensor data results in low target detection efficiency, which fails to meet real-time requirements and makes it difficult to improve detection efficiency while ensuring accuracy.
By acquiring high-level bird's-eye view features from LiDAR data, two-dimensional convolution and pyramid pooling are performed. Feature enhancement is then achieved by combining two-dimensional convolution kernels of different sizes and an attention mechanism.
It improves the accuracy and efficiency of target detection, meets the real-time requirements of autonomous driving, and is particularly suitable for the detection of small or long-distance targets.
Smart Images
Figure CN119478361A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of target detection, and in particular to a target detection method, device, equipment and readable storage medium. Background Art
[0002] Smart cars often encounter various complex driving environments during driving, such as lighting changes, bad weather, road congestion, etc. How to autonomously avoid obstacles in such a complex driving environment to ensure driving safety is very important. Among them, object detection technology is widely used in the field of smart driving to identify and avoid obstacles because it can accurately detect obstacles including pedestrians, vehicles, buildings, etc. and determine the location, size, and moving speed of obstacles, so as to avoid collision accidents and improve driving safety.
[0003] In related technologies, in order to improve the accuracy of target detection, target detection is often performed by fusing different sensor data (such as lidar data, millimeter-wave radar data, and camera data, etc.); however, the fusion of multiple sensor data usually requires higher computing resources, which increases the system response time, and then reduces the efficiency of feature fusion, resulting in lower target detection efficiency and failure to meet the real-time requirements of autonomous driving. It can be seen that how to improve the efficiency of target detection while ensuring the accuracy of target detection is a problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a target detection method, apparatus, device and readable storage medium, which can effectively improve the efficiency of target detection while ensuring the accuracy of target detection.
[0005] In a first aspect, an embodiment of the present application provides a target detection method, the target detection method comprising:
[0006] Obtain high-level bird's-eye view features corresponding to target LiDAR data;
[0007] Perform two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features respectively to obtain the first bird's-eye view features and the second bird's-eye view features;
[0008] Performing feature enhancement on the high-level bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature;
[0009] Based on the attention mechanism, feature enhancement is performed on the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature to obtain a second enhanced bird's-eye view feature;
[0010] Performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a target enhanced feature;
[0011] Target detection is performed based on the target enhancement feature to output a target detection result.
[0012] In combination with the first aspect, in one implementation, the step of enhancing the high-level bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature includes:
[0013] Performing convolution calculations on the high-level bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a third bird's-eye view feature and a fourth bird's-eye view feature;
[0014] The third bird's-eye view feature and the fourth bird's-eye view feature are spliced to obtain a fifth bird's-eye view feature;
[0015] The fifth bird's-eye view feature and the high-level bird's-eye view feature are spliced to obtain a first enhanced bird's-eye view feature.
[0016] In combination with the first aspect, in one implementation, the step of performing feature enhancement on the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature based on the attention mechanism to obtain a second enhanced bird's-eye view feature includes:
[0017] using the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature as a query bird's-eye view feature, a key bird's-eye view feature, and a value bird's-eye view feature, respectively;
[0018] Based on the multi-head attention mechanism, the correlation of the query bird's-eye view feature, the key bird's-eye view feature and the value bird's-eye view feature is calculated to obtain a second enhanced bird's-eye view feature.
[0019] In combination with the first aspect, in one implementation, the step of performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a target enhanced feature includes:
[0020] splicing the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature to obtain a spliced bird's-eye view feature;
[0021] Performing convolution calculations on the spliced bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a sixth bird's-eye view feature and a seventh bird's-eye view feature;
[0022] The sixth bird's-eye view feature and the seventh bird's-eye view feature are spliced to obtain an eighth bird's-eye view feature;
[0023] The eighth bird's-eye view feature and the spliced bird's-eye view feature are spliced to obtain a target enhanced BEV feature.
[0024] In combination with the first aspect, in one implementation, obtaining high-level bird's-eye view features corresponding to the target laser radar data includes:
[0025] Obtaining lidar bird's-eye view features corresponding to target lidar data;
[0026] The laser radar bird's-eye view features are subjected to multi-layer two-dimensional convolution processing to obtain high-level bird's-eye view features.
[0027] In combination with the first aspect, in one implementation, before the step of acquiring high-level bird's-eye view features corresponding to the target laser radar data, the method further includes:
[0028] Get initial lidar data;
[0029] Rotating the target obstacle in the initial laser radar data to obtain new laser radar data;
[0030] The initial lidar data and the new lidar data are used as target lidar data.
[0031] In combination with the first aspect, in one implementation, before the step of acquiring high-level bird's-eye view features corresponding to the target laser radar data, the method further includes:
[0032] Get initial lidar data;
[0033] Filtering out a plurality of target laser radar data from the initial laser radar data, wherein the diagonal length of the target object in the target laser radar data is greater than a preset diagonal length threshold;
[0034] For each target laser radar data, obtaining a background area from the target laser radar data;
[0035] Filtering a preset number of obstacle labels from the initial laser radar data, and adding the obstacle labels to the background area to generate new laser radar data;
[0036] The initial lidar data and the new lidar data are used as target lidar data.
[0037] In a second aspect, an embodiment of the present application provides a target detection device, the target detection device comprising:
[0038] A feature acquisition module, which is used to acquire high-level bird's-eye view features corresponding to the target lidar data;
[0039] A feature processing module, which is used to perform two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features to obtain a first bird's-eye view feature and a second bird's-eye view feature;
[0040] A feature enhancement module, which is used to perform feature enhancement on the high-level bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature; perform feature enhancement on the high-level bird's-eye view feature, the first bird's-eye view feature and the second bird's-eye view feature based on an attention mechanism to obtain a second enhanced bird's-eye view feature; perform feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a target enhanced feature;
[0041] The target detection module is used to perform target detection based on the target enhancement feature to output a target detection result.
[0042] In conjunction with the second aspect, in one implementation, the feature enhancement module is specifically used to:
[0043] Performing convolution calculations on the high-level bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a third bird's-eye view feature and a fourth bird's-eye view feature;
[0044] The third bird's-eye view feature and the fourth bird's-eye view feature are spliced to obtain a fifth bird's-eye view feature;
[0045] The fifth bird's-eye view feature and the high-level bird's-eye view feature are spliced to obtain a first enhanced bird's-eye view feature.
[0046] In conjunction with the second aspect, in one implementation, the feature enhancement module is further configured to:
[0047] using the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature as a query bird's-eye view feature, a key bird's-eye view feature, and a value bird's-eye view feature, respectively;
[0048] Based on the multi-head attention mechanism, the correlation of the query bird's-eye view feature, the key bird's-eye view feature and the value bird's-eye view feature is calculated to obtain a second enhanced bird's-eye view feature.
[0049] In conjunction with the second aspect, in one implementation, the feature enhancement module is further configured to:
[0050] splicing the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature to obtain a spliced bird's-eye view feature;
[0051] Performing convolution calculations on the spliced bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a sixth bird's-eye view feature and a seventh bird's-eye view feature;
[0052] The sixth bird's-eye view feature and the seventh bird's-eye view feature are spliced to obtain an eighth bird's-eye view feature;
[0053] The eighth bird's-eye view feature and the spliced bird's-eye view feature are spliced to obtain a target enhanced BEV feature.
[0054] In conjunction with the second aspect, in one implementation, the feature acquisition module is specifically used to:
[0055] Obtaining lidar bird's-eye view features corresponding to target lidar data;
[0056] The laser radar bird's-eye view features are subjected to multi-layer two-dimensional convolution processing to obtain high-level bird's-eye view features.
[0057] In conjunction with the second aspect, in one implementation, the feature acquisition module is further used to:
[0058] Get initial lidar data;
[0059] Rotating the target obstacle in the initial laser radar data to obtain new laser radar data;
[0060] The initial lidar data and the new lidar data are used as target lidar data.
[0061] In conjunction with the second aspect, in one implementation, the feature acquisition module is further used to:
[0062] Get initial lidar data;
[0063] Filtering out a plurality of target laser radar data from the initial laser radar data, wherein the diagonal length of the target object in the target laser radar data is greater than a preset diagonal length threshold;
[0064] For each target laser radar data, obtaining a background area from the target laser radar data;
[0065] Filtering a preset number of obstacle labels from the initial laser radar data, and adding the obstacle labels to the background area to generate new laser radar data;
[0066] The initial lidar data and the new lidar data are used as target lidar data.
[0067] In a third aspect, an embodiment of the present application provides a target detection device, which includes a processor, a memory, and a target detection program stored in the memory and executable by the processor, wherein when the target detection program is executed by the processor, the steps of the target detection method as described above are implemented.
[0068] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a target detection program is stored, wherein when the target detection program is executed by a processor, the steps of the target detection method as described above are implemented.
[0069] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:
[0070] By obtaining high-resolution high-level bird's-eye view features corresponding to the target laser radar data, and performing two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features respectively, a first bird's-eye view feature and a second bird's-eye view feature with better local perception ability are obtained; then the high-level bird's-eye view features are enhanced by two-dimensional convolution kernels of different sizes to obtain the first enhanced bird's-eye view feature that integrates different feature extraction capabilities, and the high-level bird's-eye view features, the first bird's-eye view features and the second bird's-eye view features are enhanced based on the attention mechanism to obtain the second enhanced bird's-eye view feature, that is, the feature enhancement is performed with the help of the correlation between different BEV features to improve the feature expression ability; then the first enhanced bird's-eye view features and the second enhanced bird's-eye view features are enhanced by two-dimensional convolution kernels of different sizes, and the target enhanced features that integrate more different feature extraction capabilities are further obtained, and finally accurate target detection is achieved based on the target enhanced features. It can be seen that the present application uses convolution calculations with different two-dimensional convolution kernels and attention mechanisms to perform feature enhancement to improve the accuracy of target detection, and only realizes target detection based on laser radar data without the need to fuse different sensor data, thereby effectively improving the efficiency of target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 This is a flow chart of an embodiment of the target detection method of the present application;
[0072] Figure 2 A schematic diagram of label data rotation involved in the embodiment of the present application;
[0073] Figure 3 A schematic diagram for generating background area labels involved in the embodiment of the present application;
[0074] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the target detection device of the present application;
[0075] Figure 5 This is a schematic diagram of the hardware structure of the target detection device involved in the embodiment of the present application. DETAILED DESCRIPTION
[0076] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0077] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.
[0078] In a first aspect, an embodiment of the present application provides a target detection method.
[0079] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the target detection method of this application. Figure 1 As shown, the target detection method includes:
[0080] Step S10: Obtain high-level bird's-eye view features corresponding to the target lidar data.
[0081] Exemplarily, it should be understood that the target LiDAR data refers to the LiDAR data used to achieve target detection. In this embodiment, in order to reduce the computing resources required for data fusion, target detection will be achieved only based on the LiDAR data to reduce the system response time, thereby effectively improving the efficiency of target detection. Therefore, this embodiment only needs to obtain the BEV (Bird's Eyes View) features corresponding to the LiDAR data; however, in order to further improve the accuracy of target detection, high-level BEV features with high resolution will be obtained as the basic data for target detection. It should be noted that the method and principle of how to generate BEV features based on LiDAR data are common knowledge in the field, and for the sake of brevity of description, they will not be repeated here.
[0082] Further, in one embodiment, the step of obtaining high-level bird's-eye view features corresponding to the target laser radar data includes:
[0083] Obtaining lidar bird's-eye view features corresponding to target lidar data;
[0084] The laser radar bird's-eye view features are subjected to multi-layer two-dimensional convolution processing to obtain high-level bird's-eye view features.
[0085] Exemplarily, for the target LiDAR data, this embodiment may preferably use a lightweight PIXOR network to process the target LiDAR data to obtain compressed LiDAR BEV features, thereby improving computing efficiency and effectively reducing the number of parameters; it should be noted that PIXOR is a network architecture for real-time 3D target detection, which is suitable for processing point cloud data such as lidar, and mainly includes a backbone network responsible for extracting features from the input point cloud data and a detection head responsible for target detection based on the features extracted by the backbone network. Among them, the specific method and principle of how to generate BEV features after processing point cloud data such as lidar through the PIXOR network is common knowledge in the field, and for the sake of brevity of description, it will not be repeated here.
[0086] Then, multi-layer two-dimensional convolution is performed on the LiDAR BEV features to extract spatial features and semantic features layer by layer. That is, low-level spatial features (such as edges, textures, etc.) are mainly extracted in the early convolution layers, and high-level semantic features (such as object categories, positions, etc.) are gradually extracted in the later convolution layers, thereby obtaining LiDAR BEV features with high resolution. This part of the features is called high-level BEV features and is defined as B hight It should be noted that the specific method and principle of how to obtain high-resolution features through multi-layer two-dimensional convolution calculations are common knowledge in the art, and for the sake of brevity, they will not be repeated here.
[0087] Step S20: performing two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features respectively to obtain a first bird's-eye view feature and a second bird's-eye view feature.
[0088] Exemplarily, for high-level BEV feature B hight , this embodiment will continue to use two-bit convolution for encoding to obtain new BEV features, but in this process, the BEV features will gradually blur the degree of attention to objects in the scene; therefore, considering this problem, this embodiment will use pyramid pooling (PP) to pool the new BEV features to obtain pooled BEV features, so as to ensure that this part of the features has better local perception ability and can reduce the amount of calculation during feature interaction; it can be understood that the local perception ability can be understood as having a stronger degree of attention to the local area of the BEV feature, rather than paying attention to the entire BEV feature like the high-level BEV features mentioned above.
[0089] Specifically, in this embodiment, the new BEV features are subjected to pyramid pooling twice to obtain two groups of pooled BEV features, that is, the first BEV feature is obtained after pyramid pooling of the new BEV features. At the same time, the new BEV feature is pyramid pooled to obtain the second BEV feature. Only the pooled features are subsequently calculated to effectively reduce the amount of calculation. It should be noted that the working method and principle of pyramid pooling are common knowledge in the art, and for the sake of brevity, they will not be repeated here.
[0090] Step S30: Enhance the high-level bird's-eye view features by using two-dimensional convolution kernels of different sizes to obtain first enhanced bird's-eye view features.
[0091] For example, in this embodiment, in order to improve the perception ability of BEV features as much as possible, the high-level BEV features B are respectively fused by two-dimensional convolution kernels of different sizes. hight Perform two-dimensional convolution processing to obtain BEV features with different feature extraction capabilities, and concatenate the BEV features to obtain the first enhanced BEV feature that combines different feature extraction capabilities. In order to achieve feature enhancement, thus laying the foundation for improving target detection accuracy. It should be noted that the size of different two-dimensional convolution kernels can be determined according to actual needs and is not limited here.
[0092] Further, in one embodiment, the step of enhancing the high-level bird's-eye view features by using two-dimensional convolution kernels of different sizes to obtain the first enhanced bird's-eye view features includes:
[0093] Performing convolution calculations on the high-level bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a third bird's-eye view feature and a fourth bird's-eye view feature;
[0094] The third bird's-eye view feature and the fourth bird's-eye view feature are spliced to obtain a fifth bird's-eye view feature;
[0095] The fifth bird's-eye view feature and the high-level bird's-eye view feature are spliced to obtain a first enhanced bird's-eye view feature.
[0096] For example, in this embodiment, one of the core ideas of feature enhancement is to use two sets of convolution calculations with different two-dimensional convolution kernels to obtain BEV features with different feature extraction capabilities, and then splice the two sets of features again to complete feature enhancement. Specifically, two sets of two-dimensional convolution kernels of different sizes are used to perform convolution calculations on the high-level BEV features B and B respectively. hight Convolution calculation is performed to obtain two sets of third BEV features and fourth BEV features with different feature extraction capabilities; the third BEV features and the fourth BEV features are then concatenated to obtain enhanced BEV features (i.e., the fifth BEV features) to complete the enhancement of high-level BEV features; finally, the fifth BEV features and the high-level BEV features are concatenated to obtain the final first enhanced BEV features.
[0097] Step S40: Based on the attention mechanism, feature enhancement is performed on the high-level bird's-eye view feature, the first bird's-eye view feature and the second bird's-eye view feature to obtain a second enhanced bird's-eye view feature.
[0098] Exemplarily, it can be understood that after completing the above feature extraction, the high-level BEV feature B hight and the first BEV features of the two groups after pyramid pooling With the second BEV features This embodiment will fuse these three sets of features; among them, for these three sets of features, in order to improve the perception ability of BEV features as much as possible, not only the convolution calculation method of two sets of two-dimensional convolution kernels of different sizes is adopted, but also the attention mechanism is used to enhance the attention of BEV features. Specifically, based on the attention mechanism, the high-level BEV feature B hight 1. BEV Features and second BEV features The correlation between the three is calculated, that is, the feature enhancement is performed with the help of the correlation between different BEV features to obtain the second enhanced BEV feature It can be seen that when using attention calculation, this embodiment does not calculate all features, but only performs attention calculation on the pooled features, so the amount of calculation during attention calculation can be effectively reduced, thereby effectively improving the efficiency of feature fusion.
[0099] Further, in one embodiment, the feature enhancement of the high-level bird's-eye view feature, the first bird's-eye view feature and the second bird's-eye view feature based on the attention mechanism to obtain the second enhanced bird's-eye view feature includes:
[0100] using the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature as a query bird's-eye view feature, a key bird's-eye view feature, and a value bird's-eye view feature, respectively;
[0101] Based on the multi-head attention mechanism, the correlation of the query bird's-eye view feature, the key bird's-eye view feature and the value bird's-eye view feature is calculated to obtain a second enhanced bird's-eye view feature.
[0102] For example, in this embodiment, another core idea of feature enhancement is to enhance the attention of features through the attention mechanism, wherein the MultiHeadSelfAttention (mhsa) mechanism in the Transformer network model can be preferably used to achieve attention enhancement. Specifically, firstly, the high-level BEV feature B hightDefined as the query BEV feature Query and the first BEV feature in the Transformer network model Defined as the key BEV feature Key and the second BEV feature in the Transformer network model Defined as the value BEV feature Value in the Transformer network model; then use multi-head self-attention to calculate the correlation between Query, Key and Value to obtain the second enhanced BEV feature
[0103]
[0104] In the formula, MultiHeadSelfAttention() represents multi-head attention calculation. It should be noted that its specific working method and principle are common knowledge in the field and will not be repeated here for the sake of brevity.
[0105] Step S50: performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain target enhanced features.
[0106] Exemplarily, in this embodiment, for the first enhanced BEV characteristic calculated in the above steps and second enhanced BEV features The perception ability of BEV features will be further improved through convolution calculations of two-dimensional convolution kernels of different sizes, that is, the first enhanced BEV feature is and second enhanced BEV features Perform two-dimensional convolution processing to obtain the target enhanced feature B that integrates more different feature extraction capabilities final , in order to further enhance the features.
[0107] Further, in one embodiment, the step of performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a target enhanced feature includes:
[0108] splicing the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature to obtain a spliced bird's-eye view feature;
[0109] Performing convolution calculations on the spliced bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a sixth bird's-eye view feature and a seventh bird's-eye view feature;
[0110] The sixth bird's-eye view feature and the seventh bird's-eye view feature are spliced to obtain an eighth bird's-eye view feature;
[0111] The eighth bird's-eye view feature and the spliced bird's-eye view feature are spliced to obtain a target enhanced BEV feature.
[0112] Exemplarily, in this embodiment, the first enhanced BEV characteristic is obtained and second enhanced BEV features After that, the two are first concatenated to obtain the concatenated BEV feature; then the concatenated BEV feature is convolved by two groups of two-dimensional convolution kernels of different sizes to obtain two groups of sixth BEV features and seventh BEV features with different feature extraction capabilities; the sixth BEV feature and the seventh BEV feature are then concatenated to obtain the enhanced feature (i.e., the eighth BEV feature), and finally the eighth BEV feature and the concatenated BEV feature are concatenated to calculate the final target enhanced BEV feature B final It can be seen that this embodiment only uses a simple splicing and convolution method to perform feature enhancement, thereby effectively improving the efficiency of feature fusion.
[0113] Step S60: performing target detection based on the target enhancement feature to output a target detection result.
[0114] Exemplarily, in this embodiment, after the target enhanced BEV feature B is finally enhanced final After that, the BEV feature B can be enhanced through this target final To accurately detect the target, and then output high-precision target detection results. In addition, in order to achieve efficient lidar detection, this embodiment can preferably use the detection head of the YoloX target detection algorithm to enhance the BEV feature B of the target. final Perform the final object detection prediction; it should be noted that the specific working method and principle of the detection head of the YoloX target detection algorithm are common knowledge in the field, and for the sake of brevity of description, they will not be repeated here.
[0115] It can be seen that this embodiment realizes target detection based only on lidar data without fusing different sensor data, thereby effectively improving the efficiency of target detection; and through optimized feature fusion and attention mechanism, that is, through convolution calculation and attention mechanism with different two-dimensional convolution kernels, feature enhancement is performed to improve the overall 3D and other target detection accuracy, which is especially suitable for small-sized or long-distance target detection; in addition, with the help of efficient YoloX detection head prediction method to optimize the detection framework, the detection speed can be effectively improved to improve real-time performance, thereby meeting the needs of real-time processing of autonomous driving and enhancing the responsiveness of the system. It should be noted that the feature enhancement method proposed in this embodiment is not only applicable to radar BEV features, but also to BEV features such as cameras. It can be adapted and integrated into most of the current mainstream BEV perception frameworks, and has high compatibility and application potential.
[0116] Furthermore, in one embodiment, before the step of obtaining high-level bird's-eye view features corresponding to the target laser radar data, the step further includes:
[0117] Get initial lidar data;
[0118] Rotating the target obstacle in the initial laser radar data to obtain new laser radar data;
[0119] The initial lidar data and the new lidar data are used as target lidar data.
[0120] By way of example, it is understandable that the current target detection methods based on lidar often have the problem of sample imbalance, and sample imbalance usually leads to low detection accuracy and overall imbalance in data distribution; therefore, this embodiment specifically designs a method based on rotated annotation data expansion to optimize the sample imbalance problem, that is, by rotating samples of target obstacles such as pedestrians and objects around the center of the lidar coordinate system to generate new and diverse samples. This method not only increases the number of obstacle samples, but also enhances the model's generalization ability to obstacle posture and position changes by introducing rotation changes in the samples.
[0121] For details, see Figure 2 As shown, the rotation annotation data expansion method provided in this embodiment preferably adopts a symmetrical division method for rotation, wherein: Figure 2 φ1 and φ2 represent the angular deviation of the original target obstacle relative to the origin and the rotation of the x-axis of the LiDAR coordinate system, respectively; φ3 and φ4 represent the angles between the original target obstacle and the rotation target obstacle and the y-axis of the LiDAR coordinate system; φ5 and φ6 represent the angles between the extension of the horizontal reference line of the original target obstacle and the rotation target obstacle and the y-axis of the LiDAR coordinate system, respectively. According to the similar angle calculation principle of geometric mathematics, the following expression can be obtained:
[0122] φ1+φ2=φ3+φ4=φ5+φ6
[0123] After geometric symmetry analysis, the rotated label data and rotated point data can be calculated using the following rotation plane calculation method, and the rotated label data and point data can be used for data expansion:
[0124]
[0125] In the formula, x and y in (x, y, 1) represent the horizontal and vertical coordinates before rotation, respectively, "1" means no rotation in the height direction, and (x′, y′) represents the horizontal and vertical coordinates after rotation.
[0126] Therefore, after acquiring the initial LiDAR data, this embodiment will rotate the target obstacle in the initial LiDAR data according to the above-mentioned rotation principle to obtain the rotated LiDAR data and the corresponding rotated label data to form new LiDAR data; then the newly obtained LiDAR data is added to the original dataset, that is, the initial LiDAR data and the new LiDAR data are used as target LiDAR data to increase the sample multi-sample, thereby realizing data enhancement and expansion, obtaining a more comprehensive data distribution, avoiding the problem of sample imbalance, thereby effectively improving the detection accuracy, especially for small targets such as pedestrians, and effectively improving the generalization ability of the model.
[0127] It can be seen that after obtaining the rotated LiDAR data and the corresponding rotated label data, the newly obtained data is added to the original data set to complete the data augmentation.
[0128] This embodiment increases the diversity of samples through the rotated ground truth sample enhancement (RG-Aug) method, effectively balancing the number of samples of different categories, especially increasing the samples of small targets such as pedestrians, reducing the overfitting phenomenon of the model for vehicles, and effectively improving the detection performance of small targets, thereby improving the problem of insufficient detection performance caused by sample imbalance in the scene.
[0129] Furthermore, in one embodiment, before the step of obtaining high-level bird's-eye view features corresponding to the target laser radar data, the step further includes:
[0130] Get initial lidar data;
[0131] Filtering out a plurality of target laser radar data from the initial laser radar data, wherein the diagonal length of the target object in the target laser radar data is greater than a preset diagonal length threshold;
[0132] For each target laser radar data, obtaining a background area from the target laser radar data;
[0133] Filtering a preset number of obstacle labels from the initial laser radar data, and adding the obstacle labels to the background area to generate new laser radar data;
[0134] The initial lidar data and the new lidar data are used as target lidar data.
[0135] Exemplarily, it is understandable that LiDAR data sampling is completed by emitting laser and obtaining laser reflected by the object, and for the background area of the LiDAR sampling area, the existing methods ignore the data enhancement and data augmentation of the background area. Based on this, this embodiment designs a label sample generation strategy for the background area to achieve the augmentation of the background area data to help the network recognize the objects behind the detected object. It should be noted that when performing target detection processing, only one of the rotation annotation data expansion method and the label sample generation strategy for the background area can be implemented, or both can be implemented, and the implementation order of the two can be determined according to actual needs and is not limited here.
[0136] Specifically, after acquiring the initial LiDAR data, the present embodiment includes labeled data such as cars, trucks, and engineering vehicles. Therefore, the target LiDAR data can be screened out according to the diagonal size of the target objects such as cars, trucks, and engineering vehicles. That is, only the LiDAR data of the target objects whose diagonal length is greater than the diagonal length threshold are used as the target LiDAR data. It should be noted that the specific value of the diagonal length threshold can be determined according to actual needs and is not limited here. For example, by statistically analyzing the data of pedestrians in the data set, the diagonal length threshold is set to 1.5 meters, which can ensure that pedestrians can be in the background area without causing data pollution to the initial data set; for example, see Figure 3 As shown, since the diagonal length of the truck is greater than 1.5 meters, the LiDAR data where the truck is located is used as the target LiDAR data; and since the diagonal length of the smaller car is less than 1.5 meters, the LiDAR data where the car is located is not used as the target LiDAR data.
[0137] Then, the background area is acquired and labels are generated for each target LiDAR data. For example, background area 1 where the truck is located is acquired and labels are generated, but background area 2 where the car is located is not acquired, that is, background area 2 is discarded, that is, no labels are generated for background area 2. When generating background area labels, it is necessary to first summarize the data of pedestrians and other obstacles that have been annotated in the current data set to count the size of the labels and generate pedestrian and other obstacle labels that conform to the data distribution; then a preset number of obstacle labels (the specific value can be determined according to actual needs) are selected from them, and the obstacle labels are added to the background area to generate new LiDAR data; for example, see Figure 3 As shown, 2 to 3 pedestrian labels are randomly selected and added to the background area 2 to generate new label data.
[0138] Finally, the new LiDAR data is added to the original data set, that is, the new LiDAR data and the initial LiDAR data are used as the target LiDAR data to complete the generation of label samples in the background area, thereby achieving data enhancement and data augmentation in the background area. It can be seen that this embodiment completes the LiDAR-based autonomous driving scene perception detection from three aspects: data enhancement method, feature extraction and fusion, and efficient detection framework based on LiDAR, which significantly improves the performance of the detection system in terms of real-time response, accuracy and robustness, and especially optimizes the detection capabilities of small targets and complex scenes, meeting the strict requirements of the field of autonomous driving for high-precision environmental perception.
[0139] In a second aspect, an embodiment of the present application also provides a target detection device.
[0140] In one embodiment, referring to Figure 4 , Figure 4 Schematic diagram of the functional modules of the target detection device embodiment of the present application. Figure 4 As shown, the target detection device includes:
[0141] A feature acquisition module, which is used to acquire high-level bird's-eye view features corresponding to the target lidar data;
[0142] A feature processing module, which is used to perform two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features to obtain a first bird's-eye view feature and a second bird's-eye view feature;
[0143] A feature enhancement module, which is used to perform feature enhancement on the high-level bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature; perform feature enhancement on the high-level bird's-eye view feature, the first bird's-eye view feature and the second bird's-eye view feature based on an attention mechanism to obtain a second enhanced bird's-eye view feature; perform feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a target enhanced feature;
[0144] The target detection module is used to perform target detection based on the target enhancement feature to output a target detection result.
[0145] Furthermore, in one embodiment, the feature enhancement module is specifically used to:
[0146] Performing convolution calculations on the high-level bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a third bird's-eye view feature and a fourth bird's-eye view feature;
[0147] The third bird's-eye view feature and the fourth bird's-eye view feature are spliced to obtain a fifth bird's-eye view feature;
[0148] The fifth bird's-eye view feature and the high-level bird's-eye view feature are spliced to obtain a first enhanced bird's-eye view feature.
[0149] Furthermore, in one embodiment, the feature enhancement module is further configured to:
[0150] using the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature as a query bird's-eye view feature, a key bird's-eye view feature, and a value bird's-eye view feature, respectively;
[0151] Based on the multi-head attention mechanism, the correlation of the query bird's-eye view feature, the key bird's-eye view feature and the value bird's-eye view feature is calculated to obtain a second enhanced bird's-eye view feature.
[0152] Furthermore, in one embodiment, the feature enhancement module is further configured to:
[0153] splicing the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature to obtain a spliced bird's-eye view feature;
[0154] Performing convolution calculations on the spliced bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a sixth bird's-eye view feature and a seventh bird's-eye view feature;
[0155] The sixth bird's-eye view feature and the seventh bird's-eye view feature are spliced to obtain an eighth bird's-eye view feature;
[0156] The eighth bird's-eye view feature and the spliced bird's-eye view feature are spliced to obtain a target enhanced BEV feature.
[0157] Furthermore, in one embodiment, the feature acquisition module is specifically used for:
[0158] Obtaining lidar bird's-eye view features corresponding to target lidar data;
[0159] The laser radar bird's-eye view features are subjected to multi-layer two-dimensional convolution processing to obtain high-level bird's-eye view features.
[0160] Furthermore, in one embodiment, the feature acquisition module is also used for:
[0161] Get initial lidar data;
[0162] Rotating the target obstacle in the initial laser radar data to obtain new laser radar data;
[0163] The initial lidar data and the new lidar data are used as target lidar data.
[0164] Furthermore, in one embodiment, the feature acquisition module is also used for:
[0165] Get initial lidar data;
[0166] Filtering out a plurality of target laser radar data from the initial laser radar data, wherein the diagonal length of the target object in the target laser radar data is greater than a preset diagonal length threshold;
[0167] For each target laser radar data, obtaining a background area from the target laser radar data;
[0168] Filtering a preset number of obstacle labels from the initial laser radar data, and adding the obstacle labels to the background area to generate new laser radar data;
[0169] The initial lidar data and the new lidar data are used as target lidar data.
[0170] Among them, the functional implementation of each module in the above-mentioned target detection device corresponds to the various steps in the above-mentioned target detection method embodiment, and its functions and implementation processes are no longer repeated here one by one.
[0171] In a third aspect, an embodiment of the present application provides a target detection device, which may be a personal computer (PC), a laptop computer, a server, or other device with a data processing function.
[0172] Reference Figure 5 , Figure 5 Schematic diagram of the hardware structure of the target detection device involved in the embodiment of the present application. In the embodiment of the present application, the target detection device may include a processor, a memory, a communication interface and a communication bus.
[0173] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.
[0174] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the target detection device, and the interface used to realize the interconnection between the target detection device and other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.
[0175] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0176] The processor may be a general-purpose processor, and the general-purpose processor may call the target detection program stored in the memory and execute the target detection method provided by the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the target detection program is called may refer to the various embodiments of the target detection method of the present application, and will not be repeated here.
[0177] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0178] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.
[0179] The readable storage medium of the present application stores a target detection program, wherein when the target detection program is executed by a processor, the steps of the target detection method as described above are implemented.
[0180] Among them, the method implemented when the target detection program is executed can refer to the various embodiments of the target detection method of the present application, and will not be repeated here.
[0181] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.
[0182] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.
[0183] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.
[0184] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.
[0185] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.
[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.
[0187] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A target detection method, characterized in that: The target detection method comprises: Obtain high-level bird's-eye view features corresponding to target LiDAR data; Perform two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features respectively to obtain the first bird's-eye view features and the second bird's-eye view features; Performing feature enhancement on the high-level bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature; Based on the attention mechanism, feature enhancement is performed on the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature to obtain a second enhanced bird's-eye view feature; Performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a target enhanced feature; Target detection is performed based on the target enhancement feature to output a target detection result.
2. The target detection method according to claim 1, characterized in that: The step of enhancing the high-level bird's-eye view features by using two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature includes: Performing convolution calculations on the high-level bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a third bird's-eye view feature and a fourth bird's-eye view feature; The third bird's-eye view feature and the fourth bird's-eye view feature are spliced to obtain a fifth bird's-eye view feature; The fifth bird's-eye view feature and the high-level bird's-eye view feature are spliced to obtain a first enhanced bird's-eye view feature.
3. The target detection method according to claim 1, characterized in that: The step of performing feature enhancement on the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature based on the attention mechanism to obtain a second enhanced bird's-eye view feature includes: using the high-level bird's-eye view feature, the first bird's-eye view feature, and the second bird's-eye view feature as a query bird's-eye view feature, a key bird's-eye view feature, and a value bird's-eye view feature, respectively; Based on the multi-head attention mechanism, the correlation of the query bird's-eye view feature, the key bird's-eye view feature and the value bird's-eye view feature is calculated to obtain a second enhanced bird's-eye view feature.
4. The target detection method according to claim 1, characterized in that: The step of performing feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature by using two-dimensional convolution kernels of different sizes to obtain a target enhanced feature includes: splicing the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature to obtain a spliced bird's-eye view feature; Performing convolution calculations on the spliced bird's-eye view features respectively through two groups of two-dimensional convolution kernels of different sizes to obtain a sixth bird's-eye view feature and a seventh bird's-eye view feature; The sixth bird's-eye view feature and the seventh bird's-eye view feature are spliced to obtain an eighth bird's-eye view feature; The eighth bird's-eye view feature and the spliced bird's-eye view feature are spliced to obtain a target enhanced BEV feature.
5. The target detection method according to claim 1, characterized in that: The step of obtaining high-level bird's-eye view features corresponding to target laser radar data includes: Obtaining lidar bird's-eye view features corresponding to target lidar data; The laser radar bird's-eye view features are subjected to multi-layer two-dimensional convolution processing to obtain high-level bird's-eye view features.
6. The target detection method according to claim 1, characterized in that: Before the step of obtaining high-level bird's-eye view features corresponding to the target laser radar data, the method further includes: Get initial lidar data; Rotating the target obstacle in the initial laser radar data to obtain new laser radar data; The initial lidar data and the new lidar data are used as target lidar data.
7. The target detection method according to claim 1, characterized in that: Before the step of obtaining high-level bird's-eye view features corresponding to the target laser radar data, the method further includes: Get initial lidar data; Filtering out a plurality of target laser radar data from the initial laser radar data, wherein the diagonal length of the target object in the target laser radar data is greater than a preset diagonal length threshold; For each target laser radar data, obtaining a background area from the target laser radar data; Filtering a preset number of obstacle labels from the initial laser radar data, and adding the obstacle labels to the background area to generate new laser radar data; The initial lidar data and the new lidar data are used as target lidar data.
8. A target detection device, characterized in that: The target detection device comprises: A feature acquisition module, which is used to acquire high-level bird's-eye view features corresponding to the target lidar data; A feature processing module, which is used to perform two-dimensional convolution and pyramid pooling processing on the high-level bird's-eye view features to obtain a first bird's-eye view feature and a second bird's-eye view feature; A feature enhancement module, which is used to perform feature enhancement on the high-level bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a first enhanced bird's-eye view feature; perform feature enhancement on the high-level bird's-eye view feature, the first bird's-eye view feature and the second bird's-eye view feature based on an attention mechanism to obtain a second enhanced bird's-eye view feature; perform feature enhancement on the first enhanced bird's-eye view feature and the second enhanced bird's-eye view feature through two-dimensional convolution kernels of different sizes to obtain a target enhanced feature; The target detection module is used to perform target detection based on the target enhancement feature to output a target detection result.
9. A target detection device, characterized in that: The target detection device includes a processor, a memory, and a target detection program stored in the memory and executable by the processor, wherein when the target detection program is executed by the processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a target detection program, wherein when the target detection program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.