Driving scene-based target detection method, device, equipment, medium and program product

Abstract

This application provides a target detection method, apparatus, device, medium, and program product based on driving scenarios, relating to the field of 3D target detection technology. The method includes: acquiring 3D point cloud data of a driving scenario and preprocessing the 3D point cloud data; establishing a training dataset based on the preprocessed 3D point cloud data and an autonomous driving scenario target detection database; training and validating the training dataset to obtain a feature extraction network model; performing target detection and feature extraction on the 3D point cloud data by fusing multiple feature extraction network models and combining multi-scale feature fusion; proposing a network feature fusion method to enhance the ability to extract detailed features of targets, which helps the network learn more discriminative features and can refine the accuracy of position prediction, thereby improving the overall detection effect.

Description

Technical Field

[0001] This application relates to the field of three-dimensional target detection technology, and more specifically, to a target detection method, device, equipment, medium, and program product based on driving scenarios. Background Technology

[0002] 3D point cloud data can typically be acquired directly from LiDAR cameras, with the overall dataset built through manual data annotation. However, existing voxel-based convolutional methods suffer from feature loss during point cloud voxelization, affecting the network's feature extraction for smaller objects. Furthermore, these models rely on manually set voxel sampling volumes; larger volumes can lead to loss of detail information, impacting object position estimation. Therefore, many methods require further refinement of voxel volumes to improve the network's representational capabilities, but this also increases computational complexity. Point-based learning methods, due to feature extraction for each point, have high overall computational costs and are time-consuming to implement. Summary of the Invention

[0003] The purpose of this application is to provide a target detection method, device, equipment, medium, and program product based on driving scenarios, so as to solve the problems of poor feature extraction and prediction accuracy and high computational cost of existing target detection methods.

[0004] In a first aspect, embodiments of this application provide a target detection method based on a driving scenario, including:

[0005] Acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data;

[0006] A training dataset is established based on the preprocessed 3D point cloud data and the target detection database for autonomous driving scenarios.

[0007] The feature extraction network model is obtained by training and validating the training dataset.

[0008] By fusing multiple feature extraction network models and combining multi-scale feature fusion, target detection and feature extraction are performed on 3D point cloud data.

[0009] In the above implementation process, this application embodiment acquires three-dimensional point cloud data of a driving scene and preprocesses the three-dimensional point cloud data; based on the preprocessed three-dimensional point cloud data and combined with an autonomous driving scene target detection database, a training dataset is established; the training dataset is trained and validated to obtain a feature extraction network model; through the fusion of multiple feature extraction network models and combined with multi-scale feature fusion, target detection and feature extraction are performed on the three-dimensional point cloud data; a network feature fusion method is proposed to enhance the ability to extract detailed features of the target, which helps the network learn more discriminative features and can refine the accuracy of position prediction, thereby improving the overall detection effect.

[0010] Furthermore, the acquisition of 3D point cloud data of the driving scene and the preprocessing of the 3D point cloud data include:

[0011] Three-dimensional point cloud data of the driving scene is acquired, and the farthest point sampling technology is used to perform preliminary sampling on the three-dimensional point cloud data.

[0012] In the above implementation process, by using the farthest point sampling technique, data points are filtered while ensuring the correct depiction of the object's shape. This can better preserve the object's shape characteristics, reduce the loss of texture detail information, and avoid destroying the object's spatial structure and orientation information.

[0013] Furthermore, the method of fusing multiple feature extraction network models and combining multi-scale feature fusion to perform target detection and feature extraction on 3D point cloud data includes:

[0014] A voxelized convolutional network model is used to fuse a point cloud feature learning network model to extract key point features from 3D point cloud data. This is combined with multi-scale feature fusion to perform feature fusion.

[0015] In the above implementation process, the feature extraction of the voxelized convolutional network can provide certain prior information for the point cloud feature learning network, thus accelerating the computation speed; the effective combination of deep and shallow neural networks reduces the influence of background interference and scale.

[0016] Furthermore, the method employs a voxelized convolutional network model fused with a point cloud feature learning network model to extract key point features from the 3D point cloud data, and combines multi-scale feature fusion to perform feature fusion, including:

[0017] The 3D point cloud data is processed to obtain voxelized point cloud data;

[0018] A voxelized convolutional network model is obtained by training on voxelized point cloud data;

[0019] The convolutional features trained by the voxelized convolutional network model are processed by the point cloud feature learning network model and combined with multi-scale feature fusion to extract key point features.

[0020] The key point features are fused based on the voxelized convolutional network model, and the shallow feature information is fused with the deep features of the deep network.

[0021] In the above implementation process, the feature extraction of the voxelized convolutional network can provide certain prior information for the point cloud feature learning network, thus accelerating the computation speed; the effective combination of deep and shallow neural networks reduces the influence of background interference and scale.

[0022] Furthermore, the step of training and validating the training dataset to obtain the feature extraction network model includes:

[0023] Using single-stage detection technology as the basic framework, the feature extraction network model is obtained by improving the location box generation method, utilizing prior 3D object location information as an aid, and adopting a multi-scale target prediction method to train the training dataset.

[0024] The training dataset is validated to generate candidate location boxes at multiple scales.

[0025] In the above implementation process, a single-stage detection method is used as the basic framework, that is, only one regression and classification calculation of the location box is performed. However, by improving the location box generation method, prior 3D object position information is used as an aid, and candidate location boxes are generated in the 3D data space to reduce feature loss during feature transformation. A multi-scale target prediction method is adopted to make the overall detection model more sensitive to changes in object scale and improve detection accuracy.

[0026] Furthermore, it also includes:

[0027] The feature extraction network model is tested in real-world scenarios, and feedback results are received. The feature extraction network model is then adjusted based on the feedback results.

[0028] In the above implementation process, the feature extraction network model was tested in real-world scenarios to verify the model, and the method was optimized and adjusted based on the feedback results.

[0029] Secondly, embodiments of this application provide a target detection device based on a driving scenario, comprising:

[0030] The data acquisition module is used to acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data;

[0031] The data processing module is used to build a training dataset based on the preprocessed 3D point cloud data and the autonomous driving scene target detection database.

[0032] The model training module is used to train and validate the training dataset to obtain the feature extraction network model;

[0033] The target detection module is used to perform target detection and feature extraction on 3D point cloud data by fusing multiple feature extraction network models and combining multi-scale feature fusion.

[0034] Thirdly, embodiments of this application provide an electronic device, including:

[0035] The system includes a processor, a memory, and a bus. The processor is connected to the memory via the bus. The memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the target detection method based on the driving scenario as described above.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a server, implements the target detection method based on a driving scenario as described above.

[0037] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including instructions, which, when executed by a computer, cause the computer to implement the target detection method based on the driving scenario as described above. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a target detection method based on a driving scenario provided in this application embodiment;

[0040] Figure 2 This application provides a schematic diagram of the feature fusion process for a target detection method based on a driving scenario, as illustrated in an embodiment of the present application.

[0041] Figure 3 This is a schematic diagram of a cross-layer multi-scale feature fusion process for a target detection method based on a driving scene, provided in an embodiment of this application.

[0042] Figure 4This is a schematic diagram of the structure of a target detection device based on a driving scenario provided in an embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0045] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] To address the challenges of processing and acquiring 3D point cloud data, this application proposes a more reasonable data sampling and processing method to ensure model training efficiency and improve the performance of deep learning detection models. Considering that target detection tasks in autonomous driving scenarios are affected by different environments, this application establishes a 3D target database with broader environmental coverage to improve the robustness of deep learning-based 3D target detection models, providing a prerequisite for future practical applications. Furthermore, current 3D target detection databases in autonomous driving scenarios suffer from diverse data types, disorganized annotation methods, and a lack of unified standards, hindering the scalability of various detection models for practical applications. To ensure the robustness and scalability of 3D detection models, this application addresses the issues of establishing standards for training and testing databases, unifying annotation formats, and enhancing multi-model verification capabilities. Simultaneously, this application designs a reasonable and effective 3D object point cloud data annotation method, which is also an effective guarantee for improving the discriminative ability of deep learning detection models.

[0047] The problem with voxelization-based convolution methods is the loss of feature information during point cloud voxelization, affecting the network's feature extraction for small objects. Furthermore, this model relies on manually set voxel sampling volumes; larger volumes can lead to loss of detail information, impacting object position estimation. Therefore, many methods require further refinement of voxel volumes to improve the network's representational ability, but this also increases computational complexity. Point-based learning methods, due to feature extraction for each point, have high overall computational costs and are time-consuming. To address these issues, this application proposes integrating these two types of feature learning methods. This helps the network learn more discriminative features and refines position prediction accuracy, making it more suitable for practical applications in autonomous driving object detection methods.

[0048] For two-stage object detection frameworks, these methods require two location regression calculations and two classification calculations, resulting in high overall accuracy. However, since the number of features in 3D point cloud data or RGB-D data is much greater than that in 2D images, the computational cost of two-stage methods is relatively high, leading to a decrease in overall speed. For single-stage object detection frameworks, the overall computational load is significantly reduced, improving the overall detection speed. However, this method also incurs considerable accuracy loss; although it has good practicality, the detection accuracy often falls short of application requirements. Addressing current 3D object detection methods, this application's embodiments, from the perspective of balancing detection accuracy and speed, optimize object position estimation regression and classification calculations based on the single-stage detection algorithm, enabling the overall model to maintain a fast detection speed while improving position prediction and classification accuracy.

[0049] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a target detection method based on a driving scenario, provided as an embodiment of this application. (Refer to...) Figure 1 This target detection method based on driving scenarios includes:

[0050] 100. Acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data.

[0051] Specifically, three-dimensional point cloud data of the driving scene is acquired, and the farthest point sampling technology is used to perform preliminary sampling on the three-dimensional point cloud data.

[0052] Optionally, 3D point cloud data can be acquired using sensors such as lidar, radar, sonar, or depth cameras. In driving scenarios, these sensors measure the distance and position of objects by emitting light or sound waves and receiving the reflected signals, thereby constructing a 3D model of the environment.

[0053] For example, embodiments of this application use LiDAR to acquire precise three-dimensional spatial location information, presenting environmental data in the form of point clouds. A camera is responsible for capturing rich texture and color information for object identification, and millimeter-wave radar is used to supplement target detection capabilities in adverse weather conditions. Additionally, the vehicle is equipped with various types of cameras, such as front-view cameras, rear-view cameras, and surround-view cameras, to collect visual information from all directions.

[0054] The sensors operate in real time while the vehicle is in motion, controlling the lidar to scan periodically and the camera to capture images at a certain frame rate, such as the common 30 frames per second or 60 frames per second, to adapt to environmental changes. Simultaneously, the system controls the vehicle to collect data under different road conditions (highways, city roads, etc.), weather conditions (sunny, rainy, snowy, etc.), and time conditions (daytime, nighttime, etc.), improving the system's adaptability.

[0055] Optionally, farthest-point sampling is a greedy algorithm used to select a representative set of points from the point cloud, which are distributed as widely as possible in space. Specifically, a starting point is randomly selected from the point cloud as the first sampling point; in each iteration, the algorithm calculates the distance from all points to the nearest point in the current sampling set, selects the point with the farthest distance as the next sampling point; the newly selected point is added to the sampling set; the process of selecting the next sampling point and adding it to the sampling set is repeated until a predetermined number of sampling points is reached.

[0056] Therefore, by using the farthest sampling technique, while ensuring the accurate depiction of the object's shape, the selection of data points can better preserve the object's shape characteristics, reduce the loss of texture detail information, and avoid destroying the object's spatial structure and orientation information.

[0057] For example, this application embodiment first analyzes existing autonomous driving databases. In deep learning algorithms, the quality of data has a certain impact on the accuracy of the detection model. Therefore, this application analyzes existing target detection databases, summarizes the characteristics and testing conditions of each database, analyzes the categories and feature information of objects in the data, summarizes the challenges and difficulties, and then, based on the existing databases, establishes an autonomous driving scene target detection database that can be used in this application embodiment according to requirements.

[0058] For example, this application embodiment includes the KITTI database, NuScenes database, and Waymo Open Dataset database for various databases. The KITTI database is currently the most widely used autonomous driving scene object detection database internationally, containing 14,999 2D images and point cloud data, with annotations covering eight categories, such as vehicles, pedestrians, and bicycles. This dataset lays the foundation for building a larger autonomous driving scene object detection database in the future. The NuScenes database contains 1.4 million images and 1.1 million 3D bounding boxes, posing a more severe challenge to autonomous driving scene object detection algorithms and promoting the improvement and development of deep learning-based object detection algorithms. The Waymo Open Dataset database addresses the problem of the current data's singular driving environment. Furthermore, this database not only addresses weather and driving scene issues but also incorporates temporal variations, such as night, day, and dusk, making the data more realistic. The database contains 3,000 driving segments, covering not only detection problems but also tracking and segmentation tasks. This application embodiment intends to analyze the characteristics and methods of constructing autonomous driving object detection data from these databases. Since these data cover too broad a range of scenarios, this application's embodiments refine the challenges and conditions of the target requirements, focusing only on driving scenarios, to reduce data collection costs and improve overall efficiency.

[0059] For example, after acquiring the data, it needs to be processed. The data points collected by the LiDAR camera are numerous, typically 40,000-50,000 points. Direct data input would result in high computational costs and low efficiency. Therefore, this embodiment proposes to perform preliminary sampling based on the data characteristics to reduce the number of data points and improve overall computational efficiency. For the sampling method, this embodiment proposes to use Farthest Point Sampling (FPS) to filter data points while ensuring accurate depiction of the object's shape. The Farthest Point Sampling method calculates the distance from all data points in space to the initial point, selects the maximum value as the next sampling point, and iterates to obtain the number of points that can cover the entire object's shape. On one hand, this method can better preserve the object's shape characteristics and reduce the loss of texture detail information. On the other hand, this method does not destroy the object's spatial structure and orientation information. Furthermore, this embodiment will also use more methods for comparison during experimental verification and then perform related optimizations.

[0060] 200. Based on the preprocessed 3D point cloud data and combined with the autonomous driving scene target detection database, establish a training dataset.

[0061] Optionally, after acquiring the 3D point cloud data, further preprocessing of the data can be performed, including data cleaning, annotation and classification, and data augmentation.

[0062] Data cleaning includes removing outliers from LiDAR point cloud data and invalid frames (blurred, abnormally exposed, etc.) from camera images, and processing duplicate information to reduce data redundancy. Labeling and classification involve manually labeling object categories, locations, and bounding boxes, and then organizing them into training, validation, and test sets. Data augmentation includes operations such as rotation, scaling, translation, flipping, and color transformation on image data, and random point sampling, rotation, translation, and noise addition on point cloud data to increase data diversity and model generalization ability.

[0063] 300. Train and validate the training dataset to obtain the feature extraction network model.

[0064] Specifically, using single-stage detection technology as the basic framework, the method of generating bounding boxes is improved by using prior 3D object position information as an aid and adopting a multi-scale target prediction method to train the training dataset to obtain a feature extraction network model; the training dataset is then validated to generate candidate bounding boxes at multiple scales.

[0065] For example, single-stage detection technology can use methods such as YOLO-6D and PointPillars to divide all data into multiple groups, with each group having multiple rectangular candidate boxes preset. The location and category of the target can be directly predicted through candidate box regression and classification within the group, which greatly reduces the overall computational load and improves the overall detection speed.

[0066] Therefore, a single-stage detection method is used as the basic framework, that is, only one regression and classification calculation of the location box is performed. However, by improving the location box generation method, prior 3D object position information is used as an aid, and candidate location boxes are generated in the 3D data space to reduce feature loss during feature transformation. A multi-scale target prediction method is adopted to make the overall detection model more sensitive to changes in object scale and improve detection accuracy.

[0067] For example, the embodiments of this application fuse multiple feature extraction network models to perform more detailed representation of 3D objects, and introduce the concept of multi-scale feature fusion to effectively combine deep and shallow neural networks, thereby reducing the influence of background interference and scale.

[0068] For example, current deep learning-based 3D object detection methods commonly employ a two-stage detection approach based on region generation. This involves generating candidate bounding boxes on the feature map output by the feature extraction network, filtering out boxes with background regions, and then performing regression and object category classification calculations on the bounding boxes to output the final detection result. For 3D object detection, candidate bounding boxes are typically provided on the generated bird's-eye view or top-down view, with typical methods including Point R-CNN and Part-A2Net. However, literature review and practical testing have shown that using 2D feature maps to generate candidate regions results in spatial location loss, affecting subsequent regression calculations of 3D bounding boxes. Furthermore, the two-stage approach is computationally intensive and difficult to implement in practice. Therefore, this application proposes a single-stage detection method as the basic framework, performing only one bounding box regression and classification calculation. However, by improving the bounding box generation method, prior 3D object position information is used as an aid, and candidate bounding boxes are generated in the 3D data space to reduce feature loss during feature transformation. Furthermore, this application's embodiments aim to address the problems of object scale variation and small-sized object detection by employing a multi-scale object prediction approach. This generates candidate bounding boxes at multiple scales to handle scale changes in 3D objects, making the overall detection model more sensitive to these changes and improving detection accuracy. In addition, single-stage detection methods offer faster execution speeds and lower computational costs, providing advantages for practical application and subsequent method expansion and improvement.

[0069] 400. By fusing multiple feature extraction network models and combining multi-scale feature fusion, target detection and feature extraction are performed on 3D point cloud data.

[0070] Specifically, a voxelized convolutional network model is used to fuse a point cloud feature learning network model to extract key point features from 3D point cloud data, and then multi-scale feature fusion is performed.

[0071] Therefore, feature extraction from voxelized convolutional networks can provide certain prior information for point cloud feature learning networks, thus accelerating computation; and the effective combination of deep and shallow neural networks reduces the influence of background interference and scale.

[0072] Among them, the voxelized convolutional network model transforms point clouds into three-dimensional voxel matrices. To address the problem of the disorder of point clouds, it sets rectangular voxel blocks to transform disordered point clouds into spatial matrices. Then, it uses convolutional neural networks (CNNs) to extract and represent features, ultimately achieving the position estimation and classification of three-dimensional objects. Common methods such as VoxelNet and Voxel-FPN all utilize voxelization methods to enable point clouds to use convolutional neural networks for feature extraction and combine them with detection frameworks for position prediction and category estimation.

[0073] Among them, the point cloud feature learning network model directly processes point cloud data. It uses fully connected neural networks to construct a multi-layer perceptron (MLP) to extract the relationships between points, thereby completing the target representation of 3D objects. Common methods include PoinNet, PointNet++, and Frustum PointNet. Compared to voxelized convolution methods, this method can refine the feature relationships between points more finely while preserving the spatial positional relationships of each point, thus improving the overall position estimation accuracy.

[0074] In some embodiments, the step of employing a voxelized convolutional network model fused with a point cloud feature learning network model to extract key point features from 3D point cloud data, and combining this with multi-scale feature fusion, includes:

[0075] 410. Process the 3D point cloud data to obtain voxelized point cloud data.

[0076] 420. Train the voxelized point cloud data to obtain a voxelized convolutional network model.

[0077] 430. The convolutional features trained by the voxelized convolutional network model are processed by the point cloud feature learning network model and combined with multi-scale feature fusion to extract key point features.

[0078] 440. Based on the voxelized convolutional network model, feature fusion processing is performed on key point features, and shallow feature information is fused with deep features of deep network.

[0079] In the above implementation process, the feature extraction of the voxelized convolutional network can provide certain prior information for the point cloud feature learning network, thus accelerating the computation speed; the effective combination of deep and shallow neural networks reduces the influence of background interference and scale.

[0080] For example, in multi-model fusion, current voxelized convolutional network models suffer from significant loss of detail during 3D point cloud feature extraction, affecting the determination of object bounding boxes and categories. Therefore, this application embodiment combines a point cloud feature learning network (PointNet) to compensate for the losses in the current feature extraction process, effectively improving the representation ability of 3D objects. Simultaneously, although the point cloud feature learning network has high computational cost and complexity, utilizing only a portion of the network maintains a low computational load, meeting the corresponding speed requirements. As follows... Figure 2 As shown, this is the process of feature fusion between two models. The feature extraction of the voxelized convolutional network can provide certain prior information for the point cloud feature learning network and also speeds up the computation.

[0081] For example, regarding multi-scale feature fusion, according to the concept of neural networks, shallow neural networks can acquire geometric information such as texture, color, and shape of objects. As the network layers deepen, background information of the image is gradually filtered out. In deep networks, abstract and semantic information of objects can be acquired, at which point object classification and bounding box prediction can be completed. However, for 3D objects, excessively deep network layers can easily lead to the loss of detailed information, especially for point cloud data, where an excessively deep network can affect subsequent bounding box prediction. Furthermore, 3D data contains more scale variations, and networks are not sensitive to scale changes in objects. Therefore, scale changes in objects in image space can significantly increase the false detection rate. Therefore, this application proposes a cross-layer feature fusion method suitable for 3D objects, based on currently popular multi-scale feature extraction methods. This method effectively fuses shallow feature information with deep network features, explores fusion methods, and aims to achieve optimal 3D object target feature extraction and representation, laying the foundation for feature modeling in subsequent target detection tasks. The following is an example. Figure 3 This is an example diagram of the cross-layer multi-scale feature fusion method to be used in the embodiments of this application.

[0082] To address the practical limitations of deep learning-based 3D object detection algorithms, a fast detection framework based on candidate bounding boxes is proposed. This framework improves speed while maintaining detection accuracy, and also enhances accuracy under harsh environmental conditions. 3D object detection requires utilizing object feature information at different scales. Since 3D point cloud data lacks information such as color and spatial structure, it is necessary to combine local and global object features for analysis. Therefore, during the modeling process of the detection algorithm, better determining the location and semantic category information of candidate objects based on feature information directly affects the overall performance of the method. The essence of a 3D object detection model lies in establishing a mapping relationship between 3D feature information and the semantic category and location of objects. However, the complexity of this mapping relationship directly affects the detection accuracy and speed of the model. The key technologies of the 3D object detection model involved in this application are linear regression of the bounding box and semantic category discrimination. This can ensure that feature information of different scales is introduced in the modeling process, and also ensure that the whole model has a clear linear function calculation method (bounding box regression calculation). Therefore, in order to solve the problem of balancing accuracy and speed in the detection task, this application can not only apply the 3D object detection model to actual products, but also provide a solution for the acceleration of deep model calculation strategies.

[0083] As described above, this application embodiment acquires 3D point cloud data of a driving scene and preprocesses the 3D point cloud data; based on the preprocessed 3D point cloud data and combined with an autonomous driving scene target detection database, a training dataset is established; the training dataset is trained and validated to obtain a feature extraction network model; through the fusion of multiple feature extraction network models and combined with multi-scale feature fusion, target detection and feature extraction are performed on the 3D point cloud data; a network feature fusion method is proposed to enhance the ability to extract detailed features of the target, which helps the network learn more discriminative features and can refine the accuracy of position prediction, thereby improving the overall detection effect.

[0084] Based on the above embodiments, the embodiments of this application can be further specified as follows: They also include:

[0085] The feature extraction network model is tested in real-world scenarios, and feedback results are received. The feature extraction network model is then adjusted based on the feedback results.

[0086] In the above implementation process, the feature extraction network model was tested in real-world scenarios to verify the model, and the method was optimized and adjusted based on the feedback results.

[0087] For example, regarding core technologies for object detection, on the one hand, there are various ways to utilize data, including methods based on point cloud data, multi-sensor fusion (such as LiDAR cameras, depth cameras, structured light scanners, ToF cameras, etc.), visual-laser fusion, and visual depth cognition to acquire and process data, providing rich information for object detection. For instance, the uniqueness of point cloud data can be used for object detection, or the advantages of multiple sensor data can be integrated to improve detection accuracy. On the other hand, there are advanced network architectures and algorithms, involving deep learning architectures such as PointNet, PointNet++, convolutional neural networks (CNNs), Transformer networks, and graph convolutions, as well as methods combined with traditional machine learning methods (such as SVM and decision trees). These architectures and algorithms are used for feature extraction, pose estimation, and other stages, such as achieving 3D object detection based on visual-laser fusion using graph convolutions, and learning pose estimation directly from data using deep learning.

[0088] For example, regarding integration with autonomous driving, in terms of target detection related to intelligent driving, the emphasis is on combining 3D target detection and intelligent driving methods. The target detection results are applied to the driving strategies of autonomous driving to ensure its safety and efficiency. For instance, after detecting a target, the vehicle can adjust its speed and direction based on the target information. In terms of autonomous driving system capability assessment and assurance, this includes autonomous driving capability testing methods, the construction of a driving task test scenario library, and autonomous driving simulation testing methods. By testing the autonomous driving system in different scenarios, quantitatively evaluating its capabilities, and simulating various situations in a simulation environment, the reliability and stability of the autonomous driving system in actual operation are ensured.

[0089] For example, regarding support for related devices and equipment, in terms of target detection devices and equipment, this includes hardware such as devices, controllers, and electronic devices corresponding to various target detection methods, as well as media used for storing and running related algorithms. This hardware and media constitute the physical foundation for the implementation of target detection technology in autonomous vehicles, ensuring the efficient operation of detection algorithms. In terms of software platforms and hardware acceleration technologies, this includes operating systems such as Ubuntu and ROS, robot development frameworks such as MoveIt and RoboticsMiddleware, and hardware acceleration technologies such as FPGAs and dedicated AI chips. Appropriate software platforms and hardware acceleration can improve computational efficiency and optimize the overall performance of target detection and autonomous driving systems.

[0090] For example, regarding data processing and model training optimization, in addition to farthest point sampling, downsampling methods such as random sampling consensus and Voxel Grid filters are also used in data processing and downsampling techniques. At the same time, data augmentation techniques such as rotation, scaling, and shearing are used to increase dataset diversity, reduce overfitting, and improve data quality and model generalization ability. In terms of target detection model training, attention is paid to the training methods and devices of target detection models. By optimizing the training process, such as adjusting parameters and combining pseudo-label information, the model can better adapt to the complex and ever-changing target features and environmental conditions in autonomous driving scenarios, thereby improving target detection performance.

[0091] Based on deep learning theory, this application proposes a more robust 3D object detection method and lightweights the model for practical application. Addressing the difficulties in processing and acquiring 3D point cloud data, it constructs a more suitable autonomous driving database and proposes a more reasonable data sampling and training method to ensure processing efficiency and improve the performance of the deep learning detection model. To address the current difficulty in representing 3D data, it proposes a network feature fusion method to enhance the ability to extract detailed features of targets and improve the overall detection effect. Finally, to address the limitations of the practicality of deep learning-based 3D detection algorithms, it proposes a fast detection framework based on candidate boxes, improving speed while maintaining detection accuracy.

[0092] The steps described above are not strictly performed in the order of their numbers; they should be understood as a whole.

[0093] Secondly, based on the above embodiments, Figure 4 This is a schematic diagram of a target detection device based on a driving scenario, provided as an embodiment of this application. (Reference) Figure 4 The target detection device based on driving scenario provided in this embodiment specifically includes: a data acquisition module 401, a data processing module 402, a model training module 403, and a target detection module 404.

[0094] The data acquisition module 401 is used to acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data; the data processing module 402 is used to establish a training dataset based on the preprocessed 3D point cloud data and the autonomous driving scene target detection database; the model training module 403 is used to train and validate the training dataset to obtain a feature extraction network model; and the target detection module 404 is used to perform target detection and feature extraction on the 3D point cloud data by fusing multiple feature extraction network models and combining multi-scale feature fusion.

[0095] As described above, this application embodiment acquires 3D point cloud data of a driving scene and preprocesses the 3D point cloud data; based on the preprocessed 3D point cloud data and combined with an autonomous driving scene target detection database, a training dataset is established; the training dataset is trained and validated to obtain a feature extraction network model; through the fusion of multiple feature extraction network models and combined with multi-scale feature fusion, target detection and feature extraction are performed on the 3D point cloud data; a network feature fusion method is proposed to enhance the ability to extract detailed features of the target, which helps the network learn more discriminative features and can refine the accuracy of position prediction, thereby improving the overall detection effect.

[0096] The target detection device based on driving scenarios provided in this application embodiment can be used to execute the target detection method based on driving scenarios provided in the above embodiment, and has corresponding functions and beneficial effects.

[0097] Thirdly, embodiments of this application also provide an electronic device that can integrate the target detection device based on driving scenarios provided in embodiments of this application. Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. (Reference) Figure 5 The electronic device includes an input device 43, an output device 44, a memory 42, and one or more processors 41. The memory 42 stores one or more programs. When the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the target detection method based on a driving scenario as provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0098] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned target detection method based on driving scenarios.

[0099] The electronic device provided above can be used to execute the target detection method based on driving scenarios provided in the above embodiments, and has corresponding functions and beneficial effects.

[0100] Fourthly, embodiments of this application also provide a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the target detection method based on the driving scenario as described above, and can achieve the same beneficial effects.

[0101] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the target detection method based on driving scenarios as described above, but can also execute related operations in the target detection method based on driving scenarios provided in any embodiment of this application.

[0102] Fifthly, embodiments of this application also provide a computer program product. The methods described in the various embodiments of this application can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the various embodiments of this application are executed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM (Open Application Model), or other programmable devices.

[0103] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0105] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0106] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A target detection method based on driving scenarios, characterized in that, include: Acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data; A training dataset is established based on the preprocessed 3D point cloud data and the target detection database for autonomous driving scenarios. The feature extraction network model is obtained by training and validating the training dataset. By fusing multiple feature extraction network models and combining multi-scale feature fusion, target detection and feature extraction are performed on 3D point cloud data. The method involves fusing multiple feature extraction network models and combining multi-scale feature fusion to perform target detection and feature extraction on 3D point cloud data, including: The 3D point cloud data is processed to obtain voxelized point cloud data; A voxelized convolutional network model is obtained by training on voxelized point cloud data; The convolutional features trained by the voxelized convolutional network model are processed by the point cloud feature learning network model and combined with multi-scale feature fusion to extract key point features. The key point features are fused based on a voxelized convolutional network model, which also fuses the shallow features of the shallow neural network with the deep features of the deep network. The shallow neural network is used to obtain the geometric information of the object, while the deep network is used to obtain the abstract and semantic information of the object.

2. The target detection method based on driving scenarios according to claim 1, characterized in that, The acquisition of 3D point cloud data of the driving scene and the preprocessing of the 3D point cloud data include: Three-dimensional point cloud data of the driving scene is acquired, and the farthest point sampling technology is used to perform preliminary sampling on the three-dimensional point cloud data.

3. The target detection method based on driving scenarios according to claim 1, characterized in that, The process of training and validating the training dataset to obtain the feature extraction network model includes: Using single-stage detection technology as the basic framework, the feature extraction network model is obtained by improving the location box generation method, utilizing prior 3D object location information as an aid, and adopting a multi-scale target prediction method to train the training dataset. The training dataset is validated to generate candidate location boxes at multiple scales.

4. The target detection method based on driving scenarios according to claim 1, characterized in that, Also includes: The feature extraction network model is tested in real-world scenarios, and feedback results are received. The feature extraction network model is then adjusted based on the feedback results.

5. A target detection device based on driving scenarios, characterized in that, The apparatus is used to perform the method as described in claim 1, comprising: The data acquisition module is used to acquire 3D point cloud data of the driving scene and preprocess the 3D point cloud data; The data processing module is used to build a training dataset based on the preprocessed 3D point cloud data and the autonomous driving scene target detection database. The model training module is used to train and validate the training dataset to obtain the feature extraction network model; The target detection module is used to perform target detection and feature extraction on 3D point cloud data by fusing multiple feature extraction network models and combining multi-scale feature fusion.

6. An electronic device, characterized in that, include: The processor, memory, and bus are provided, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions that, when executed by the processor, are used to implement the target detection method based on a driving scenario as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a server, implements the target detection method based on any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computer, cause the computer to implement the target detection method based on any one of claims 1-4.

Images