Three-dimensional target detection method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a three-dimensional target detection method and device, a storage medium and electronic equipment, and the method comprises the steps: converting to-be-detected point cloud data into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, the three-dimensional target detection model is a model used for performing target detection on the input three-dimensional point cloud data; the target point cloud features are input into a multi-scale feature fusion network of a three-dimensional target detection model, map view features under an aerial view angle output by the multi-scale feature fusion network are obtained, and the multi-scale feature fusion network is used for performing depth separable convolution on the input point cloud features through convolution kernels of multiple scales to obtain map view features under the aerial view angle; the network is used for carrying out convolution and fusing multi-scale features obtained by convolution; the map view features are input into a detection head of the three-dimensional target detection model, a target detection result output by the detection head is obtained, and the target detection result is used for indicating a three-dimensional target detected from the point cloud data to be detected.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and specifically, to a three-dimensional target detection method and device, a storage medium, and an electronic device. Background Art

[0002] 3D point cloud data is widely used in computer vision, robotics, autonomous driving and other fields due to its high precision, rich information and strong sense of reality. The objects identified from 3D point cloud data can be applied to subsequent processing procedures, such as obstacle avoidance, driving route planning, etc.

[0003] However, point cloud data is usually composed of sparse points, which means that there are a lot of blank areas in the space, lacking sufficient information to accurately describe and analyze the shape and characteristics of the target. In addition, the size of the target varies greatly, and the sparse point cloud may cause the missing or incomplete key feature points, which will reduce the accuracy of target detection.

[0004] It can be seen that the three-dimensional target detection method in the related art has the problem of inaccurate target detection results due to the sparse point cloud data. Summary of the invention

[0005] The embodiments of the present application provide a three-dimensional target detection method and device, a storage medium and an electronic device to at least solve the problem that the three-dimensional target detection method in the related art has inaccurate target detection results due to sparse point cloud data.

[0006] According to one aspect of an embodiment of the present application, a three-dimensional target detection method is provided, comprising: converting point cloud data to be detected into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on input three-dimensional point cloud data; inputting the target point cloud features into a multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under a bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features through convolution kernels of multiple scales, and fusing the multi-scale features obtained by the convolution; inputting the map view features into a detection head of the three-dimensional target detection model to obtain a target detection result output by the detection head, wherein the target detection result is used to indicate a three-dimensional target detected from the point cloud data to be detected.

[0007] According to another aspect of an embodiment of the present application, a three-dimensional target detection device is provided, comprising: a conversion unit, used to convert point cloud data to be detected into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on input three-dimensional point cloud data; a first input unit, used to input the target point cloud features into a multi-scale feature fusion network of the three-dimensional target detection model, to obtain map view features under a bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features through convolution kernels of multiple scales, and fusing the multi-scale features obtained by the convolution; a second input unit, used to input the map view features into a detection head of the three-dimensional target detection model, to obtain a target detection result output by the detection head, wherein the target detection result is used to indicate a three-dimensional target detected from the point cloud data to be detected.

[0008] In an exemplary embodiment, the multi-scale feature fusion network includes M groups of parallel cascaded convolution kernels, wherein the i-th group of cascaded convolution kernels in the M groups of cascaded convolution kernels includes cascaded 1*m i The convolution kernel and m i *1, M is a positive integer greater than or equal to 2, 1≤i≤M; the first input unit includes: a first input module, used to input the target point cloud feature into the multi-scale feature fusion network, so that the multi-scale feature fusion network performs the following processing operations on the target point cloud feature to obtain the map view feature: a convolution module, used to perform depth-separable convolution on the target point cloud feature through each group of cascaded convolution kernels in the M groups of cascaded convolution kernels, respectively, to obtain the features to be fused with the output of each group of cascaded convolution kernels; a feature fusion module, used to perform feature fusion on the features to be fused with the output of each group of cascaded convolution kernels, to obtain the map view feature.

[0009] In an exemplary embodiment, the number of channels of the target point cloud feature is N, wherein N is a positive integer greater than or equal to 2; the device further includes: a first convolution unit, configured to perform a first 1*1 convolution on the target point cloud feature before inputting the target point cloud feature into the multi-scale feature fusion network of the three-dimensional target detection model to obtain the updated target point cloud feature, wherein the first 1*1 convolution is used to reduce the number of channels of the target point cloud feature; a second convolution unit, configured to perform a second 1*1 convolution on the map view feature after inputting the target point cloud feature into the multi-scale feature fusion network of the three-dimensional target detection model to obtain the updated map view feature, wherein the second 1*1 convolution is used to restore the number of channels of the map view feature to N.

[0010] In an exemplary embodiment, the point cloud data to be detected is radar point cloud data; the conversion unit includes: a second input module, used to input the point cloud data to be detected into the three-dimensional backbone network, so that the three-dimensional backbone network uses a three-dimensional encoder to preprocess and extract features of the point cloud data to be detected to obtain the target point cloud features, wherein the preprocessing includes data compression.

[0011] In an exemplary embodiment, the device also includes: a first acquisition unit, used to acquire first training point cloud data and first annotation data corresponding to the first training point cloud data before the three-dimensional backbone network of the three-dimensional target detection model converts the point cloud data to be detected into target point cloud features, wherein the first annotation data contains an angle true value label corresponding to the annotated target in the first training point cloud data, and the angle true value label is a K-dimensional classification vector, each element position in the K-dimensional classification vector is associated with an angle value, and the element value in the angle true value label satisfies a Gaussian distribution according to the associated angle value, and K is a positive integer greater than or equal to 2; a first training unit, used to use the first training point cloud data and the first annotation data to perform model training on the initial three-dimensional target detection model to obtain the trained three-dimensional target detection model.

[0012] In an exemplary embodiment, the first acquisition unit includes: an acquisition module, used to acquire the angle one-hot label corresponding to the annotation target, wherein the angle one-hot label is a K-dimensional classification vector, the element value at one element position in the angle one-hot label is 1, and the element values ​​at the other element positions are 0, and in the angle one-hot label corresponding to the annotation target, the angle associated with the element position with the element value of 1 is the target angle; an adding module, used to add a cyclic Gaussian window to the angle one-hot label corresponding to the annotation target to obtain the angle true value label corresponding to the annotation target, wherein the cyclic Gaussian window is obtained by cyclically shifting a basic Gaussian window with the target angle, and the basic Gaussian window is a normalized Gaussian window with 0 degrees as the center position and an angle range of [-90, 89], and the cyclic shift refers to: angle value shift, window function value cycle.

[0013] In an exemplary embodiment, the device also includes: a second acquisition unit, used to acquire second training point cloud data and second annotation data corresponding to the training point cloud data before the three-dimensional backbone network of the three-dimensional target detection model converts the point cloud data to be detected into target point cloud features, wherein the second annotation data includes an annotation heat map, in which the thermal values ​​of feature points in a specified area that does not contain a point cloud are set to zero; a second training unit, used to use the second training point cloud data and the second annotation data to perform model training on the initial three-dimensional target detection model to obtain the trained three-dimensional target detection model.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above method embodiments when executing the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any of the above method embodiments.

[0017] In an embodiment of the present application, a method of performing three-dimensional target detection on input point cloud data using a three-dimensional target detection model is adopted. First, the point cloud data to be detected is converted into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model. Based on this, the high-dimensional point cloud data is converted into low-dimensional abstract features through the three-dimensional backbone network; then, the target point cloud features are input into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under the bird's-eye view perspective output by the multi-scale feature fusion network. Here, the multi-scale feature fusion network is a network for performing deep separable convolution on the input point cloud features through convolution kernels of multiple scales, and fusing the multi-scale features obtained by the convolution. Since the input point cloud features are convolved with convolution kernels of multiple scales, the receptive field of the model can be increased, the perception ability of different targets can be improved, and the possible information loss in the point cloud data can be effectively compensated. At the same time, the convolution performed on the input point cloud features is a depth-separable convolution, which can take into account the resource consumption of the algorithm calculation; finally, the map view features are input into the detection head of the three-dimensional target detection model to obtain the target detection results output by the detection head. At this point, the three-dimensional target detection in the point cloud data to be detected can be realized, and the technical effect of improving the accuracy of target detection can be achieved, thereby solving the problem that the target detection results of the three-dimensional target detection method in the related technology are not accurate enough due to the sparse point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 It is a schematic diagram of an application scenario of a three-dimensional target detection method according to an embodiment of the present application.

[0020] Figure 2 It is a flowchart of an optional three-dimensional target detection method according to an embodiment of the present application.

[0021] Figure 3 It is a schematic diagram of an optional point cloud 3D detection network according to an embodiment of the present application.

[0022] Figure 4 It is a schematic diagram of an optional depth-separable multi-scale fusion according to an embodiment of the present application.

[0023] Figure 5 It is a schematic diagram of an optional cyclic Gaussian-based angle regression according to an embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of an optional thermal map according to an embodiment of the present application.

[0025] Figure 7 It is a flowchart of another optional three-dimensional target detection method according to an embodiment of the present application.

[0026] Figure 8 It is a structural block diagram of an optional three-dimensional target detection device according to an embodiment of the present application.

[0027] Fig. 9 It is a block diagram of a computer system structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work (for example, feature splitting in the same embodiment and combining features in different embodiments, if there is no contradiction) shall fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to one aspect of an embodiment of the present application, a three-dimensional object detection method is provided. Optionally, in this embodiment, the three-dimensional object detection method can be applied to Figure 1 In the hardware environment composed of the vehicle terminal 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the vehicle-mounted terminal 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104.

[0031] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth.

[0032] The three-dimensional object detection method of the embodiment of the present application can be executed by the server 104, or by the vehicle terminal 102, or by both the server 104 and the vehicle terminal 102. The vehicle terminal 102 can execute the three-dimensional object detection method of the embodiment of the present application, or the client installed thereon.

[0033] Taking the three-dimensional object detection method in this embodiment executed by the server 104 as an example, Figure 2 is a flow chart of an optional three-dimensional target detection method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method includes the following steps:

[0034] Step S202, converting the point cloud data to be detected into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model;

[0035] Step S204, inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model to obtain the map view features under the bird's-eye view perspective output by the multi-scale feature fusion network;

[0036] Step S206, inputting the map view features into the detection head of the three-dimensional target detection model to obtain the target detection result output by the detection head, wherein the target detection result is used to indicate the three-dimensional target detected from the point cloud data to be detected.

[0037] The three-dimensional target detection method in this embodiment can be applied to the field of artificial intelligence technology, and can be applied to the scene of target detection on three-dimensional point cloud data. The technology for target detection on three-dimensional point cloud data is three-dimensional (3D, 3Dimensions) point cloud technology. Three-dimensional point cloud technology is a method of obtaining three-dimensional information of the surrounding environment through a sensing device. Point cloud data is the direct output result of this technology. It consists of a series of points in three-dimensional space. Each point represents the coordinate position of a surface point of an object in the environment that can be detected by the sensing device. In addition, point cloud data can also carry additional information, such as reflection intensity, color, etc.

[0038] In the progress of science and technology and industrial development, 3D point cloud technology, as an important way to obtain and process information, is gradually showing its wide application prospects in many fields, such as intelligent driving. In the field of intelligent driving, the 3D point cloud data detected by the sensing equipment can be used to perceive and understand the environment around the vehicle, identify pedestrians, vehicles and other obstacles, etc.

[0039] Optionally, the above-mentioned sensing device may include but is not limited to at least one of the following: millimeter wave radar, ultrasonic sensor, laser radar, etc. Taking laser radar as a sensing device as an example, the laser radar can detect the distance of an object by emitting laser pulses and measuring the reflected light, thereby generating three-dimensional point cloud data and sending it to the vehicle terminal; correspondingly, the vehicle terminal can obtain the three-dimensional point cloud data, and the obtained three-dimensional point cloud data can be transmitted to the server side, and the server side performs point cloud three-dimensional target detection on the three-dimensional point cloud data.

[0040] With the advancement of sensor technology and the improvement of computing power, the ability to collect and process large-scale 3D point cloud data has also been significantly enhanced, which provides strong support for the development of 3D detection and recognition algorithms. However, in related technologies, there are still many challenges and difficulties in the processing of 3D point cloud data, including data noise, point cloud sparsity, scene complexity, and algorithm efficiency. These have limited its further promotion and application in practical applications. It is necessary to effectively extract valuable information from massive 3D point cloud data and to develop efficient and accurate target detection and recognition solutions.

[0041] Among them, point cloud sparsity means that point cloud data is usually composed of sparse points, which means that there are a large number of blank areas in space and there is a lack of sufficient information to accurately describe and analyze the shape and characteristics of the target. Target detection algorithms often face the difficulty of inferring the overall target shape and characteristics from limited information. In addition, sparse point clouds may lead to missing or incomplete key feature points, such as the boundaries, surface details or structural features of objects, which will lead to unsatisfactory target detection results. Especially for large targets, the blank area in the middle is large, and sometimes the point cloud data is only distributed on one side, resulting in the lack of features in the main area of ​​the target, affecting the detection effect.

[0042] It should be noted that when detecting three-dimensional objects, the End-to-End Learning for Point Cloud Based 3D Object Detection method or the Center-based 3D Object Detection and Tracking method can be used. End-to-End Learning for Point Cloud Based 3D Object Detection is specially designed to accurately and efficiently detect three-dimensional objects from point cloud data: VoxelNet (voxel network) first converts point cloud data into a voxel grid. The voxel grid is a three-dimensional cube structure that divides the entire space into voxels of uniform size, and each voxel contains the point cloud information within the range of the voxel; after extracting point cloud features through voxels, 3D convolution is performed to further extract point cloud depth features, and finally a three-dimensional convolution layer network is used to process the voxel grid data, followed by a fully connected layer and a classification / regression layer. Center-based 3D Object Detection and Tracking is an algorithm for detecting and locating the center of objects from 3D point cloud data. It focuses on efficiently extracting the accurate position and size information of objects from sparse and dense point clouds. The algorithm performs target detection and position regression by predicting the center point of the object, thereby simplifying the post-processing process. The above two 3D target detection methods in related technologies both have the problem of low target detection accuracy.

[0043] In order to at least partially solve the above technical problems, the present embodiment provides a point cloud three-dimensional target detection solution based on deep learning, and uses a three-dimensional target detection model to perform three-dimensional target detection on the input point cloud data. The three-dimensional target detection model includes a multi-scale feature fusion network. The multi-scale feature fusion network can perform deep separable convolution on the input point cloud features through convolution kernels of multiple scales. Since the input point cloud features are convolved with convolution kernels of multiple scales, the receptive field of the model can be increased, the perception ability of different targets can be improved, and the possible information missing in the point cloud data can be effectively compensated, thereby improving the accuracy of target detection. At the same time, the convolution performed on the input point cloud features is a deep separable convolution, which can take into account the resource consumption of algorithm calculation.

[0044] Optionally, the 3D target detection model can be used to detect targets on the input 3D point cloud data, which can be an end-to-end Center-based point cloud 3D target detection model, where the end-to-end model means that the entire 3D target detection model can be trained and optimized as a whole, the input layer of the model can directly receive the raw point cloud data (i.e., the point cloud data to be detected) without preprocessing, and the output is the direct target detection result, without the need for manually designed feature extraction or post-processing steps in the middle. The overall structure of the above-mentioned 3D target detection model can include: a 3D backbone network (3Dbackbone), a multi-scale feature fusion network and a detection head, wherein the 3D backbone network is used to convert high-dimensional point cloud data into low-dimensional abstract features, the multi-scale feature fusion network is used to perform depth-separable convolution on the input point cloud features through convolution kernels of multiple scales, and fuse the multi-scale features obtained by convolution, and the detection head is used to regress all or part of the target category, position, size and rotation angle of the target frame.

[0045] For example, Figure 3 As shown, the point cloud 3D detection network (i.e., the point cloud 3D target detection model) can take the point cloud data (Point cloud) of the perception device as input, and after being converted into low-dimensional abstract features through the three-dimensional backbone network, it is input into the multi-scale feature fusion module (corresponding to the multi-scale feature fusion network) for depth wise (depth separable) multi-scale convolution feature fusion (i.e., depth wise separable convolution is performed through convolution kernels of multiple scales, and the features obtained by the convolution are fused) to obtain the map view feature (Mapview feature) from the perspective of BEV (Bird's Eye View), and use the BEV perspective feature for detection head learning and regression.

[0046] For the point cloud data to be detected (i.e., the point cloud data to be detected in three dimensions, which may be radar point cloud data), the point cloud data to be detected may be converted into target point cloud features through a three-dimensional backbone network. The target point cloud features may be low-dimensional abstract features converted by the three-dimensional backbone network from the point cloud data to be detected (high-dimensional point cloud data). Through the above conversion, the original, sparse point cloud data may be converted into ordered, easy-to-analyze point cloud features. Optionally, the target point cloud features may be represented in the form of a set of feature maps, and different feature maps may be feature maps of different channels or the same channel. Through a set of feature maps, the spatial structure information of the point cloud data may be retained.

[0047] In image or point cloud target detection networks, a neck is generally used between the backbone network and the detection head to fuse features. Common feature fusion algorithms include FPN (Feature Pyramid Networks) and PANnet (Path Aggregation Network). These algorithms fuse features of different scales and depths to obtain semantic features that are more suitable for target detection and regression. In addition, feature fusion algorithms based on attention mechanisms and transformers are also common. However, due to the sparsity of point clouds, point cloud feature points are often missing, especially for large targets, where the central feature is more seriously missing. Therefore, a relatively large receptive field is required. The way to increase the receptive field is to increase the size of the convolution kernel, but a large convolution kernel will lead to a sudden increase in the number of parameters and calculations.

[0048] In this embodiment, in order to take into account the detection of targets of different scales and the needs of receptive fields of different sizes, a method of multi-scale convolution fusion based on depth wise is adopted. A multi-scale convolution feature fusion module based on depthwise (depth separable) (i.e., a multi-scale feature fusion network) is added after the backbone. The multi-scale convolution feature fusion module performs multi-scale feature fusion based on depth wise on the input point cloud features. After multi-scale feature fusion, the Map view feature under the BEV perspective is obtained, and the BEV perspective feature is used for head learning and regression. Through the above method, while increasing the receptive field, the amount of parameters and resource calculation can be effectively reduced, wherein the multi-scale convolution fusion based on depth wise can be realized by the aforementioned multi-scale feature fusion network. The multi-scale feature fusion network uses convolution kernels of multiple scales to convolve the input point cloud features to increase the receptive field of the model, improve the perception ability of different targets, effectively make up for the possible information loss in the point cloud data, and improve the accuracy of target detection; at the same time, the convolution performed on the input point cloud features is a depth separable convolution, which can take into account the resource consumption of algorithm calculation.

[0049] Here, convolution kernels of multiple scales refer to convolution kernels of multiple different sizes. Convolution kernels of different sizes are used to capture features at different scales. For example, larger convolution kernels can capture more macroscopic features, while smaller convolution kernels can capture local details in detail. By using these convolution kernels of different sizes, target information can be perceived through a variety of different receptive fields, avoiding the problem of feature loss caused by the inability of a single-scale convolution kernel to fully capture the features of the target due to the sparsity of the point cloud and the size difference of the target.

[0050] For the target point cloud features, the target point cloud features can be input into the multi-scale feature fusion network, which performs depth-separable convolution on the target point cloud features through convolution kernels of multiple scales, and fuses the multi-scale features obtained by the convolution to obtain the map view features under the bird's-eye view perspective output by the multi-scale feature fusion network. The fusion of the multi-scale features obtained by the convolution can be achieved in a variety of ways, such as splicing, weighted fusion, etc. This is not limited in the present embodiment. The bird's-eye view perspective is an image perspective that observes vertically downward from above, which can project targets in three-dimensional space onto a two-dimensional plane, thereby simplifying the expression of features and the detection of targets.

[0051] In this embodiment, the number of scales of convolution kernels included in the multi-scale feature fusion network affects the model's perception of objects of different sizes and the efficiency of feature fusion. Generally, a larger number means that the network can capture detailed features of the target through convolution kernels of more scales. However, as the scale of the convolution kernel increases, the computational cost will also increase accordingly. Therefore, it is necessary to weigh the numerical selection of the number of scales of the convolution kernel according to specific needs and computing resource limitations to achieve a balance between performance and efficiency.

[0052] After obtaining the map view features, the map view features can be input into the detection head of the 3D object detection model, and the detection head performs learning and regression to obtain the object detection result, which is used to indicate the 3D object detected from the point cloud data to be detected. Here, the detection head can analyze the input feature map through the regression network to obtain accurate object information, such as all or part of the object category, object position, object size, and rotation angle of the object box.

[0053] Here, the target detection result can not only indicate the presence of a three-dimensional target, but also provide an accurate bounding box of the three-dimensional target, that is, a target box, which can be described by at least some of the following parameters: position, size, rotation angle (direction). In the field of intelligent driving, based on the target detection results, intelligent driving vehicles can accurately perceive objects in the surrounding environment and make corresponding decisions and plans.

[0054] Through the embodiments provided by the present application, the point cloud data to be detected is converted into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on the input three-dimensional point cloud data; the target point cloud features are input into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features respectively through convolution kernels of multiple scales and fusing the multi-scale features obtained by the convolution; the map view features are input into the detection head of the three-dimensional target detection model to obtain the target detection result output by the detection head, wherein the target detection result is used to indicate the three-dimensional target detected from the point cloud data to be detected, which can solve the problem that the target detection result of the three-dimensional target detection method in the related art is not accurate enough due to the sparse point cloud data, thereby improving the accuracy of target detection.

[0055] In an exemplary embodiment, the convolution kernels of multiple scales in the multi-scale feature fusion network can be multiple convolution kernels in parallel, for example, K convolution kernels in parallel, and the size of the i-th convolution kernel is k i *k i , K is a positive integer greater than or equal to 2, 1≤i≤K. At this time, the computational complexity of the depth-separable convolution based on the convolution kernel of each size is large. To this end, in this embodiment, the convolution kernels of multiple scales can be M groups of parallel cascaded convolution kernels, wherein the i-th group of cascaded convolution kernels in the M groups of cascaded convolution kernels includes cascaded 1*m i The convolution kernel and m i *1, M is a positive integer greater than or equal to 2, 1≤i≤M. Here, M and m i The value of can be set according to the experience value. For different detection targets and different detection scenarios, M and m i The value of can be different. For example, the value of M can be 3, 5 or other values. i The value of can be 7, 9, 11, 13, 15 or other values.

[0056] Correspondingly, the target point cloud features are input into the multi-scale feature fusion network of the 3D target detection model to obtain the map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, including:

[0057] S11, inputting the target point cloud features into the multi-scale feature fusion network, so that the multi-scale feature fusion network performs the following processing operations on the target point cloud features to obtain map view features:

[0058] S12, performing depth-separable convolution on the target point cloud features through each group of cascade convolution kernels in the M groups of cascade convolution kernels, and obtaining features to be fused with the output of each group of cascade convolution kernels;

[0059] S13, performing feature fusion on the features to be fused output by each group of cascaded convolution kernels to obtain map view features.

[0060] For the target point cloud data features input into the multi-scale feature fusion network, each group of cascade convolution kernels can be used to perform depth-separable convolution on the target point cloud features. For each i-th group of cascade convolution kernels, 1*m i The convolution kernel of m is used for depth-separable convolution, and then m is used i *1 Perform depth-separable convolution to obtain the features to be fused with the output of each set of cascaded convolution kernels. Perform feature fusion on the features to be fused output by each set of cascaded convolution kernels. There are many ways to fusion features, such as feature splicing, feature superposition or other methods. The features finally output by the multi-scale feature fusion network are map view features.

[0061] For example, the multi-scale feature fusion network may include three parallel groups of cascaded convolution kernels, the first group of cascaded convolution kernels includes cascaded 1*7 convolution kernels and 7*1 convolution kernels, the second group of cascaded convolution kernels includes cascaded 1*11 convolution kernels and 11*1 convolution kernels, and the third group of cascaded convolution kernels includes cascaded 1*15 convolution kernels and 15*1 convolution kernels. For the above multi-scale feature fusion network, it can first perform three sizes of separable convolutions, and the convolution kernels are: 1*7, 1*11 and 1*15. The convolution kernels in the same direction can only increase the receptive field of a single dimension. Therefore, 7*1, 11*1 and 15*1 convolutions are cascaded later, and the features after convolution can be spliced.

[0062] Through this embodiment, by performing deep separable convolution on the input point cloud features through multiple groups of cascaded convolution kernels and performing feature fusion, the receptive field can be increased to improve the accuracy of target detection, while the model calculation amount can be reduced and the model processing efficiency can be improved.

[0063] In an exemplary embodiment, the number of channels of the target point cloud feature is N, where N is a positive integer greater than or equal to 2. For the point cloud features output by the backbone network, the number of channels of the feature map may be very large, which will result in a very large amount of computation for the multi-scale feature fusion network.

[0064] To this end, before inputting the target point cloud features into the multi-scale feature fusion network of the 3D target detection model, the method further includes:

[0065] S21, performing the first 1*1 convolution on the target point cloud feature to obtain an updated target point cloud feature, wherein the first 1*1 convolution is used to reduce the number of channels of the target point cloud feature.

[0066] In this embodiment, in order to further reduce the amount of calculation, the target point cloud features can be subjected to a first 1*1 convolution before being input into the multi-scale feature fusion network to reduce the number of channels of the target point cloud features, thereby reducing model parameters and computational complexity. Here, by convolving the target point cloud features with a 1*1 convolution kernel to reduce the number of channels of the target point cloud features, more feature information can be retained while reducing the number of channels of the target point cloud features. After the first 1*1 convolution, the number of channels of the target point cloud features can be reduced to N / 2, N / 4 or other numbers. The updated target point cloud features can be input into the multi-scale feature fusion network for subsequent processing.

[0067] Correspondingly, after inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model, the method further includes:

[0068] S22, performing a second 1*1 convolution on the map view feature to obtain an updated map view feature, wherein the second 1*1 convolution is used to restore the number of channels of the map view feature to N.

[0069] For the map view features output by the multi-scale feature fusion network, in order to enable the subsequent detection head to obtain input that matches the original number of channels to maintain sufficient information for accurate detection, a second 1*1 convolution can be performed on the map view features output by the multi-scale feature fusion network to restore the number of channels of the map view features to N, thereby obtaining updated map view features. The updated map view features can be input to the detection head for subsequent processing. Here, the convolution kernels of the first 1*1 convolution and the second 1*1 convolution can be merged with the multi-scale feature fusion network, that is, the first 1*1 convolution and the second 1*1 convolution are both performed by the multi-scale feature fusion network.

[0070] For example, Figure 4 For example, we can first use 1*1 convolution to reduce the number of channels, and then use three sizes of separable convolution, with convolution kernels of 1*7, 1*11 and 1*15 respectively, and then cascade 7*1, 11*1 and 15*1 convolutions, concatenate the features, and finally restore the original number of channels through 1*1 convolution.

[0071] Through this embodiment, the target point cloud features can be reduced in dimension before multi-scale feature fusion, and then the target point cloud features can be restored to the original number of channels after multi-scale feature fusion, which can reduce the computing resource consumption of multi-scale feature fusion.

[0072] In an exemplary embodiment, the point cloud data to be detected may be point cloud data sensed by a radar, that is, the point cloud data to be detected is radar point cloud data. The radar that senses the point cloud data to be detected may be a millimeter wave radar, a laser radar (LiDAR) or other types of radars, which are not limited in this embodiment. The above-mentioned radar may be a radar in a vehicle-mounted radar system, which can obtain point cloud data for characterizing the environment around the vehicle, and by processing these point cloud data, information such as the position, distance, angle, etc. of the target in the environment around the vehicle can be obtained.

[0073] Optionally, in this embodiment, the point cloud data to be detected is converted into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, including:

[0074] S31, inputting the point cloud data to be detected into the three-dimensional backbone network, so that the three-dimensional backbone network uses a three-dimensional encoder to preprocess and extract features of the point cloud data to be detected, and obtain target point cloud features, wherein the preprocessing includes data compression.

[0075] When converting high-dimensional point cloud data into low-dimensional abstract features, the 3D backbone network (3D backbone) can use a 3D encoder to preprocess and extract features of the input point cloud features (e.g., point cloud data to be detected) to obtain corresponding point cloud features. The preprocessing of the input point cloud data can be one or more, for example, data compression to reduce the amount of data to be processed, data cleaning to remove noise points or abnormal points to improve the accuracy of target detection, and standardization to improve the convenience of data processing.

[0076] In this embodiment, the 3D encoder can be a deep neural network, such as VoxelNet, SECOND or PointPillar, etc., which is not limited in this embodiment. They can directly process 3D point cloud data and convert point cloud data into feature maps, i.e., point cloud features, through a series of operations such as convolution and full connection.

[0077] For example, the 3D backbone network uses a 3D encoder to perform point cloud preprocessing (e.g., data compression) and feature extraction on the input radar point cloud data, and converts the high-dimensional point cloud data into low-dimensional abstract features through the 3D backbone.

[0078] Through this embodiment, by preprocessing (for example, data compression) and feature extraction of the input point cloud data through the three-dimensional backbone network, it is possible to reduce the complexity of calculation and improve the efficiency of data processing while converting high-dimensional point cloud data into low-dimensional abstract features.

[0079] In an exemplary embodiment, before converting the to-be-detected point cloud data into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, the method further includes:

[0080] S41, obtaining first training point cloud data and first annotation data corresponding to the first training point cloud data, wherein the first annotation data includes an angle truth label corresponding to an annotation target in the first training point cloud data, the angle truth label is a K-dimensional classification vector, each element position in the K-dimensional classification vector is associated with an angle value, the element value in the angle truth label satisfies a Gaussian distribution according to the associated angle value, and K is a positive integer greater than or equal to 2;

[0081] S42: Perform model training on the initial three-dimensional object detection model using the first training point cloud data and the first annotation data to obtain a trained three-dimensional object detection model.

[0082] Before actually applying the three-dimensional target detection model to detect the target, the three-dimensional target detection model can be trained to optimize the model parameters of the three-dimensional target detection model, thereby improving the accuracy and reliability of target detection. In order to perform model training on the three-dimensional target detection model, training point cloud data and annotation data corresponding to the training point cloud data can be obtained, and the initial three-dimensional target detection model can be trained using the training point cloud data and the annotation data to obtain a trained three-dimensional target detection model. The method of model training on the three-dimensional target detection model can be: inputting the training point cloud data into the three-dimensional target detection model to obtain the three-dimensional target predicted by the three-dimensional target detection model, based on the difference between the predicted three-dimensional target and the actual three-dimensional target indicated by the annotation data, determining the loss value according to the set loss function, and adjusting the model parameters of the three-dimensional target detection model based on the determined loss value to reduce the corresponding loss value. Here, the loss function refers to a mathematical function used to measure the degree of inconsistency between the model prediction value and the actual value. The above-mentioned training point cloud data can be the first training point cloud data, and the above-mentioned annotation data can be the first annotation data.

[0083] Optionally, while regressing the target category, target position and target size in point cloud three-dimensional target detection, the rotation angle of the target box can also be regressed, and the corresponding target can be a rotated target. Rotating target detection faces multiple challenges in the field of computer vision, which limit the effectiveness and accuracy of traditional target detection algorithms when dealing with rotating objects. The target detection algorithms in related technologies are usually based on bounding boxes or feature extraction methods in fixed directions, which are difficult to remain unchanged for rotating objects. The deep learning-based methods generally obtain the specific angle values ​​of the rotating targets through classification or regression. The loss convergence of the neural network for angle value learning is more difficult than the ordinary target box position regression, resulting in inaccuracy in angle learning. Therefore, the rotating target detection method in the related technology has the problem of inaccurate target angle regression.

[0084] Here, in the rotating target detection task, the regression problem of the target angle has always been a difficult point. In addition, angle regression algorithms are often needed in the fields of posture estimation and action recognition. Common angle regression algorithms include angle classification method, trigonometric function regression method, etc. The direct classification method needs to be divided into several intervals according to the different angle intervals of the task. It is more common to directly divide it into 180 or 360 categories for category learning. This method is simple and direct, but the directionality of angle regression is unclear. The trigonometric function regression method can accurately regress the angle value, but the angle interval is compressed to the interval of [-1, 1]. The regression angle error is often large, it is difficult to meet the angle value accuracy requirements, and the convergence effect is poor.

[0085] In this embodiment, by optimizing the annotation data, for the first training point cloud data, the corresponding annotation data is the first annotation data. The first annotation data includes an angle truth label corresponding to the annotation target in the first training point cloud data, and the angle truth label is a K-dimensional classification vector. Each element position in the K-dimensional classification vector is associated with an angle value. The element value in the angle truth label satisfies the Gaussian distribution according to the associated angle value, and K is a positive integer greater than or equal to 2. Optionally, the K value can be selected and adjusted according to the design needs of the model. For example, K=180 can be set to represent all angle values ​​from 0 to 179 degrees, and each angle value corresponds to an element position in the angle truth label (180-dimensional classification vector). For another example, K=360 can be set to represent all angle values ​​from 0 to 359 degrees, and each angle value corresponds to an element position in the angle truth label (360-dimensional classification vector).

[0086] The element values ​​in the angle truth label satisfy the Gaussian distribution according to the associated angle value, which means that the element values ​​in the angle truth label are centered at the true angle (the actual rotation angle) and satisfy the Gaussian distribution according to the associated angle value. Compared with simple one-hot encoding (i.e., the element value at the element position corresponding to one angle is 1, and the element values ​​at the other element positions are 0), the loss function of the model is more continuous and smooth (the closer the angle values ​​are, the smaller the loss value between them), and it can converge more stably during the training process, which can not only meet the accuracy requirements of the angle value (the dimension of the angle truth label can be set based on the accuracy requirements), but also effectively improve the training and convergence efficiency of the model.

[0087] For example, the true value angle is 45 degrees, and the K value is 180. In the 180-dimensional classification vector, the element value of the element position corresponding to 45 degrees is the largest, which can be 1. The closer the corresponding angle value is to 45 degrees, the closer the element value is to 1. The farther the corresponding angle value is from 45 degrees, the closer the element value is to 0. At the same time, due to the periodicity of the rotation angle, the element values ​​of the element positions corresponding to 0 to 45 degrees gradually increase to 1, the element values ​​of the element positions corresponding to 45 to 135 degrees gradually decrease from 1 to the minimum value, and the element values ​​of the element positions corresponding to 135 to 179 degrees gradually increase from the minimum value to a certain value.

[0088] Through this embodiment, by configuring the element values ​​in the angle true value labels in the labeled data to satisfy the Gaussian distribution according to the associated angle values, the labeled data can be optimized, and while meeting the accuracy requirements of the angle values, the model training effect and convergence efficiency can be effectively improved, and the periodicity problem of angle regression can be solved.

[0089] In an exemplary embodiment, obtaining first training point cloud data and first annotation data corresponding to the first training point cloud data includes:

[0090] S51, obtaining an angle one-hot label corresponding to the labeled target, wherein the angle one-hot label is a K-dimensional classification vector, the element value at one element position in the angle one-hot label is 1, and the element values ​​at the other element positions are 0, and in the angle one-hot label corresponding to the labeled target, the angle associated with the element position with the element value of 1 is the target angle;

[0091] S52, adding a cyclic Gaussian window to the angle one-hot label corresponding to the labeled target to obtain the angle true value label corresponding to the labeled target.

[0092] In this embodiment, a 3D angle regression method based on cyclic Gaussian can be used to regress the target angle. Different from directly performing angle classification, when obtaining the true value label of the angle corresponding to the labeled target in the first labeled data, the category one-hot label (i.e., the angle one-hot label) can be added to a cyclic Gaussian window. The cyclic Gaussian window is obtained by cyclically shifting the basic Gaussian window with the target angle. The basic Gaussian window is a normalized Gaussian window with 0 degrees as the center position and an angle range of [-90, 89]. The cyclic shift means: the angle value is shifted, and the window function value is cycled. The cyclic Gaussian window can be understood as a circular Gaussian. The circular Gaussian has a total of 180 angles with an interval of [-90, 89]. Figure 5 See Figure 5 After adding the cyclic Gaussian window, the true value label changes from the original one-hot format to a smooth Gaussian function shape. If the true value angle is N, the true value is a smooth curve with N as the maximum value. The cyclic Gaussian window takes into account the periodicity of the rotation angle and the characteristic that the closer the angle is, the smaller the loss is, and the convergence effect is more ideal.

[0093] In order to obtain the true angle label corresponding to the labeled target, we can first obtain the angle one-hot label corresponding to the labeled target. Here, the angle one-hot label is a K-dimensional classification vector. The element value at one element position in the angle one-hot label is 1, and the element values ​​at the other element positions are 0. In the angle one-hot label corresponding to the labeled target, the angle associated with the element position with the element value of 1 is the target angle; then, the above-mentioned cyclic Gaussian window is added to the angle one-hot label corresponding to the labeled target to obtain the true angle label corresponding to the labeled target.

[0094] Through this embodiment, by adding a cyclic Gaussian window to the angle one-hot label corresponding to the labeled target to obtain the angle true value label corresponding to the labeled target, the convenience of information acquisition can be improved.

[0095] In an exemplary embodiment, before converting the to-be-detected point cloud data into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, the method further includes:

[0096] S61, obtaining second training point cloud data and second annotation data corresponding to the training point cloud data, wherein the second annotation data includes an annotation heat map, in which the thermal values ​​of feature points in a specified area not including the point cloud are set to zero;

[0097] S62: Use the second training point cloud data and the second annotation data to perform model training on the initial three-dimensional object detection model to obtain a trained three-dimensional object detection model.

[0098] In this embodiment, second training point cloud data and second annotation data corresponding to the training point cloud data can be obtained, and the second training point cloud data and the second annotation data can be used to perform model training on the initial three-dimensional target detection model to obtain a trained three-dimensional target detection model. The method of using the training data and the corresponding annotation information to train the initial three-dimensional target detection model is similar to that in the aforementioned embodiment and will not be repeated here.

[0099] When performing three-dimensional target detection, an anchor free target detection algorithm can be used. An anchor free target detection algorithm generally uses a heat map to make and regress the true value of the target category. The heat map can be implemented based on a two-dimensional Gaussian function window. The closer the feature point is to the center of the target, the greater the probability value of the target. The heat map can include a heat value, that is, the value of each point in the heat map, which usually represents the probability of the target existing at that position. In the process of model training, the heat map is used as the target label (ground-truth) to compare with the heat map predicted by the network. The network parameters are adjusted through the loss function (for example, mean square error) to make the prediction result as close as possible to the real key point position. In the point cloud target detection task, unlike images, point cloud features are sparse. For large targets, the features are missing at the center. The heat map algorithm has high probability values ​​at the center and low probability values ​​at the edge, which makes the target detection algorithm unsatisfactory for large targets.

[0100] For example, Figure 6 As shown in the figure, the target is located below the feature map. At this time, the point cloud is only distributed on the upper side of the target. If the algorithm is made according to the true value of the heat map, all feature points within the target range have the probability of the existence of the target, and the probability of the center position is the highest. However, the real point cloud only exists in the dark area feature map at the top. In this case, the heatmap value of the light area below is higher, which is obviously inconsistent with the actual situation and may lead to inaccurate target detection.

[0101] In order to solve at least some of the above problems, in this embodiment, a mask can be added to the heat map according to the location of the point cloud distribution (whether there is point cloud data within the feature point), the heat map of the specified area without real point cloud is set to zero, and only the location with point cloud features is used to regress the target, which can effectively solve the problem of missing features at the center of large targets and unsatisfactory detection effect, thereby improving the accuracy of target detection.

[0102] It should be noted that the above process is completed before the model training of the 3D target detection model. That is to say, before the model training of the 3D target detection model, the thermal values ​​of the feature points in the specified area that does not contain the point cloud in the annotated thermal map can be set to zero. The specified area can be a vehicle perception blind spot, such as below the front of the vehicle, behind the rear of the vehicle, and near points on both sides, so as to optimize the second annotated data. When the predicted thermal map of the model is compared with the optimized annotated thermal map, the error calculated by the loss function will reflect the degree of deviation between the model prediction result and the actual result. The model adjusts its internal parameters according to the error. Then, the initial 3D target detection model can learn more accurate target detection capabilities in subsequent training, thereby improving the accuracy of the 3D target detection model.

[0103] Through this embodiment, the accuracy of target detection can be improved by setting the thermal values ​​of feature points in a specified area of ​​the annotated thermal map that does not contain a point cloud to zero.

[0104] The following is an explanation of the three-dimensional target detection method in the embodiment of the present application in conjunction with an optional example. This optional example provides a point cloud 3D target detection method based on deep learning, which belongs to the field of artificial intelligence technology. By analyzing the characteristics and structure of point cloud data, combined with machine learning and computer vision technology, it can achieve accurate positioning and recognition of targets in complex scenes, which can not only improve the accuracy and robustness of the detection algorithm, but also ensure the real-time and scalability of target detection, and improve the overall performance of the algorithm. The 3D point cloud detection network provided by this optional example supports the detection of multiple target types and can meet the requirements of real-time detection.

[0105] The process of 3D object detection model from training to use can be described as follows: Figure 7 As shown, in the model training stage, the training point cloud data and the labeled data can be used to train the initial three-dimensional target detection model to obtain a trained three-dimensional target detection model, wherein the entire model structure of the three-dimensional target detection model adopts an end-to-end design network, and the rotation target angle is learned based on the cyclic Gaussian during model training, and the target category is learned based on the heat map with the added mask, which takes into account both the model accuracy and the reasoning speed, and can meet the deployment requirements of the embedded platform. In the model reasoning stage, the point cloud data to be detected (radar point cloud data) can be input into the trained three-dimensional target detection model, and the trained three-dimensional target detection model performs target detection on the point cloud data to be detected to obtain the target detection result. When performing target detection, a multi-scale fusion module based on depthwise is used to increase the receptive field of the model, while taking into account the resource consumption of the algorithm calculation.

[0106] Through this optional example, after using the improved point cloud 3D target detection algorithm, the detection effect of the model is improved, especially the detection effect of large-size targets is significantly improved.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, or by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROMRAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a vehicle-mounted terminal) to execute the methods of various embodiments of the present invention.

[0108] According to another aspect of the embodiments of the present application, a three-dimensional target detection device is also provided, which is used to implement the three-dimensional target detection method provided in the above-mentioned embodiments, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0109] Figure 8 is a structural block diagram of an optional three-dimensional object detection device according to an embodiment of the present application, such as Figure 8 As shown in , the device includes: a conversion unit 802 , a first input unit 804 and a second input unit 806 .

[0110] The conversion unit 802 is used to convert the point cloud data to be detected into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on the input three-dimensional point cloud data.

[0111] The first input unit 804 is used to input the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model to obtain the map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network used to perform deep separable convolution on the input point cloud features through convolution kernels of multiple scales, and to fuse the multi-scale features obtained by the convolution.

[0112] The second input unit 806 is used to input the map view features into the detection head of the three-dimensional target detection model to obtain the target detection result output by the detection head, wherein the target detection result is used to indicate the three-dimensional target detected from the point cloud data to be detected.

[0113] It should be noted that the conversion unit 802 in this embodiment can be used to execute the above step S202, the first input unit 804 in this embodiment can be used to execute the above step S204, and the second input unit 806 in this embodiment can be used to execute the above step S202.

[0114] Through the embodiments provided by the present application, the point cloud data to be detected is converted into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on the input three-dimensional point cloud data; the target point cloud features are input into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features respectively through convolution kernels of multiple scales and fusing the multi-scale features obtained by the convolution; the map view features are input into the detection head of the three-dimensional target detection model to obtain the target detection result output by the detection head, wherein the target detection result is used to indicate the three-dimensional target detected from the point cloud data to be detected, which can solve the technical problem of inaccurate target detection caused by sparse point clouds in the three-dimensional target detection method in the related art, thereby improving the accuracy of target detection.

[0115] In an exemplary embodiment, the multi-scale feature fusion network includes M groups of cascaded convolution kernels in parallel, wherein the i-th group of cascaded convolution kernels in the M groups of cascaded convolution kernels includes cascaded 1*m i The convolution kernel and m i *1, M is a positive integer greater than or equal to 2, 1≤i≤M; the first input unit includes: a first input module, used to input the target point cloud features into the multi-scale feature fusion network, so that the multi-scale feature fusion network performs the following processing operations on the target point cloud features to obtain map view features: a convolution module, used to perform depth-separable convolution on the target point cloud features through each group of cascaded convolution kernels in M ​​groups of cascaded convolution kernels, respectively, to obtain features to be fused output by each group of cascaded convolution kernels; a feature fusion module, used to perform feature fusion on the features to be fused output by each group of cascaded convolution kernels to obtain map view features.

[0116] In an exemplary embodiment, the number of channels of the target point cloud feature is N, where N is a positive integer greater than or equal to 2; the above-mentioned device also includes: a first convolution unit, used to perform a first 1*1 convolution on the target point cloud feature before inputting the target point cloud feature into the multi-scale feature fusion network of the three-dimensional target detection model to obtain an updated target point cloud feature, wherein the first 1*1 convolution is used to reduce the number of channels of the target point cloud feature; a second convolution unit, used to perform a second 1*1 convolution on the map view feature after inputting the target point cloud feature into the multi-scale feature fusion network of the three-dimensional target detection model to obtain an updated map view feature, wherein the second 1*1 convolution is used to restore the number of channels of the map view feature to N.

[0117] In an exemplary embodiment, the point cloud data to be detected is radar point cloud data; the conversion unit includes: a second input module, used to input the point cloud data to be detected into a three-dimensional backbone network, so that the three-dimensional backbone network uses a three-dimensional encoder to preprocess and extract features of the point cloud data to be detected to obtain target point cloud features, wherein the preprocessing includes data compression.

[0118] In an exemplary embodiment, the above-mentioned device also includes: a first acquisition unit, which is used to acquire first training point cloud data and first annotation data corresponding to the first training point cloud data before converting the point cloud data to be detected into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, wherein the first annotation data includes an angle true value label corresponding to the annotated target in the first training point cloud data, and the angle true value label is a K-dimensional classification vector, and each element position in the K-dimensional classification vector is associated with an angle value, and the element value in the angle true value label satisfies the Gaussian distribution according to the associated angle value, and K is a positive integer greater than or equal to 2; a first training unit, which is used to use the first training point cloud data and the first annotation data to perform model training on the initial three-dimensional target detection model to obtain a trained three-dimensional target detection model.

[0119] In an exemplary embodiment, the first acquisition unit includes: an acquisition module, which is used to acquire an angle one-hot label corresponding to the labeled target, wherein the angle one-hot label is a K-dimensional classification vector, the element value at one element position in the angle one-hot label is 1, and the element values ​​at the remaining element positions are 0, and in the angle one-hot label corresponding to the labeled target, the angle associated with the element position with an element value of 1 is the target angle; an adding module, which is used to add a cyclic Gaussian window to the angle one-hot label corresponding to the labeled target to obtain an angle true value label corresponding to the labeled target, wherein the cyclic Gaussian window is obtained by cyclically shifting a basic Gaussian window with a target angle, and the basic Gaussian window is a normalized Gaussian window with 0 degrees as the center position and an angle range of [-90, 89], and the cyclic shift refers to: angle value shift, window function value cycle.

[0120] In an exemplary embodiment, the above-mentioned device also includes: a second acquisition unit, which is used to acquire second training point cloud data and second annotation data corresponding to the training point cloud data before converting the point cloud data to be detected into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, wherein the second annotation data includes an annotation heat map, in which the thermal values ​​of feature points in a specified area that does not contain a point cloud are set to zero; a second training unit, which is used to use the second training point cloud data and the second annotation data to perform model training on the initial three-dimensional target detection model to obtain a trained three-dimensional target detection model.

[0121] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0122] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the steps of any of the above method embodiments are executed when the program is run.

[0123] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0124] According to another aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the steps in any of the above method embodiments through the computer program. In an exemplary embodiment, the electronic device may further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0126] According to another aspect of the embodiment of the present application, a computer program product is also provided, which includes a computer program / instruction, and the computer program / instruction contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit 901, various functions provided by the embodiment of the present application are executed. The above-mentioned serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0127] Fig. 9 The computer system structure block diagram of the electronic device used to implement the embodiment of the present application is schematically shown. Fig. 9 As shown, the computer system 900 includes a CPU (Central Processing Unit) 901, which can perform various appropriate actions and processes according to the program stored in the ROM 902 or the program loaded from the storage part 908 to the RAM 903. Various programs and data required for system operation are also stored in the random access memory 903. The central processing unit 901, the read-only memory 902 and the random access memory 903 are connected to each other through a bus 904. An I / O (Input / Output) interface 905 is also connected to the bus 904.

[0128] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

[0129] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processor 901, various functions defined in the system of the present application are executed.

[0130] It should be noted that Fig. 9 The computer system 900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0132] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional target detection method, characterized in that: include: The point cloud data to be detected is converted into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on input three-dimensional point cloud data; Inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features respectively through convolution kernels of multiple scales, and fusing the multi-scale features obtained by the convolution; The map view feature is input into a detection head of the three-dimensional target detection model to obtain a target detection result output by the detection head, wherein the target detection result is used to indicate a three-dimensional target detected from the point cloud data to be detected.

2. The method according to claim 1, characterized in that: The multi-scale feature fusion network includes M groups of parallel cascade convolution kernels, wherein the i-th group of cascade convolution kernels in the M groups of cascade convolution kernels includes cascaded 1*m i The convolution kernel and m i *1 convolution kernel, M is a positive integer greater than or equal to 2, 1≤i≤M; The step of inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features output by the multi-scale feature fusion network from a bird's-eye view perspective includes: The target point cloud feature is input into the multi-scale feature fusion network, so that the multi-scale feature fusion network performs the following processing operations on the target point cloud feature to obtain the map view feature: Performing depth-separable convolution on the target point cloud features respectively through each group of cascade convolution kernels in the M groups of cascade convolution kernels to obtain features to be fused with the output of each group of cascade convolution kernels; Feature fusion is performed on the features to be fused output by each group of cascaded convolution kernels to obtain the map view features.

3. The method according to claim 1, characterized in that: The number of channels of the target point cloud feature is N, where N is a positive integer greater than or equal to 2; Before inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model, the method further includes: performing a first 1*1 convolution on the target point cloud features to obtain the updated target point cloud features, wherein the first 1*1 convolution is used to reduce the number of channels of the target point cloud features; After inputting the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model, the method further includes: performing a second 1*1 convolution on the map view features to obtain the updated map view features, wherein the second 1*1 convolution is used to restore the number of channels of the map view features to N.

4. The method according to claim 1, characterized in that: The point cloud data to be detected is radar point cloud data; The three-dimensional backbone network of the three-dimensional target detection model converts the point cloud data to be detected into target point cloud features, including: The point cloud data to be detected is input into the three-dimensional backbone network, so that the three-dimensional backbone network uses a three-dimensional encoder to preprocess and extract features of the point cloud data to be detected to obtain the target point cloud features, wherein the preprocessing includes data compression.

5. The method according to any one of claims 1 to 4, characterized in that Before converting the to-be-detected point cloud data into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, the method further includes: Acquire first training point cloud data and first annotation data corresponding to the first training point cloud data, wherein the first annotation data includes an angle truth label corresponding to an annotation target in the first training point cloud data, the angle truth label is a K-dimensional classification vector, each element position in the K-dimensional classification vector is associated with an angle value, and the element value in the angle truth label satisfies a Gaussian distribution according to the associated angle value, and K is a positive integer greater than or equal to 2; The initial three-dimensional object detection model is trained using the first training point cloud data and the first annotation data to obtain the trained three-dimensional object detection model.

6. The method according to claim 5, characterized in that The acquiring of first training point cloud data and first annotation data corresponding to the first training point cloud data includes: Obtain an angle one-hot label corresponding to the labeled target, wherein the angle one-hot label is a K-dimensional classification vector, the element value at one element position in the angle one-hot label is 1, and the element values ​​at the other element positions are 0, and in the angle one-hot label corresponding to the labeled target, the angle associated with the element position with the element value of 1 is the target angle; A cyclic Gaussian window is added to the angle unique hot label corresponding to the labeled target to obtain the angle true value label corresponding to the labeled target, wherein the cyclic Gaussian window is obtained by cyclically shifting a basic Gaussian window with the target angle, and the basic Gaussian window is a normalized Gaussian window with 0 degrees as the center position and an angle range of [-90, 89]. The cyclic shift refers to: angle value shift and window function value cycle.

7. The method according to any one of claims 1 to 4, characterized in that Before converting the to-be-detected point cloud data into target point cloud features through the three-dimensional backbone network of the three-dimensional target detection model, the method further includes: Acquire second training point cloud data and second annotated data corresponding to the training point cloud data, wherein the second annotated data comprises an annotated heat map, in which the heat values ​​of feature points in a specified area not including the point cloud are set to zero; The initial three-dimensional object detection model is trained using the second training point cloud data and the second annotation data to obtain the trained three-dimensional object detection model.

8. A three-dimensional target detection device, characterized in that: include: A conversion unit, used to convert the point cloud data to be detected into target point cloud features through a three-dimensional backbone network of a three-dimensional target detection model, wherein the three-dimensional target detection model is a model for performing target detection on input three-dimensional point cloud data; A first input unit is used to input the target point cloud features into the multi-scale feature fusion network of the three-dimensional target detection model to obtain map view features under the bird's-eye view perspective output by the multi-scale feature fusion network, wherein the multi-scale feature fusion network is a network for performing depth-separable convolution on the input point cloud features respectively through convolution kernels of multiple scales, and fusing the multi-scale features obtained by the convolution; The second input unit is used to input the map view feature into the detection head of the three-dimensional target detection model to obtain the target detection result output by the detection head, wherein the target detection result is used to indicate the three-dimensional target detected from the point cloud data to be detected.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of any one of the methods of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images