Target detection method and device, vehicle, processor and electronic device

Abstract

The application provides a target detection method and device, a vehicle, a processor and an electronic device. The original point cloud is obtained, voxelization processing is performed on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix; the voxel feature matrix and the voxel coordinate matrix are input into a sparse convolution backbone network, the sparse convolution backbone network is controlled to classify the voxel feature matrix, and the sparse convolution backbone network is controlled to output target features, wherein the target features are features obtained by removing non-target features, and the target features are features corresponding to the target; dense processing is performed on the target features to obtain a dense feature map, wherein the dense feature map does not contain features corresponding to non-targets; and the target is detected according to the dense feature map. The technical problem that in the related art, in a relatively complex scene, the SECOND algorithm is often used, green belts are often misdetected, and the detection accuracy of the target is not high is solved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and more specifically, to a method and apparatus for detecting objects, a vehicle, a processor, and electronic equipment.

[0002] Non-target technology

[0003] Among related technologies, deep learning has developed rapidly and is increasingly applied to fields such as computer vision. As an important branch of computer vision, deep learning has also been widely used in 3D object detection algorithms. Laser-based 3D object detection algorithms are divided into voxel-based methods (which typically process the original point cloud into regular voxel bounding boxes, and then process the regular voxel bounding boxes) and original point cloud-based methods (the algorithm directly processes the original point cloud). A typical example of a voxel-based algorithm is SECOND (Sparsely Embedded Convolutional Detection). The SECOND algorithm's processing flow is as follows: First, the point cloud is voxelized. Then, the voxelized point cloud data is fed into a sparse convolution backbone network composed of 3D sub-stream sparse convolutions, 3D sparse convolutions, batch regularization layers, and activation layers. The output sparse features and corresponding coordinates are then processed through a sparse-to-dense module to obtain a dense feature map. The dense feature map is then fed into a subsequent RPN layer to obtain the 3D detection output. For simple scenarios, such as highway scenarios, the SECOND algorithm can achieve good results. The reason is that on highways, only vehicles are present, with few pedestrians or bicycles. Furthermore, highway medians are relatively regular, resulting in fewer false detections. However, in urban scenarios, the traffic environment becomes more complex, with frequent bicycles and pedestrians, and median shapes vary across different road sections, increasing the difficulty of detection. Therefore, the SECOND algorithm frequently misdetects medians. For example, it often misdetects a median at an intersection as a vehicle, or a road sign within a median as a vehicle.

[0004] No effective solutions have yet been proposed to address the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] The main objective of this application is to provide a method and apparatus, vehicle, processor and electronic device for detecting targets, in order to solve the technical problem in the related art that the SECOND algorithm often results in false detection of green belts in relatively complex scenarios, leading to low detection accuracy of targets.

[0006] According to one aspect of the present invention, a method for detecting a target object is provided, comprising: acquiring an original point cloud; performing voxelization processing on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix; inputting the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network; controlling the sparse convolutional backbone network to classify the voxel feature matrix and controlling the sparse convolutional backbone network to output target features, wherein the target features are features obtained after removing non-target features, and the target features are features corresponding to a target object; performing densification processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features not corresponding to the target; and detecting the target object based on the dense feature map.

[0007] Further, the original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, including: determining the size of the voxel bounding box, wherein the size of the voxel bounding box includes the length, width and height of the voxel bounding box; determining the point cloud range of the original point cloud, wherein the point cloud range includes at least the size range of the original point cloud in three directions corresponding to a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel spatial region; and determining the voxel feature matrix and the voxel coordinate matrix based on the size of the voxel bounding box and the point cloud range.

[0008] Further, based on the size of the voxel bounding box and the point cloud range, the voxel feature matrix and voxel coordinate matrix are determined, including: dividing the original point cloud into multiple voxel blocks according to the size of the voxel bounding box and the point cloud range corresponding to the original point cloud; identifying blank voxel blocks that do not contain the original point cloud, removing the blank voxel blocks, and obtaining target voxel blocks containing the original point cloud; determining at least one original point cloud contained in each target voxel block, and determining at least one point cloud coordinate of at least one original point cloud in a preset spatial coordinate system; determining the voxel block coordinates of each target voxel block in the preset spatial coordinate system; and determining the voxel feature matrix and voxel coordinate matrix based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates.

[0009] Further, based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates, the voxel feature matrix and the voxel coordinate matrix are determined, including: determining the average coordinates corresponding to at least one point cloud coordinate contained in the target voxel block, and determining the average coordinates as the voxel feature corresponding to the target voxel block, and determining multiple voxel features corresponding to multiple target voxel blocks, and determining the voxel feature matrix based on the multiple voxel features; obtaining multiple voxel block coordinates corresponding to multiple target voxel blocks in a preset spatial coordinate system, and obtaining the voxel coordinate matrix based on the multiple voxel block coordinates.

[0010] Furthermore, the sparse convolutional backbone network is controlled to classify the voxel feature matrix, including: controlling the voxel feature matrix and voxel coordinate matrix to be input sequentially into multiple convolutional modules, and controlling the multiple convolutional modules to output the first voxel feature matrix and the first voxel coordinate matrix sequentially to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the coordinates of the multiple first voxel matrices decrease sequentially; based on the multiple first voxel matrices and the first voxel coordinate matrices, multiple new features corresponding to the voxel feature matrix are obtained; the new features are controlled to be input into the fully connected layer in the sparse convolutional backbone network for classification, and the fully connected layer is controlled to output the classification result, wherein the classification result is a non-target feature or a target feature.

[0011] Furthermore, based on multiple first voxel matrices and first voxel coordinate matrices, multiple new features corresponding to the voxel feature matrix are obtained, including: determining the target voxel feature and target voxel coordinates, wherein the target voxel feature is the first voxel feature matrix, and the target voxel coordinate is any coordinate in the first voxel coordinate matrix; controlling the target voxel coordinates to be sequentially input into a multi-layer dense feature space, and whenever any dense feature space receives the target voxel coordinates, controlling the dense feature space to output multiple target sparse features, wherein the sparse convolutional backbone network contains multiple layers of dense feature spaces; and obtaining new features based on the multiple target sparse features.

[0012] Furthermore, based on multiple target sparse features, new features are obtained, including: determining the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of multiple dense feature spaces; within the target dense feature space, taking the target voxel coordinates as the center and the spatial region size as the bounding range, determining multiple sparse features existing within the target dense feature space; calculating the average value corresponding to multiple target sparse features for each dense feature space output, to obtain multiple average values ​​corresponding to multiple dense feature spaces, wherein there is a one-to-one correspondence between the dense feature spaces and the average values; and concatenating the multiple average values ​​to obtain new features.

[0013] Furthermore, after inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network and controlling the sparse convolutional backbone network to classify the voxel feature matrix, the method further includes: when the classification result is a target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the first label; when the classification result is a non-target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the second label.

[0014] Furthermore, before inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network, the method further includes: constructing an initial sparse convolutional backbone network; determining the loss function used by the sparse convolutional backbone network to classify voxel features as a preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, which is: L all =αL 3Ddetection +βL sparse-feature-classification α takes the value of 10, β takes the value of 1, and L 3Ddetection Let L be the 3D target loss function. sparse-feature-classification The sparse feature classification loss is used; an initial sparse convolutional backbone network is trained using a preset loss function to obtain the sparse convolutional backbone network.

[0015] Furthermore, within the target dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding range, multiple sparse features existing within the target dense feature space are determined, including: when the target dense feature space is a first-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding range, 8 sparse features are determined within the first-layer dense feature space; when the target dense feature space is a second-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding range, 4 sparse features are determined within the second-layer dense feature space; and when the target dense feature space is a third-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding range, 2 sparse features are determined within the third-layer dense feature space.

[0016] According to another aspect of the present invention, a target object detection device is also provided, comprising: a first processing unit, configured to acquire an original point cloud, perform voxelization processing on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix; a first control unit, configured to input the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network, control the sparse convolutional backbone network to classify the voxel feature matrix, and control the sparse convolutional backbone network to output target features, wherein the target features are features obtained after removing non-target features, and the target features are features corresponding to the target object; a densification processing unit, configured to perform densification processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features not corresponding to the target; and a detection unit, configured to detect the target object based on the dense feature map.

[0017] According to another aspect of the present invention, a vehicle is also provided, comprising: at least one depth camera disposed at a preset position on the vehicle, a target detection device, and a target detection method for performing any one of the above.

[0018] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes any one of the target detection methods.

[0019] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes any one of the target detection methods.

[0020] According to another aspect of the present invention, an electronic device is also provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for detecting a target for performing any one of the above.

[0021] In this embodiment of the invention, the original point cloud is acquired and voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix. The voxel feature matrix and the voxel coordinate matrix are input into a sparse convolutional backbone network, which is controlled to classify the voxel feature matrix and output target features, wherein the target features are the features obtained after removing non-target features and are the features corresponding to the target object. The target features are densified to obtain a dense feature map, wherein the dense feature map does not contain features not corresponding to the target. Based on the dense feature map, the target object is detected. This solves the technical problem in related technologies where the SECOND algorithm often results in false detections of green belts in complex scenes, leading to low detection accuracy of target objects. Thus, by classifying target features and background features, the detection efficiency of target objects is improved. Attached Figure Description

[0022] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0023] Figure 1 This is a flowchart of a target detection method provided according to an embodiment of this application; and

[0024] Figure 2 A schematic diagram showing the determination of voxel features and voxel coordinates is shown;

[0025] Figure 3 A schematic diagram illustrating the classification of voxel features and voxel coordinates using an improved sparse convolutional backbone network.

[0026] Figure 4 This is a schematic diagram of a target detection device provided according to an embodiment of this application. Detailed Implementation

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be an intermediate element present. Furthermore, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element via a third element.

[0031] As mentioned in the non-target technology section, in complex scenarios, the SECOND algorithm in the prior art often results in false detections of green belts, leading to low detection accuracy of target objects. In order to solve the above problems, in a typical embodiment of this application, a target object detection method and device, vehicle, processor and electronic equipment are provided.

[0032] According to an embodiment of this application, a method for detecting a target object is provided.

[0033] Figure 1This is a flowchart of a target object detection method provided according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0034] Step S101: Obtain the original point cloud and perform voxelization on the original point cloud to obtain the voxel feature matrix and the voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix.

[0035] Step S102: Input the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network, control the sparse convolutional backbone network to classify the voxel feature matrix, and control the sparse convolutional backbone network to output the target features, wherein the target features are the features obtained after removing non-target features, and the target features are the features corresponding to the target object.

[0036] Step S103: Perform densification processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features that are not corresponding to the target.

[0037] Step S104: Detect the target object based on the dense feature map.

[0038] This application proposes a target detection method. To address the false detection problem of green belts in the SECOND algorithm, this method, based on the SECOND algorithm, adds a sparse feature classification module to the sparse convolutional backbone network. This module classifies the sparse features output by the sparse convolutional backbone network into target features and background features. Then, only the target features are densified to generate a feature map, thereby removing interference from background features such as green belts and buildings. Since background features such as green belts and buildings are not input into subsequent RPN layers, false detection of green belts is avoided.

[0039] In one optional embodiment, the original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, including: determining the size of the voxel bounding box, wherein the size of the voxel bounding box includes the length, width, and height of the voxel bounding box; determining the point cloud range of the original point cloud, wherein the point cloud range includes at least the size range of the original point cloud in three directions corresponding to a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel spatial region; and determining the voxel feature matrix and the voxel coordinate matrix based on the size of the voxel bounding box and the point cloud range.

[0040] Based on the size of the voxel bounding box and the extent of the point cloud, the voxel feature matrix and voxel coordinate matrix are determined, including: dividing the original point cloud into multiple voxel blocks according to the size of the voxel bounding box and the extent of the point cloud corresponding to the original point cloud; identifying blank voxel blocks that do not contain the original point cloud, removing the blank voxel blocks, and obtaining target voxel blocks containing the original point cloud; determining at least one original point cloud contained in each target voxel block, and determining at least one point cloud coordinate of at least one original point cloud in a preset spatial coordinate system; determining the voxel block coordinates of each target voxel block in the preset spatial coordinate system; and determining the voxel feature matrix and voxel coordinate matrix based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates.

[0041] The above-mentioned places, Figure 2 A schematic diagram for determining voxel features and voxel coordinates, such as Figure 2 As shown, Figure 2 The example illustrates a voxel space of (3m, 3m, 3m). Based on this voxel space, a preset spatial coordinate system is established. In this coordinate system, the range of the overall point cloud is (0, 3m) along the x-axis, (0, 3m) along the y-axis, and (0, 3m) along the z-axis. Figure 2 In the example shown, the size of the voxel bounding box used to divide the point cloud is (1m, 1m, 1m). Therefore, dividing the entire point cloud using this voxel bounding box yields multiple closely adjacent voxel blocks, where... Figure 2 The four voxel blocks shown are four of a plurality of voxel blocks, and the size of each voxel block is the same as the size of the voxel frame. Figure 2 As can be seen from the voxel blocks shown, some voxel blocks contain multiple original point clouds, while others contain only one original point cloud. The voxel feature matrix and voxel coordinate matrix can be determined by dividing the voxel blocks.

[0042] Furthermore, through Figure 2 As can be seen from the example shown, Figure 2 The diagram shows four voxel blocks. Voxel block 1 contains two original point clouds, voxel block 2 does not contain any original point clouds and is a blank voxel block, and voxel blocks 3 and 4 each contain one original point cloud. Voxel block 2 is removed, and the voxel feature matrix and voxel coordinate matrix are determined from the remaining voxel blocks.

[0043] Furthermore, based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates, the voxel feature matrix and the voxel coordinate matrix are determined, including: determining the average coordinates corresponding to at least one point cloud coordinate contained in the target voxel block, and determining the average coordinates as the voxel feature corresponding to the target voxel block; determining multiple voxel features corresponding to multiple target voxel blocks; and determining the voxel feature matrix based on the multiple voxel features; obtaining the multiple voxel block coordinates corresponding to the multiple target voxel blocks in a preset spatial coordinate system; and obtaining the voxel coordinate matrix based on the multiple voxel block coordinates. Figure 2 In the example shown, voxel block 1 contains point cloud 1 and point cloud 2. Voxel block 2 is a blank voxel block that has been removed. Voxel block 3 contains point cloud 3, and voxel block 4 contains point cloud 4. The coordinates of point cloud 1 in the spatial coordinate system are (0.4, 0.5, 0.5), the coordinates of point cloud 2 in the spatial coordinate system are (0.5, 0.8, 0.5), the coordinates of point cloud 3 in the spatial coordinate system are (2.5, 2.6, 1.5), and the coordinates of point cloud 4 in the spatial coordinate system are (2.5, 2.8, 0.7). Since voxel block 1 contains two original point clouds, the average value of the coordinates of point cloud 1 and point cloud 2 is calculated as (0.4 + 0.5) / 2, (0.5 + 0.8 ... Therefore, the average coordinates of voxel block 1 are (0.45, 0.65, 0.65), and the corresponding voxel feature is (0.45, 0.65, 0.65). Voxel block 3 contains only one point cloud, and its average coordinates are (2.5, 2.6, 1.5). The corresponding voxel feature is (2.5, 2.6, 1.5). Similarly, the average coordinates of voxel block 4 are (2.5, 2.8, 0.7), and the corresponding voxel feature is (2.5, 2.8, 0.7). Therefore, based on the above deduction, the average coordinates of the point cloud corresponding to each voxel block in the spatial region are calculated, and multiple coordinates are obtained. The matrix formed by multiple coordinates is determined as the voxel feature matrix. The voxel coordinate matrix is ​​a matrix formed by multiple voxel coordinates, such as... Figure 2 As shown, the voxel space coordinates of voxel block 1 in voxel space are (0, 0, 0), the coordinates of voxel block 3 in the space coordinate system are (2, 2, 1), and the coordinates of voxel block 4 in the space coordinate system are (2, 2, 0). The coordinates of each voxel block in the space coordinate system are determined as the voxel coordinates corresponding to each voxel block, and the matrix formed by multiple voxel coordinates is determined as the voxel coordinate matrix.

[0044] In one optional embodiment, controlling the sparse convolutional backbone network to classify the voxel feature matrix includes: controlling the voxel feature matrix and the voxel coordinate matrix to be sequentially input into multiple convolutional modules, and controlling the multiple convolutional modules to sequentially output the first voxel feature matrix and the first voxel coordinate matrix to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the coordinates of the multiple first voxel matrices decrease sequentially; obtaining multiple new features corresponding to the voxel feature matrix based on the multiple first voxel matrices and the first voxel coordinate matrices; controlling the new features to be input into the fully connected layer in the sparse convolutional backbone network for classification, and controlling the fully connected layer to output the classification result, wherein the classification result is a non-target feature or a target feature.

[0045] As described above, this application mainly modifies the sparse convolutional backbone network layer based on the SECOND algorithm. By adding a sparse feature classification module, the sparse features output by the sparse convolutional backbone network are filtered. Only target features are retained, and dense feature maps are generated using the filtered features. Since background features are not added to the dense feature map, false detections of background areas such as green belts will not occur in subsequent detection stages. Specifically, as follows... Figure 3 As shown, a point cloud frame is first voxelized and then input into a sparse convolutional backbone network to obtain voxel features and corresponding voxel coordinates, such as... Figure 3The voxel space is calculated by first determining the N4x64 bounding box and its corresponding N4x4 voxel coordinates, and then calculating the spatial region of the voxel feature in the real point cloud space based on the coordinates and the size of the voxel bounding box. In this application, the length, width, and height of the voxel bounding box used for point cloud voxelization are (0.05, 0.05, 0.1), and the point cloud range is (x: 0~70.4, y: -40~40, z: -3~1), in meters. Therefore, the voxel space after voxelization is 41x1600x1408 (zyx). After passing through a sparse convolutional backbone network, the voxel space becomes 2x200x176, and the length, width, and height of the voxel bounding box become (0.4, 0.4, 2.0). Taking a coordinate value (0.1, 100, 50) in an N4x4 coordinate matrix as an example, let's illustrate how to calculate the spatial region of a voxel feature in the real point cloud space: Using (0, 1, 100, 50) and the dimensions (0.4, 0.4, 2.0) of the voxel bounding box, the center point of the bounding box is calculated as (0+50X0.4+0.5X0.4, -40+100X0.4+0.5x0.4, -3+1x2.0+0.5x2.0) = (20.2, 0.2, 0)(xyz), and the dimensions of the bounding box are (0.4, 0.4, 2.0). Using this method, the voxel feature spatial region corresponding to all coordinate points in the N4x4 matrix is ​​calculated. If a voxel feature spatial region contains the target point cloud (the point cloud within the labeled bounding box), then the classification ground truth value of this voxel feature is marked as 1; otherwise, it is marked as 0. Using the above method, we can obtain the classification labels of the sparse features output by the sparse convolutional backbone network.

[0046] Furthermore, based on multiple first voxel matrices and first voxel coordinate matrices, multiple new features corresponding to the voxel feature matrix are obtained, including: determining the target voxel feature and target voxel coordinates, wherein the target voxel feature is the first voxel feature matrix, and the target voxel coordinate is any coordinate in the first voxel coordinate matrix; controlling the target voxel coordinates to be sequentially input into a multi-layer dense feature space, and whenever any dense feature space receives the target voxel coordinates, controlling the dense feature space to output multiple target sparse features, wherein the sparse convolutional backbone network contains multiple layers of dense feature spaces; and obtaining new features based on the multiple target sparse features. Based on multiple target sparse features, new features are obtained, including: determining the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of multiple dense feature spaces; within the target dense feature space, taking the target voxel coordinates as the center and the spatial region size as the bounding range, determining multiple sparse features existing within the target dense feature space; calculating the average value corresponding to multiple target sparse features for each dense feature space output, to obtain multiple average values ​​corresponding to multiple dense feature spaces, wherein there is a one-to-one correspondence between the dense feature spaces and the average values; and concatenating the multiple average values ​​to obtain new features.

[0047] The above-mentioned places, specifically as follows Figure 3 As shown, Figure 3 The diagram illustrates the improved sparse convolutional backbone network. This patent primarily adds a sparse feature classification module beneath the sparse convolutional backbone network. The input to the sparse feature classification model is the output of each 3D sparse convolutional layer of the backbone network, i.e., voxel features and their corresponding coordinates. The function of the sparse feature classification module is to classify the output features of the backbone network, that is, to classify... Figure 3 Classification is performed using the mid-voxel features N4x64 and their corresponding coordinates N4x4. Then, background features are filtered out, retaining only target features, to construct new voxel features N5x64 and N5x4. Using these new voxel features and their corresponding coordinates, a dense feature map is obtained and input into the subsequent detection algorithm for target detection.

[0048] Specifically, with Figure 3 Taking the coordinates {0, 1, 2, 0} in the N4x4 coordinate matrix as an example, we can illustrate how to classify sparse features. First, the position of the coordinates in the real point cloud space is calculated based on the coordinates {0, 1, 2, 0} and the size of the voxel space, as described above. Then, the voxel features and coordinates output by each 3D sparse convolution in the backbone network are densified into the point cloud space. The densification method is the same as the original SECOND algorithm, using the real position calculated by the coordinates {0, 1, 2, 0} as the center, a spatial region with length, width, and height of (0.4, 0, 4, 2.0) is bounded in the dense feature space of each layer. Figure 3 (In the image, the yellow dashed boxes in each dense feature space). Figure 3As shown, in the first dense feature space, there are 8 sparse features within the yellow dashed box; in the second dense feature space, there are 4 sparse features within the yellow dashed box; and in the third dense feature space, there are 2 sparse features within the yellow dashed box. The average value of the sparse features within each layer is calculated to obtain features M1, M2, and M3. Then, M1, M2, M3, and M0 are concatenated to obtain a new feature M. M is input into a fully connected layer, and the output is a binary classification result: 0 indicates that the feature is a background feature, and 1 indicates that the feature is a target feature. The reason for using concatenated features from different layers as classification features is that the last layer features are deep features, containing only abstract information and insufficient discriminative information, resulting in poor classification performance. Shallow features, on the other hand, represent the shape and texture structure of point clouds, providing more discriminative information. Therefore, this patent uses the output features of each layer of the 3D sparse convolution in the backbone network, incorporating the features from each layer of the backbone network into the classification features for classification, thereby making the classification results more accurate. The coordinates in the N4x4 array are processed one by one to obtain the classification results of all sparse features. Then, the predicted results are used to filter the N4x64 features, removing background features, resulting in new voxel features N5x64 and coordinates N5x4. Finally, the filtered voxel features and coordinates are input into the subsequent algorithm processing flow.

[0049] In an optional embodiment, before inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network, the method further includes: constructing an initial sparse convolutional backbone network; determining the loss function corresponding to the classification of voxel features by the sparse convolutional backbone network as a preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, which is:

[0050] L all =αL 3Ddetection +βL sparse-feature-classification α takes the value of 10, β takes the value of 1, and L 3Ddetection Let L be the 3D target loss function. sparse-feature-classification The sparse feature classification loss is used; an initial sparse convolutional backbone network is trained using a preset loss function to obtain the sparse convolutional backbone network. The reason for using focal loss instead of conventional cross-entropy loss in this application is that the proportion of positive class samples in sparse features is relatively small, resulting in far fewer positive samples than negative samples.

[0051] The target object detection method provided in this application classifies the sparse features output by the sparse convolutional backbone network into target features and background features by adding a sparse feature classification module. Then, only the target features are densified to generate a feature map, thereby removing interference from background features such as green belts and buildings. Since the features of backgrounds such as green belts and buildings are not input into subsequent RPN layers, false detections of green belts are avoided. To make the classification results more accurate, this patent uses features from different layers in the middle of the sparse convolutional module concatenated with the features of the last layer as input to the feature classification module. Because shallow features carry more geometric shape, edge, and other texture information, while deep features carry more semantic information, concatenating features from different layers makes the features more discriminative, thus making the classification more accurate and improving the detection efficiency of target objects.

[0052] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0053] This application also provides a target object detection device. It should be noted that the target object detection device of this application can be used to execute the target object detection method provided in this application. The following describes the target object detection device provided in this application.

[0054] Figure 4 This is a schematic diagram of a target object detection device provided according to an embodiment of this application. Figure 4 As shown, the device includes: a first processing unit 401, used to acquire the original point cloud, perform voxelization processing on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix; a first control unit 402, used to input the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network, control the sparse convolutional backbone network to classify the voxel feature matrix, and control the sparse convolutional backbone network to output target features, wherein the target features are the features obtained after removing non-target features, and the target features are the features corresponding to the target object; a densification processing unit 403, used to perform densification processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features corresponding to non-target objects; and a detection unit 404, used to detect the target object based on the dense feature map.

[0055] In an optional embodiment, the first processing unit 401 includes: a first determining subunit for determining the size of the voxel frame, wherein the size of the voxel frame includes the length, width, and height of the voxel frame; a second determining subunit for determining the point cloud range of the original point cloud, wherein the point cloud range includes at least the size range of the original point cloud in three directions of a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel spatial region; and a third determining subunit for determining the voxel feature matrix and the voxel coordinate matrix based on the size of the voxel frame and the point cloud range.

[0056] In an optional embodiment, the third determining subunit includes: a partitioning module, used to divide the original point cloud into multiple voxel blocks according to the size of the voxel frame and the point cloud range corresponding to the original point cloud; a first determining module, used to determine blank voxel blocks that do not contain the original point cloud, remove the blank voxel blocks, and obtain target voxel blocks containing the original point cloud; a second determining module, used to determine at least one original point cloud contained in each target voxel block, and determine at least one point cloud coordinate in a preset spatial coordinate system; a third determining module, used to determine the voxel block coordinates in the preset spatial coordinate system for each target voxel block; and a fourth determining module, used to determine the voxel feature matrix and the voxel coordinate matrix based on the at least one point cloud coordinate in the target voxel block and the voxel block coordinates.

[0057] In an optional embodiment, the fourth determining module includes: a first determining submodule, configured to determine the average coordinates corresponding to at least one point cloud coordinate contained within the target voxel block, and determine the average coordinates as the voxel feature corresponding to the target voxel block, and determine multiple voxel features corresponding to multiple target voxel blocks, and determine a voxel feature matrix based on the multiple voxel features; and a first obtaining submodule, configured to obtain multiple voxel block coordinates corresponding to multiple target voxel blocks in a preset spatial coordinate system, and obtain a voxel coordinate matrix based on the multiple voxel block coordinates.

[0058] In an optional embodiment, the first control unit 402 includes: a first control subunit, configured to control the voxel feature matrix and the voxel coordinate matrix to be sequentially input into multiple convolutional modules, and to control the multiple convolutional modules to sequentially output the first voxel feature matrix and the first voxel coordinate matrix to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the coordinates of the multiple first voxel matrices decrease sequentially; a first obtaining subunit, configured to obtain multiple new features corresponding to the voxel feature matrix based on the multiple first voxel matrices and the first voxel coordinate matrix; and a second control subunit, configured to control the new features to be input into the fully connected layer in the sparse convolutional backbone network for classification, and to control the fully connected layer to output the classification result, wherein the classification result is a non-target feature or a target feature.

[0059] In an optional embodiment, the first obtaining subunit includes: a fifth determining module, used to determine target voxel features and target voxel coordinates, wherein the target voxel features are a first voxel feature matrix, and the target voxel coordinates are any coordinates in the first voxel coordinate matrix; a first controlling module, used to control the target voxel coordinates to be sequentially input into a multi-layer dense feature space, and whenever any dense feature space receives the target voxel coordinates, control the dense feature space to output multiple target sparse features, wherein the sparse convolutional backbone network contains multiple layers of dense feature spaces; and an obtaining module, used to obtain new features based on the multiple target sparse features.

[0060] In an optional embodiment, the acquisition module includes: a second determining submodule, configured to determine the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of multiple dense feature spaces; a third determining submodule, configured to determine multiple sparse features existing within the target dense feature space, centered on the target voxel coordinates and bounded by the size of the spatial region; a calculation submodule, configured to calculate the average value corresponding to the multiple target sparse features output by each dense feature space, to obtain multiple average values ​​corresponding to the multiple dense feature spaces, wherein the dense feature spaces correspond one-to-one with the average values; and an acquisition submodule, configured to control the concatenation of the multiple average values ​​to obtain new features.

[0061] In an optional embodiment, the device further includes: a first determining unit, configured to, after inputting the voxel feature matrix and the voxel coordinate matrix into the sparse convolutional backbone network and controlling the sparse convolutional backbone network to classify the voxel feature matrix, determine the classification ground value of the voxel feature corresponding to the target voxel coordinate as a first label when the classification result is a target feature; and a second determining unit, configured to, when the classification result is a non-target feature, determine the classification ground value of the voxel feature corresponding to the target voxel coordinate as a second label.

[0062] In an optional embodiment, the device further includes: a construction unit, configured to construct an initial sparse convolutional backbone network before inputting the voxel feature matrix and the voxel coordinate matrix into the sparse convolutional backbone network; and a third determining unit, configured to determine that the loss function corresponding to the classification of voxel features by the sparse convolutional backbone network is a preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, which is: L all =αL 3Ddetection +βL sparse-feature-classification α takes the value of 10, β takes the value of 1, and L 3Ddetection Let L be the 3D target loss function. sparse-feature-classification The loss function is used for sparse feature classification; the acquisition unit is used to train an initial sparse convolutional backbone network through a preset loss function to obtain the sparse convolutional backbone network.

[0063] In an optional embodiment, the third determining submodule includes: a fourth determining submodule, used to determine, when the target dense feature space is a first layer dense feature space, that there are 8 sparse features in the first layer dense feature space, with the target voxel coordinates as the center and the size of the spatial region as the bounding range; a fifth determining submodule, used to determine, when the target dense feature space is a second layer dense feature space, that there are 4 sparse features in the second layer dense feature space, with the target voxel coordinates as the center and the size of the spatial region as the bounding range; and a sixth determining submodule, used to determine, when the target dense feature space is a third layer dense feature space, that there are 2 sparse features in the third layer dense feature space, with the target voxel coordinates as the center and the size of the spatial region as the bounding range.

[0064] A target detection device includes a processor and a memory. The first processing unit 401 and the like are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0065] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, the technical problem of low detection accuracy of targets caused by false detections of green belts in complex scenarios using the SECOND algorithm can be addressed.

[0066] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0067] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for detecting a target object.

[0068] This invention provides a processor for running a program, wherein the program executes a method for detecting a target object.

[0069] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: acquiring an original point cloud; performing voxelization on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix; inputting the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network; controlling the sparse convolutional backbone network to classify the voxel feature matrix and controlling the sparse convolutional backbone network to output target features, wherein the target features are features obtained after removing non-target features, and the target features are features corresponding to the target object; performing densification processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features not corresponding to the target; and detecting the target object based on the dense feature map.

[0070] Optionally, the original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, including: determining the size of the voxel bounding box, wherein the size of the voxel bounding box includes the length, width, and height of the voxel bounding box; determining the point cloud range of the original point cloud, wherein the point cloud range includes at least the size range of the original point cloud in three directions corresponding to a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel spatial region; and determining the voxel feature matrix and the voxel coordinate matrix based on the size of the voxel bounding box and the point cloud range.

[0071] Optionally, based on the size of the voxel bounding box and the point cloud range, the voxel feature matrix and voxel coordinate matrix are determined, including: dividing the original point cloud into multiple voxel blocks based on the size of the voxel bounding box and the point cloud range corresponding to the original point cloud; identifying blank voxel blocks that do not contain the original point cloud, removing the blank voxel blocks, and obtaining target voxel blocks containing the original point cloud; determining at least one original point cloud contained in each target voxel block, and determining at least one point cloud coordinate of at least one original point cloud in a preset spatial coordinate system; determining the voxel block coordinates of each target voxel block in the preset spatial coordinate system; and determining the voxel feature matrix and voxel coordinate matrix based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates.

[0072] Optionally, based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates, the voxel feature matrix and the voxel coordinate matrix are determined, including: determining the average coordinates corresponding to at least one point cloud coordinate contained in the target voxel block, and determining the average coordinates as the voxel feature corresponding to the target voxel block, and determining multiple voxel features corresponding to multiple target voxel blocks, and determining the voxel feature matrix based on the multiple voxel features; obtaining multiple voxel block coordinates corresponding to multiple target voxel blocks in a preset spatial coordinate system, and obtaining the voxel coordinate matrix based on the multiple voxel block coordinates.

[0073] Optionally, controlling the sparse convolutional backbone network to classify the voxel feature matrix includes: controlling the voxel feature matrix and voxel coordinate matrix to be sequentially input into multiple convolutional modules, and controlling the multiple convolutional modules to sequentially output the first voxel feature matrix and the first voxel coordinate matrix to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the coordinates of the multiple first voxel matrices decrease sequentially; obtaining multiple new features corresponding to the voxel feature matrix based on the multiple first voxel matrices and the first voxel coordinate matrices; controlling the new features to be input into the fully connected layer in the sparse convolutional backbone network for classification, and controlling the fully connected layer to output the classification result, wherein the classification result is a non-target feature or a target feature.

[0074] Optionally, based on multiple first voxel matrices and first voxel coordinate matrices, multiple new features corresponding to the voxel feature matrix are obtained, including: determining the target voxel feature and target voxel coordinates, wherein the target voxel feature is the first voxel feature matrix, and the target voxel coordinates are any coordinates in the first voxel coordinate matrix; controlling the target voxel coordinates to be sequentially input into a multi-layer dense feature space, and whenever any dense feature space receives the target voxel coordinates, controlling the dense feature space to output multiple target sparse features, wherein the sparse convolutional backbone network contains multiple layers of dense feature spaces; and obtaining new features based on the multiple target sparse features.

[0075] Optionally, new features are obtained based on multiple target sparse features, including: determining the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of multiple dense feature spaces; within the target dense feature space, taking the target voxel coordinates as the center and the spatial region size as the bounding range, determining multiple sparse features existing within the target dense feature space; calculating the average value corresponding to the multiple target sparse features output by each dense feature space to obtain multiple average values ​​corresponding to multiple dense feature spaces, wherein there is a one-to-one correspondence between the dense feature spaces and the average values; and concatenating the multiple average values ​​to obtain new features.

[0076] Optionally, after inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network and controlling the sparse convolutional backbone network to classify the voxel feature matrix, the method further includes: if the classification result is a target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the first label; if the classification result is a non-target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the second label.

[0077] Optionally, before inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network, the method further includes: constructing an initial sparse convolutional backbone network; determining the loss function corresponding to the classification of voxel features by the sparse convolutional backbone network as a preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, which is: L all =αL 3Ddetection +βL sparse-feature-classification α takes the value of 10, β takes the value of 1, and L 3Ddetection Let L be the 3D target loss function. sparse-feature-classification The sparse feature classification loss is used; an initial sparse convolutional backbone network is trained using a preset loss function to obtain the sparse convolutional backbone network.

[0078] Optionally, within the target dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding area, multiple sparse features existing within the target dense feature space are determined. These include: if the target dense feature space is a first-layer dense feature space, then using the target voxel coordinates as the center and the spatial region size as the bounding area, 8 sparse features are determined within the first-layer dense feature space; if the target dense feature space is a second-layer dense feature space, then using the target voxel coordinates as the center and the spatial region size as the bounding area, 4 sparse features are determined within the second-layer dense feature space; and if the target dense feature space is a third-layer dense feature space, then using the target voxel coordinates as the center and the spatial region size as the bounding area, 2 sparse features are determined within the third-layer dense feature space. The device mentioned in this paper can be a server, PC, PAD, mobile phone, etc.

[0079] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: acquiring an original point cloud; performing voxelization on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix; inputting the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network; controlling the sparse convolutional backbone network to classify the voxel feature matrix; and controlling the sparse convolutional backbone network to output target features, wherein the target features are features obtained after removing non-target features, and the target features are features corresponding to the target object; performing densification on the target features to obtain a dense feature map, wherein the dense feature map does not contain features not corresponding to the target; and detecting the target object based on the dense feature map.

[0080] Optionally, the original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, including: determining the size of the voxel bounding box, wherein the size of the voxel bounding box includes the length, width, and height of the voxel bounding box; determining the point cloud range of the original point cloud, wherein the point cloud range includes at least the size range of the original point cloud in three directions corresponding to a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel spatial region; and determining the voxel feature matrix and the voxel coordinate matrix based on the size of the voxel bounding box and the point cloud range.

[0081] Optionally, based on the size of the voxel bounding box and the point cloud range, the voxel feature matrix and voxel coordinate matrix are determined, including: dividing the original point cloud into multiple voxel blocks based on the size of the voxel bounding box and the point cloud range corresponding to the original point cloud; identifying blank voxel blocks that do not contain the original point cloud, removing the blank voxel blocks, and obtaining target voxel blocks containing the original point cloud; determining at least one original point cloud contained in each target voxel block, and determining at least one point cloud coordinate of at least one original point cloud in a preset spatial coordinate system; determining the voxel block coordinates of each target voxel block in the preset spatial coordinate system; and determining the voxel feature matrix and voxel coordinate matrix based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates.

[0082] Optionally, based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates, the voxel feature matrix and the voxel coordinate matrix are determined, including: determining the average coordinates corresponding to at least one point cloud coordinate contained in the target voxel block, and determining the average coordinates as the voxel feature corresponding to the target voxel block, and determining multiple voxel features corresponding to multiple target voxel blocks, and determining the voxel feature matrix based on the multiple voxel features; obtaining multiple voxel block coordinates corresponding to multiple target voxel blocks in a preset spatial coordinate system, and obtaining the voxel coordinate matrix based on the multiple voxel block coordinates.

[0083] Optionally, controlling the sparse convolutional backbone network to classify the voxel feature matrix includes: controlling the voxel feature matrix and voxel coordinate matrix to be sequentially input into multiple convolutional modules, and controlling the multiple convolutional modules to sequentially output the first voxel feature matrix and the first voxel coordinate matrix to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the coordinates of the multiple first voxel matrices decrease sequentially; obtaining multiple new features corresponding to the voxel feature matrix based on the multiple first voxel matrices and the first voxel coordinate matrices; controlling the new features to be input into the fully connected layer in the sparse convolutional backbone network for classification, and controlling the fully connected layer to output the classification result, wherein the classification result is a non-target feature or a target feature.

[0084] Optionally, based on multiple first voxel matrices and first voxel coordinate matrices, multiple new features corresponding to the voxel feature matrix are obtained, including: determining the target voxel feature and target voxel coordinates, wherein the target voxel feature is the first voxel feature matrix, and the target voxel coordinates are any coordinates in the first voxel coordinate matrix; controlling the target voxel coordinates to be sequentially input into a multi-layer dense feature space, and whenever any dense feature space receives the target voxel coordinates, controlling the dense feature space to output multiple target sparse features, wherein the sparse convolutional backbone network contains multiple layers of dense feature spaces; and obtaining new features based on the multiple target sparse features.

[0085] Optionally, new features are obtained based on multiple target sparse features, including: determining the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of multiple dense feature spaces; within the target dense feature space, taking the target voxel coordinates as the center and the spatial region size as the bounding range, determining multiple sparse features existing within the target dense feature space; calculating the average value corresponding to the multiple target sparse features output by each dense feature space to obtain multiple average values ​​corresponding to multiple dense feature spaces, wherein there is a one-to-one correspondence between the dense feature spaces and the average values; and concatenating the multiple average values ​​to obtain new features.

[0086] Optionally, after inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network and controlling the sparse convolutional backbone network to classify the voxel feature matrix, the method further includes: if the classification result is a target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the first label; if the classification result is a non-target feature, determining the classification ground value of the voxel feature corresponding to the target voxel coordinate as the second label.

[0087] Optionally, before inputting the voxel feature matrix and voxel coordinate matrix into the sparse convolutional backbone network, the method further includes: constructing an initial sparse convolutional backbone network; determining the loss function corresponding to the classification of voxel features by the sparse convolutional backbone network as a preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, which is: L all =αL 3Ddetection +βL sparse-feature-classification α takes the value of 10, β takes the value of 1, and L 3Ddetection Let L be the 3D target loss function. sparse-feature-classification The sparse feature classification loss is used; an initial sparse convolutional backbone network is trained using a preset loss function to obtain the sparse convolutional backbone network.

[0088] Optionally, within the target dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding area, multiple sparse features existing within the target dense feature space are determined, including: if the target dense feature space is a first-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding area, 8 sparse features are determined within the first-layer dense feature space; if the target dense feature space is a second-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding area, 4 sparse features are determined within the second-layer dense feature space; if the target dense feature space is a third-layer dense feature space, using the target voxel coordinates as the center and the spatial region size as the bounding area, 2 sparse features are determined within the third-layer dense feature space.

[0089] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0092] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0094] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0095] 1) By adding a sparse feature classification module, the sparse features output by the sparse convolutional backbone network are classified into target features and background features. Then, only the target features are densified to generate a feature map, thereby removing interference from background features such as green belts and buildings. Since the features of background elements such as green belts and buildings are not input into subsequent RPN layers, false detections of green belts are avoided, achieving the technical effect of improving target object detection efficiency.

[0096] 2) To make the classification results more accurate, this patent uses features from different layers in the middle of the sparse convolution module and the features from the last layer to concatenate as input to the feature classification module. Since shallow features carry more geometric shape, edge and other texture information, and deep features carry more semantic information, concatenating features from different layers can make the features more discriminative, thereby making the classification more accurate and improving the accuracy and efficiency of background information recognition.

[0097] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting a target object, characterized in that, include: The original point cloud is obtained, and the original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix. The voxel feature matrix and the voxel coordinate matrix are input into a sparse convolutional backbone network. The sparse convolutional backbone network is controlled to classify the voxel feature matrix and output target features. The target features are the features obtained after removing non-target features and are the features corresponding to the target object. The target features are densified to obtain a dense feature map, wherein the dense feature map does not contain features that do not correspond to the target. The target object is detected based on the dense feature map. Controlling the sparse convolutional backbone network to classify the voxel feature matrix includes: controlling the voxel feature matrix and voxel coordinate matrix to be sequentially input into multiple convolutional modules, and controlling the multiple convolutional modules to sequentially output a first voxel feature matrix and a first voxel coordinate matrix to obtain multiple first voxel matrices and first voxel coordinate matrices, wherein the dimensions of the multiple first voxel matrices and the multiple first voxel coordinate matrices decrease sequentially; based on the multiple first voxel matrices and the first voxel coordinate matrices, obtaining multiple new features corresponding to the voxel feature matrix includes: determining target voxel features and target voxel coordinates, wherein the target voxel features are the... A first voxel feature matrix is ​​used, and the target voxel coordinates are any coordinates in the first voxel coordinate matrix. The target voxel coordinates are sequentially input into multiple dense feature spaces. Whenever any of the dense feature spaces receives the target voxel coordinates, the dense feature spaces output multiple target sparse features. The sparse convolutional backbone network contains multiple dense feature spaces. New features are obtained based on the multiple target sparse features. The new features are input into the fully connected layers of the sparse convolutional backbone network for classification, and the fully connected layers output classification results, where the classification results are either non-target features or target features. Based on multiple target sparse features, a new feature is obtained, including: determining the size of the spatial region corresponding to the target dense feature space, wherein the target dense feature space is any one of the multiple dense feature spaces; within the target dense feature space, taking the target voxel coordinates as the center and the spatial region size as the bounding range, determining multiple sparse features existing within the target dense feature space; calculating the average value corresponding to the multiple target sparse features for each dense feature space to obtain multiple average values ​​corresponding to the multiple dense feature spaces, wherein the dense feature spaces correspond one-to-one with the average values; and concatenating the multiple average values ​​to obtain the new feature.

2. The method according to claim 1, characterized in that, The original point cloud is voxelized to obtain a voxel feature matrix and a voxel coordinate matrix, including: Determine the size of the voxel frame, wherein the size of the voxel frame includes the length, width, and height of the voxel frame; The point cloud range of the original point cloud is determined, wherein the point cloud range includes at least the size range of the original point cloud in three directions of a preset spatial coordinate system, wherein the preset spatial coordinate system is a spatial coordinate system constructed based on a preset voxel space region; Based on the size of the voxel bounding box and the range of the point cloud, the voxel feature matrix and the voxel coordinate matrix are determined.

3. The method according to claim 2, characterized in that, Based on the size of the voxel bounding box and the range of the point cloud, the voxel feature matrix and the voxel coordinate matrix are determined, including: Based on the size of the voxel frame and the point cloud range corresponding to the original point cloud, the original point cloud is divided into multiple voxel blocks; Identify blank voxel blocks that do not contain the original point cloud, remove the blank voxel blocks, and obtain the target voxel block that contains the original point cloud. Determine at least one original point cloud contained in each target voxel block, and determine at least one point cloud coordinate of at least one original point cloud in the preset spatial coordinate system; Determine the voxel coordinates of each target voxel in the preset spatial coordinate system; The voxel feature matrix and the voxel coordinate matrix are determined based on at least one of the point cloud coordinates in the target voxel block and the voxel block coordinates.

4. The method according to claim 3, characterized in that, Determining the voxel feature matrix and the voxel coordinate matrix based on at least one point cloud coordinate in the target voxel block and the voxel block coordinates includes: The average coordinates corresponding to at least one point cloud coordinate contained in the target voxel block are determined, and the average coordinates are determined as the voxel feature corresponding to the target voxel block. Multiple voxel features corresponding to multiple target voxel blocks are determined, and the voxel feature matrix is ​​determined based on the multiple voxel features. Obtain the coordinates of multiple target voxels in the preset spatial coordinate system, and obtain the voxel coordinate matrix based on the coordinates of the multiple voxels.

5. The method according to claim 1, characterized in that, After inputting the voxel feature matrix and the voxel coordinate matrix into a sparse convolutional backbone network and controlling the sparse convolutional backbone network to classify the voxel feature matrix, the method further includes: If the classification result is the target feature, the classification ground value of the voxel feature corresponding to the target voxel coordinates is determined as the first label; If the classification result is the non-target feature, the classification truth value of the voxel feature corresponding to the target voxel coordinates is determined as the second label.

6. The method according to claim 1, characterized in that, Before inputting the voxel feature matrix and the voxel coordinate matrix into the sparse convolutional backbone network, the method further includes: Construct the initial sparse convolutional backbone network; The loss function corresponding to the classification of the voxel features by the sparse convolutional backbone network is determined to be the preset loss function, wherein the formula corresponding to the preset loss function is Formula 1, and Formula 1 is: , The value is 10. The value is 1. For 3D target loss function, For sparse feature classification loss; The initial sparse convolutional backbone network is trained using the preset loss function to obtain the sparse convolutional backbone network.

7. The method according to claim 1, characterized in that, Within the target dense feature space, taking the target voxel coordinates as the center and the size of the spatial region as the bounding area, determine multiple sparse features existing within the target dense feature space, including: When the target dense feature space is the first layer dense feature space, with the target voxel coordinates as the center and the size of the space region as the bounding range, it is determined that there are 8 sparse features in the first layer dense feature space. When the target dense feature space is the second layer dense feature space, with the target voxel coordinates as the center and the size of the spatial region as the bounding range, it is determined that there are 4 sparse features in the second layer dense feature space. When the target dense feature space is a third-layer dense feature space, with the target voxel coordinates as the center and the size of the spatial region as the bounding range, it is determined that there are 2 sparse features in the third-layer dense feature space.

8. A target object detection device, characterized in that, A method for detecting a target object according to any one of claims 1 to 7, comprising: The first processing unit is used to acquire the original point cloud and perform voxelization processing on the original point cloud to obtain a voxel feature matrix and a voxel coordinate matrix, wherein the voxel features in the voxel feature matrix correspond one-to-one with the coordinates in the voxel coordinate matrix. The first control unit is configured to input the voxel feature matrix and the voxel coordinate matrix into the sparse convolutional backbone network, control the sparse convolutional backbone network to classify the voxel feature matrix, and control the sparse convolutional backbone network to output target features, wherein the target features are features obtained after removing non-target features, and the target features are features corresponding to the target object. A dense processing unit is used to perform dense processing on the target features to obtain a dense feature map, wherein the dense feature map does not contain features that are not corresponding to the target; The detection unit is used to detect the target object based on the dense feature map.

9. A vehicle, characterized in that, include: At least one depth camera, wherein the depth camera is disposed at a preset position on the vehicle, and a target detection device is used to perform a target detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes a method for detecting a target object as described in any one of claims 1 to 7.

11. A processor, characterized in that, The processor is used to run a program, wherein the program executes a target detection method according to any one of claims 1 to 7.

12. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising a method for performing a target detection method according to any one of claims 1 to 7.

Images