Target detection method, device, electronic device and storage medium

Abstract

The application provides a target detection method and device, electronic equipment and storage medium, wherein the method comprises: determining a foreground image and an overhead view based on point cloud data to be detected; performing feature extraction on the foreground image to obtain semantic features of the point cloud data; performing two-dimensional grid division on the overhead view, and determining geometric features of the point cloud data based on the two-dimensional grid obtained by the division; and performing target detection based on the semantic features and the geometric features of the point cloud data. The target detection method, device, electronic equipment and storage medium provided by the application fuse the semantic features and the geometric features of the point cloud data, supplement rich feature information, and improve the precision and accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target detection method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence and computer vision, tasks in the field of 2D image vision, such as classification, object detection, and tracking, have been well solved and widely applied. However, 3D visual perception tasks based on point clouds have not been well solved, often losing image feature information, resulting in inaccurate 3D object detection results.

[0003] At present, a mainstream 3D visual perception method based on point cloud is to use the top view to extract features to complete target detection. However, this method ignores the semantic features and has poor performance in the top view. Figure 2 In the process of 3D grid division, the relationship between adjacent point pairs is ignored, and there is lost feature information, which leads to inaccurate 3D target detection results. Summary of the Invention

[0004] The present invention provides a target detection method, device, electronic device and storage medium, which are used to solve the problem of low target detection accuracy in the prior art.

[0005] The present invention provides a target detection method, comprising:

[0006] Determine a foreground image and a top view based on the point cloud data to be detected;

[0007] Performing feature extraction on the foreground image to obtain semantic features of the point cloud data;

[0008] Performing two-dimensional grid division on the top view, and determining geometric features of the point cloud data based on the two-dimensional grid obtained by the division;

[0009] Target detection is performed based on the semantic features and geometric features of the point cloud data.

[0010] According to a target detection method provided by the present invention, target detection is performed based on the semantic features and geometric features of the point cloud data, including:

[0011] Determining a fusion feature of each point in the point cloud data based on the semantic features and geometric features of the point cloud data;

[0012] Clustering the center points of the non-empty two-dimensional grids obtained by division to obtain multiple center point sets;

[0013] Based on the multiple center point sets, performing aggregation extraction and dispersion operations on the fusion features of each point in the point cloud data to obtain a bird's-eye view feature map;

[0014] Target detection is performed based on the top-view feature map.

[0015] According to a target detection method provided by the present invention, target detection is performed based on the top view feature map, including:

[0016] Performing multi-scale feature extraction on the top-view feature map to obtain features of the top-view feature map at multiple scales;

[0017] Determining fusion weights of the multiple scales based on mean values ​​of features of the top-view feature map at the multiple scales;

[0018] Aggregating the features of the top-view feature map at the multiple scales based on the fusion weights of the multiple scales to obtain aggregated features;

[0019] Target detection is performed based on the aggregated features.

[0020] According to a target detection method provided by the present invention, extracting features from the foreground image to obtain semantic features of the point cloud data includes:

[0021] constructing a vector representation of the foreground image based on the three-dimensional coordinates, reflection intensity, distance, and vertical angle of each point in the foreground image;

[0022] Multi-scale feature extraction is performed on the vector representation of the foreground image to obtain multi-scale features, and the multi-scale features are fused to obtain a foreground feature map, where the foreground feature map represents the semantic features of the point cloud data.

[0023] According to a target detection method provided by the present invention, determining the geometric features of the point cloud data based on the two-dimensional grid obtained by division includes:

[0024] The geometric features of the point cloud data are determined based on the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the average coordinates of all points in the grid in which it is located, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid.

[0025] According to a target detection method provided by the present invention, determining a foreground image based on point cloud data to be detected includes:

[0026] The point cloud data is projected onto a front view, and the closest points of the point cloud data at the same pixel position are retained in the front view to obtain the foreground image.

[0027] The present invention also provides a target detection device, comprising:

[0028] An image determination unit, configured to determine a foreground image and a top view based on the point cloud data to be detected;

[0029] a semantic feature determination unit, configured to extract features from the foreground image to obtain semantic features of the point cloud data;

[0030] a geometric feature determination unit, configured to divide the top view into two-dimensional grids, and determine the geometric features of the point cloud data based on the two-dimensional grids obtained by the division;

[0031] The target detection unit is used to perform target detection based on the semantic features and geometric features of the point cloud data.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the target detection methods described above is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the target detection methods described above when executed by a processor.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the target detection methods described above.

[0035] The present invention provides a target detection method, device, electronic device and storage medium, which determine the fusion features of each point in the point cloud data by fusing the semantic features of the obtained point cloud data with the geometric features of the point cloud data. The fusion features here contain richer feature information. The fusion features are then combined with multiple center point sets obtained by clustering the center points of the non-empty two-dimensional grids obtained by division, and aggregation extraction and dispersion operations are performed to obtain a bird's-eye view feature map. Target detection is performed based on the bird's-eye view feature map. The target detection performed in this way will not lose feature information and can effectively improve the precision and accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the target detection method provided by the present invention;

[0038] Figure 2 It is a schematic diagram of the process of extracting the top view feature map provided by the present invention;

[0039] Figure 3 It is a schematic diagram of the process of selecting the center point of the top view provided by the present invention;

[0040] Figure 4 Schematic diagram of the structure of the attention network provided by the present invention;

[0041] Figure 5 This is the second flow chart of the target detection method provided by the present invention;

[0042] Figure 6 It is a schematic diagram of the process of extracting the foreground feature map provided by the present invention;

[0043] Figure 7 It is a structural schematic diagram of the target detection device provided by the present invention;

[0044] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] An embodiment of the present invention provides a target detection method. Figure 1 This is one of the flow charts of the target detection method provided by the present invention, such as Figure 1 As shown, the method includes:

[0047] Step 110 : determining a foreground image and a top view based on the point cloud data to be detected.

[0048] Specifically, by projecting the point cloud data to be detected in the three-dimensional space from a forward perspective, a foreground image of the point cloud data to be detected can be obtained; by projecting the point cloud data to be detected in the three-dimensional space from a downward perspective, a top view of the point cloud data to be detected can be obtained.

[0049] Step 120 : performing feature extraction on the foreground image to obtain semantic features of the point cloud data.

[0050] Specifically, the foreground image can reflect the form and features of the target itself from the perspective of the foreground, and the foreground image covers rich semantic information of the target. Therefore, the foreground image can be feature extracted to obtain the semantic features of the point cloud data. The semantic features here can reflect the positional relationship between each point in the point cloud data from the foreground perspective, thereby reflecting the semantic information of the target contained in the point cloud data. The feature extraction here can be implemented using convolutional neural networks (CNN), or using VGG (Visual Geometry Group) based on convolutional neural networks, or using residual neural networks (ResNet). The embodiment of the present invention does not specifically limit this. The feature extraction here can extract features of one scale or features of multiple scales.

[0051] Step 130 : performing two-dimensional grid division on the top view, and determining geometric features of the point cloud data based on the two-dimensional grid obtained by division.

[0052] Specifically, by performing a two-dimensional grid division on the top view, each point in the top view can be divided into a corresponding two-dimensional grid, obtaining the two-dimensional grid where each point in the top view is located. Based on the two-dimensional grid obtained by the division, the geometric characteristics of the point cloud data can be determined. The geometric characteristics here can reflect the characteristics of the two-dimensional grid where each point is located, and can also reflect the characteristics of each point and the grid as a whole. For example, it can reflect the difference between each point and the mean of all coordinates of the network in which it is located, and can also reflect the deviation of each point from the center of the grid. Therefore, the geometric characteristics of the point cloud data determined based on the two-dimensional grid obtained by the division can reflect the positional characteristics of the geometric distribution from a top-down perspective.

[0053] Step 140: Perform target detection based on the semantic features and geometric features of the point cloud data.

[0054] Specifically, by combining the semantic features obtained from the foreground perspective and the geometric features obtained from the bird's-eye view to perform target detection, the accuracy of target detection can be improved.

[0055] Compared with the traditional method of completing target detection by extracting features only from the top view, the method provided by the embodiment of the present invention integrates the semantic features and geometric features of point cloud data, supplements rich feature information, and improves the precision and accuracy of target detection.

[0056] Based on the above embodiments, Figure 2 FIG. 1 is a flow chart of extracting a top view feature map according to an embodiment of the present invention. Figure 2 As shown, step 140 includes:

[0057] Step 141 : determining a fusion feature of each point in the point cloud data based on the semantic features and geometric features of the point cloud data.

[0058] Specifically, the semantic features and geometric features of point cloud data are feature information in these two dimensions for each point in the point cloud data. The feature information in these two dimensions are spliced ​​or weighted to obtain the fusion features of each point in the point cloud data. The fusion features here can reflect the semantic features from the foreground perspective and the positional features of the geometric distribution from the bird's-eye view.

[0059] Step 142 : clustering the center points of the non-empty two-dimensional grids obtained by division to obtain a plurality of center point sets.

[0060] Specifically, Figure 3 This is a flow chart of selecting the center point of the top view provided by the present invention, such as Figure 3 As shown, the top view is divided into two-dimensional grids to obtain multiple two-dimensional grids of the top view. The multiple two-dimensional grids here can be obtained by judging whether the grid contains points, wherein the two-dimensional grid containing points is a non-empty two-dimensional grid, and the non-empty two-dimensional grid can determine its center point. A set is constructed for the center points of the non-empty two-dimensional grids obtained by division, and the center points in the set are clustered. The clustering here can be implemented by the K-Means Clustering Algorithm (K-Means), or by the Density-Based Spatial Clustering of Applications with Noise (DBSCAN), or by the AClustering Algorithm based on Randomized Search (CLARANS), which is not specifically limited in the embodiment of the present invention.

[0061] Specifically, the clustering results obtained by clustering the center points of the divided non-empty two-dimensional grids may include multiple center point sets, each center point set includes one or more center points. It can be understood that one center point set may correspond to one or more non-empty two-dimensional grids.

[0062] Step 143 : Based on the multiple center point sets, perform aggregation extraction and dispersion operations on the fusion features of each point in the point cloud data to obtain a bird's-eye view feature map.

[0063] Specifically, the division of the center point set realizes the clustering of non-empty two-dimensional grids in the top view, thereby clustering the points in the point cloud data from the top view, that is, the points under the non-empty two-dimensional grid corresponding to each center point in a center point set can be regarded as a set or a class. Based on multiple center point sets, the fusion features of each point in the point cloud data are aggregated, extracted and dispersed. The aggregation operation here refers to the use of the point set feature aggregation network to aggregate the two-dimensional grid point set features. The dispersion operation here refers to dividing the points of the point cloud data in the three-dimensional space into the grid corresponding to its top view according to the coordinates of each point in the point cloud data, and performing aggregation, extraction and dispersion operations on the fusion features of each point in the point cloud data to obtain a top view feature map.

[0064] Step 144: perform target detection based on the top-view feature map.

[0065] Specifically, after obtaining the overhead feature map, target detection can be performed based on the overhead feature map. The target detection here can use the detection head to output the target detection result, and can also perform target detection on the overhead feature map after feature extraction by performing feature extraction on the overhead feature map.

[0066] The method provided by the embodiment of the present invention clusters the center points of the non-empty two-dimensional grids obtained by division to obtain multiple center point sets, and aggregates, extracts and disperses the fusion features of each point in the point cloud data based on the multiple center point sets. This method is a dynamic operation process, thereby making the semantic features and geometric features extracted by the embodiment of the present invention richer, and can effectively improve the precision and accuracy of target detection.

[0067] Based on the above embodiment, step 144 includes:

[0068] Step 144 - 1 , performing multi-scale feature extraction on the top-view feature map to obtain features of the top-view feature map at multiple scales.

[0069] Specifically, multi-scale feature extraction is performed on the top-view feature map. The multi-scale feature extraction here can use a multi-layer convolutional network and perform downsampling to obtain features of the top-view feature map at multiple scales.

[0070] Step 144 - 2 : Determine fusion weights of the multiple scales based on the mean values ​​of the features of the top-view feature map at the multiple scales.

[0071] Specifically, Figure 4 is a schematic diagram of the structure of the attention network provided by the present invention, such as Figure 4 As shown in the figure, an attention network is used to learn the feature weights of the top view feature map at multiple scales, and a global average pooling layer (GAP) is used to calculate the mean of the features of the top view feature map at multiple scales, that is, abk =∑ x，y，c M bk , k=1,2,3, where M b It is a top view feature map, and then the average value [a b1 , a b2 , a b3 ] Input to a fully connected layer to learn weights w k ,pass Dynamically adjust multi-scale features and perform upsampling to obtain fusion weights of multiple scales.

[0072] Step 144 - 3 : Aggregate the features of the top-view feature map at the multiple scales based on the fusion weights of the multiple scales to obtain aggregated features.

[0073] Specifically, based on the fusion weights of multiple scales, the fusion weights here can reflect the importance of features at different scales, and the features of the top-view feature map at multiple scales are aggregated. The aggregation here is to perform weighted fusion on the features of the top-view feature map at multiple scales, thereby obtaining aggregated features.

[0074] Step 144 - 4 , performing target detection based on the aggregated features.

[0075] Specifically, the features of the top-view feature map at multiple scales are aggregated to obtain aggregated features for target detection.

[0076] Based on the above embodiment, step 120 includes:

[0077] Step 121 : constructing a vector representation of the foreground image based on the three-dimensional coordinates, reflection intensity, distance, and vertical angle of each point in the foreground image.

[0078] Specifically, the three-dimensional coordinates, reflection intensity, distance and vertical angle of each point in the foreground image are represented by [x, y, z], [i], [diatance], and [azimuth], respectively. Based on the three-dimensional coordinates, reflection intensity, distance and vertical angle of each point in the foreground image, a vector representation of the foreground image is constructed, namely, a six-dimensional feature vector [x, y, z, i, diatance, azimuth].

[0079] Step 122 : performing multi-scale feature extraction on the vector representation of the foreground image to obtain multi-scale features, and fusing the multi-scale features to obtain a foreground feature map, wherein the foreground feature map represents the semantic features of the point cloud data.

[0080] Specifically, a multi-layer convolutional network is used to extract multi-scale features from the vector representation of the foreground image to obtain multi-scale features. After obtaining the multi-scale features, the multi-scale features are fused to obtain a foreground feature map, which is used to represent the semantic features of the point cloud data.

[0081] Based on the above embodiment, step 130, determining the geometric features of the point cloud data based on the two-dimensional grid obtained by division, includes:

[0082] Step 131 , determining the geometric features of the point cloud data based on the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the mean coordinates of all points in the grid in which it is located, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid.

[0083] Specifically, the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the mean coordinates of all points in the grid, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid are represented by [x, y, z], [r], [x c ,y c , z c ]、[x p ,y p ] indicates that the geometric features of the point cloud data are obtained based on the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the mean coordinates of all points in the grid, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid, that is, [x, y, z, r, x c ,y c , z c , x p ,y p ], where the geometric features of the point cloud data are the feature vectors of each point in the point cloud data in nine dimensions.

[0084] Based on the above embodiment, determining a foreground image based on the point cloud data to be detected includes:

[0085] Step 210 : Projecting the point cloud data into a front view, retaining the closest points of the point cloud data at the same similar point position in the front view, and obtaining the foreground image.

[0086] Specifically, the point cloud data is projected into the front view, and the nearest point of the point cloud data at the same pixel position is retained in the front view. The nearest point here refers to the nearest point of the point cloud data at the same pixel position in the front view, thereby obtaining the foreground image, and constructing a six-dimensional feature vector [x, y, z, i, diatance, azimuth] for each point in the foreground image.

[0087] Based on the above embodiments, Figure 5This is the second flow chart of the target detection method provided by the present invention, such as Figure 5 As shown, the method includes:

[0088] First, project the point cloud data onto the foreground image using the formula P c = R0T0P, convert the coordinate P of the point in the point cloud data in the lidar coordinate system to the coordinate P in the image plane c , where T0 is the transformation matrix from the lidar coordinate system to the camera coordinate system, R0 is the camera parameter matrix, and for the point set with overlapping two-dimensional coordinates in the foreground image, only the point closest to the lidar is retained. For each point in the foreground image, a six-dimensional feature vector [x, y, z, i, diatance, azimuth] is constructed, including the three-dimensional coordinates, reflection intensity, distance, and vertical angle of each point. Figure 6 This is a flow chart of the foreground feature map extraction process provided by the present invention, such as Figure 6 As shown in Figure 2, three foreground network modules are composed of three multi-layer convolutional networks, namely Figure 6 Modules 1, 2, and 3 shown in the figure extract semantic feature information of images of different scales, and then fuse the semantic features at different scales to obtain a foreground feature map. And fix the size of the foreground image to [H, W], where H and W are the length and width of the foreground image respectively, C f is the size of the feature in the foreground image.

[0089] In addition, the point cloud data is projected into the top view and two-dimensional grid division is performed. The top view of the point cloud data is divided into two-dimensional grids of fixed size, and the center points of the non-empty two-dimensional grids are taken to form a set S c , for S c Any point c i , the K nearest neighbor algorithm is used to find its corresponding point set, the maximum distance threshold is set to 0.5m, the number of neighbor points is set to 64, and the corresponding point set of the selected non-empty two-dimensional grid center is established to reduce the information loss between point pairs of point cloud data in the rigid voxelization process. Voxelization here refers to converting the geometric form representation of an object into the voxel representation form closest to the object, generating a volume data set, which not only contains the surface information of the model, but also can describe the internal properties of the model.

[0090] It can be understood that after completing the dynamic division of the point set, a nine-dimensional feature vector is constructed, which includes the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the mean value of the coordinates of all points in the grid, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid, that is, [x, y, z, r, x c ,y c , z c , x p ,yp ] geometric eigenvector, where the difference between the coordinates of each point and the mean coordinates of all points in the grid is given by [x c ,y c , z c ] represents that the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid is represented by [x p ,y p ]express.

[0091] The semantic features and geometric features of the point cloud data are fused, and each point in the foreground feature map is restored to the three-dimensional coordinate system of the lidar. The semantic feature vector and the geometric feature vector are directly spliced ​​and fused at each point to obtain the fusion features of each point in the point cloud data. The center points of the non-empty two-dimensional grids obtained by the fusion feature division are clustered, and the point set feature aggregation network is used to aggregate the two-dimensional grid point set features, and the feature vector of the cubic column layer is extracted. The top-view feature map M is constructed using a scatter operation. b .

[0092] Then, a multi-layer convolutional network is used to map the top view feature map M b Encode, and then use the attention network to dynamically aggregate multi-scale features, use the attention network to learn the feature weights of the top view feature map at multiple scales, and use a global average pooling layer (Global Average Pooling, GAP) to calculate the mean of the features of the top view feature map at multiple scales, that is, a bk =∑ x，y，c M bk , k=1,2,3, where M b It is a top view feature map, and then the average value [a b1 , a b2 , a b3 ] Input to a fully connected layer to learn weights w k ,pass Dynamically adjust multi-scale features, perform upsampling, complete feature fusion, and finally use the detection head to output the target detection results.

[0093] The target detection device provided by the present invention is described below. The target detection device described below and the target detection method described above can be referenced to each other.

[0094] Figure 7 is a schematic diagram of the structure of the target detection device provided by the present invention, such as Figure 7 As shown, the device includes:

[0095] An image determination unit 710 is configured to determine a foreground image and a top view based on the point cloud data to be detected;

[0096] A semantic feature determination unit 720 is configured to perform feature extraction on the foreground image to obtain semantic features of the point cloud data;

[0097] a geometric feature determination unit 730 configured to divide the top view into two-dimensional grids and determine geometric features of the point cloud data based on the two-dimensional grids obtained by the division;

[0098] The target detection unit 740 is configured to perform target detection based on the semantic features and geometric features of the point cloud data.

[0099] The device provided by the embodiment of the present invention fuses the semantic features of the point cloud data determined by the semantic feature determination unit and the geometric features of the point cloud data determined by the geometric feature determination unit, supplements the rich feature information, and improves the precision and accuracy of target detection.

[0100] Based on any of the foregoing embodiments, the performing of target detection based on the semantic features and geometric features of the point cloud data includes:

[0101] Determining a fusion feature of each point in the point cloud data based on the semantic features and geometric features of the point cloud data;

[0102] Clustering the center points of the non-empty two-dimensional grids obtained by division to obtain multiple center point sets;

[0103] Based on the multiple center point sets, performing aggregation extraction and dispersion operations on the fusion features of each point in the point cloud data to obtain a bird's-eye view feature map;

[0104] Target detection is performed based on the top-view feature map.

[0105] Based on any of the above embodiments, target detection is performed based on the top view feature map, including:

[0106] Performing multi-scale feature extraction on the top-view feature map to obtain features of the top-view feature map at multiple scales;

[0107] Determining fusion weights of the multiple scales based on mean values ​​of features of the top-view feature map at the multiple scales;

[0108] Aggregating the features of the top-view feature map at the multiple scales based on the fusion weights of the multiple scales to obtain aggregated features;

[0109] Target detection is performed based on the aggregated features.

[0110] Based on any of the above embodiments, extracting features from the foreground image to obtain semantic features of the point cloud data includes:

[0111] constructing a vector representation of the foreground image based on the three-dimensional coordinates, reflection intensity, distance, and vertical angle of each point in the foreground image;

[0112] Multi-scale feature extraction is performed on the vector representation of the foreground image to obtain multi-scale features, and the multi-scale features are fused to obtain a foreground feature map, where the foreground feature map represents the semantic features of the point cloud data.

[0113] Based on any of the foregoing embodiments, determining the geometric features of the point cloud data based on the two-dimensional grid obtained by division includes:

[0114] The geometric features of the point cloud data are determined based on the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the average coordinates of all points in the grid in which it is located, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid.

[0115] Based on any of the above embodiments, determining a foreground image based on the point cloud data to be detected includes:

[0116] The point cloud data is projected onto a front view, and the closest points of the point cloud data at the same pixel position are retained in the front view to obtain the foreground image.

[0117] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a target detection method, which includes: determining a foreground image and a top view based on the point cloud data to be detected; performing feature extraction on the foreground image to obtain semantic features of the point cloud data; performing two-dimensional grid division on the top view, and determining geometric features of the point cloud data based on the two-dimensional grid obtained by the division; and performing target detection based on the semantic features and geometric features of the point cloud data.

[0118] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target detection method provided by the above methods, which includes: determining a foreground image and a top view based on the point cloud data to be detected; performing feature extraction on the foreground image to obtain the semantic features of the point cloud data; performing two-dimensional grid division on the top view, and determining the geometric features of the point cloud data based on the two-dimensional grid obtained by the division; and performing target detection based on the semantic features and geometric features of the point cloud data.

[0120] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the target detection method provided by the above-mentioned methods, the method comprising: determining a foreground image and a top view based on point cloud data to be detected; performing feature extraction on the foreground image to obtain semantic features of the point cloud data; performing two-dimensional grid division on the top view, and determining the geometric features of the point cloud data based on the two-dimensional grid obtained by the division; and performing target detection based on the semantic features and geometric features of the point cloud data.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A target detection method, characterized in that: include: Determine a foreground image and a top view based on the point cloud data to be detected; Performing feature extraction on the foreground image to obtain semantic features of the point cloud data; Performing two-dimensional grid division on the top view, and determining geometric features of the point cloud data based on the two-dimensional grid obtained by the division; Target detection is performed based on the semantic features and geometric features of the point cloud data; the geometric features are used to reflect the features of the two-dimensional grid in which each point is located, or to reflect the features of each point and the grid as a whole; The performing target detection based on the semantic features and geometric features of the point cloud data includes: Determining, based on the semantic features and geometric features of the point cloud data, a fusion feature of each point in the point cloud data; the fusion feature reflects the semantic features from a foreground perspective and the positional features of the geometric distribution from a bird's-eye view; Clustering the center points of the non-empty two-dimensional grids obtained by division to obtain multiple center point sets; Based on the multiple center point sets, performing aggregation extraction and dispersion operations on the fused features of each point in the point cloud data to obtain a bird's-eye view feature map; the aggregation extraction refers to aggregating the features of the two-dimensional grid point set using a point set feature aggregation network; the dispersion operation refers to dividing each point of the point cloud data in the three-dimensional space into a grid corresponding to its bird's-eye view according to the coordinates of each point in the point cloud data; Performing target detection based on the top-view feature map; The performing target detection based on the top view feature map includes: Performing multi-scale feature extraction on the top-view feature map to obtain features of the top-view feature map at multiple scales; Determining fusion weights of the multiple scales based on mean values ​​of features of the top-view feature map at the multiple scales; Aggregating the features of the top-view feature map at the multiple scales based on the fusion weights of the multiple scales to obtain aggregated features; Target detection is performed based on the aggregated features.

2. The target detection method according to claim 1, wherein: The extracting features of the foreground image to obtain semantic features of the point cloud data includes: constructing a vector representation of the foreground image based on the three-dimensional coordinates, reflection intensity, distance, and vertical angle of each point in the foreground image; Multi-scale feature extraction is performed on the vector representation of the foreground image to obtain multi-scale features, and the multi-scale features are fused to obtain a foreground feature map, where the foreground feature map represents the semantic features of the point cloud data.

3. The target detection method according to claim 1, wherein: The determining of the geometric features of the point cloud data based on the two-dimensional grid obtained by division includes: The geometric features of the point cloud data are determined based on the coordinates of each point in the point cloud data, the reflectivity, the difference between the coordinates of each point and the average coordinates of all points in the grid in which it is located, and the difference between the coordinates of each point and the coordinates of the center of the two-dimensional grid.

4. The target detection method according to claim 1, wherein: The determining of the foreground image based on the point cloud data to be detected includes: The point cloud data is projected onto a front view, and the closest points of the point cloud data at the same pixel position are retained in the front view to obtain the foreground image.

5. A target detection device, characterized in that: include: An image determination unit, configured to determine a foreground image and a top view based on the point cloud data to be detected; a semantic feature determination unit, configured to extract features from the foreground image to obtain semantic features of the point cloud data; a geometric feature determination unit, configured to divide the top view into two-dimensional grids, and determine the geometric features of the point cloud data based on the two-dimensional grids obtained by the division; An object detection unit is configured to perform object detection based on semantic features and geometric features of the point cloud data; the geometric features are configured to reflect features of the two-dimensional grid in which each point is located, or to reflect features of each point and the grid as a whole; The target detection unit is specifically used to: Determining, based on the semantic features and geometric features of the point cloud data, a fusion feature of each point in the point cloud data; the fusion feature reflects the semantic features from a foreground perspective and the positional features of the geometric distribution from a bird's-eye view; Clustering the center points of the non-empty two-dimensional grids obtained by division to obtain multiple center point sets; Based on the multiple center point sets, performing aggregation extraction and dispersion operations on the fused features of each point in the point cloud data to obtain a bird's-eye view feature map; the aggregation extraction refers to aggregating the features of the two-dimensional grid point set using a point set feature aggregation network; the dispersion operation refers to dividing each point of the point cloud data in the three-dimensional space into a grid corresponding to its bird's-eye view according to the coordinates of each point in the point cloud data; Performing target detection based on the top-view feature map; The performing target detection based on the top view feature map includes: Performing multi-scale feature extraction on the top-view feature map to obtain features of the top-view feature map at multiple scales; Determining fusion weights of the multiple scales based on mean values ​​of features of the top-view feature map at the multiple scales; Aggregating the features of the top-view feature map at the multiple scales based on the fusion weights of the multiple scales to obtain aggregated features; Target detection is performed based on the aggregated features.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the target detection method according to any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 4 is implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 4 is implemented.

Images