Three-dimensional target detection method and device

Abstract

The present invention relates to the field of image processing technology, and provides a method and device for detecting three-dimensional targets. The method comprises: downsampling the original target point cloud image of the three-dimensional target a preset number of times and performing feature extraction respectively to obtain first point cloud features of each characteristic scale; upsampling the point cloud image of each characteristic scale and performing feature extraction respectively to obtain second point cloud features of each characteristic scale; splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale and performing three-dimensional target detection to determine the target detection result of the three-dimensional target. The method and device for detecting three-dimensional targets provided by the present invention splices the point cloud features of each characteristic scale obtained by multiple downsampling with the point cloud features obtained after the corresponding upsampling, and performs three-dimensional target detection, so that the original point cloud image retains more feature information of the three-dimensional target during the feature extraction process, thereby improving the detection accuracy of the three-dimensional target.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional target detection method and device. Background Art

[0002] The current mainstream three-dimensional target detection method is to convert the raw data obtained by sensors with depth perception capabilities into point clouds in three-dimensional space, and perform three-dimensional target detection through two three-dimensional target detection methods: grid-based or raw point cloud-based.

[0003] Grid-based methods generally use voxels to divide the space into a dense three-dimensional grid for feature extraction using convolution operations. However, the point cloud is compressed when dividing the voxels, resulting in the loss of some spatial information. Three-dimensional object detection algorithms based on raw point clouds have two problems due to the continuous downsampling of the point cloud. First, the receptive field of a single point continues to expand, resulting in the loss of local features. Second, as the point cloud decreases, the number of points in the target box also decreases, resulting in information loss. In both existing detection methods, feature information is lost during the feature extraction process, resulting in low accuracy in three-dimensional object detection. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a three-dimensional object detection method that enables the original point cloud image to retain more feature information of the three-dimensional object during the feature extraction process, thereby improving the detection accuracy of the three-dimensional object.

[0005] The present invention also provides a three-dimensional target detection device.

[0006] The present invention also provides an electronic device.

[0007] The present invention also provides a non-transitory computer-readable storage medium.

[0008] The present invention also provides a computer program product.

[0009] According to an embodiment of the first aspect of the present invention, a three-dimensional target detection method includes: downsampling an original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein, one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other; performing feature extraction on the point cloud images of each characteristic scale to obtain a first point cloud feature of each characteristic scale; upsampling the point cloud images of each characteristic scale, and performing feature extraction on each upsampled point cloud image to obtain a second point cloud feature of each characteristic scale; splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale; performing three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target.

[0010] According to an embodiment of the present invention, a method for detecting three-dimensional objects is performed by extracting features from point cloud images of each characteristic scale obtained after multiple downsampling, thereby obtaining first point cloud features at each characteristic scale. The point cloud images of each characteristic scale obtained after multiple downsampling are then upsampled and feature extracted, thereby obtaining second point cloud features at each characteristic scale. The first point cloud features of each characteristic scale are then concatenated with the second point cloud features of the corresponding characteristic scale, and three-dimensional object detection is performed on the concatenated features. This allows the original point cloud image to retain more characteristic information of the three-dimensional object during the feature extraction process, thereby improving the accuracy of three-dimensional object detection.

[0011] According to one embodiment of the present invention, feature extraction is performed on the point cloud images at each characteristic scale to obtain the first point cloud features at each characteristic scale, including: obtaining the global features of each point in the point cloud images at each characteristic scale based on a self-attention propagation algorithm; aggregating neighboring point clouds within a preset radius of each point in the point cloud images at each characteristic scale and obtaining the local features of each point in the point cloud images at each characteristic scale based on the self-attention propagation algorithm; and determining the first point cloud features at each characteristic scale based on the global features and the local features. The self-attention propagation algorithm is used to obtain the global and local features of each point in the point cloud images at each characteristic scale, extracting the scene features and target detail features of each point in the point cloud images, so that the feature extraction retains more feature information.

[0012] According to one embodiment of the present invention, three-dimensional object detection is performed on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional object, including: inputting the spliced ​​point cloud features of each characteristic scale into a fully connected layer to obtain a candidate center point of the three-dimensional object and determining the local features of the center candidate point; inputting the local features of the center candidate point into a classifier to obtain a classification result of the three-dimensional object; inputting the local features of the center candidate point into a target box regressor to obtain a target box of the three-dimensional object and the orientation of the three-dimensional object; and using the classification result of the three-dimensional object, the target box of the three-dimensional object, and the orientation of the three-dimensional object as the detection result of the three-dimensional object. The classifier is obtained by training an initial classifier based on three-dimensional object point cloud image samples and category labels corresponding to the three-dimensional object point cloud image samples; the target box regressor is obtained by training the initial target box regressor based on three-dimensional object point cloud image samples and target box labels corresponding to the three-dimensional object point cloud image samples. By inputting the spliced ​​point cloud features of each characteristic scale into the trained classifier and target box regressor, the three-dimensional object detection result is obtained, thereby achieving target detection of the three-dimensional object. At the same time, the spliced ​​point clouds of each characteristic scale retain more feature information and improve the accuracy of target detection.

[0013] According to one embodiment of the present invention, determining the local features of the center candidate point includes aggregating neighboring point clouds within a preset radius of the center candidate point and obtaining the local features of the center candidate point using a self-attention propagation algorithm. By determining the center candidate point of a three-dimensional object, aggregating the neighboring point clouds of the center candidate point, and obtaining the local features of the center candidate point using the self-attention propagation algorithm, the point cloud feature information of the three-dimensional object is enhanced.

[0014] According to one embodiment of the present invention, the local features of the center candidate point are input into a target box regressor to obtain the target box of the three-dimensional target and the orientation of the three-dimensional target, including: inputting the local features of the center candidate point into the target box regressor to obtain the detection box of the three-dimensional target and the orientation of the three-dimensional target; and filtering the detection box of the three-dimensional target according to a pre-set target box size threshold and a non-maximum suppression algorithm to obtain the target box of the three-dimensional target. The multiple detection boxes output by the target box regressor are filtered by the pre-set target box size threshold and the non-maximum suppression algorithm, thereby further realizing the determination of the three-dimensional target.

[0015] According to one embodiment of the present invention, downsampling an original target point cloud image of a three-dimensional target a preset number of times to obtain point cloud images at various characteristic scales includes: performing balanced farthest point downsampling on the original target point cloud image of the three-dimensional target a preset number of times to obtain a first target point cloud image at each characteristic scale; performing characteristic distance downsampling on the original target point cloud image of the three-dimensional target a preset number of times to obtain a second target point cloud image at each characteristic scale; and concatenating the first target point cloud image at each characteristic scale with the second target point cloud image at the corresponding characteristic scale to obtain a point cloud image at each characteristic scale. By combining balanced farthest point downsampling and characteristic distance downsampling to obtain a downsampled point cloud image at the current characteristic scale, the data volume is reduced to alleviate the computational burden.

[0016] According to the second aspect of the present invention, the three-dimensional target detection device includes: a point cloud downsampling module, which is used to downsample the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein, one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other; a first point cloud feature extraction module, which is used to perform feature extraction on the point cloud images of each characteristic scale respectively to obtain first point cloud features of each characteristic scale; a second point cloud feature extraction module, which is used to upsample the point cloud images of each characteristic scale respectively, and perform feature extraction on each upsampled point cloud image respectively to obtain second point cloud features of each characteristic scale; a feature splicing module, which is used to splice the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain point cloud features of each spliced ​​characteristic scale; and a three-dimensional target detection module, which is used to perform three-dimensional target detection on the point cloud features of each spliced ​​characteristic scale to determine the detection result of the three-dimensional target.

[0017] According to an embodiment of the present invention, a three-dimensional target detection device performs feature extraction on each characteristic scale point cloud image obtained after multiple downsampling to obtain first point cloud features at each characteristic scale. The point cloud images obtained after multiple downsampling are then upsampled and feature extracted to obtain second point cloud features at each characteristic scale. The first point cloud features at each characteristic scale are concatenated with the second point cloud features at the corresponding characteristic scale, and three-dimensional target detection is performed on the concatenated features. This allows the original point cloud image to retain more characteristic information of the three-dimensional target during the feature extraction process, thereby improving the accuracy of three-dimensional target detection.

[0018] According to an embodiment of the third aspect of the present invention, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any one of the above-described three-dimensional target detection methods.

[0019] According to the non-transitory computer-readable storage medium of the fourth aspect of the present invention, a computer program is stored thereon, and when the computer program is executed by a processor, it implements any of the three-dimensional target detection methods described above.

[0020] According to an embodiment of the fifth aspect of the present invention, a computer program product includes a computer program, which, when executed by a processor, implements any of the three-dimensional target detection methods described above.

[0021] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0022] The first point cloud features at each characteristic scale are obtained by performing feature extraction on each point cloud image obtained after multiple downsampling. The second point cloud features at each characteristic scale are obtained by upsampling and feature extraction on each point cloud image obtained after multiple downsampling. The first point cloud features at each characteristic scale are then concatenated with the second point cloud features at the corresponding characteristic scale, and 3D object detection is performed on the concatenated features. This allows the original point cloud image to retain more characteristic information of the 3D object during the feature extraction process, thereby improving the accuracy of 3D object detection.

[0023] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 1 is a flow chart of a three-dimensional target detection method provided by an embodiment of the present invention;

[0026] Figure 2 1 is a flow chart of the inverted point cloud adaptive propagation method provided by an embodiment of the present invention;

[0027] Figure 3 2 is a schematic diagram of the structure of a global point cloud propagation module provided by an embodiment of the present invention;

[0028] Figure 4 Schematic diagram of the structure of a local point cloud propagation module provided by an embodiment of the present invention;

[0029] Figure 51 is a flow chart of a method for detecting a three-dimensional object according to an embodiment of the present invention;

[0030] Figure 6 Schematic diagram of the structure of a point cloud pyramid module provided by an embodiment of the present invention;

[0031] Figure 7 1 is a schematic structural diagram of a three-dimensional target detection device provided by an embodiment of the present invention;

[0032] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0034] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0035] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0036] Figure 1 Schematic diagram of the process of detecting a three-dimensional target provided by an embodiment of the present invention. Figure 1 The three-dimensional target detection method provided by the embodiment of the present invention may include:

[0037] Step 110: downsampling the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein each downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other;

[0038] Step 120: performing feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale;

[0039] Step 130 , upsampling the point cloud images at each characteristic scale, and performing feature extraction on each upsampled point cloud image to obtain second point cloud features at each characteristic scale;

[0040] Step 140: splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale;

[0041] Step 150 : Perform three-dimensional target detection on the stitched point cloud features of each characteristic scale to determine a detection result of the three-dimensional target.

[0042] The execution subject of the three-dimensional target detection method provided in the embodiment of the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), or a personal computer (PC), etc., which is not specifically limited in the embodiment of the present invention.

[0043] The following describes the technical solution of the embodiment of the present invention in detail by taking a computer executing the three-dimensional target detection method provided by the embodiment of the present invention as an example.

[0044] In step 110, after obtaining an original point cloud image of the three-dimensional object, the original point cloud image of the three-dimensional object is downsampled a predetermined number of times to obtain point cloud images of different characteristic scales. Each downsampling step produces a point cloud image of a different characteristic scale, and each characteristic scale is different from another.

[0045] If the preset number of times is three, for three downsamplings, each downsampling obtains a point cloud image of a characteristic scale, and the characteristic scales are different from each other, and finally three point cloud images of different characteristic scales are obtained.

[0046] For example, if the original target point cloud image is a point cloud image composed of 2048 points, after three downsamplings with different sampling feature scales, point cloud images with three different feature scales of 1024 points, 512 points, and 256 points can be obtained.

[0047] Optionally, the original target point cloud image can be acquired through an acquisition device such as a depth camera or a laser radar. Before downsampling the original target point cloud image of the three-dimensional target a preset number of times, the coordinate system of the acquired original target point cloud image can be converted into point cloud coordinates in the autonomous coordinate system before subsequent processing. Specifically, for the acquisition device laser radar, after acquiring the point cloud coordinates, the coordinates can be converted into point cloud coordinates in the autonomous coordinate system based on the sensor intrinsic parameters. For the depth camera, the acquired depth map can be converted into point cloud coordinates in the autonomous coordinate system based on the sensor intrinsic parameters.

[0048] In step 120 , after obtaining the point cloud images of each characteristic scale, feature extraction is performed on the point cloud images of each characteristic scale respectively, and features of the point cloud images of each characteristic scale are respectively extracted to obtain first point cloud features of each characteristic scale.

[0049] It can be understood that if three downsampling operations are performed to obtain point cloud images of three characteristic scales, feature extraction is performed on the three point cloud images of different characteristic scales respectively to obtain the first point cloud features corresponding to the three point cloud images of different characteristic scales.

[0050] In step 130, after downsampling the original point cloud image to obtain point cloud images at each characteristic scale, the point cloud images at each characteristic scale are upsampled to obtain upsampled point cloud images at each characteristic scale. Feature extraction is performed on the upsampled point cloud images at each characteristic scale to obtain second point cloud features at each characteristic scale.

[0051] It is understood that if three downsampling steps are performed to obtain point cloud images of three characteristic scales, then the point cloud images of the three different characteristic scales are upsampled separately to obtain upsampled point cloud images of three different characteristic scales. Feature extraction is performed on the upsampled point cloud images of the three different characteristic scales to obtain second point cloud features corresponding to the upsampled point cloud images of the three different characteristic scales.

[0052] Optionally, the second point cloud features of each feature scale obtained after upsampling can also be obtained by constructing an inverse point cloud adaptive propagation. Figure 2 As shown in the flowchart of the inverted point cloud adaptive propagation method provided by an embodiment of the present invention, the first point cloud features of each feature scale obtained after downsampling are passed through two sets of fully connected layers to obtain a query vector and a value vector. The point cloud before downsampling is passed through a fully connected layer to obtain a key vector. Subsequently, the return features are spliced ​​together through the attention propagation algorithm to serve as the second point cloud features of the corresponding feature scale:

[0053]

[0054] Among them, O is the return feature, φ is the softmax function, Q is the query vector, V is the value vector, K is the key vector, Δ is the normalized parameter, It is the point cloud feature before downsampling, used for residual stitching.

[0055] In step 140 , the first point cloud features of each characteristic scale obtained in step 120 are spliced ​​with the second point cloud features of the corresponding characteristic scale obtained in step 130 to obtain spliced ​​point cloud features of each characteristic scale.

[0056] Specifically, concatenating the first point cloud features at each characteristic scale with the second point cloud features at the corresponding characteristic scale means that, for each characteristic scale, the first point cloud features at the current characteristic scale are concatenated with the second point cloud features at the current characteristic scale to obtain the concatenated point cloud features at the current characteristic scale. This operation is repeated for all characteristic scales to obtain point cloud features at each characteristic scale.

[0057] In step 150 , the spliced ​​point cloud features of each characteristic scale are used for three-dimensional object detection to obtain a target detection result of the three-dimensional target.

[0058] Optionally, three-dimensional target detection can be performed on the point cloud features of each characteristic scale after splicing by using a trained category classifier and a target box regressor to perform target detection on the input point cloud features of each characteristic scale after splicing, thereby obtaining a target detection result of the three-dimensional target.

[0059] The three-dimensional target detection method provided by an embodiment of the present invention performs feature extraction on each characteristic scale point cloud image obtained after multiple downsampling to obtain a first point cloud feature at each characteristic scale. The point cloud images obtained after multiple downsampling are then upsampled and feature extracted to obtain a second point cloud feature at each characteristic scale. The first point cloud features at each characteristic scale are then concatenated with the second point cloud features at the corresponding characteristic scale, and three-dimensional target detection is performed on the concatenated features. This allows the original point cloud image to retain more characteristic information of the three-dimensional target during the feature extraction process, thereby improving the accuracy of three-dimensional target detection.

[0060] In one embodiment, feature extraction is performed on the point cloud images of each characteristic scale respectively to obtain the first point cloud features of each characteristic scale, including: obtaining the global features of each point in the point cloud images of each characteristic scale according to a self-attention propagation algorithm; aggregating the adjacent point clouds within a preset radius of each point in the point cloud images of each characteristic scale, and obtaining the local features of each point in the point cloud images of each characteristic scale according to the self-attention propagation algorithm; and determining the first point cloud features of each characteristic scale according to the global features and the local features.

[0061] Optional, such as Figure 3 As shown in the structural diagram of the global point cloud propagation module provided by the embodiment of the present invention, a global point cloud propagation module can be constructed to perform feature extraction and obtain the global features of each point in the point cloud image of each characteristic scale.

[0062] The self-attention weight generation module 310 is used to obtain the query matrix, key matrix and value matrix of each point in the point cloud image of each feature scale through three sets of fully connected layers. The formula is:

[0063] Q,K,V=(b Q ,b K ,b V )+∑ i (W Q ,W K ,W V )·F (2)

[0064] Where Q is the query matrix, K is the key matrix, and V is the value matrix; b Q ,b K ,b V Corresponding to the bias of three groups of fully connected layers, W Q ,W K ,W V are three sets of fully connected layer parameters; i is a point in the point cloud image point set of each feature scale, and F is the feature matrix of the point cloud image point set of each feature scale.

[0065] The query matrix and the key matrix are operated to obtain the self-attention weight matrix and determine the self-attention weight.

[0066] The point cloud eigenvalue generation module 320 is used to determine the point cloud eigenvalue according to the value matrix calculated by formula (2).

[0067] The adaptive feature propagation module 330 is used to enable each point to obtain the required global features based on the determined self-attention weights and point cloud feature values ​​through the self-attention propagation algorithm. The specific CPAP (Complete Point Adaptive Propagation, global point cloud adaptive propagation) algorithm formula is as follows:

[0068]

[0069] Among them, A i is the correlation coefficient of all points in the point cloud image of each feature scale, φ is the softmax activation function, q i is the query vector of point i, Δ is the normalized parameter, AP is the method for obtaining the global adaptive feature of a single point, f i is the eigenvector of point i, Ψ represents the compression of the matrix into a one-dimensional vector, CPAP is the global point cloud adaptive propagation algorithm, is the point set of each characteristic scale point cloud image, c i is the coordinate of point i, and n is the number of all points in the point cloud image of each characteristic scale.

[0070] The point cloud images of each characteristic scale are combined with the global features required by each point obtained by the adaptive feature propagation module to perform point cloud feature residual splicing to obtain the global features of the point cloud images of each characteristic scale.

[0071] Optional, such as Figure 4 As shown in the structural diagram of the local point cloud propagation module provided by an embodiment of the present invention, a local point cloud propagation module can be constructed to perform feature extraction and obtain the local features of each point in the point cloud image of each characteristic scale.

[0072] The point cloud downsampling module 410 downsamples the original point cloud image of the three-dimensional object multiple times according to a preset number of times to obtain point cloud images of various characteristic scales. The point cloud downsampling method may include balanced farthest point downsampling and characteristic distance downsampling.

[0073] The local point cloud grouping module 420 groups the points within a sphere formed within a preset radius into a group of local point clouds, with the point cloud image of each characteristic scale obtained after downsampling as the center.

[0074] The local point adaptive cloud propagation module 430 uses local point cloud adaptive propagation to generate a query vector for the grouped local point cloud through a fully connected layer and a maximum pooling layer. It then uses the attention propagation algorithm to obtain the local point cloud feature vector, and ultimately determines the local features of each point in the point cloud image at each feature scale. The specific calculation process is as follows:

[0075]

[0076]

[0077] in, The maximum pooling operation results in a one-dimensional vector. is the feature concatenation matrix of the point cloud within the group, MaxPool is the maximum pooling operation, MLP is the fully connected layer, GPAP (Grouped Point Adaptive Propagation) algorithm, and m is the number of all points in the point cloud image at each feature scale.

[0078] After determining the local features and global features of each point in the point cloud image of each characteristic scale, each point in the point cloud image of each characteristic scale has both global features and local features, which are the first point cloud features of each characteristic scale.

[0079] The three-dimensional target detection method provided by the embodiment of the present invention obtains the global features and local features of each point in the point cloud image of each characteristic scale through the self-attention propagation algorithm, extracts the scene features and target detail features of each point in the point cloud image, so that the feature extraction retains more feature information.

[0080] In one embodiment, three-dimensional target detection is performed on the point cloud features of each characteristic scale after splicing to determine the detection result of the three-dimensional target, including: inputting the point cloud features of each characteristic scale after splicing into a fully connected layer to obtain a center candidate point of the three-dimensional target, and determining the local features of the center candidate point; inputting the local features of the center candidate point into a classifier to obtain a classification result of the three-dimensional target; inputting the local features of the center candidate point into a target box regressor to obtain a target box of the three-dimensional target and an orientation of the three-dimensional target; and using the classification result of the three-dimensional target, the target box of the three-dimensional target, and the orientation of the three-dimensional target as the detection result of the three-dimensional target. The classifier is obtained by training an initial classifier based on three-dimensional target point cloud image samples and category labels corresponding to the three-dimensional target point cloud image samples; the target box regressor is obtained by training the initial target box regressor based on three-dimensional target point cloud image samples and target box labels corresponding to the three-dimensional target point cloud image samples.

[0081] Specifically, after obtaining the concatenated point cloud features at each characteristic scale, they are passed through a fully connected layer to obtain a center candidate point close to the center of the 3D object. Based on the center candidate point, the local features of the center candidate point are determined and input into a classifier and a target bounding box regressor to obtain the 3D object classification result, the 3D object orientation, and the 3D object target bounding box, i.e., the 3D object detection result.

[0082] Among them, the classifier is obtained by training the initial classifier based on the three-dimensional target point cloud image samples and the category labels corresponding to the three-dimensional target point cloud image samples. The target frame regressor is obtained by training the initial target frame regressor based on the three-dimensional target point cloud image samples and the target frame labels corresponding to the three-dimensional target point cloud image samples. The training of the classifier and the target frame regressor enables the trained classifier and target frame regressor to detect the three-dimensional target and determine the classification result of the three-dimensional target, the orientation of the three-dimensional target and the target frame of the three-dimensional target. The three-dimensional target detection method provided by the embodiment of the present invention determines the center candidate point of the three-dimensional target based on the point cloud features of each characteristic scale after splicing and obtains the detection result of the three-dimensional target based on the classifier and the target frame regressor, thereby realizing the target detection of the three-dimensional target. At the same time, the point clouds of each characteristic scale after splicing retain more feature information, thereby improving the accuracy of target detection.

[0083] In one embodiment, determining the local features of the center candidate point includes: aggregating neighboring point clouds within a preset radius of the center candidate point, and obtaining the local features of the center candidate point based on a self-attention propagation algorithm.

[0084] After obtaining the candidate center point of the 3D object based on the fully connected layer, the local features of the center candidate point are determined. The adjacent point clouds within a sphere with a preset radius are aggregated around the center candidate point, and the local features of the center candidate point are obtained using the self-attention propagation algorithm.

[0085] The three-dimensional target detection method provided by an embodiment of the present invention determines the center candidate point of the three-dimensional target, aggregates the point cloud adjacent to the center candidate point, and obtains the local features of the center candidate point based on the self-attention propagation algorithm, thereby improving the point cloud feature information of the three-dimensional target.

[0086] In one embodiment, the local features of the center candidate point are input into a target frame regressor to obtain the target frame of the three-dimensional target and the orientation of the three-dimensional target, including: inputting the local features of the center candidate point into a target frame regressor to obtain the detection frame of the three-dimensional target and the orientation of the three-dimensional target; and filtering the detection frame of the three-dimensional target according to a pre-set target frame size threshold and a non-maximum suppression algorithm to obtain the target frame of the three-dimensional target.

[0087] The local features of the center candidate point are fed into the target bounding box regressor. This not only determines the orientation of the 3D object, but also generates multiple 3D object detection boxes. Therefore, it is necessary to screen these multiple 3D object detection boxes and determine the unique and most appropriate detection box as the 3D object's target bounding box.

[0088] The detection frames of multiple 3D objects are filtered using a pre-set target frame size, and the detection frames of the 3D objects with the required size are selected. At the same time, the detection frames of the 3D objects are further filtered using the non-maximum suppression algorithm to finally obtain the target frame of the 3D object.

[0089] The three-dimensional target detection method provided by the embodiment of the present invention further realizes the determination of the three-dimensional target by screening multiple detection boxes output by the target box regressor through a pre-set target box size threshold and a non-maximum suppression algorithm.

[0090] In one embodiment, an original target point cloud image of a three-dimensional target is downsampled a preset number of times to obtain point cloud images of each characteristic scale, including: performing balanced farthest point downsampling on the original target point cloud image of the three-dimensional target a preset number of times to obtain a first target point cloud image of each characteristic scale; performing characteristic distance downsampling on the original target point cloud image of the three-dimensional target a preset number of times to obtain a second target point cloud image of each characteristic scale; and splicing the first target point cloud image of each characteristic scale with the second target point cloud image of the corresponding characteristic scale to obtain a point cloud image of each characteristic scale.

[0091] Specifically, after obtaining the original target point cloud image, the original target point cloud image of the three-dimensional target is downsampled a preset number of times. Each downsampling process can be combined by balancing the farthest point downsampling and the feature distance downsampling to obtain the final result of this downsampling. For example, when downsampling the original target point cloud image from 2048 points to 1024 points, 512 points can be sampled by balancing the farthest point downsampling, and then 512 points can be sampled by the feature distance downsampling. The points obtained by the two downsampling methods are spliced ​​to obtain the point cloud image of the current feature scale downsampling.

[0092] Optionally, the balanced farthest point downsampling method in the downsampling method calculates the length, width, and height of the sampling range of the entire three-dimensional scene to determine the balance weight of each dimension. The calculation formula is as follows:

[0093]

[0094] Among them, X d is the sampling range of the X axis in space, Y d is the sampling range of the Y axis in space, Z d is the sampling range of the Z axis in space, α x is the weight value of the X axis, α y is the weight value of the Y axis, α z is the weight value of the Z axis.

[0095] By randomly selecting a point in the point set of the original point cloud target image, and then calculating the Euclidean space distance to other points, the balance weight required for each dimension is calculated. The balance weight is used to balance the different ranges of the three axes. The calculation formula is as follows:

[0096]

[0097] Among them, D B1 (P i ,P j ) is point P i and point P j The similarity, α τis the weight value of the current axis, τ is the specific coordinate axis, τ i and τ j Point P i and P j The farthest point is added to the subset of the sampled point cloud and the above method is iterated until the number of points in the subset reaches the requirement.

[0098] Optionally, the feature distance downsampling method is used to downsample the point set of the entire original point cloud target image, and feature extraction is performed on the point set. By randomly selecting a point in the point set, the similarity with the features of other points is calculated. The calculation formula is as follows:

[0099]

[0100] Among them, D B2 (P i ,P j ) is point P i and point P j The similarity, f i is the eigenvector of the point, and n is the number of points in the point set.

[0101] The point with the lowest similarity is added to the subset of the sampled point cloud and the above method is iterated until the number of points in the subset reaches the downsampling requirement.

[0102] The three-dimensional target detection method provided by the embodiment of the present invention combines the two downsampling methods of balanced farthest point downsampling and feature distance downsampling to obtain a point cloud image of the current feature scale downsampling, thereby reducing the amount of data to alleviate the computational burden.

[0103] The following is a flow chart of a method for detecting a three-dimensional target provided by an embodiment of the present invention. Figure 5 Take the following as an example to illustrate the technical solution provided by the embodiment of the present invention:

[0104] In step 510, an original target point cloud image is acquired by a laser radar or a depth camera, and the original target point cloud image is subjected to coordinate system transformation to be converted into a point cloud image in an autonomous coordinate system.

[0105] In step 520 , the original target point cloud image in the autonomous coordinate system is downsampled. The downsampling may be performed by a mixed downsampling method of balancing the farthest point downsampling and the feature distance downsampling.

[0106] In step 530 , feature extraction is performed on the downsampled point cloud image obtained by downsampling in step 520 .

[0107] After completing downsampling and feature extraction once, the downsampling and feature extraction process is repeated according to a preset number of times. The original target point cloud image is downsampled a preset number of times to obtain point cloud images of each feature scale and the first point cloud features of each feature scale.

[0108] In step 540, a point cloud pyramid module is constructed. Figure 6 The schematic diagram of the point cloud pyramid module structure provided by an embodiment of the present invention is shown. The preset number of times is set to three. According to the downsampling and feature extraction module 610, the original target point cloud image of the three-dimensional target is downsampled three times to obtain the first point cloud features corresponding to the three-layer point cloud images of L1, L2 and L3. The three-layer point cloud images of L1, L2 and L3 are respectively input into the upsampling and feature extraction module 620 to obtain the second point cloud features corresponding to the three-layer point cloud images. The first point cloud features corresponding to the three-layer point cloud images of L1, L2 and L3 are spliced ​​with the second point cloud features corresponding to the three-layer point cloud images of L1, L2 and L3 to obtain the spliced ​​point cloud features of each characteristic scale.

[0109] In step 550, the spliced ​​point cloud features of each feature scale are input into the fully connected layer to obtain the center candidate point of the three-dimensional target, aggregate the adjacent point clouds within a preset radius of the center candidate point, and obtain the local features of the center candidate point based on the self-attention propagation algorithm.

[0110] In step 560, the local features of the center candidate point are input into the trained classifier and target box regressor, and the classification result of the three-dimensional target, the target box of the three-dimensional target, and the orientation of the three-dimensional target are output, and this output is used as the target detection result of the three-dimensional target.

[0111] The three-dimensional target detection method provided by the embodiment of the present invention combines the two downsampling methods of balanced farthest point downsampling and feature distance downsampling to obtain a point cloud image of the current feature scale downsampling, thereby reducing the amount of data to alleviate the computational burden.

[0112] Figure 7 A schematic diagram of the structure of a three-dimensional target detection device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device includes:

[0113] The point cloud downsampling module 710 is configured to downsample the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of various characteristic scales; wherein each downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from another;

[0114] A first point cloud feature extraction module 720 is configured to perform feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale;

[0115] A second point cloud feature extraction module 730 is configured to upsample the point cloud images at each characteristic scale, and perform feature extraction on each upsampled point cloud image to obtain second point cloud features at each characteristic scale;

[0116] A feature stitching module 740 is configured to stitch the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain stitched point cloud features of each characteristic scale;

[0117] The three-dimensional object detection module 750 is configured to perform three-dimensional object detection on the spliced ​​point cloud features of each characteristic scale to determine a detection result of the three-dimensional object.

[0118] The three-dimensional target detection device provided in an embodiment of the present invention performs feature extraction on each characteristic scale point cloud image obtained after multiple downsampling to obtain first point cloud features at each characteristic scale. The point cloud images obtained after multiple downsampling are then upsampled and feature extracted to obtain second point cloud features at each characteristic scale. The first point cloud features at each characteristic scale are concatenated with the second point cloud features at the corresponding characteristic scale, and three-dimensional target detection is performed on the concatenated features. This allows the original point cloud image to retain more characteristic information of the three-dimensional target during the feature extraction process, thereby improving the accuracy of three-dimensional target detection.

[0119] In one embodiment, the first point cloud feature extraction module 720 is specifically configured to:

[0120] Feature extraction is performed on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale, including:

[0121] Obtaining the global features of each point in the point cloud image at each feature scale according to a self-attention propagation algorithm;

[0122] Aggregating adjacent point clouds within a preset radius of each point in the point cloud images of each characteristic scale, and obtaining local features of each point in the point cloud images of each characteristic scale based on a self-attention propagation algorithm;

[0123] Determine first point cloud features at each feature scale based on the global features and the local features.

[0124] In one embodiment, the 3D object detection module 750 is specifically configured to:

[0125] Performing three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target includes:

[0126] Inputting the spliced ​​point cloud features of each characteristic scale into a fully connected layer to obtain a candidate center point of the three-dimensional object, and determining the local features of the candidate center point;

[0127] Inputting the local features of the center candidate point into a classifier to obtain a classification result of the three-dimensional object;

[0128] Inputting the local features of the center candidate point into a target box regressor to obtain the target box of the three-dimensional object and the orientation of the three-dimensional object;

[0129] taking the classification result of the three-dimensional object, the target frame of the three-dimensional object, and the orientation of the three-dimensional object as the detection result of the three-dimensional object;

[0130] The classifier is obtained by training an initial classifier based on three-dimensional target point cloud image samples and category labels corresponding to the three-dimensional target point cloud image samples;

[0131] The target frame regressor is obtained by training an initial target frame regressor based on three-dimensional target point cloud image samples and target frame labels corresponding to the three-dimensional target point cloud image samples.

[0132] In one embodiment, the 3D object detection module 750 is further configured to:

[0133] Determining the local features of the center candidate point includes:

[0134] Aggregate the neighboring point clouds within a preset radius of the center candidate point, and obtain the local features of the center candidate point based on the self-attention propagation algorithm.

[0135] In one embodiment, the 3D object detection module 750 is further configured to:

[0136] Inputting the local features of the center candidate point into a target box regressor to obtain the target box of the three-dimensional object and the orientation of the three-dimensional object, including:

[0137] Inputting the local features of the center candidate point into the target box regressor to obtain the detection box of the three-dimensional target and the orientation of the three-dimensional target;

[0138] According to a preset target frame size threshold and a non-maximum suppression algorithm, the detection frame of the three-dimensional target is screened to obtain the target frame of the three-dimensional target.

[0139] In one embodiment, the point cloud downsampling module 710 is specifically configured to:

[0140] The original target point cloud image of the three-dimensional target is downsampled a preset number of times to obtain point cloud images of various characteristic scales, including:

[0141] Performing a preset number of balanced farthest point downsampling on the original target point cloud image of the three-dimensional target to obtain a first target point cloud image of each characteristic scale;

[0142] Performing a preset number of feature distance downsampling on the original target point cloud image of the three-dimensional target to obtain a second target point cloud image of each feature scale;

[0143] The first target point cloud image of each characteristic scale and the second target point cloud image of the corresponding characteristic scale are spliced ​​to obtain a point cloud image of each characteristic scale.

[0144] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the following method:

[0145] Downsampling the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other;

[0146] Perform feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale;

[0147] Upsampling the point cloud images of each characteristic scale respectively, and performing feature extraction on each upsampled point cloud image respectively to obtain second point cloud features of each characteristic scale;

[0148] Splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale;

[0149] Perform three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target.

[0150] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0151] Furthermore, an embodiment of the present invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the three-dimensional object detection method provided by the above-mentioned method embodiments, for example, including:

[0152] Downsampling the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other;

[0153] Perform feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale;

[0154] Upsampling the point cloud images of each characteristic scale respectively, and performing feature extraction on each upsampled point cloud image respectively to obtain second point cloud features of each characteristic scale;

[0155] Splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale;

[0156] Perform three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target.

[0157] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the three-dimensional object detection method provided in each of the above embodiments is implemented, for example, including:

[0158] Downsampling the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other;

[0159] Perform feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale;

[0160] Upsampling the point cloud images of each characteristic scale respectively, and performing feature extraction on each upsampled point cloud image respectively to obtain second point cloud features of each characteristic scale;

[0161] Splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale;

[0162] Perform three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0164] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

[0166] The above embodiments are intended to illustrate the present invention only and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be encompassed by the scope of the claims of the present invention.

Claims

1. A three-dimensional target detection method, characterized in that: include: Downsampling the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of each characteristic scale; wherein one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other; Perform feature extraction on the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale; Upsampling the point cloud images of each characteristic scale respectively, and performing feature extraction on each upsampled point cloud image respectively to obtain second point cloud features of each characteristic scale; Splicing the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain spliced ​​point cloud features of each characteristic scale; Performing three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine a detection result of the three-dimensional target; The downsampling of the original target point cloud image of the three-dimensional target by a preset number of times to obtain point cloud images of each characteristic scale includes: Performing a preset number of balanced farthest point downsampling on the original target point cloud image of the three-dimensional target to obtain a first target point cloud image of each characteristic scale; Performing a preset number of feature distance downsampling on the original target point cloud image of the three-dimensional target to obtain a second target point cloud image of each feature scale; The first target point cloud image of each characteristic scale and the second target point cloud image of the corresponding characteristic scale are spliced ​​to obtain a point cloud image of each characteristic scale.

2. The three-dimensional target detection method according to claim 1, characterized in that: The step of extracting features from the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale includes: Obtaining the global features of each point in the point cloud image at each feature scale according to a self-attention propagation algorithm; Aggregating adjacent point clouds within a preset radius of each point in the point cloud images of each characteristic scale, and obtaining local features of each point in the point cloud images of each characteristic scale based on a self-attention propagation algorithm; Determine first point cloud features at each feature scale based on the global features and the local features.

3. The three-dimensional target detection method according to claim 1, characterized in that: The performing three-dimensional target detection on the spliced ​​point cloud features of each characteristic scale to determine the detection result of the three-dimensional target includes: Inputting the spliced ​​point cloud features of each characteristic scale into a fully connected layer to obtain a candidate center point of the three-dimensional object, and determining the local features of the candidate center point; Inputting the local features of the center candidate point into a classifier to obtain a classification result of the three-dimensional object; Inputting the local features of the center candidate point into a target box regressor to obtain the target box of the three-dimensional object and the orientation of the three-dimensional object; taking the classification result of the three-dimensional object, the target frame of the three-dimensional object, and the orientation of the three-dimensional object as the detection result of the three-dimensional object; The classifier is obtained by training an initial classifier based on three-dimensional target point cloud image samples and category labels corresponding to the three-dimensional target point cloud image samples; The target frame regressor is obtained by training an initial target frame regressor based on three-dimensional target point cloud image samples and target frame labels corresponding to the three-dimensional target point cloud image samples.

4. The three-dimensional target detection method according to claim 3, characterized in that: The determining of the local features of the candidate center point includes: Aggregate the neighboring point clouds within a preset radius of the center candidate point, and obtain the local features of the center candidate point based on the self-attention propagation algorithm.

5. The three-dimensional target detection method according to claim 3, characterized in that: Inputting the local features of the center candidate point into a target frame regressor to obtain the target frame of the three-dimensional object and the orientation of the three-dimensional object includes: Inputting the local features of the center candidate point into the target box regressor to obtain the detection box of the three-dimensional target and the orientation of the three-dimensional target; According to a preset target frame size threshold and a non-maximum suppression algorithm, the detection frame of the three-dimensional target is screened to obtain the target frame of the three-dimensional target.

6. A three-dimensional target detection device, characterized in that: include: A point cloud downsampling module is used to downsample the original target point cloud image of the three-dimensional target a preset number of times to obtain point cloud images of various characteristic scales; wherein, one downsampling obtains a point cloud image of a characteristic scale, and each characteristic scale is different from each other; A first point cloud feature extraction module is used to extract features from the point cloud images of each characteristic scale to obtain first point cloud features of each characteristic scale; A second point cloud feature extraction module is used to upsample the point cloud images of each characteristic scale, and perform feature extraction on each upsampled point cloud image to obtain a second point cloud feature of each characteristic scale; A feature stitching module, configured to stitch the first point cloud features of each characteristic scale with the second point cloud features of the corresponding characteristic scale to obtain stitched point cloud features of each characteristic scale; A three-dimensional target detection module is used to perform three-dimensional target detection on the point cloud features of each characteristic scale after the splicing, and determine the detection result of the three-dimensional target; The downsampling of the original target point cloud image of the three-dimensional target by a preset number of times to obtain point cloud images of each characteristic scale includes: Performing a preset number of balanced farthest point downsampling on the original target point cloud image of the three-dimensional target to obtain a first target point cloud image of each characteristic scale; Performing a preset number of feature distance downsampling on the original target point cloud image of the three-dimensional target to obtain a second target point cloud image of each feature scale; The first target point cloud image of each characteristic scale and the second target point cloud image of the corresponding characteristic scale are spliced ​​to obtain a point cloud image of each characteristic scale.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional target detection method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional target detection method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional target detection method according to any one of claims 1 to 5 is implemented.

Images