Target detection method, target detection model training method, device and electronic device

Abstract

The present disclosure provides a target detection method, a target detection model training method, a device and an electronic device, which relate to the field of artificial intelligence technology, specifically the field of deep learning, image processing, and computer vision technology. The method includes: voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected to obtain a voxelized result after temporal aggregation; extracting features from the voxelized result after temporal aggregation to obtain aggregated features of the scene to be detected; performing target detection based on the aggregated features of the scene to be detected to obtain an initial detection result of the object to be detected in the scene to be detected; performing target detection based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected to obtain a target detection result of the object to be detected. The above technical solution can improve the target detection performance.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically deep learning and computer vision technology, and can be applied to scenes such as augmented reality, robot navigation and autonomous driving, and specifically to a target detection method, a target detection model training method, a device and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence (AI), AI has been widely used in computer vision technology, speech recognition technology, natural language processing technology, deep learning, big data processing technology and other fields.

[0003] Computer vision technology based on artificial intelligence has also been widely used in different scenarios. Among them, the object detection technology based on artificial intelligence is used to determine the object bounding box and object category in the point cloud. How to improve the performance of object detection is very important. Summary of the invention

[0004] The present disclosure provides a target detection method, a target detection model training method, a device and an electronic device.

[0005] According to one aspect of the present disclosure, there is provided a target detection method, comprising:

[0006] voxelize and temporally aggregate at least two frames of point clouds in the scene to be detected to obtain a voxelized result after temporal aggregation;

[0007] Perform feature extraction on the voxelized results after temporal aggregation to obtain aggregated features of the scene to be detected;

[0008] Performing target detection according to the aggregated features of the scene to be detected to obtain an initial detection result of the object to be detected in the scene to be detected;

[0009] Target detection is performed based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected to obtain a target detection result of the object to be detected.

[0010] According to another aspect of the present disclosure, a method for training a target detection model is provided, comprising:

[0011] voxelize and temporally aggregate at least two frames of point clouds in the sample scene to obtain a voxelized result after temporal aggregation;

[0012] Perform feature extraction on the time-aggregated voxelization results to obtain aggregated features of the sample scene;

[0013] Performing target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0014] Performing target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain a target detection result of the sample object;

[0015] The target detection model is trained according to the target detection result of the sample object and the true value detection result of the sample object.

[0016] According to another aspect of the present disclosure, there is provided a target detection device, comprising:

[0017] An aggregation voxelization module, used for voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected, to obtain a voxelization result after temporal aggregation;

[0018] An aggregation feature module is used to extract features from the voxelization results after temporal aggregation to obtain the aggregation features of the scene to be detected;

[0019] An initial detection module, used to perform target detection according to the aggregated features of the scene to be detected, and obtain an initial detection result of the object to be detected in the scene to be detected;

[0020] The aggregate detection module is used to perform target detection based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected, so as to obtain the target detection result of the object to be detected.

[0021] According to another aspect of the present disclosure, there is provided a target detection model training device, comprising:

[0022] An aggregation voxelization module, used for voxelizing and temporally aggregating at least two frames of point clouds in a sample scene to obtain a voxelization result after temporal aggregation;

[0023] An aggregation feature module is used to extract features from the voxelized results after temporal aggregation to obtain the aggregation features of the sample scene;

[0024] An initial detection module, used to perform target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0025] An aggregate detection module, used to perform target detection based on the initial detection result of the sample object and the aggregate features of the sample scene to obtain a target detection result of the sample object;

[0026] The model training module is used to train the target detection model according to the target detection result of the sample object and the true value detection result of the sample object.

[0027] at least one processor; and

[0028] a memory communicatively connected to the at least one processor; wherein,

[0029] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method provided by any embodiment of the present disclosure.

[0030] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by any embodiment of the present disclosure.

[0031] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0033] Figure 1a is a flow chart of a target detection method provided according to an embodiment of the present disclosure;

[0034] Figure 1b is a schematic diagram of the structure of a target detection model provided according to an embodiment of the present disclosure;

[0035] Figure 2a is a flow chart of another target detection method provided according to an embodiment of the present disclosure;

[0036] Figure 2b It is a structural schematic diagram of a three-dimensional feature extractor provided in the related art;

[0037] Figure 2c is a structural schematic diagram of a three-dimensional feature extractor provided by an embodiment of the present disclosure;

[0038] Figure 3a is a flowchart of another target detection method provided according to an embodiment of the present disclosure;

[0039] Figure 3b is a structural diagram of another target detection model provided according to an embodiment of the present disclosure;

[0040] Figure 3c is a structural diagram of another target detection model provided according to an embodiment of the present disclosure;

[0041] Figure 4 is a flow chart of a target detection model training method provided according to an embodiment of the present disclosure;

[0042] Figure 5 is a flowchart of another target detection model training method provided according to an embodiment of the present disclosure;

[0043] Figure 6 is a flowchart of another target detection model training method provided according to an embodiment of the present disclosure;

[0044] Figure 7 is a structural schematic diagram of a target detection device provided according to an embodiment of the present disclosure;

[0045] Figure 8 is a structural schematic diagram of a target detection model training device provided according to an embodiment of the present disclosure;

[0046] Fig. 9 It is a block diagram of an electronic device used to implement the target detection method or target detection model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Figure 1a 1 is a flow chart of a target detection method provided according to an embodiment of the present disclosure. The method is applicable to the case of performing target detection on continuous point cloud frame data. The method can be executed by a target detection device, which can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 1a As shown, the target detection method of this embodiment may include:

[0048] S101, voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected, to obtain a voxelized result after temporal aggregation;

[0049] S102, performing feature extraction on the time-aggregated voxelization result to obtain aggregated features of the scene to be detected;

[0050] S103, performing target detection according to the aggregated features of the scene to be detected, and obtaining an initial detection result of the object to be detected in the scene to be detected;

[0051] S104, performing target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, to obtain a target detection result of the object to be detected.

[0052] Among them, the scene to be detected is a scene where target detection is required, for example, it can be a scene in which point cloud data is collected by sensors in technologies such as augmented reality technology, robot navigation technology or autonomous driving, and the sensor can be a lidar or a millimeter wave radar. The object to be detected is an object in the scene to be detected. It should be noted that the objects in the scene to be detected can be in motion and / or static state, for example, part of the object is in motion and the other part is static. Each frame of the point cloud of the scene to be detected has a timestamp.

[0053] Among them, the voxelization result after time aggregation includes not only the voxelization data of each frame point cloud, but also the encoded data of each frame point cloud in the time dimension, that is, not only each frame point cloud is voxelized to obtain voxelized data, but also the time dimension is integrated into the voxelized data of the point cloud. For each frame point cloud, it is also possible to encode the encoded data of the frame point cloud in the time dimension according to the timestamps of at least two frames of point cloud, and add the encoded data in the time dimension to the voxelized data of the frame point cloud to obtain the voxelization result after time aggregation. By independently encoding the timestamps of each frame point cloud in the time dimension, more point cloud timing information can be retained, thereby improving the target detection performance of objects in motion.

[0054] Specifically, obtain M frames of point cloud of the scene to be detected, perform motion compensation on the M frames of point cloud, and unify the M frames of point cloud into the same coordinate system. Perform voxelization on the M frames of point cloud respectively to obtain voxel data of the M frames of point cloud; and determine the time interval to which the scene to be detected belongs according to the timestamp of each frame of point cloud, divide the time interval into N sub-intervals, that is, divide the time dimension into N parts, determine the encoding data of the frame of point cloud in the time dimension according to the sub-interval to which the timestamp of each frame of point cloud belongs, introduce the encoding data of the frame of point cloud in the time dimension into the voxel data, and obtain the voxelization result after time aggregation. Among them, M and N are both fixed arrays greater than 2, and the values ​​of M and N can be the same or different.

[0055] In the disclosed embodiment, feature extraction can also be performed on the time-aggregated voxelization result based on sparse convolution to obtain the aggregated features of the scene to be detected. The time-aggregated voxelization result is obtained by voxelizing and time-aggregating at least two consecutive frames of point clouds, and feature extraction is performed on the time-aggregated voxelization result so that the aggregated features of the scene to be detected incorporate richer time information of the scene to be detected, which facilitates the subsequent two-stage target detection of the object to be detected based on the aggregated features of the scene to be detected.

[0056] Figure 1b is a schematic diagram of a target detection model provided according to an embodiment of the present disclosure, with reference to Figure 1b , the object detection model may include a two-stage object detector. Figure 1b , the aggregated features of the scene to be detected are input into the target detector of the first stage to obtain the initial detection result of the object to be detected; wherein the initial detection result may include the following: initial position, initial detection frame, initial orientation angle, initial speed or initial category. In addition, the target detector of the second stage can also be used to perform the second stage target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, and obtain the target detection result of the object to be detected. By adopting a two-stage target detector and using the motion information contained in at least two frames of point clouds as constraints, the spatiotemporal information of long time series can be fully mined, thereby improving the performance and robustness of target detection.

[0057] The technical solution provided by the embodiment of the present disclosure obtains a time-aggregated voxelization result by voxelizing and temporally aggregating at least two frames of point clouds, and performs feature extraction on the time-aggregated voxelization result to obtain the aggregated features of the scene to be detected, so that the aggregated features of the scene to be detected retain more point cloud temporal information; and, by performing two-stage target detection on the aggregated features of the scene to be detected, it is able to fully exploit long-term spatiotemporal information, thereby improving the performance and robustness of target detection.

[0058] In an optional embodiment, the voxelization and time aggregation of at least two frames of point clouds in the scene to be detected to obtain a voxelization result after time aggregation includes: voxelizing at least two frames of point clouds in the scene to be detected separately to obtain three-dimensional voxelization results of at least two frames of point clouds; splicing the three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; merging the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a voxelization result after time aggregation.

[0059] For each frame of point cloud, the frame of point cloud can be voxelized in the following way to obtain the three-dimensional voxelization result of the frame of point cloud: traverse all points in the frame of point cloud set, voxelize each point, and obtain the voxelization result of each point; take the average of the voxelization results of each point to obtain the three-dimensional voxelization result of the frame of point cloud. The three-dimensional voxelization result of the frame of point cloud can be expressed as (H, W, D, C), where H, W, D, C correspond to height, width, depth and channel respectively. The channel dimension can include the following attributes: the three-dimensional coordinates of the voxel, the reflection intensity of the voxel and which line beam it belongs to, that is, the size of the channel dimension can be 5.

[0060] In the disclosed embodiment, the four-dimensional voxelization result can be obtained in the following manner: encoding the time dimension according to the timestamps of the two frames of point cloud to obtain the encoded data of each frame of point cloud in the time dimension T; splicing the three-dimensional voxelization results (H, W, D, C) of each frame of point cloud according to the encoded data in the time dimension T to obtain the four-dimensional voxelization result (H, W, D, T, C). In addition, the time dimension in the four-dimensional voxelization result can be merged into the channel dimension to obtain the time-aggregated voxelization result (H, W, D, T×C).

[0061] The three-dimensional voxelization results of each frame point cloud are determined respectively, and the three-dimensional voxelization results of each frame point cloud are spliced ​​according to the time dimension to obtain a four-dimensional voxelization result, and the time dimension in the four-dimensional voxelization result is merged into the channel dimension to obtain a time-aggregated voxelization result as a four-dimensional voxelization result. Compared with splicing multiple frames of point cloud into one frame and performing three-dimensional voxelization, the above technical solution can retain more time sequence information, thereby improving the target detection performance of moving objects.

[0062] Figure 2a is a flow chart of another target detection method provided according to an embodiment of the present disclosure. Figure 2a , the target detection method of this embodiment may include:

[0063] S201, voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected, to obtain a voxelized result after temporal aggregation;

[0064] S202, performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features;

[0065] S203, processing the time-aggregated three-dimensional features to obtain aggregated features of the scene to be detected;

[0066] S204, performing target detection according to the aggregated features of the scene to be detected, and obtaining an initial detection result of the object to be detected in the scene to be detected;

[0067] S205 , performing target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, to obtain a target detection result of the object to be detected.

[0068] In the embodiments of the present disclosure, the dimensions of the aggregated features of the scene to be detected may be different from the dimensions of the three-dimensional features aggregated over time. For example, the aggregated features of the scene to be detected may be two-dimensional features, which facilitates the subsequent use of a two-dimensional target detector to perform target detection on the objects to be detected in the scene to be detected.

[0069] Specifically, a three-dimensional feature extractor can be used to extract three-dimensional features from the time-aggregated voxelization results to obtain time-aggregated three-dimensional features; the time-aggregated three-dimensional features are processed to obtain aggregated features of the scene to be detected. By using a three-dimensional feature extractor to extract features from the time-aggregated voxelization results to obtain time-aggregated three-dimensional features; and determining the aggregated features of the detection scene based on the time-aggregated three-dimensional features, the aggregated features of the scene to be detected contain the long-term spatiotemporal information of the scene to be detected, thereby improving the performance of subsequent target detection based on the aggregated features of the scene to be detected.

[0070] In an optional embodiment, the three-dimensional feature extraction is performed on the voxelization result after time aggregation to obtain the three-dimensional feature after time aggregation, including: using the first convolution subunit in the three-dimensional feature extractor to perform feature extraction on the voxelization result after time aggregation to obtain a first feature; using the second convolution subunit in the three-dimensional feature extractor to downsample the first feature to obtain a second feature; using the third convolution subunit in the three-dimensional feature extractor to perform feature extraction on the second feature to obtain a third feature; using the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as the three-dimensional feature after time aggregation; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

[0071] Figure 2b It is a schematic diagram of the structure of a three-dimensional feature extractor provided in the related art. Figure 2b , the three-dimensional feature extractor in the related art includes at least two feature extraction units 21, one feature extraction unit 21 may include at least two convolution subunits, and the receptive fields of the convolution subunits in one feature extraction unit 21 are the same. After the time aggregation of multi-frame point clouds, the moving objects are not aligned in space, but cover a wide spatial range, that is, a tail is formed. The three-dimensional feature extractor in the related art cannot solve the tailing problem of dynamic objects, which impairs the target detection performance of dynamic objects.

[0072] Figure 2c is a schematic diagram of the structure of a three-dimensional feature extractor provided by an embodiment of the present disclosure, with reference to Figure 2c, the three-dimensional feature extractor in the embodiment of the present disclosure includes at least two feature extraction units 22, and one feature extraction unit 22 includes at least a first convolution subunit 221, a second convolution subunit 222, a third convolution subunit 223 and a fourth convolution subunit 224. Specifically, the voxelization result after time aggregation is input into the first convolution subunit 221 for feature extraction to obtain a first feature; the second convolution subunit 222 is used to downsample the first feature to obtain a second feature; the third convolution subunit 223 is used to extract the second feature to obtain a third feature; and the fourth convolution subunit 224 is used to upsample the third feature to obtain a fourth feature as a three-dimensional feature after time aggregation. It should be noted that one feature extraction unit 22 may include multiple first convolution subunits 221 and multiple third convolution subunits 223.

[0073] The size of the first feature is the same as the size of the fourth feature, the size of the second feature is the same as the size of the third feature, and the second feature is a scaling result of the first feature. In the embodiment of the present disclosure, a preset reduction ratio can be used to downsample the first feature to obtain the second feature, for example, the reduction ratio can be 0.5.

[0074] Specifically, the first feature is downsampled to obtain the second feature, and the third convolution subunit is sampled to extract the second feature to obtain the third feature, so that the receptive field in the feature extraction process can be increased, thereby effectively capturing long-term dependencies, especially for objects in motion, richer features can be extracted. Through the above processing, part of the calculation in each sparse convolution block is transferred to a smaller resolution, which can reduce the amount of calculation on the one hand; on the other hand, a larger receptive field can be obtained, further improving the target detection performance of moving objects.

[0075] In an optional implementation, the second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

[0076] The hash table of the sparse convolution is based on the input hash table and the output hash table. Figure 2c, the hash table of the sparse convolution used by the second convolution subunit 222 and the fourth convolution subunit 224 is the same. Among them, the hash table may include an input hash table and an output hash table. Since the voxels participating in the convolution in the second convolution subunit and the fourth convolution subunit remain unchanged, the time consumption of constructing the hash table of the sparse convolution can be reduced by sharing the hash table, which can further improve the feature extraction efficiency compared to constructing hash tables for the second convolution subunit and the fourth convolution subunit respectively. It should be noted that, when a feature extraction unit 22 includes multiple first convolution subunits 221, the hash table can also be shared between the third convolution subunits 223; when a feature extraction unit 22 includes multiple third convolution subunits 223, the hash table can also be shared between the third convolution subunits 223.

[0077] The technical solution provided by the embodiment of the present disclosure determines the voxelization result after time aggregation based on at least two frames of point cloud through a time aggregation mechanism, thereby aggregating richer time information; in terms of feature extraction, a larger receptive field is obtained at a negligible additional time cost, which can solve the tailing problem of moving objects, thereby further improving the performance of target detection.

[0078] In an optional embodiment, the processing of the time-aggregated three-dimensional features to obtain the aggregated features of the scene to be detected includes: performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain the aggregated features of the scene to be detected; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

[0079] In the disclosed embodiment, the height dimension in the time-aggregated three-dimensional features is also merged into the channel dimension to obtain time-aggregated two-dimensional features; and the two-dimensional features aggregated in time are subjected to two-dimensional feature extraction through a two-dimensional feature extractor to obtain aggregated features of the scene to be detected. Taking the size (H', W', D', C') of the time-aggregated three-dimensional features as an example, the size of the time-aggregated two-dimensional features obtained by merging the height dimension into the channel dimension is (H', W', D'×C'), and the size of the aggregated features of the scene to be detected obtained through the two-dimensional feature extraction is (H', W', D'×C'). By converting the time-aggregated three-dimensional features into aggregated features of the scene to be detected, it is convenient to subsequently use a two-dimensional feature extractor to perform two-dimensional target detection, which can further improve the target detection efficiency compared to three-dimensional target detection.

[0080] Figure 3a is a flow chart of another target detection method provided according to an embodiment of the present disclosure. Figure 3a , the target detection method of this embodiment may include:

[0081] S301, voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected, to obtain a voxelized result after temporal aggregation;

[0082] S302, extracting features from the time-aggregated voxelization results to obtain aggregated features of the scene to be detected;

[0083] S303, performing target detection according to the aggregated features of the scene to be detected, and obtaining an initial detection result of the object to be detected in the scene to be detected;

[0084] S304, performing feature fusion on the object to be detected according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, to determine the fusion features of the object to be detected;

[0085] S305, extracting an initial speed of the object to be detected from the initial detection result of the object to be detected, and determining a state of the object to be detected according to the initial speed of the object to be detected;

[0086] When the object to be detected is in motion, continue to execute S306; when the object to be detected is in a stationary state, continue to execute S307;

[0087] S306, using a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the object to be detected, to obtain a target detection result of the object to be detected;

[0088] S307, using a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the object to be detected, to obtain a target detection result of the object to be detected.

[0089] Combination Figure 3b and Figure 3c , voxelize and temporally aggregate at least two frames of point clouds in the scene to be detected to obtain a voxelized result after temporal aggregation; use a three-dimensional feature extractor such as 3DCNN (Convolutional Neural Networks) to extract features from the voxelized result after temporal aggregation to obtain a three-dimensional feature after temporal aggregation; use a two-dimensional feature extractor such as 2D CNN to process the three-dimensional features after temporal aggregation to obtain the aggregated features of the scene to be detected.

[0090] Furthermore, an initial detection is performed on the aggregated features of the scene to be detected to obtain an initial detection result of the object to be detected. The initial detection result of the object to be detected may include the initial position, initial detection frame, initial orientation angle, initial speed or initial category of the object to be detected; the initial position may be the center point position of the initial detection frame. The initial speed of the object to be detected is compared with the speed threshold; if the initial speed of the object to be detected is greater than the speed threshold, the object to be detected is in motion; otherwise, the object to be detected is in a stationary state. The fusion features of the object to be detected are used to represent the total feature representation of the object to be detected in at least two frames of point clouds, and the object to be detected can be determined by feature fusion based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected.

[0091] Combination Figure 3b and Figure 3c , the target detector of the second stage may include a motion detection head corresponding to the motion state and a stationary detection head corresponding to the stationary state; when the object to be detected is in motion, the fusion features of the object to be detected are input into the motion detection head corresponding to the motion state to perform target detection of the second stage, and the target detection result of the object to be detected is obtained; when the object to be detected is a static object, the fusion features of the object to be detected are input into the stationary detection head corresponding to the stationary state to perform target detection of the second stage, and the target detection result of the object to be detected is obtained. When the scene to be detected includes both moving objects and stationary objects, the motion detection head is used to perform target detection of the moving objects in the second stage, and the stationary detection head is used to perform target detection of the stationary objects in the second stage.

[0092] By providing a motion detection head and a static detection head respectively in the second stage, determining the state of the object to be detected according to the initial speed of the object to be detected, using the motion detection head to detect the object in motion, and using the static detection head to detect the object in static state, that is, based on the static-dynamic separation mechanism, objects with different motion states can be explicitly separated, and the corresponding detection head is used for target detection according to the motion state, which can fully learn the states of static objects and dynamic objects, thereby further improving the performance of target detection. In particular, for static objects, the static detection head can better learn the orientation angle of the object, and for dynamic objects, the dynamic detection head can better learn the speed of the object, thereby solving the bottleneck problem of objects in different motion states in the target detection process, thereby improving the performance of target detection for objects in different states.

[0093] In an optional embodiment, feature processing is performed on the object to be detected based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected to obtain the aggregated features of the object to be detected, including: encoding the initial detection result of the object to be detected to obtain the encoded features of the object to be detected; extracting the initial detection frame of the object to be detected from the initial detection result of the object to be detected, and determining the aggregated features of the object to be detected based on the initial detection frame and the aggregated features of the scene to be detected; and fusing the encoded features of the object to be detected and the aggregated features of the object to be detected to obtain the fused features of the object to be detected.

[0094] refer to Figure 3c , input the initial detection result of the object to be detected into the detection frame encoder to obtain the encoding feature of the object to be detected. Referring to the following formula (1), the encoding feature of the object to be detected can be determined according to the initial detection result of the object to be detected:

[0095] fbox =MLP[Concat(pos,dim,rot,vel,score,y)]; (1)

[0096] Among them, MLP represents perceptron, Concat represents concatenation, fbox represents the encoding feature of the object to be detected, pos represents the initial position, dim represents the length, width and height of the initial detection box, rot represents the initial orientation angle, vel represents the initial velocity, score represents the confidence score, and y represents the initial category, where the initial category can use a one-hot code.

[0097] Moreover, an initial detection frame of the object to be detected is extracted from the initial detection result of the object to be detected, and the aggregated features of the object to be detected are extracted from the aggregated features of the scene to be detected according to the position of the initial detection frame; the coded features of the object to be detected and the aggregated features of the object to be detected are fused to obtain the fused features of the object to be detected. Through the above processing, the fused features of the object to be detected not only contain the semantic information of the initial detection result, but also contain the abstract feature representation of the aggregated features of the object to be detected, thereby improving the performance of subsequent target detection based on the fused features.

[0098] In an optional embodiment, determining the aggregate features of the object to be detected based on the initial detection frame and the aggregate features of the scene to be detected includes: extracting key points of the object to be detected from the initial detection frame when the object to be detected is in a stationary state; and extracting the aggregate features of the object to be detected from the aggregate features of the scene to be detected based on the key points of the object to be detected.

[0099] When the object to be detected is in a stationary state, the aggregated features of the object to be detected can be extracted according to the following formula (2):

[0100] fsta =MLP[Concat(l,r,f,b,c)]; (2)

[0101] Among them, MLP represents perceptron, Concat represents concatenation, fsta is the aggregated feature of the object to be detected, l, r, f, b, c are the aggregated features at the midpoint of the left frame, the midpoint of the right frame, the midpoint of the front frame, the midpoint of the back frame in the initial detection frame respectively, and the aggregated features at the center point of the initial detection frame, that is, the midpoint of the left frame, the midpoint of the right frame, the midpoint of the front frame, and the midpoint of the back frame are the key points of the object to be detected; the aggregated features at each point are extracted from the aggregated features of the scene to be detected according to the position of each point.

[0102] In addition, the fusion features of the object to be detected can be determined by the following formula (3):

[0103] Os=MLP[Concat(fsta,fbox)]; (3)

[0104] Among them, MLP represents perceptron, Concat represents concatenation, Os represents the fused features of the object to be detected, fbox and fsta represent the encoding features and aggregation features of the object to be detected respectively.

[0105] For static objects, by extracting the key points of the object to be detected from the initial detection frame, and extracting the aggregate features of the object to be detected from the aggregate features of the scene to be detected based on the key points, the accuracy of the aggregate features of the object to be detected can be improved, laying the foundation for subsequent target detection.

[0106] In an optional embodiment, determining the aggregate features of the object to be detected based on the initial detection frame and the aggregate features of the scene to be detected includes: when the object to be detected is in motion, also extracting the initial position and initial orientation angle of the object to be detected from the initial detection result of the object to be detected; determining an auxiliary detection frame of the object to be detected based on the initial speed, the initial position, the initial orientation angle and the initial detection frame; respectively extracting key points of the object to be detected from the initial detection frame and the auxiliary detection frame; and extracting the aggregate features of the object to be detected from the aggregate features of the scene to be detected based on the key points of the object to be detected.

[0107] When the object to be detected is in motion, the initial detection result of the object to be detected is the initial detection result of the object to be detected in the specified frame point cloud, and the specified frame point cloud belongs to one of the at least two frame point clouds, which can be preset to a fixed frame number. The other frame point clouds in the at least two frame point clouds except the specified frame point cloud can be called auxiliary frame point clouds.

[0108] Specifically, the auxiliary detection frame of the object to be detected in the auxiliary frame point cloud can be determined according to the initial speed, initial position, initial orientation angle and initial detection frame of the object to be detected in the specified frame point cloud; the key points of the object to be detected are respectively extracted from the initial detection frame and the auxiliary detection frame. For example, the midpoint of the left frame, the midpoint of the right frame, the midpoint of the front frame, the midpoint of the rear frame, and the center point of the frame in the initial detection and auxiliary detection frames can all be used as key points, and the key points are used to extract the aggregated features of the object to be detected from the aggregated features of the scene to be detected.

[0109] When the object to be detected is in motion, the aggregate features of the object to be detected can be extracted according to the following formulas (4) and (5):

[0110] X i =MLP[Concat(l i ,r i ,f i , b i ,c i )]; (4)

[0111] fmov =MLP[Concat(X0, X1,…, X T-1 ) )]; (5)

[0112] Among them, MLP represents perceptron, Concat represents concatenation, and X i is the aggregated feature of the object to be detected in the i-th detection frame. The i-th detection frame can be the initial detection frame or the auxiliary detection frame. i 、r i 、f i 、b i 、c i They are the aggregate features at the midpoint of the left frame, the midpoint of the right frame, the midpoint of the front frame, and the midpoint of the back frame in the corresponding detection frame; fmov is the aggregate feature of the object to be detected, which is obtained based on the aggregate features of the object to be detected in the initial detection frame and the aggregate features in each auxiliary detection frame.

[0113] For dynamic objects, the auxiliary detection frame of the object to be detected is determined by the initial speed, initial position, initial orientation angle and initial detection frame; the key points of the object to be detected are extracted from the initial detection frame and the auxiliary detection frame respectively; based on the key points of the object to be detected, the aggregate features of the object to be detected are extracted from the aggregate features of the scene to be detected, which can improve the accuracy of the detection frame and speed prediction of dynamic objects.

[0114] In actual applications, there are generally objects in different motion states in the scene to be detected. If dynamic targets and static targets are treated indiscriminately, the differences in network features between dynamic and static targets will cause the network to only achieve suboptimal performance. The disclosed embodiment introduces a dynamic and static separation mechanism, which is of great significance for point cloud time series 3D detection. It allows the network to learn better features for dynamic and static targets, and has broad practical value.

[0115] The technical solution provided by the embodiments of the present disclosure, by proposing a time aggregation mechanism, aggregates richer time information and obtains a larger receptive field with negligible additional time cost; it also proposes a dynamic-static separation mechanism, which optimizes the 3D bounding boxes of static and moving objects respectively by utilizing the predicted object motion information; and, based on the above-mentioned time aggregation mechanism and dynamic-static separation mechanism, a two-stage point cloud 3D object detector is proposed, which uses the motion information contained in multi-frame point clouds as a constraint, can fully mine the spatiotemporal information of long time series, and improve the detection performance and robustness of the point cloud 3D detection model.

[0116] Figure 4 1 is a flow chart of a method for training a target detection model according to an embodiment of the present disclosure. The method is applicable to the case of training a target detection model. The method can be executed by a target detection model training device, which can be implemented in software and / or hardware and can be integrated into an electronic device. Figure 4 As shown, the target detection model training method of this embodiment may include:

[0117] S401, voxelizing and temporally aggregating at least two frames of point clouds in a sample scene to obtain a voxelized result after temporal aggregation;

[0118] S402, extracting features from the time-aggregated voxelized results to obtain aggregated features of the sample scene;

[0119] S403, performing target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0120] S404, performing target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain a target detection result of the sample object;

[0121] S405: Training an object detection model according to the object detection result of the sample object and the true value detection result of the sample object.

[0122] Specifically, the true detection results of the sample scenes in the training samples and the sample objects in the sample scenes are obtained. The sample scene may be a scene in which point cloud data is collected by sensors in technologies such as augmented reality technology, robot navigation technology or autonomous driving, and the sensor may be a lidar or a millimeter wave radar. The sample object is an object in the sample scene. It should be noted that the objects in the sample scene may be in motion and / or stationary state, for example, one part of the object is in motion and the other part is stationary. Each frame of the point cloud of the sample scene has a timestamp. Among them, the true detection result of the sample object may include the true position, true detection frame, true orientation angle, true speed or true category of the sample object.

[0123] Among them, the voxelization result after time aggregation includes not only the voxelization data of each frame point cloud, but also the encoded data of each frame point cloud in the time dimension, that is, not only each frame point cloud is voxelized to obtain voxelized data, but also the time dimension is integrated into the voxelized data of the point cloud. For each frame point cloud, it is also possible to encode the encoded data of the frame point cloud in the time dimension according to the timestamps of at least two frames of point cloud, and add the encoded data in the time dimension to the voxelized data of the frame point cloud to obtain the voxelization result after time aggregation. By independently encoding the timestamps of each frame point cloud in the time dimension, more point cloud timing information can be retained, thereby improving the target detection performance of objects in motion.

[0124] Specifically, M frames of point clouds of the sample scene are obtained, and motion compensation is performed on the M frames of point clouds to unify the M frames of point clouds into the same coordinate system. The M frames of point clouds are voxelized respectively to obtain voxel data of the M frames of point clouds; and the time interval to which the sample scene belongs is determined according to the timestamp of each frame of point cloud, and the time interval is divided into N subintervals, that is, the time dimension is divided into N parts, and the encoding data of the frame of point cloud in the time dimension is determined according to the subinterval to which the timestamp of each frame of point cloud belongs, and the encoding data of the frame of point cloud in the time dimension is introduced into the voxel data to obtain the voxelization result after time aggregation.

[0125] In the disclosed embodiment, feature extraction can also be performed on the time-aggregated voxelization result based on sparse convolution to obtain the aggregated features of the sample scene. The time-aggregated voxelization result is obtained by voxelizing and time-aggregating at least two consecutive frames of point clouds, and feature extraction is performed on the time-aggregated voxelization result, so that the aggregated features of the sample scene incorporate richer time information of the sample scene, which facilitates the subsequent two-stage target detection of the sample object based on the aggregated features of the sample scene.

[0126] The target detection model to be trained may include a two-stage target detector. The aggregated features of the sample scene are input into the target detector of the first stage to obtain the initial detection result of the sample object; wherein the initial detection result may include the following: initial position, initial detection frame, initial orientation angle, initial speed or initial category. In addition, the target detector of the second stage may be used to perform the second stage target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain the target detection result of the sample object. A loss function is constructed according to the target detection result of the sample object and the true value detection result of the sample object, and the loss function is used to update the network parameters to be trained in the target detection model to obtain a trained target detection model.

[0127] By adopting a two-stage target detector and using the motion information contained in at least two frames of point clouds as constraints, the spatiotemporal information of long time series can be fully mined, thereby improving the performance and robustness of target detection.

[0128] The technical solution provided by the embodiment of the present disclosure obtains a time-aggregated voxelization result by voxelizing and temporally aggregating at least two frames of point clouds, and obtains aggregated features of a sample scene by performing feature extraction on the time-aggregated voxelization result, so that the aggregated features of the sample scene retain more time series information of the point cloud; and, by performing two-stage target detection on the aggregated features of the sample scene, it is able to fully mine long-term spatiotemporal information, thereby improving the performance and robustness of the target detection model.

[0129] In an optional embodiment, the voxelization and time aggregation of at least two frames of point clouds in the sample scene to obtain a time-aggregated voxelization result includes: voxelizing at least two frames of point clouds in the sample scene separately to obtain three-dimensional voxelization results of at least two frames of point clouds; splicing the three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; and merging the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a time-aggregated voxelization result.

[0130] For each frame of point cloud, the three-dimensional voxelization of the frame point cloud can be performed in the following manner to obtain the three-dimensional voxelization result of the frame point cloud: traverse all points in the frame point cloud set, perform voxel quantization on each point, and obtain the voxelization result of each point; take the average of the voxelization results of each point to obtain the three-dimensional voxelization result of the frame point cloud. The three-dimensional voxelization result of the frame point cloud can be expressed as (H, W, D, C), where H, W, D, and C correspond to height, width, depth, and channel respectively. The channel dimension may include the following attributes: the three-dimensional coordinates of the voxel, the reflection intensity of the voxel, and which line beam it belongs to, that is, the size of the channel dimension can be 5.

[0131] In the disclosed embodiment, the four-dimensional voxelization result can be obtained in the following manner: encoding the time dimension according to the timestamps of the two frames of point cloud to obtain the encoded data of each frame of point cloud in the time dimension T; splicing the three-dimensional voxelization results (H, W, D, C) of each frame of point cloud according to the encoded data in the time dimension T to obtain the four-dimensional voxelization result (H, W, D, T, C). In addition, the time dimension in the four-dimensional voxelization result can be merged into the channel dimension to obtain the time-aggregated voxelization result (H, W, D, T×C).

[0132] The three-dimensional voxelization results of each frame point cloud are determined respectively, and the three-dimensional voxelization results of each frame point cloud are spliced ​​according to the time dimension to obtain a four-dimensional voxelization result, and the time dimension in the four-dimensional voxelization result is merged into the channel dimension to obtain a time-aggregated voxelization result as a four-dimensional voxelization result. Compared with splicing multiple frames of point cloud into one frame and performing three-dimensional voxelization, the above technical solution can retain more time sequence information, thereby improving the target detection performance of objects in motion.

[0133] Figure 5 is a flowchart of another target detection model training method provided according to an embodiment of the present disclosure. Figure 5 , the target detection model training method of this embodiment may include:

[0134] S501, voxelizing and temporally aggregating at least two frames of point clouds in a sample scene to obtain a voxelized result after temporal aggregation;

[0135] S502, extracting three-dimensional features from the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features;

[0136] S503, processing the time-aggregated three-dimensional features to obtain aggregated features of the sample scene;

[0137] S504, performing target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0138] S505, performing target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain a target detection result of the sample object;

[0139] S506: Training an object detection model according to the object detection result of the sample object and the true value detection result of the sample object.

[0140] In the embodiments of the present disclosure, the dimension of the aggregated features of the sample scene may be different from the dimension of the three-dimensional features aggregated over time. For example, the aggregated features of the sample scene may be two-dimensional features, which facilitates the subsequent use of a two-dimensional target detector to perform target detection on sample objects in the sample scene.

[0141] Specifically, a three-dimensional feature extractor can be used to extract three-dimensional features from the time-aggregated voxelization results to obtain time-aggregated three-dimensional features; the time-aggregated three-dimensional features are processed to obtain aggregated features of the sample scene. By using a three-dimensional feature extractor to extract features from the time-aggregated voxelization results to obtain time-aggregated three-dimensional features; and determining the aggregated features of the detection scene based on the time-aggregated three-dimensional features, the aggregated features of the sample scene contain the long-term spatiotemporal information of the sample scene, thereby improving the performance of subsequent target detection based on the aggregated features of the sample scene.

[0142] In an optional embodiment, the three-dimensional feature extraction is performed on the voxelization result after time aggregation to obtain the three-dimensional feature after time aggregation, including: using the first convolution subunit in the three-dimensional feature extractor to perform feature extraction on the voxelization result after time aggregation to obtain a first feature; using the second convolution subunit in the three-dimensional feature extractor to downsample the first feature to obtain a second feature; using the third convolution subunit in the three-dimensional feature extractor to perform feature extraction on the second feature to obtain a third feature; using the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as the three-dimensional feature after time aggregation; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

[0143] The three-dimensional feature extractor may include at least two feature extraction units, and one feature extraction unit includes at least a first convolution subunit, a second convolution subunit, a third convolution subunit, and a fourth convolution subunit. Specifically, the voxelization result after time aggregation is input into the first convolution subunit for feature extraction to obtain a first feature; the second convolution subunit is used to downsample the first feature to obtain a second feature; the third convolution subunit is used to extract the second feature to obtain a third feature; and the fourth convolution subunit is used to upsample the third feature to obtain a fourth feature as the three-dimensional feature after time aggregation.

[0144] The size of the first feature is the same as the size of the fourth feature, the size of the second feature is the same as the size of the third feature, and the second feature is the scaling result of the first feature. Through the above processing, part of the calculation in each sparse convolution block is transferred to a smaller resolution, which can reduce the amount of calculation on the one hand; on the other hand, a larger receptive field can be obtained, further improving the target detection performance of moving objects.

[0145] In an optional implementation, the second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

[0146] Among them, the hash table of the sparse convolution is based on the input hash table and the output hash table. Since the voxels involved in the convolution in the second convolution subunit and the fourth convolution subunit remain unchanged, the time consumption of constructing the hash table of the sparse convolution can be reduced by sharing the hash table. Compared with constructing the hash table for the second convolution subunit and the fourth convolution subunit separately, the feature extraction efficiency can be further improved.

[0147] The technical solution provided by the embodiment of the present disclosure determines the voxelization result after time aggregation based on at least two frames of point cloud through a time aggregation mechanism, thereby aggregating richer time information; in terms of feature extraction, a larger receptive field is obtained at a negligible additional time cost, which can solve the tailing problem of moving objects, thereby further improving the performance of target detection.

[0148] In an optional embodiment, the processing of the time-aggregated three-dimensional features to obtain the aggregated features of the sample scene includes: performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain the aggregated features of the sample scene; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

[0149] In the disclosed embodiment, the height dimension in the time-aggregated three-dimensional features is also merged into the channel dimension to obtain time-aggregated two-dimensional features; and the two-dimensional features aggregated in time are subjected to two-dimensional feature extraction through a two-dimensional feature extractor to obtain aggregated features of the sample scene. Taking the size (H', W', D', C') of the time-aggregated three-dimensional features as an example, the size of the time-aggregated two-dimensional features obtained by merging the height dimension into the channel dimension is (H', W', D'×C'), and the size of the aggregated features of the sample scene obtained by the two-dimensional feature extraction is (H', W', D'×C'). By converting the time-aggregated three-dimensional features into aggregated features of the sample scene, it is convenient to subsequently use a two-dimensional feature extractor to perform two-dimensional target detection, which can further improve the target detection efficiency compared to three-dimensional target detection.

[0150] Figure 6 is a flowchart of another target detection model training method provided according to an embodiment of the present disclosure. Figure 6 , the target detection model training method of this embodiment may include:

[0151] S601, voxelizing and temporally aggregating at least two frames of point clouds in a sample scene to obtain a voxelized result after temporal aggregation;

[0152] S602, extracting features from the time-aggregated voxelized results to obtain aggregated features of the sample scene;

[0153] S603, performing target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0154] S604, performing feature fusion on the sample object according to the initial detection result of the sample object and the aggregated features of the sample scene to determine the fusion features of the sample object;

[0155] S605, extracting an initial velocity of the sample object from the initial detection result of the sample object, and determining a state of the sample according to the initial velocity of the sample object;

[0156] When the sample object is in motion, continue to execute S606; when the sample object is in a stationary state, continue to execute S607;

[0157] S606, using a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the sample object, to obtain a target detection result of the sample object;

[0158] S607, using a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object;

[0159] S608: Training an object detection model according to the object detection result of the sample object and the true value detection result of the sample object.

[0160] It should be noted that after obtaining the target detection results of the sample object in S606 and S607, S608 is continued to be executed.

[0161] The initial detection result of the sample object may include the initial position, initial detection frame, initial orientation angle, initial speed or initial category of the sample object; the initial position may be the center point position of the initial detection frame. The initial speed of the sample object is compared with the speed threshold; if the initial speed of the sample object is greater than the speed threshold, the sample object is in motion; otherwise, the sample object is stationary. The fusion feature of the sample object is used to represent the total feature representation of the sample object in at least two frames of point cloud. The sample object can be determined by feature fusion based on the initial detection result of the sample object and the aggregated features of the sample scene.

[0162] The target detector of the second stage may include a motion detection head corresponding to the motion state and a stationary detection head corresponding to the stationary state; when the sample object is in motion, the fusion features of the sample object are input into the motion detection head corresponding to the motion state to perform target detection of the second stage, and obtain the target detection result of the sample object; when the sample object is a static object, the fusion features of the sample object are input into the stationary detection head corresponding to the stationary state to perform target detection of the second stage, and obtain the target detection result of the sample object. When the sample scene includes both moving objects and stationary objects, the motion detection head is used to perform the target detection of the moving objects in the second stage, and the stationary detection head is used to perform the target detection of the stationary objects in the second detection.

[0163] In the second stage, a motion detection head and a static detection head are provided respectively; the state of the sample object is determined according to the initial speed of the sample object; the motion detection head is used to detect the object in motion, and the static detection head is used to detect the object in static state, that is, based on the static-dynamic separation mechanism, objects with different motion states can be explicitly separated, and the corresponding detection head is used for target detection according to the motion state, so that the state of static objects and dynamic objects can be fully learned, thereby further improving the performance of target detection. In particular, for static objects, the static detection head can better learn the orientation angle of the object, and for dynamic objects, the dynamic detection head can better learn the speed of the object, thereby solving the bottleneck problem of objects in different motion states in the target detection process, thereby improving the performance of target detection for objects in different states.

[0164] It should be noted that the sample scene may include both static objects and moving objects. During the training of the target detection model, static object data may be used to update the network parameters in the static detection head, and moving object data may be used to update the network parameters in the moving detection head; and static object data and moving object data may be used to update the common network parameters in the target detection model.

[0165] In an optional embodiment, the method of performing feature fusion on the sample object based on the initial detection result of the sample object and the aggregated feature of the sample scene to determine the fused feature of the sample object includes: encoding the initial detection result of the sample object to obtain the encoded feature of the sample object; extracting the initial detection frame of the sample object from the initial detection result of the sample object, and determining the aggregated feature of the sample object based on the initial detection frame and the aggregated feature of the sample scene; and fusing the encoded feature of the sample object with the aggregated feature of the sample object to obtain the fused feature of the sample object.

[0166] The initial detection result of the object to be detected is input into the detection frame encoder to obtain the encoding features of the object to be detected. In addition, the initial detection frame of the object to be detected is extracted from the initial detection result of the object to be detected, and the aggregated features of the object to be detected are extracted from the aggregated features of the scene to be detected according to the position of the initial detection frame; the encoding features of the object to be detected and the aggregated features of the object to be detected are fused to obtain the fused features of the object to be detected. Through the above processing, the fused features of the object to be detected not only contain the semantic information of the initial detection results, but also contain the abstract feature representation of the aggregated features of the object to be detected, thereby improving the performance of subsequent target detection based on the fused features.

[0167] In an optional embodiment, determining the aggregate features of the sample object based on the initial detection frame and the aggregate features of the sample scene includes: extracting key points of the sample object from the initial detection frame when the sample object is in a stationary state; and extracting the aggregate features of the sample object from the aggregate features of the sample scene based on the key points of the sample object.

[0168] When the sample object is stationary, the midpoint of the left frame, the midpoint of the right frame, the midpoint of the front frame, the midpoint of the back frame and the center point of the initial detection frame can be used as key points, and the aggregate features at the key points are extracted from the aggregate features of the sample scene to obtain the aggregate features of the sample object. The fused features of the sample object are obtained by fusing the encoding features and the aggregate features of the sample object. For static objects, by extracting the key points of the sample object from the initial detection frame and extracting the aggregate features of the sample object from the aggregate features of the sample scene based on the key points, the accuracy of the aggregate features of the sample object can be improved, laying the foundation for subsequent target detection.

[0169] In an optional embodiment, determining the aggregate features of the sample object based on the initial detection frame and the aggregate features of the sample scene includes: when the sample object is in motion, extracting the initial position and initial orientation angle of the sample object from the initial detection result of the sample object; determining an auxiliary detection frame of the sample object based on the initial speed, the initial position, the initial orientation angle and the initial detection frame; extracting key points of the sample object from the initial detection frame and the auxiliary detection frame respectively; and extracting the aggregate features of the sample object from the aggregate features of the sample scene based on the key points of the sample object.

[0170] When the sample object is in motion, the initial detection result of the sample object is the initial detection result of the sample object in the specified frame point cloud, and the specified frame point cloud belongs to one of the at least two frame point clouds, which can be preset to a fixed frame number. The other frame point clouds in the at least two frame point clouds except the specified frame point cloud can be called auxiliary frame point clouds.

[0171] Specifically, the auxiliary detection frame of the sample object in the auxiliary frame point cloud can be determined according to the initial speed, initial position, initial orientation angle and initial detection frame of the sample object in the specified frame point cloud; the key points of the sample object are extracted from the initial detection frame and the auxiliary detection frame respectively, for example, the left frame midpoint, the right frame midpoint, the front frame midpoint, the rear frame midpoint and the center point of the frame in the initial detection and auxiliary detection frames can be used as key points, and the aggregate features of the sample object are extracted from the aggregate features of the sample scene using each key point. For dynamic objects, the auxiliary detection frame of the sample object is determined according to the initial speed, initial position, initial orientation angle and initial detection frame; the key points of the sample object are extracted from the initial detection frame and the auxiliary detection frame respectively; according to the key points of the sample object, the aggregate features of the sample object are extracted from the aggregate features of the sample scene, which can improve the accuracy of the detection frame and speed prediction of dynamic objects.

[0172] In actual applications, sample scenes generally contain objects in different motion states. If dynamic targets and static targets are treated indiscriminately, the differences in network features between dynamic and static targets will result in the network only being able to achieve suboptimal performance. The disclosed embodiment introduces a dynamic and static separation mechanism, which is of great significance for point cloud time series 3D detection. It allows the network to learn better features for dynamic and static targets, and has broad practical value.

[0173] The technical solution provided by the embodiments of the present disclosure, by proposing a time aggregation mechanism, aggregates richer time information and obtains a larger receptive field with negligible additional time cost; it also proposes a dynamic-static separation mechanism, which optimizes the 3D bounding boxes of static and moving objects respectively by utilizing the predicted object motion information; and, based on the above-mentioned time aggregation mechanism and dynamic-static separation mechanism, a two-stage point cloud 3D object detector is proposed, which uses the motion information contained in multi-frame point clouds as a constraint, can fully mine the spatiotemporal information of long time series, and improve the detection performance and robustness of the point cloud 3D detection model.

[0174] Figure 7 1 is a schematic diagram of a target detection device according to an embodiment of the present disclosure. This embodiment is applicable to the case where target detection is performed on continuous point cloud frame data. The device can be implemented in software and / or hardware. Figure 7 As shown, the target detection device 700 of this embodiment may include:

[0175] The aggregate voxelization module 710 is used to voxelize and temporally aggregate at least two frames of point clouds in the scene to be detected to obtain a voxelization result after temporal aggregation;

[0176] Aggregation feature module 720, used to extract features from the time-aggregated voxelization results to obtain aggregate features of the scene to be detected;

[0177] An initial detection module 730 is used to perform target detection according to the aggregated features of the scene to be detected, and obtain an initial detection result of the object to be detected in the scene to be detected;

[0178] The aggregate detection module 740 is used to perform target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, so as to obtain the target detection result of the object to be detected.

[0179] In an optional implementation, the aggregate voxelization module 710 includes:

[0180] A three-dimensional voxel unit is used to voxelize at least two frames of point clouds in the scene to be detected, respectively, to obtain three-dimensional voxelization results of at least two frames of point clouds;

[0181] A four-dimensional voxel unit is used to splice three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result;

[0182] The time merging unit is used to merge the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a time-aggregated voxelization result.

[0183] In an optional implementation, the aggregation feature module 720 includes:

[0184] A three-dimensional feature extraction unit, used for performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features;

[0185] The aggregate feature extraction unit is used to process the time-aggregated three-dimensional features to obtain the aggregate features of the scene to be detected.

[0186] In an optional implementation, the three-dimensional feature extraction unit includes:

[0187] A first feature subunit is used to extract features from the time-aggregated voxelized result using a first convolution subunit in the three-dimensional feature extractor to obtain a first feature;

[0188] A second feature subunit is used to downsample the first feature using a second convolution subunit in the three-dimensional feature extractor to obtain a second feature;

[0189] A third feature subunit is used to extract the second feature by using a third convolution subunit in the three-dimensional feature extractor to obtain a third feature;

[0190] The fourth feature subunit is used to use the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as a three-dimensional feature aggregated by time; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

[0191] In an optional implementation, the second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

[0192] In an optional implementation, the aggregate feature extraction unit is specifically used to:

[0193] Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the scene to be detected; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

[0194] In an optional implementation, the aggregation detection module 740 includes:

[0195] A fusion feature submodule, used to perform feature fusion on the object to be detected according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, and determine the fusion features of the object to be detected;

[0196] A state determination submodule, used to extract the initial speed of the object to be detected from the initial detection result of the object to be detected, and determine the state of the object to be detected according to the initial speed of the object to be detected;

[0197] A motion detection submodule, used to, when the object to be detected is in a motion state, use a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the object to be detected, and obtain a target detection result of the object to be detected;

[0198] The stationary detection submodule is used to, when the object is in a stationary state, use a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the object to be detected, so as to obtain a target detection result of the object to be detected.

[0199] In an optional implementation, the fusion feature submodule includes:

[0200] An encoding unit, used for encoding the initial detection result of the object to be detected to obtain a coding feature of the object to be detected;

[0201] An object aggregation unit, used to extract an initial detection frame of the object to be detected from the initial detection result of the object to be detected, and determine the aggregation features of the object to be detected based on the initial detection frame and the aggregation features of the scene to be detected;

[0202] The fusion feature unit is used to fuse the coding feature of the object to be detected and the aggregation feature of the object to be detected to obtain the fusion feature of the object to be detected.

[0203] In an optional implementation, the object aggregation unit includes:

[0204] A stationary key point subunit, used for extracting key points of the object to be detected from the initial detection frame when the object to be detected is in a stationary state;

[0205] The stationary aggregation subunit is used to extract the aggregated features of the object to be detected from the aggregated features of the scene to be detected according to the key points of the object to be detected.

[0206] In an optional implementation, the object aggregation unit includes:

[0207] A result extraction subunit is used to extract the initial position and initial orientation angle of the object to be detected from the initial detection result of the object to be detected when the object to be detected is in motion;

[0208] an auxiliary subunit, configured to determine an auxiliary detection frame of the object to be detected according to the initial speed, the initial position, the initial orientation angle and the initial detection frame;

[0209] A motion key point subunit, used to extract key points of the object to be detected from the initial detection frame and the auxiliary detection frame respectively;

[0210] The motion aggregation subunit is used to extract the aggregated features of the object to be detected from the aggregated features of the scene to be detected according to the key points of the object to be detected.

[0211] The technical solution of the disclosed embodiment can make full use of the additional information provided by multi-frame point clouds to effectively improve the accuracy and robustness of point cloud 3D target detection, especially for moving targets, it can improve the accuracy of 3D box and speed estimation. In actual scenes, there are generally objects in different motion states. If the network treats dynamic and static targets indiscriminately, the differences in network features between dynamic and static targets will cause the network to only obtain suboptimal performance. Therefore, the introduction of a dynamic and static separation mechanism is of great significance for point cloud time series 3D detection, which can allow the network to learn better features for dynamic and static targets.

[0212] Figure 8 1 is a schematic diagram of a target detection model training device according to an embodiment of the present disclosure. This embodiment is applicable to the case of training a target detection model. The device can be implemented in software and / or hardware. Figure 8As shown, the target detection model training device 800 of this embodiment may include:

[0213] The aggregate voxelization module 810 is used to voxelize and temporally aggregate at least two frames of point clouds in the sample scene to obtain a temporally aggregated voxelization result;

[0214] Aggregation feature module 820, used to extract features from the time-aggregated voxelization results to obtain aggregate features of the sample scene;

[0215] An initial detection module 830, configured to perform target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene;

[0216] The aggregate detection module 840 is used to perform target detection based on the initial detection results of the sample object and the aggregate features of the sample scene to obtain the target detection results of the sample object; the model training module 850 is used to train the target detection model based on the target detection results of the sample object and the true value detection results of the sample object.

[0217] In an optional implementation, the aggregate voxelization module 810 includes:

[0218] A three-dimensional voxel unit, used to voxelize at least two frames of point clouds in the sample scene respectively, to obtain three-dimensional voxelization results of at least two frames of point clouds;

[0219] A four-dimensional voxel unit is used to splice three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result;

[0220] The time merging unit is used to merge the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a time-aggregated voxelization result.

[0221] In an optional implementation, the aggregation feature module 820 includes:

[0222] A three-dimensional feature extraction unit, used for performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features;

[0223] The aggregate feature extraction unit is used to process the time-aggregated three-dimensional features to obtain the aggregate features of the sample scene.

[0224] In an optional implementation, the three-dimensional feature extraction unit includes:

[0225] A first feature subunit is used to extract features from the time-aggregated voxelized result using a first convolution subunit in the three-dimensional feature extractor to obtain a first feature;

[0226] A second feature subunit is used to downsample the first feature using a second convolution subunit in the three-dimensional feature extractor to obtain a second feature;

[0227] A third feature subunit is used to extract the second feature by using a third convolution subunit in the three-dimensional feature extractor to obtain a third feature;

[0228] The fourth feature subunit is used to use the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as a three-dimensional feature aggregated by time; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

[0229] In an optional implementation, the second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

[0230] In an optional implementation, the aggregate feature extraction unit is specifically used to:

[0231] Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the sample scene; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

[0232] In an optional implementation, the aggregation detection module 840 includes:

[0233] A fusion feature submodule, used to perform feature fusion on the sample object according to the initial detection result of the sample object and the aggregated features of the sample scene, and determine the fusion features of the sample object;

[0234] A state determination submodule, used to extract the initial speed of the sample object from the initial detection result of the sample object, and determine the state of the sample according to the initial speed of the sample object;

[0235] A motion detection submodule, configured to, when the sample object is in a motion state, use a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object;

[0236] The stationary detection submodule is used to, when the object is in a stationary state, use a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object.

[0237] In an optional implementation, the fusion feature submodule includes:

[0238] An encoding unit, used for encoding the initial detection result of the sample object to obtain a coding feature of the sample object;

[0239] An object aggregation unit, configured to extract an initial detection frame of the sample object from the initial detection result of the sample object, and determine an aggregation feature of the sample object according to the initial detection frame and the aggregation feature of the sample scene;

[0240] The fusion feature unit is used to fuse the coding feature of the sample object and the aggregation feature of the sample object to obtain the fusion feature of the sample object.

[0241] In an optional implementation, the object aggregation unit includes:

[0242] A stationary key point subunit, configured to extract key points of the sample object from the initial detection frame when the sample object is in a stationary state;

[0243] The stationary aggregation subunit is used to extract the aggregate features of the sample object from the aggregate features of the sample scene according to the key points of the sample object.

[0244] In an optional implementation, the object aggregation unit includes:

[0245] A result extraction subunit, configured to extract an initial position and an initial orientation angle of the sample object from an initial detection result of the sample object when the sample object is in motion;

[0246] an auxiliary subunit, configured to determine an auxiliary detection frame of the sample object according to the initial speed, the initial position, the initial orientation angle and the initial detection frame;

[0247] A motion key point subunit, used to extract key points of sample objects from the initial detection frame and the auxiliary detection frame respectively;

[0248] The motion aggregation subunit is used to extract the aggregate features of the sample object from the aggregate features of the sample scene according to the key points of the sample object.

[0249] The technical solution of the disclosed embodiment can make full use of the additional information provided by multi-frame point clouds to effectively improve the accuracy and robustness of point cloud 3D target detection, especially for moving targets, it can improve the accuracy of 3D box and speed estimation. In actual scenes, there are generally objects in different motion states. If the network treats dynamic and static targets indiscriminately, the differences in network features between dynamic and static targets will cause the network to only obtain suboptimal performance. Therefore, the introduction of a dynamic and static separation mechanism is of great significance for point cloud time series 3D detection, which can allow the network to learn better features for dynamic and static targets.

[0250] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0251] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0252] Fig. 9 It is a block diagram of an electronic device used to implement the target detection method or target detection model training method of the embodiment of the present disclosure.

[0253] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0254] like Fig. 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0255] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0256] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a target detection method or a target detection model training method. For example, in some embodiments, the target detection method or the target detection model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the target detection method or the target detection model training method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the target detection method or the target detection model training method in any other suitable manner (for example, by means of firmware).

[0257] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0258] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0259] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0260] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0261] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0262] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0263] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0264] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0265] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0266] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, comprising: voxelize and temporally aggregate at least two frames of point clouds in the scene to be detected to obtain a voxelized result after temporal aggregation; Perform feature extraction on the voxelized results after temporal aggregation to obtain aggregated features of the scene to be detected; Performing target detection according to the aggregated features of the scene to be detected to obtain an initial detection result of the object to be detected in the scene to be detected; Performing target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected to obtain a target detection result of the object to be detected; The step of voxelizing and temporally aggregating at least two frames of point clouds in the scene to be detected to obtain a voxelized result after temporal aggregation includes: voxelize at least two frames of point clouds in the scene to be detected, respectively, to obtain three-dimensional voxelized results of at least two frames of point clouds; splicing the three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; The time dimension in the four-dimensional voxelization result is merged into the channel dimension to obtain a time-aggregated voxelization result.

2. The method according to claim 1, wherein: The step of extracting features from the temporally aggregated voxelized results to obtain aggregated features of the scene to be detected includes: Performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional feature; The time-aggregated three-dimensional features are processed to obtain aggregated features of the scene to be detected.

3. The method according to claim 2, wherein: The step of extracting three-dimensional features from the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features includes: Using the first convolution subunit in the three-dimensional feature extractor, extracting features from the temporally aggregated voxelized result to obtain a first feature; Downsampling the first feature using a second convolution subunit in a three-dimensional feature extractor to obtain a second feature; Using a third convolution subunit in a three-dimensional feature extractor, extracting features from the second feature to obtain a third feature; The fourth convolution subunit in the three-dimensional feature extractor is used to upsample the third feature to obtain a fourth feature as a time-aggregated three-dimensional feature; wherein the first feature and the fourth feature have the same size; and the convolution kernels of each convolution subunit have the same size.

4. The method according to claim 3, wherein: The second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

5. The method according to claim 2, wherein: The processing of the time-aggregated three-dimensional features to obtain aggregated features of the scene to be detected includes: Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the scene to be detected; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

6. The method according to claim 1, wherein: The performing target detection according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected to obtain the target detection result of the object to be detected includes: Performing feature fusion on the object to be detected according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected to determine the fusion features of the object to be detected; Extracting an initial speed of the object to be detected from the initial detection result of the object to be detected, and determining the state of the object to be detected according to the initial speed of the object to be detected; When the object to be detected is in motion, a motion detection head corresponding to the motion state is used to perform target detection according to the aggregated features of the object to be detected to obtain a target detection result of the object to be detected; When the object is in a stationary state, a stationary detection head corresponding to the stationary state is used to perform target detection according to the aggregated features of the object to be detected, so as to obtain a target detection result of the object to be detected.

7. The method according to claim 6, wherein: The step of fusing features of the object to be detected based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected to determine the fused features of the object to be detected includes: Encoding the initial detection result of the object to be detected to obtain a coding feature of the object to be detected; Extracting an initial detection frame of the object to be detected from the initial detection result of the object to be detected, and determining aggregate features of the object to be detected based on the initial detection frame and aggregate features of the scene to be detected; The coding feature of the object to be detected and the aggregation feature of the object to be detected are fused to obtain the fusion feature of the object to be detected.

8. The method according to claim 7, wherein: The step of determining the aggregate features of the object to be detected according to the initial detection frame and the aggregate features of the scene to be detected includes: When the object to be detected is in a stationary state, extracting key points of the object to be detected from the initial detection frame; According to the key points of the object to be detected, aggregate features of the object to be detected are extracted from aggregate features of the scene to be detected.

9. The method according to claim 7, wherein: The step of determining the aggregate features of the object to be detected according to the initial detection frame and the aggregate features of the scene to be detected includes: In the case where the object to be detected is in motion, an initial position and an initial orientation angle of the object to be detected are also extracted from the initial detection result of the object to be detected; Determining an auxiliary detection frame of the object to be detected according to the initial speed, the initial position, the initial orientation angle, and the initial detection frame; Extract key points of the object to be detected from the initial detection frame and the auxiliary detection frame respectively; According to the key points of the object to be detected, aggregate features of the object to be detected are extracted from aggregate features of the scene to be detected.

10. A target detection model training method, comprising: voxelize and temporally aggregate at least two frames of point clouds in the sample scene to obtain a voxelized result after temporal aggregation; Perform feature extraction on the time-aggregated voxelization results to obtain aggregated features of the sample scene; Performing target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene; Performing target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain a target detection result of the sample object; Training an object detection model according to the object detection result of the sample object and the true value detection result of the sample object; The step of voxelizing and temporally aggregating at least two frames of point clouds in the sample scene to obtain a temporally aggregated voxelized result includes: voxelize at least two frames of point clouds in the sample scene respectively to obtain three-dimensional voxelized results of at least two frames of point clouds; splicing the three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; The time dimension in the four-dimensional voxelization result is merged into the channel dimension to obtain a time-aggregated voxelization result.

11. The method according to claim 10, wherein: The step of extracting features from the temporally aggregated voxelized results to obtain aggregated features of the sample scene includes: Performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional feature; The time-aggregated three-dimensional features are processed to obtain aggregated features of the sample scene.

12. The method according to claim 11, wherein: The step of extracting three-dimensional features from the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features includes: Using the first convolution subunit in the three-dimensional feature extractor, extracting features from the temporally aggregated voxelized result to obtain a first feature; Downsampling the first feature using a second convolution subunit in a three-dimensional feature extractor to obtain a second feature; Using a third convolution subunit in a three-dimensional feature extractor, extracting features from the second feature to obtain a third feature; The fourth convolution subunit in the three-dimensional feature extractor is used to upsample the third feature to obtain a fourth feature as a time-aggregated three-dimensional feature; wherein the first feature and the fourth feature have the same size; and the convolution kernels of each convolution subunit have the same size.

13. The method according to claim 12, wherein: The second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

14. The method according to claim 11, wherein: The processing of the time-aggregated three-dimensional features to obtain aggregated features of the sample scene includes: Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the sample scene; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

15. The method according to claim 10, wherein: The performing target detection according to the initial detection result of the sample object and the aggregated features of the sample scene to obtain the target detection result of the sample object includes: Performing feature fusion on the sample object according to the initial detection result of the sample object and the aggregated features of the sample scene to determine the fusion features of the sample object; Extracting an initial velocity of the sample object from an initial detection result of the sample object, and determining a state of the sample according to the initial velocity of the sample object; When the sample object is in a moving state, a motion detection head corresponding to the moving state is used to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object; When the object is in a stationary state, a stationary detection head corresponding to the stationary state is used to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object.

16. The method according to claim 15, wherein: The step of fusing features of the sample object according to the initial detection result of the sample object and the aggregated features of the sample scene to determine the fused features of the sample object includes: Encoding the initial detection result of the sample object to obtain a coding feature of the sample object; Extracting an initial detection frame of the sample object from the initial detection result of the sample object, and determining aggregate features of the sample object according to the initial detection frame and aggregate features of the sample scene; The coding feature of the sample object and the aggregation feature of the sample object are fused to obtain a fused feature of the sample object.

17. The method according to claim 16, wherein: The determining the aggregate features of the sample object according to the initial detection frame and the aggregate features of the sample scene includes: When the sample object is in a stationary state, extracting key points of the sample object from the initial detection frame; According to the key points of the sample object, the aggregate features of the sample object are extracted from the aggregate features of the sample scene.

18. The method according to claim 16, wherein: The determining the aggregate features of the sample object according to the initial detection frame and the aggregate features of the sample scene includes: In the case where the sample object is in motion, an initial position and an initial orientation angle of the sample object are also extracted from the initial detection result of the sample object; Determining an auxiliary detection frame of the sample object according to the initial speed, the initial position, the initial orientation angle, and the initial detection frame; Extract key points of sample objects from the initial detection frame and the auxiliary detection frame respectively; According to the key points of the sample object, the aggregate features of the sample object are extracted from the aggregate features of the sample scene.

19. A target detection device, comprising: An aggregation voxelization module, used for voxelizing and temporally aggregating at least two frames of point clouds in a scene to be detected, to obtain a voxelization result after temporal aggregation; An aggregation feature module is used to extract features from the voxelized results after temporal aggregation to obtain the aggregation features of the scene to be detected; An initial detection module, used to perform target detection according to the aggregated features of the scene to be detected, and obtain an initial detection result of the object to be detected in the scene to be detected; An aggregate detection module, used to perform target detection based on the initial detection result of the object to be detected and the aggregated features of the scene to be detected, to obtain a target detection result of the object to be detected; Wherein, the aggregate voxelization module comprises: A three-dimensional voxel unit is used to voxelize at least two frames of point clouds in the scene to be detected, respectively, to obtain three-dimensional voxelization results of at least two frames of point clouds; A four-dimensional voxel unit is used to splice three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; The time merging unit is used to merge the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a time-aggregated voxelization result.

20. The device according to claim 19, wherein The aggregation feature module includes: A three-dimensional feature extraction unit, used for performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features; The aggregate feature extraction unit is used to process the time-aggregated three-dimensional features to obtain the aggregate features of the scene to be detected.

21. The device according to claim 20, wherein: The three-dimensional feature extraction unit comprises: A first feature subunit is used to extract features from the time-aggregated voxelized result using a first convolution subunit in the three-dimensional feature extractor to obtain a first feature; A second feature subunit is used to downsample the first feature using a second convolution subunit in the three-dimensional feature extractor to obtain a second feature; A third feature subunit is used to extract the second feature by using a third convolution subunit in the three-dimensional feature extractor to obtain a third feature; The fourth feature subunit is used to use the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as a three-dimensional feature aggregated by time; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

22. The device according to claim 21, wherein The second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

23. The device according to claim 20, wherein: The aggregate feature extraction unit is specifically used for: Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the scene to be detected; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

24. The device according to claim 19, wherein The aggregation detection module comprises: A fusion feature submodule, used to perform feature fusion on the object to be detected according to the initial detection result of the object to be detected and the aggregated features of the scene to be detected, and determine the fusion features of the object to be detected; A state determination submodule, used to extract the initial speed of the object to be detected from the initial detection result of the object to be detected, and determine the state of the object to be detected according to the initial speed of the object to be detected; A motion detection submodule, used to, when the object to be detected is in a motion state, use a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the object to be detected, and obtain a target detection result of the object to be detected; The stationary detection submodule is used to, when the object is in a stationary state, use a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the object to be detected, so as to obtain a target detection result of the object to be detected.

25. The device according to claim 24, wherein: The fusion feature submodule includes: An encoding unit, used for encoding the initial detection result of the object to be detected to obtain a coding feature of the object to be detected; An object aggregation unit, used to extract an initial detection frame of the object to be detected from the initial detection result of the object to be detected, and determine the aggregation features of the object to be detected based on the initial detection frame and the aggregation features of the scene to be detected; The fusion feature unit is used to fuse the coding feature of the object to be detected and the aggregation feature of the object to be detected to obtain the fusion feature of the object to be detected.

26. The device according to claim 25, wherein The object aggregation unit comprises: A stationary key point subunit, used for extracting key points of the object to be detected from the initial detection frame when the object to be detected is in a stationary state; The stationary aggregation subunit is used to extract the aggregated features of the object to be detected from the aggregated features of the scene to be detected according to the key points of the object to be detected.

27. The device according to claim 25, wherein: The object aggregation unit comprises: A result extraction subunit is used to extract the initial position and initial orientation angle of the object to be detected from the initial detection result of the object to be detected when the object to be detected is in motion; an auxiliary subunit, configured to determine an auxiliary detection frame of the object to be detected according to the initial speed, the initial position, the initial orientation angle and the initial detection frame; A motion key point subunit, used to extract key points of the object to be detected from the initial detection frame and the auxiliary detection frame respectively; The motion aggregation subunit is used to extract the aggregated features of the object to be detected from the aggregated features of the scene to be detected according to the key points of the object to be detected.

28. A target detection model training device, comprising: An aggregation voxelization module, used for voxelizing and temporally aggregating at least two frames of point clouds in a sample scene to obtain a voxelization result after temporal aggregation; An aggregation feature module is used to extract features from the voxelized results after temporal aggregation to obtain the aggregation features of the sample scene; An initial detection module, used to perform target detection according to the aggregated features of the sample scene to obtain an initial detection result of a sample object in the sample scene; An aggregate detection module, used to perform target detection based on the initial detection result of the sample object and the aggregate features of the sample scene to obtain a target detection result of the sample object; A model training module, used to train the target detection model according to the target detection result of the sample object and the true value detection result of the sample object; Wherein, the aggregate voxelization module comprises: A three-dimensional voxel unit, used to voxelize at least two frames of point clouds in the sample scene respectively, to obtain three-dimensional voxelization results of at least two frames of point clouds; A four-dimensional voxel unit is used to splice three-dimensional voxelization results of at least two frames of point clouds according to the time dimension to obtain a four-dimensional voxelization result; The time merging unit is used to merge the time dimension in the four-dimensional voxelization result into the channel dimension to obtain a time-aggregated voxelization result.

29. The device according to claim 28, wherein The aggregation feature module includes: A three-dimensional feature extraction unit, used for performing three-dimensional feature extraction on the time-aggregated voxelization result to obtain the time-aggregated three-dimensional features; The aggregate feature extraction unit is used to process the time-aggregated three-dimensional features to obtain the aggregate features of the sample scene.

30. The device according to claim 29, wherein: The three-dimensional feature extraction unit comprises: A first feature subunit is used to extract features from the time-aggregated voxelized result using a first convolution subunit in the three-dimensional feature extractor to obtain a first feature; A second feature subunit is used to downsample the first feature using a second convolution subunit in the three-dimensional feature extractor to obtain a second feature; A third feature subunit is used to extract the second feature by using a third convolution subunit in the three-dimensional feature extractor to obtain a third feature; The fourth feature subunit is used to use the fourth convolution subunit in the three-dimensional feature extractor to upsample the third feature to obtain a fourth feature as a three-dimensional feature aggregated by time; wherein the first feature and the fourth feature have the same size; and the convolution kernel size of each convolution subunit is the same.

31. The device according to claim 30, wherein The second convolution subunit and the fourth convolution subunit use the same hash table for sparse convolution.

32. The device according to claim 29, wherein: The aggregate feature extraction unit is specifically used for: Performing two-dimensional feature extraction on the time-aggregated two-dimensional features to obtain aggregated features of the sample scene; wherein the time-aggregated two-dimensional features are obtained by merging the height dimension in the time-aggregated three-dimensional features into the channel dimension.

33. The apparatus of claim 28, wherein: The aggregation detection module comprises: A fusion feature submodule, used to perform feature fusion on the sample object according to the initial detection result of the sample object and the aggregated features of the sample scene, and determine the fusion features of the sample object; A state determination submodule, used to extract the initial speed of the sample object from the initial detection result of the sample object, and determine the state of the sample according to the initial speed of the sample object; A motion detection submodule, configured to, when the sample object is in a motion state, use a motion detection head corresponding to the motion state to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object; The stationary detection submodule is used to, when the object is in a stationary state, use a stationary detection head corresponding to the stationary state to perform target detection according to the aggregated features of the sample object to obtain a target detection result of the sample object.

34. The device according to claim 33, wherein The fusion feature submodule includes: An encoding unit, used for encoding the initial detection result of the sample object to obtain a coding feature of the sample object; An object aggregation unit, configured to extract an initial detection frame of the sample object from the initial detection result of the sample object, and determine an aggregation feature of the sample object according to the initial detection frame and the aggregation feature of the sample scene; The fusion feature unit is used to fuse the coding feature of the sample object and the aggregation feature of the sample object to obtain the fusion feature of the sample object.

35. The device according to claim 34, wherein The object aggregation unit comprises: A stationary key point subunit, configured to extract key points of a sample object from the initial detection frame when the sample object is in a stationary state; The stationary aggregation subunit is used to extract the aggregate features of the sample object from the aggregate features of the sample scene according to the key points of the sample object.

36. The device according to claim 35, wherein The object aggregation unit comprises: A result extraction subunit, configured to extract an initial position and an initial orientation angle of the sample object from an initial detection result of the sample object when the sample object is in motion; an auxiliary subunit, configured to determine an auxiliary detection frame of the sample object according to the initial speed, the initial position, the initial orientation angle and the initial detection frame; A motion key point subunit, used to extract key points of sample objects from the initial detection frame and the auxiliary detection frame respectively; The motion aggregation subunit is used to extract the aggregate features of the sample object from the aggregate features of the sample scene according to the key points of the sample object.

37. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-9; or, execute the method according to any one of claims 10-18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 9; or to execute the method according to any one of claims 10 to 18.

39. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9; or implements the method according to any one of claims 10 to 18.

Images