Point cloud detection method, system and device based on time sequence anchor frame and storage medium

Abstract

The invention provides a point cloud detection method, system and device based on a time sequence anchor frame, and a storage medium. The method comprises the following steps: collecting time sequence point cloud data of different time sequence frames; establishing a time step point cloud group according to each time sequence point cloud and n continuous time sequence point clouds before the time sequence point cloud, and performing point cloud fusion; inputting the point cloud of the current time sequence and the time step point cloud group into a sparse convolutional backbone network to obtain a first aerial view feature matrix and a second aerial view feature matrix; generating a proposed anchor frame according to the second aerial view feature matrix corresponding to each time step point cloud group; performing alignment and weighted aggregation on m proposed anchor frames with continuous time sequences to obtain a proposed anchor frame matrix; inputting the proposed anchor frame matrix and the first aerial view feature matrix into a neural network to obtain a proposed anchor frame residual sequence; and adding the attribute sequence of the proposed anchor frame matrix and the proposed anchor frame residual sequence, and outputting a target category and a confidence score thereof from the neural network. According to the invention, the accuracy of point cloud identification, especially detection of a far-end target, can be enhanced.

Description

Background Art

[0002] In the autonomous driving system, 3D target detection is a basic and important perception task, and the quality of its detection performance will directly affect the downstream planning and decision-making tasks. Since the point cloud generated by LiDAR is highly robust to environmental conditions such as lighting and weather, most existing autonomous driving vehicles use LiDAR to detect 3D targets in the surrounding environment. With the development of deep learning technology, more and more detection methods use end-to-end methods for detection, and the detection accuracy has been greatly improved. However, in some special cases, such as: the detection target is far away, the relative scale of the object is small or it is partially blocked, which will cause the target point cloud to be too sparse, which will cause fluctuations in detection performance. If the number of light sources of the LiDAR is simply increased in order to accurately identify distant targets, the cost of the LiDAR will be greatly increased, but the benefits will be very limited.

[0003] Therefore, the present invention provides a point cloud detection method, system, device and storage medium based on a temporal anchor frame. Summary of the invention

[0004] In response to the problems in the prior art, the purpose of the present invention is to provide a point cloud detection method, system, device and storage medium based on temporal anchor frames, which overcomes the difficulties of the prior art and can enhance target features that are weakened due to sparse point clouds, thereby improving the overall detection accuracy, especially for remote targets.

[0005] An embodiment of the present invention provides a point cloud detection method based on a temporal anchor frame, comprising the following steps:

[0006] S110, collecting time-series point cloud data of different time-series frames and converting them to the point cloud coordinate system of the current time sequence;

[0007] S120, establishing a time step point cloud group according to each time sequence point cloud and n consecutive time sequence point clouds before the time sequence point cloud, where n≥1, and performing point cloud fusion;

[0008] S130, after voxelizing and averaging the point cloud of the current time series and the time step point cloud group, the point cloud of the current time series is input into a sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and the time step point cloud group is input into a sparse convolution backbone network to obtain a second bird's-eye view feature matrix;

[0009] S140, generating a proposed anchor frame according to the second bird's-eye view feature matrix corresponding to each time step point cloud group, wherein the attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame;

[0010] S150, aligning and weighted aggregation of the proposed anchor frames corresponding to the m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix, wherein the closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6;

[0011] S160, inputting the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence;

[0012] S170, respectively adding the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputting the target category and its confidence score from the neural network to obtain a detection result.

[0013] Preferably, the step S110 includes:

[0014] S111, collecting time-series point cloud data and vehicle posture information of different time-series frames;

[0015] S112, converting the point cloud data of the previous time series to the current time series point cloud coordinate system of the point cloud of the current time series according to the external parameter matrix obtained from the vehicle posture information. Preferably, the step S120 includes:

[0016] S121, according to the sequential order of the time series frames, a time step point cloud group is established based on each time series point cloud and the n consecutive time series point clouds before the time series point cloud, where n≥1;

[0017] S122, fusing the point clouds in each of the time step point cloud groups, and adding a time series attribute corresponding to the time step point cloud group to each point in the point cloud.

[0018] Preferably, the step S130 includes:

[0019] S131, voxelizing and averaging the point cloud of the current time series and the point cloud group of the time step respectively;

[0020] S132, inputting the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix;

[0021] S133. Input the time step point cloud group into a sparse convolution backbone network to obtain a second bird's-eye view feature matrix, wherein the first bird's-eye view feature matrix and the second bird's-eye view feature matrix are both plane matrices, and each point in the plane matrix has a one-dimensional tensor.

[0022] Preferably, in step S140, a preset detection head is used respectively according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step to generate a proposed anchor frame, and the attribute sequence of each proposed anchor frame is: (x, y, z, dx, dy, dz, r, s), wherein x, y, z are the spatial coordinates of the center of the proposed anchor frame in the temporal point cloud coordinate system, dx, dy, dz are the length, width, and height of the proposed anchor frame, r is the heading angle corresponding to the proposed anchor frame, and s is the maximum value of all prediction scores corresponding to the proposed anchor frame.

[0023] Preferably, the step S150 includes:

[0024] S151, aligning the proposed anchor frames corresponding to m temporally consecutive time step point cloud groups, 2≤M≤6;

[0025] S152. Aggregate the proposed anchor frames according to a preset weight sequence, where the preset weight sequence includes m weight elements, and the weight element corresponding to the proposed anchor frame closer to the current time sequence is larger. Multiply the weight element corresponding to the proposed anchor frame by the confidence score in the attribute sequence, and aggregate the proposed anchor frames corresponding to the m time step point cloud groups to form a proposed anchor frame matrix.

[0026] Preferably, in step S150, m is 4, and the preset weight sequence is (1, 0.9, 0.8, 0.7).

[0027] Preferably, the step S160 includes:

[0028] S161, inputting the aggregated proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition;

[0029] S162, encoding the aggregated matrix of the proposed anchor boxes into a query vector, fusing it with a preset learnable query vector, and then sending it to a self-attention model for feature interaction;

[0030] S163. Input the output of the self-attention model and the first bird's-eye view feature matrix into a deformable cross-attention model to obtain the proposed anchor box residual sequence and the updated learnable query vector, and use the updated learnable query vector to replace the preset learnable query vector.

[0031] Preferably, the step S170 includes:

[0032] S171, respectively adding the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame;

[0033] S172. The predicted label with the highest confidence score among the confidence scores output by the deformable cross-attention model and whose confidence score is greater than a preset threshold is used as the obstacle label of the proposed anchor box.

[0034] An embodiment of the present invention further provides a point cloud detection system based on a temporal anchor frame, which is used to implement the above-mentioned point cloud detection method based on a temporal anchor frame. The point cloud detection system based on a temporal anchor frame includes:

[0035] Point cloud acquisition module, which collects time-series point cloud data of different time frames and converts them into the point cloud coordinate system of the current time sequence;

[0036] The point cloud fusion module establishes a time step point cloud group according to each time sequence point cloud and the n consecutive time sequence point clouds before the time sequence point cloud, where n ≥ 1, and performs point cloud fusion;

[0037] The bird's-eye view feature module voxelizes and averages the point cloud of the current time series and the time step point cloud group, respectively, and then inputs the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and inputs the time step point cloud group into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix;

[0038] The proposed anchor frame module generates a proposed anchor frame according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step, and the attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame;

[0039] Anchor frame aggregation module aligns and weighted aggregates the proposed anchor frames corresponding to m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix. The closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6;

[0040] An anchor box residual module, inputting the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence;

[0041] The detection and recognition module adds the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputs the target category and its confidence score from the neural network to obtain the detection result.

[0042] An embodiment of the present invention further provides a point cloud detection device based on a temporal anchor frame, comprising:

[0043] processor;

[0044] a memory storing executable instructions of the processor;

[0045] The processor is configured to perform the steps of the above-mentioned point cloud detection method based on temporal anchor frames by executing the executable instructions.

[0046] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which, when executed, implements the steps of the above-mentioned point cloud detection method based on temporal anchor boxes.

[0047] The purpose of the present invention is to provide a point cloud detection method, system, device and storage medium based on a temporal anchor frame, which can enhance the target features weakened by the sparse point cloud, thereby improving the overall detection accuracy, especially the detection accuracy of remote targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0049] Figure 1 It is a flow chart of the point cloud detection method based on temporal anchor frames of the present invention.

[0050] Figure 2 It is a schematic diagram of the operating principle of the point cloud detection method based on the temporal anchor frame of the present invention.

[0051] Figure 3 It is a schematic diagram of the process of implementing the point cloud detection method based on the temporal anchor frame of the present invention.

[0052] Figure 4 It is a structural schematic diagram of the point cloud detection system based on the temporal anchor frame of the present invention.

[0053] Figure 5 It is a structural schematic diagram of a point cloud detection device based on a temporal anchor frame of the present invention.

[0054] Figure 6 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation methods, and the details in the present application can also be modified or changed in various ways according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0056] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in a variety of different forms and is not limited to the embodiments described herein.

[0057] In the representations of the present application, the representations with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials or characteristics represented in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the represented specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples represented in the present application and the features of the different embodiments or examples, unless they contradict each other.

[0058] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the representation of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0059] In order to clearly describe the present application, components irrelevant to the description are omitted, and the same reference symbols are given to the same or similar components throughout the specification.

[0060] Throughout the specification, when a device is said to be "connected" to another device, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a device is said to "include" a certain component, unless otherwise stated, it does not exclude other components, but means that other components may be included.

[0061] When a device is said to be "on" another device, it may be directly on the other device, but there may also be other devices between it. In contrast, when a device is said to be "directly" on another device, there are no other devices between it.

[0062] Although the terms first, second, etc. are used to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first interface and the second interface, etc. are represented. Moreover, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising" and "including" indicate the existence of features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Only when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way, will there be an exception to this definition.

[0063] The technical terms used herein are only used to refer to specific embodiments and are not intended to limit the present application. The singular form used herein also includes the plural form unless the sentence clearly indicates the contrary meaning. The meaning of "including" used in the specification is to specify specific characteristics, regions, integers, steps, operations, elements and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements and / or components.

[0064] Although not defined differently, all terms, including technical and scientific terms used herein, have the same meaning as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with the relevant technical literature and the contents of the current disclosure, and unless defined, shall not be overly interpreted as ideal or very formal meanings.

[0065] Figure 1 is a flow chart of the point cloud detection method based on the temporal anchor frame of the present invention. Figure 1 As shown, the point cloud detection method based on the temporal anchor frame of the present invention includes:

[0066] S110, collecting time-series point cloud data of different time-series frames and converting them into the point cloud coordinate system of the current time sequence.

[0067] S120, establishing a time step point cloud group according to each time sequence point cloud and the n consecutive time sequence point clouds before the time sequence point cloud, where n≥1, and performing point cloud fusion.

[0068] S130, after voxelizing and average-normalizing the point cloud of the current time series and the time-step point cloud group, the point cloud of the current time series is input into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and the time-step point cloud group is input into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix.

[0069] S140, generating proposed anchor frames according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step, wherein the attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame.

[0070] S150, align and weighted aggregate the proposed anchor frames corresponding to the m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix. The closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6.

[0071] S160: Input the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence.

[0072] S170, respectively adding the position information, size and heading angle of the anchor frame in the attribute sequence of the aggregated proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputting the target category and its confidence score from the neural network to obtain the detection result.

[0073] The present invention provides a Transformer model based on temporal proposed anchor boxes for LiDAR point cloud 3D target detection, which can use temporal prior knowledge to enhance detection performance. The general process is: first, sparse convolution is used to process point cloud voxels to generate downsampled BEV feature representations. Then, the proposed anchor boxes of the previous frame and the current frame are aggregated and encoded into a query vector, which is used to extract all candidate target features in the deformable cross-attention module. The proposed method can enhance the target features weakened by the sparse point cloud, thereby improving the overall detection performance.

[0074] In a preferred embodiment, step S110 includes:

[0075] S111, collecting time-series point cloud data and vehicle posture information of different time-series frames.

[0076] S112, converting the point cloud data of the previous time series into the current time series point cloud coordinate system of the point cloud of the current time series according to the external parameter matrix obtained from the vehicle posture information.

[0077] In a preferred embodiment, step S120 includes:

[0078] S121. According to the sequential order of the time series frames, a time step point cloud group is established based on each time series point cloud and the n consecutive time series point clouds before the time series point cloud, where n≥1.

[0079] S122, fusing the point clouds in each time step point cloud group, and adding a time series attribute of the corresponding time step point cloud group to each point in the point cloud.

[0080] The time step in this embodiment is a concept used in numerical simulation, which represents the discretization of the simulation system in time. In numerical simulation, the time range of the problem is divided into continuous small time intervals, each of which is called a time step. In each time step, the state of the system is updated according to physical laws and calculation methods. The size of the time step can be determined according to the stability and accuracy requirements of the simulation system. A smaller time step can improve the accuracy of the simulation, but it will increase the complexity of the calculation.

[0081] In a preferred embodiment, step S130 includes:

[0082] S131, voxelize and average normalize the point cloud of the current time series and the time step point cloud group respectively. Among them, the voxelization of the point cloud refers to the process of converting irregular point cloud data into a regular voxel grid. A voxel is a cubic element in a three-dimensional space, similar to a pixel in a two-dimensional image. Voxelization can simplify point cloud data processing, improve computational efficiency, and support various three-dimensional analyses and applications, which will not be repeated here. Average normalization is a common data preprocessing technique used to convert data into a uniform scale or range for better analysis and comparison. This method mainly relies on calculating the average value of the data and using the average value as a benchmark for data standardization. First, the feature of each voxel is subtracted from its average value and divided by the standard deviation or range of the feature so that the data has zero mean and unit variance or is scaled to a specific range. This processing method can eliminate the effects of different dimensions or data scale differences in the data set, making the data more comparable, which will not be repeated here.

[0083] S132: Input the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix.

[0084] S133, input the time step point cloud group into the sparse convolution backbone network to obtain the second bird's-eye view feature matrix, the first bird's-eye view feature matrix and the second bird's-eye view feature matrix are both plane matrices, and each point in the plane matrix has a one-dimensional tensor. Among them, sparse convolutional networks are a convolutional neural network specifically used to process sparse data (such as 3D point clouds). Sparse data refers to high-dimensional data in which most elements are zero or non-existent, such as LiDAR point clouds. Traditional convolution operations are not efficient for sparse data processing because they need to process a large number of zero values ​​in each convolution operation. Sparse convolutional networks only perform convolution operations on non-zero elements, thereby improving computational efficiency and memory usage efficiency. Efficient and configurable sparse convolution backbone networks are the key to achieving SOTA results on various 3D scene understanding tasks. In Google's TF 3D library, Google uses submanifold sparse convolution and pooling operations, both of which are designed to process 3D sparse data more efficiently. Sparse convolutional models are at the core of most SOTA methods used in outdoor autonomous driving (such as Waymo and NuScenes) and indoor benchmarks (such as ScanNet). Google also uses various CUDA techniques to accelerate computations (such as hashing algorithms, splitting / caching filters in shared memory, and bit operations). Experiments on the Waymo Open Competition dataset show that this implementation is about 20 times faster than the implementation that uses pre-existing TensorFlow operations. The 3D submanifold sparse U-Net architecture is used in the TF 3D library to extract features for each voxel. The U-Net architecture has been proven to be very effective by allowing the network to extract sparse and subtle features and combine them to make predictions. Structurally, the U-Net network contains three modules: encoder, bottleneck layer, and decoder, all of which are composed of a large number of sparse convolution blocks with potential pooling or non-pooling operations, which will not be described here.

[0085] Moreover, the bird's eye view feature, or BEV feature (Bird's Eye View feature), is an important concept in the field of autonomous driving and robot perception, especially when using cameras or LiDAR for 3D target detection. The BEV feature provides a mapping from 3D space to a 2D plane, making it possible to process and analyze 3D data on a 2D plane. This mapping helps to simplify calculations while retaining key spatial information, which is essential for achieving efficient 3D target detection and tracking. For example, the generation of BEV features usually involves the following steps:

[0086] (1) Image feature extraction: First, a 2D convolutional neural network (such as ResNet, Swin Transformer, etc.) is used to extract multi-scale features from the input multi-view image to obtain a multi-scale feature map.

[0087] (2) Viewpoint Conversion: Next, a viewpoint conversion module (such as LSS) is used to predict the discrete depth distribution of the input multi-view image features to obtain 3D view cone features. This step maps the 2D image features to 3D space by estimating the depth of each pixel position.

[0088] (3) BEV space feature extraction: The 3D cone features are “flattened” into the BEV space through a pooling operation to form BEV features. The BEV feature encoder performs multi-scale feature extraction on the features in the BEV space to obtain BEV features.

[0089] (4) 3D object detection: Finally, the BEV features are sent to the detection head for 3D object detection. The loss function used for classification supervision is Focal loss, and the loss function used for bounding box supervision is L1 loss, but not limited to this.

[0090] The advantages of BEV features include: Simplified calculation: By projecting 3D spatial data onto a 2D plane, the calculation process can be simplified and the processing speed can be increased. Retaining key information: During the projection process, key spatial information (such as the position and size of the target) is retained, which is essential for achieving accurate 3D target detection. Applicable to a variety of sensors: BEV features are suitable for 3D perception tasks using a variety of sensors such as cameras or lidars, and have broad application prospects, which will not be elaborated here.

[0091] In a preferred embodiment, in step S140, a preset detection head (the detection head includes a series of convolutional layers and deconvolutional layers for generating detection results) is used respectively according to the second bird's-eye view feature matrix corresponding to each time step point cloud group to generate a proposed anchor frame, and the attribute sequence of each proposed anchor frame is: (x, y, z, dx, dy, dz, r, s), where x, y, z are the spatial coordinates of the center of the proposed anchor frame in the temporal point cloud coordinate system. dx, dy, dz are the length, width, and height of the proposed anchor frame. r is the heading angle corresponding to the proposed anchor frame (the heading angle can be obtained by existing technology and will not be repeated here). s is the maximum value of all prediction scores corresponding to the proposed anchor frame. In this embodiment, there are 8 types of targets, including cars, truck heads, tire cranes, trucks, quay cranes, lock stations, pedestrians, and street lights, and s includes 8 confidence scores corresponding to the above 8 types. Among them, the anchor box technology: Anchor box (anchor box), also known as prior box (Prior Box), is a technology used to improve model accuracy in the field of target detection. The target detection task involves identifying objects in the image and locating their positions. Anchor boxes play a key role in this regard. In the target detection task, the model usually needs to predict the existence of an object and its bounding box (bounding box) for each location in the image. The role of the anchor box is to provide the model with a set of predefined boxes with different sizes and aspect ratios. The model makes predictions through these anchor boxes and then adjusts them according to the position and shape of the actual target. The following are some key concepts and functions of anchor boxes:

[0092] 1. Multiple scales and aspect ratios: Anchor boxes usually cover multiple scales and aspect ratios to accommodate objects of different sizes and shapes. This enables the model to better adapt to the characteristics of various objects.

[0093] 2. Benchmark of the predicted box: The anchor box serves as the benchmark for the model to predict the target box. The model outputs a rough box, which is then compared with the corresponding anchor box to adjust and correct the final bounding box.

[0094] 3. Position sensitivity: Anchor boxes enable the model to detect objects in different areas of the image, not just in fixed locations. This improves the model's ability to adapt to changes in object positions.

[0095] 4. Reduce the amount of calculation: Anchor boxes can reduce the number of bounding boxes that the model needs to predict, thereby reducing computational complexity. Compared with predicting a box at every position in the image, using anchor boxes can handle object detection tasks more efficiently.

[0096] Anchor box technology is widely used in deep learning-based object detection methods, such as Fast R-CNN. These methods improve the model's ability to detect objects in complex scenes through anchor boxes, making it more robust and generalizable. Anchor boxes allow neural networks to focus on images at different scales to determine the best area of ​​interest for an object, which will not be described in detail here.

[0097] In a preferred embodiment, step S150 includes:

[0098] S151. Align the proposed anchor frames corresponding to m temporally consecutive time step point cloud groups, 2≤M≤6.

[0099] S152. Aggregate the proposed anchor frames according to a preset weight sequence, where the preset weight sequence contains m weight elements. The closer the proposed anchor frame is to the current time sequence, the larger the value of the weight element corresponding to the proposed anchor frame is, and the farther the proposed anchor frame is from the current time sequence, the smaller the value of the weight element corresponding to the proposed anchor frame is. Multiply the weight element corresponding to the proposed anchor frame with the corresponding confidence score in the attribute sequence, and aggregate the proposed anchor frames corresponding to the m time step point cloud groups to form a proposed anchor frame matrix.

[0100] In a preferred embodiment, in step S150, m is 4, and the preset weight sequence is (1, 0.9, 0.8, 0.7).

[0101] In a preferred embodiment, step S160 includes:

[0102] S161. Input the aggregated proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition.

[0103] S162. Encode the matrix of the aggregated proposed anchor boxes into a query vector, fuse it with the preset learnable query vector, and then send it to the self-attention model for feature interaction. Among them, the self-attention mechanism (Self-Attention), also known as internal attention, is a mechanism that allows the model to directly establish relationships between different positions within the sequence. This technology has completely changed the model architecture in fields such as natural language processing (NLP) and will not be repeated here.

[0104] S163. Input the output of the self-attention model and the first bird's-eye view feature matrix into the deformable cross attention model to obtain the proposed anchor box residual sequence and the updated learnable query vector, and use the updated learnable query vector to replace the preset learnable query vector. Among them, the deformable cross attention module (Deformable Cross Attention Module) is a neural network structure, which is usually used to process data with spatial structure, such as image or video data. This module combines the ideas of attention mechanism and deformable convolution to better capture the spatial relationship and features in the input data, which will not be repeated here.

[0105] In a preferred embodiment, step S170 includes:

[0106] S171, respectively adding the position information, size and heading angle of the anchor frame in the attribute sequence of the aggregated proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame;

[0107] S172. The predicted label with the highest confidence score among the confidence scores output by the deformable cross-attention model and whose confidence score is greater than a preset threshold is used as the obstacle label of the proposed anchor box.

[0108] The point cloud detection method based on temporal anchor frames of the present invention can enhance target features weakened by sparse point clouds, thereby improving the overall detection accuracy, especially that of remote targets.

[0109] Figure 2 It is a schematic diagram of the operating principle of the point cloud detection method based on the temporal anchor frame of the present invention. Figure 3 Schematic diagram of the process of implementing the point cloud detection method based on the time sequence anchor frame of the present invention. Figure 2 and 3As shown, a specific embodiment of the present invention mainly performs obstacle recognition during vehicle driving in closed areas such as unmanned docks, and uses port operation vehicles to collect data. The vehicle is equipped with a 32-line laser radar for collecting point cloud data. In addition, an IMU and a GPS unit are equipped to record the posture information of the vehicle at each moment. (The 32-line laser radar in the present invention is a mature and inexpensive laser radar product, but after using the method of the present invention, accurate recognition of remote obstacles can be achieved based on this type of 32-line laser radar, which greatly improves the recognition accuracy under the premise of using ordinary hardware). The collected data is sampled at a frequency of 1Hz, and every 20 consecutive frames are placed in a folder. 3D target annotation is performed based on point cloud data, and there are 8 types of targets, including cars, truck heads, tire cranes, trucks, quay cranes, lock stations, pedestrians and street lights. The data set is divided into a training set and a test set, and the proportions of the two are 80% and 20% respectively (in folders). The training data set is used to supervise model learning, and the test data set is used to evaluate the performance of the model.

[0110] Collect the time-series point cloud data and vehicle posture information of different time-series frames. Convert the point cloud data of the previous time-series to the point cloud of the current time-series according to the external parameter matrix obtained from the vehicle posture information. In the current time-series point cloud coordinate system, the point cloud at the current moment is the point cloud 11 of the T time sequence, and the previous moments correspond to the point cloud 12 at the (T-1) moment, the point cloud 13 at the (T-2) moment, and so on.

[0111] According to the sequential order of the time series frames, a time step point cloud group is established based on each time series point cloud and the n consecutive time series point clouds before the time series point cloud, n ≥ 1, see Figure 2 , 3 In the first time step, the point cloud group 21 includes the point cloud 11 at time T and the point cloud 12 at time (T-1), and, Figure 3 The number of point clouds in each time step point cloud group is not necessarily 2, but is a variable. The value range of the number of point clouds can be [2, 4]. The second time step point cloud group 22 includes the point cloud 12 at time (T-1) and the point cloud 13 at time (T-2). The point clouds in each time step point cloud group are fused (a one-dimensional feature needs to be added to each point during the fusion process to represent its corresponding time step), and the time series attributes of the corresponding time step point cloud group are added to each point in the point cloud. The first time step point cloud group 21, the second time step point cloud group 22, etc. are collected to form a time step point cloud group set 2.

[0112] The point cloud of the current time series (point cloud 11 of the T time series) and each time step point cloud group (first time step point cloud group 21, second time step point cloud group 22, etc.) in the time step point cloud group set 2 are input into the sparse convolution backbone network model 3. Voxelization and average normalization are performed respectively, and then the point cloud of the current time series is input into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix 44. The time step point cloud group is input into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix. Both the first bird's-eye view feature matrix and the second bird's-eye view feature matrix are plane matrices, and each point in the plane matrix has a one-dimensional tensor.

[0113] Generate proposed anchor frames according to the second bird's-eye view feature matrix corresponding to each time step point cloud group. The attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame. Generate proposed anchor frames using the preset detection head according to the second bird's-eye view feature matrix corresponding to each time step point cloud group. The attribute sequence of each proposed anchor frame is: (x, y, z, dx, dy, dz, r, s), where x, y, z are the spatial coordinates of the center of the proposed anchor frame in the temporal point cloud coordinate system, dx, dy, dz are the length, width and height of the proposed anchor frame, r is the heading angle corresponding to the proposed anchor frame, and s is the maximum value of all prediction scores corresponding to the proposed anchor frame.

[0114] The proposed anchor frames corresponding to the four consecutive time step point cloud groups are aligned, and the proposed anchor frames are aggregated according to the preset weight sequence. The preset weight sequence contains four weight elements. The closer the proposed anchor frame is to the current time sequence, the larger the weight element corresponding to the proposed anchor frame is. The weight element corresponding to the proposed anchor frame is multiplied by the confidence score s in the attribute sequence, and the confidence scores of each proposed anchor frame 31, 32, etc. are weighted and aggregated. The preset weight sequence is (1, 0.9, 0.8, 0.7). For example, the corresponding confidence score in the weighted first time step point cloud group 21 is s1×1=s1. The corresponding confidence score in the weighted second time step point cloud group 22 is s2×0.9=0.9s2. The corresponding confidence score in the weighted third time step point cloud group (not shown in the figure) is s3×0.8=0.8s3. The corresponding confidence score in the weighted fourth time step point cloud group (not shown in the figure) is s4×0.7=0.7s4. The attribute sequence of the aggregated proposal anchor box matrix is: (X0, Y0, Z0, L0, W0, H0, R0, S0).

[0115] The aggregated proposed anchor box matrix and the first bird's-eye view feature matrix 44 are input into the neural network 4 based on anchor box recognition. The learnable query vector corresponding to the aggregated proposed anchor box matrix is ​​fused with the preset anchor box encoding query vector 46, and then sent to the self-attention model for feature interaction. The output of the self-attention model and the first bird's-eye view feature matrix 44 are input into the deformable cross attention model to obtain the proposed anchor box residual sequence 42 (equivalent to the first output of the deformable cross attention model) and the updated anchor box encoding query vector 47 (equivalent to the second output of the deformable cross attention model), and the updated anchor box encoding query vector 47 replaces the preset anchor box encoding query vector 46.

[0116] The position information, size and heading angle of the anchor frame in the attribute sequence 41 of the proposed anchor frame matrix are added to the corresponding elements in the proposed anchor frame residual sequence 42, and the attribute sequence of the added attribute sequence 43 is: (X1, Y1, Z1, L1, W1, H1, R1), and the confidence score output by the deformable cross-attention model (the confidence score is based on the output of the deformable cross-attention module, calculated by the fully connected network layer of the neural network, which is equivalent to the third output of the deformable cross-attention model, not shown in the figure) The predicted label with the highest confidence score and a confidence score greater than a preset threshold is used as the obstacle label 45 of the proposed anchor frame.

[0117] Therefore, the point cloud detection method based on the time sequence anchor frame of the present invention makes full use of point clouds of different time sequences for recognition, thereby enhancing the target features weakened by the sparse point clouds and improving the overall detection accuracy, especially for remote targets.

[0118] In the present invention, by encoding historical prior information into a query vector, a higher attention weight is assigned to the position where the target may exist in the BEV feature at the current time step, which helps to improve the overall detection effect. Its advantages are mainly reflected in the following aspects:

[0119] 1. Use query transformer to enhance the extraction and utilization of temporal information.

[0120] 2. Allocate attention to prior information according to temporal position to speed up model convergence and improve the overall detection effect.

[0121] 3. Encode the historical proposed anchor box information into a query vector, so that the model can focus on the potential target location and improve the detection performance.

[0122] Figure 4 Schematic diagram of the structure of the point cloud detection system based on the time sequence anchor frame of the present invention. Figure 4As shown, an embodiment of the present invention further provides a point cloud detection system based on a temporal anchor frame, which is used to implement the above-mentioned point cloud detection method based on a temporal anchor frame. The point cloud detection system based on a temporal anchor frame 5 includes:

[0123] The point cloud acquisition module 51 acquires the time-series point cloud data of different time-series frames and converts them into the point cloud coordinate system of the current time sequence.

[0124] The point cloud fusion module 52 establishes a time step point cloud group according to each time sequence point cloud and the n consecutive time sequence point clouds before the time sequence point cloud, where n≥1, and performs point cloud fusion.

[0125] The bird's-eye view feature module 53 performs voxelization and average normalization on the point cloud of the current time series and the time step point cloud group, respectively, and then inputs the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and inputs the time step point cloud group into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix.

[0126] The anchor frame proposal module 54 generates an anchor frame proposal according to the second bird's-eye view feature matrix corresponding to each time step point cloud group, and the attribute sequence of each anchor frame proposal includes the position, size, heading angle and obstacle label confidence of the anchor frame.

[0127] The anchor frame aggregation module 55 aligns and weighted aggregates the proposed anchor frames corresponding to the m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix. The closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6.

[0128] The anchor box residual module 56 inputs the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence.

[0129] The detection and recognition module 57 adds the position information, size and heading angle of the anchor frame in the attribute sequence of the aggregated proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputs the target category and its confidence score from the neural network to obtain the detection result. In a preferred embodiment, the point cloud acquisition module 51 is configured to collect time-series point cloud data and vehicle posture information of different time-series frames, and convert the point cloud data of the previous time-series to the current time-series point cloud coordinate system of the current time-series point cloud according to the external parameter matrix obtained from the vehicle posture information, but is not limited to this.

[0130] In a preferred embodiment, the point cloud fusion module 52 is configured to establish a time step point cloud group based on each time step point cloud and the n consecutive time step point clouds before the time step point cloud according to the sequential order of the time step frames, where n≥1. The point clouds in each time step point cloud group are fused, and the time step attributes of the corresponding time step point cloud group are added to each point in the point cloud, but the present invention is not limited thereto.

[0131] In a preferred embodiment, the bird's-eye view feature module 53 is configured to voxelize and average normalize the point cloud of the current time series and the time step point cloud group. The point cloud of the current time series is input into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix. The time step point cloud group is input into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix. The first bird's-eye view feature matrix and the second bird's-eye view feature matrix are both plane matrices, and each point in the plane matrix has a one-dimensional tensor, but is not limited thereto.

[0132] In a preferred embodiment, the proposed anchor frame module 54 is configured to use a preset detection head to generate a proposed anchor frame according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step, and the attribute sequence of each proposed anchor frame is: (x, y, z, dx, dy, dz, r, s), wherein x, y, z are the spatial coordinates of the center of the proposed anchor frame in the temporal point cloud coordinate system, dx, dy, dz are the length, width, and height of the proposed anchor frame, r is the heading angle corresponding to the proposed anchor frame, and s is the maximum value of all prediction scores corresponding to the proposed anchor frame, but is not limited to this.

[0133] In a preferred embodiment, the anchor frame aggregation module 55 is configured to align the proposed anchor frames corresponding to m time-series continuous time step point cloud groups, 2≤M≤6. The proposed anchor frames are aggregated according to a preset weight sequence, the preset weight sequence includes m weight elements, the closer the proposed anchor frame to the current time sequence, the larger the weight element corresponding to the proposed anchor frame is, the weight element corresponding to the proposed anchor frame is multiplied by the confidence score in the attribute sequence, and the proposed anchor frames corresponding to the m time step point cloud groups are aggregated to form a proposed anchor frame matrix, but not limited thereto.

[0134] In a preferred embodiment, in the anchor frame aggregation module 55 , m is 4, and the preset weight sequence is (1, 0.9, 0.8, 0.7).

[0135] In a preferred embodiment, the anchor box residual module 56 is configured to input the aggregated proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition. The matrix of the aggregated proposed anchor box is encoded as a query vector, fused with the preset learnable query vector, and then sent to the self-attention model for feature interaction. The output of the self-attention model and the first bird's-eye view feature matrix are input into the deformable cross attention model to obtain the proposed anchor box residual sequence and the updated anchor box encoded query vector, obtain the proposed anchor box residual sequence and the updated learnable query vector, and use the updated learnable query vector to replace the preset learnable query vector, but not limited to this.

[0136] In a preferred embodiment, the detection and identification module 57 is configured to respectively add the position information, size and heading angle of the anchor frame in the attribute sequence of the aggregated proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and take the predicted label with the highest confidence score among the confidence scores output by the deformable cross-attention model and the confidence score greater than a preset threshold as the obstacle label of the proposed anchor frame, but not limited to this.

[0137] The point cloud detection system based on temporal anchor frames of the present invention can enhance target features weakened by point cloud sparseness, thereby improving the overall detection accuracy, especially that of remote targets.

[0138] The embodiment of the present invention further provides a point cloud detection device based on a temporal anchor frame, comprising a processor, a memory, in which executable instructions of the processor are stored, wherein the processor is configured to execute the steps of the point cloud detection method based on a temporal anchor frame by executing the executable instructions.

[0139] As shown above, the point cloud detection device based on the temporal anchor frame of this embodiment of the present invention can enhance the target features weakened by the sparse point cloud, thereby improving the overall detection accuracy, especially the detection accuracy of remote targets.

[0140] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0141] Figure 5 Schematic diagram of the structure of the point cloud detection device based on the time sequence anchor frame of the present invention. Figure 5 The electronic device 600 according to this embodiment of the present invention is described. Figure 5 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 5 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0143] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0144] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0145] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0146] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0147] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0148] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the point cloud detection method based on the temporal anchor box are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above method section of this specification.

[0149] As shown above, the point cloud detection system based on the temporal anchor frame of this embodiment of the present invention can enhance the target features weakened due to the sparse point cloud, thereby improving the overall detection accuracy, especially the detection accuracy of remote targets.

[0150] Figure 6 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 6 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0151] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0152] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0153] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0154] In summary, the purpose of the present invention is to provide a point cloud detection method, system, device and storage medium based on a temporal anchor frame, which can enhance the target features weakened by the sparse point cloud, thereby improving the overall detection accuracy, especially the detection accuracy of remote targets.

[0155] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A point cloud detection method based on temporal anchor frames, characterized in that: The following steps are involved: S110, collecting time-series point cloud data of different time-series frames and converting them into the point cloud coordinate system of the current time sequence; S120, establishing a time step point cloud group according to each time sequence point cloud and the n consecutive time sequence point clouds before the time sequence point cloud, where n≥1, and performing point cloud fusion; S130, after voxelizing and averaging the point cloud of the current time series and the time step point cloud group, the point cloud of the current time series is input into a sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and the time step point cloud group is input into a sparse convolution backbone network to obtain a second bird's-eye view feature matrix; S140, generating a proposed anchor frame according to the second bird's-eye view feature matrix corresponding to each time step point cloud group, wherein the attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame; S150, aligning and weighted aggregation of the proposed anchor frames corresponding to the m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix, wherein the closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6; S160, inputting the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence; S170, respectively adding the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputting the target category and its confidence score from the neural network to obtain a detection result.

2. The point cloud detection method based on temporal anchor frames according to claim 1, characterized in that: The step S110 includes: S111, collecting time-series point cloud data and vehicle posture information of different time-series frames; S112, converting the point cloud data of the previous time series into the current time series point cloud coordinate system of the point cloud of the current time series according to the external parameter matrix obtained from the vehicle posture information.

3. The point cloud detection method based on temporal anchor frames according to claim 1, characterized in that: The step S120 includes: S121, according to the sequential order of the time series frames, a time step point cloud group is established based on each time series point cloud and the n consecutive time series point clouds before the time series point cloud, where n≥1; S122, fusing the point clouds in each of the time step point cloud groups, and adding a time series attribute corresponding to the time step point cloud group to each point in the point cloud.

4. The point cloud detection method based on temporal anchor frames according to claim 1, characterized in that: The step S130 includes: S131, voxelizing and averaging the point cloud of the current time series and the point cloud group of the time step respectively; S132, inputting the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix; S133. Input the time step point cloud group into a sparse convolution backbone network to obtain a second bird's-eye view feature matrix, wherein the first bird's-eye view feature matrix and the second bird's-eye view feature matrix are both plane matrices, and each point in the plane matrix has a one-dimensional tensor.

5. The point cloud detection method based on temporal anchor frames according to claim 2, characterized in that: In the step S140, a preset detection head is used respectively according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step to generate a proposed anchor frame, and the attribute sequence of each proposed anchor frame is: (x, y, z, dx, dy, dz, r, s), wherein x, y, z are the spatial coordinates of the center of the proposed anchor frame in the temporal point cloud coordinate system, dx, dy, dz are the length, width, and height of the proposed anchor frame, r is the heading angle corresponding to the proposed anchor frame, and s is the maximum value of all prediction scores corresponding to the proposed anchor frame.

6. The point cloud detection method based on temporal anchor frames according to claim 5, characterized in that: The step S150 includes: S151, aligning the proposed anchor frames corresponding to m temporally consecutive time step point cloud groups, 2≤M≤6; S152. Aggregate the proposed anchor frames according to a preset weight sequence, wherein the preset weight sequence includes m weight elements, and the weight element corresponding to the proposed anchor frame closer to the current time sequence is larger. Multiply the weight element corresponding to the proposed anchor frame by the confidence score in the attribute sequence, and aggregate the proposed anchor frames corresponding to the m time step point cloud groups to form a proposed anchor frame matrix.

7. The point cloud detection method based on temporal anchor frames according to claim 6, characterized in that: In the step S150, m is 4, and the preset weight sequence is (1, 0.9, 0.8, 0.7).

8. The point cloud detection method based on temporal anchor frames according to claim 1, characterized in that: The step S160 includes: S161, inputting the aggregated proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition; S162, encoding the aggregated matrix of the proposed anchor boxes into a query vector, fusing it with a preset learnable query vector, and then sending it to a self-attention model for feature interaction; S163. Input the output of the self-attention model and the first bird's-eye view feature matrix into a deformable cross-attention model to obtain the proposed anchor box residual sequence and the updated learnable query vector, and use the updated learnable query vector to replace the preset learnable query vector.

9. The point cloud detection method based on temporal anchor frames according to claim 8, characterized in that: The step S170 includes: S171, respectively adding the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame; S172. The predicted label with the highest confidence score among the confidence scores output by the deformable cross-attention model and whose confidence score is greater than a preset threshold is used as the obstacle label of the proposed anchor box.

10. A point cloud detection system based on a temporal anchor frame, used to implement the point cloud detection method based on a temporal anchor frame according to claim 1, characterized in that: include: Point cloud acquisition module, which collects time-series point cloud data of different time frames and converts them into the point cloud coordinate system of the current time sequence; The point cloud fusion module establishes a time step point cloud group according to each time sequence point cloud and the n consecutive time sequence point clouds before the time sequence point cloud, where n ≥ 1, and performs point cloud fusion; The bird's-eye view feature module voxelizes and averages the point cloud of the current time series and the time step point cloud group, respectively, and then inputs the point cloud of the current time series into the sparse convolution backbone network to obtain a first bird's-eye view feature matrix, and inputs the time step point cloud group into the sparse convolution backbone network to obtain a second bird's-eye view feature matrix; The proposed anchor frame module generates a proposed anchor frame according to the second bird's-eye view feature matrix corresponding to the point cloud group at each time step, and the attribute sequence of each proposed anchor frame includes the position, size, heading angle and obstacle label confidence of the anchor frame; Anchor frame aggregation module aligns and weighted aggregates the proposed anchor frames corresponding to m time-series continuous time step point cloud groups to obtain a proposed anchor frame matrix. The closer the proposed anchor frame is to the current time sequence, the greater the corresponding weight is, 2≤M≤6; An anchor box residual module, inputting the proposed anchor box matrix and the first bird's-eye view feature matrix into a neural network based on anchor box recognition to obtain a proposed anchor box residual sequence; The detection and recognition module adds the position information, size and heading angle of the anchor frame in the aggregated attribute sequence of the proposed anchor frame matrix to the corresponding elements in the residual sequence of the proposed anchor frame, and outputs the target category and its confidence score from the neural network to obtain the detection result.

11. A point cloud detection device based on a temporal anchor frame, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the point cloud detection method based on temporal anchor frames according to any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the point cloud detection method based on temporal anchor frames described in any one of claims 1 to 9 are implemented.

Images