Point cloud target detection method and system based on self-distillation

Abstract

The invention provides a point cloud target detection method and system based on self-distillation, and relates to the technical field of point cloud target detection, and the method comprises the steps: obtaining a historical point cloud of a to-be-detected target; performing sparse processing and shielding processing on the historical point clouds to generate distorted point clouds with different distortion degrees; a target detection model is constructed, the target detection model comprises a first branch and a second branch which are parallel, and each of the first branch and the second branch comprises a preprocessing module, a global information fusion module, a foreground prior module and a detection head module; the distorted point cloud and the historical point cloud serve as a training set, a consistency loss function is combined, a target detection model is trained through a first distortion degree and a second distortion degree, and the first distortion degree is smaller than the second distortion degree; acquiring a real-time point cloud of a to-be-detected target; and inputting the real-time point cloud into the trained target detection model, and outputting a target detection result of the to-be-detected target.

Description

Technical Field

[0001] The present invention relates to the technical field of point cloud object detection, and particularly to a point cloud object detection method and system based on self-distillation. Background Art

[0002] Self-distillation refers to, when training a model, using the model itself as a "teacher" and learning by comparing with the predictions generated by itself. Point cloud object detection refers to the process of identifying and locating objects from these point cloud data, where the objects can be static or dynamic. And the point cloud object detection method based on self-distillation means that the model self-learns and is trained using distorted data (such as occluded or sparse data), avoiding over-reliance on traditional artificial labels or external teacher models, thereby enhancing the robustness of the model.

[0003] By performing data distortion operations on the original point cloud, data with higher perception difficulty is constructed, which is input into the same network together with the original data to build a self-distillation architecture to improve the model's recognition ability for occluded and sparse point clouds, reducing the dependence on external resources, which is of great significance for autonomous driving or other 3D perception applications, thus making the system more flexible and scalable and suitable for object detection tasks in complex environments and real scenarios.

[0004] However, most of the existing point cloud object detection algorithms do not provide effective training strategies for occluded and sparse point clouds, and do not notice the importance of global information for the network to distinguish noise and foreground information, resulting in a decrease in detection accuracy. They can extract sufficient local features for objects with less severe occlusion nearby to provide relatively accurate results. However, when dealing with complex scenes, the spatial constraint relationships between different objects are very important for the judgment of results. Due to the lack of global attention to foreground information, the features of some objects extracted by the current method are confused with background features, limiting the detection performance in the PCOD task. Summary of the Invention

[0005] In order to solve the technical problems that most of the existing point cloud object detection algorithms do not provide effective training strategies for occluded and sparse point clouds, and do not notice the importance of global information for the network to distinguish noise and foreground information, resulting in a decrease in detection accuracy, and the features of some objects extracted are confused with background features, limiting the detection performance in the PCOD task, the present invention provides a point cloud object detection method and system based on self-distillation.

[0006] The technical solutions provided by the embodiments of the present invention are as follows:

[0007] First aspect:

[0008] A point cloud object detection method based on self-distillation provided by an embodiment of the present invention includes:

[0009] S1: Obtain the historical point cloud of the target to be detected;

[0010] S2: Perform sparsification processing and occlusion processing on the historical point cloud to generate distorted point clouds with different degrees of distortion;

[0011] S3: Construct a target detection model, where the target detection model includes a parallel first branch and a second branch, and both the first branch and the second branch include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module;

[0012] S4: Use the distorted point cloud and the historical point cloud as the training set, and combine the consistency loss function to train the target detection model with the first distortion degree and the second distortion degree respectively, where the first distortion degree is less than the second distortion degree;

[0013] S5: Obtain the real-time point cloud of the target to be detected;

[0014] S6: Input the real-time point cloud into the trained target detection model and output the target detection result of the target to be detected.

[0015] Second aspect:

[0016] A point cloud target detection system based on self-distillation provided by an embodiment of the present invention includes:

[0017] A processor;

[0018] A memory, on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor, the method for point cloud target detection based on self-distillation as described in the first aspect is implemented.

[0019] Third aspect:

[0020] A computer-readable storage medium provided by an embodiment of the present invention, on which a computer program is stored. When the program is executed by the processor, the method for point cloud target detection based on self-distillation as described in the first aspect is implemented.

[0021] The beneficial effects brought by the technical solutions provided by the embodiments of the present invention at least include:

[0022] In an embodiment of the present invention, by acquiring the historical point cloud of the target to be detected, sparsifying and occluding the historical point cloud to generate distorted point clouds with different degrees of distortion, and then constructing a target detection model. The target detection model includes parallel first and second branches, where both the first and second branches include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module. The target detection model can better learn the impact of distorted data on target detection, and use the distorted point cloud and the historical point cloud as the training set. Combining with the consistency loss function, the target detection model is trained with the first distortion degree and the second distortion degree respectively. Finally, the real-time point cloud of the target to be detected is acquired, and the real-time point cloud is input into the trained target detection model to output the target detection result of the target to be detected, thereby improving the detection accuracy, increasing the generalization ability of the model, and enhancing the recognition ability of the model for occlusions and sparse point clouds. Description of the Drawings

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0024] Figure 1 It is a flowchart of a point cloud target detection method based on self-distillation provided by an embodiment of the present invention;

[0025] Figure 2 It is a structural diagram of a target detection model provided by an embodiment of the present invention;

[0026] Figure 3 It is a structural diagram of a point cloud target detection system based on self-distillation provided by an embodiment of the present invention. Detailed Embodiments

[0027] The following describes the technical solutions in the present invention with reference to the drawings.

[0028] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly, the use of the word "example" is intended to present concepts in a specific manner. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0029] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they convey are the same. "Of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they convey are the same.

[0030] In the embodiments of the present invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meanings they convey are the same.

[0031] To make the technical problems, technical solutions, and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0032] Refer to the attached Figure 1 , which shows a schematic flowchart of a point cloud object detection method based on self-distillation provided by an embodiment of the present invention.

[0033] The embodiments of the present invention provide a point cloud object detection method based on self-distillation. This method can be implemented by a point cloud object detection device based on self-distillation. The point cloud object detection device based on self-distillation can be a terminal or a server. The processing flow of the point cloud object detection method based on self-distillation may include the following steps:

[0034] S1: Obtain the historical point cloud of the target to be detected.

[0035] Among them, the target to be detected refers to the object that needs to be recognized and located in the system, and the historical point cloud refers to the point cloud data obtained by the system at a certain moment in the past, which is usually used for training, backtracking, or reference.

[0036] Specifically, the target to be detected is specifically common targets such as vehicles, pedestrians, and cyclists in the autonomous driving scenario.

[0037] It should be noted that, first, the historical point cloud data provides rich environmental information and reference features for the model, which can improve the model's recognition ability of different object shapes, positions, and relative relationships. Second, the historical data, as a background reference, can provide more stable context information for real-time detection, enabling the model to cope with dynamic changes in the real-time environment and improving robustness and accuracy.

[0038] S2: Perform sparse processing and occlusion processing on the historical point cloud to generate distorted point clouds with different degrees of distortion.

[0039] Among them, the sparsification process is a data processing technique that simulates the incomplete point cloud data in the actual environment due to long distances, object occlusion, or sensor limitations by reducing the number of points in the point cloud data or selectively removing some data points.

[0040] Among them, the occlusion processing is a process of compensating for the situation where some information is lost or distorted in the point cloud data due to object occlusion. By simulating the object being occluded, a distorted point cloud is generated to help the network consider these occlusion problems in the real world during training.

[0041] Among them, the distorted point cloud is the point cloud data generated after operations such as sparsification and occlusion on the historical point cloud. Different degrees of distortion mean that the generated distorted point clouds have different data qualities. For example, some point clouds may lose more information or simulate more occlusion situations.

[0042] It should be noted that by performing sparsification and occlusion processing on the historical point cloud data, the incomplete data and data loss in the real environment are simulated, and point clouds with different degrees of distortion are generated. This enables the model to be exposed to data with different degrees of distortion, learn how to handle incomplete or occluded input data, and improve its robustness in complex environments. Secondly, the model learns how to extract effective features from imperfect data in this way, so that it can still maintain high accuracy and stability when processing real-world noise, occlusion, and sparse point clouds.

[0043] Refer to the attached Figure 2 description, which shows a schematic structural diagram of an object detection model provided by an embodiment of the present invention.

[0044] Figure 2Among them, the original point cloud data refers to the unprocessed three-dimensional spatial data collected from a Light Detection and Ranging (LiDAR) or other sensors. By performing sparsification processing and occlusion processing on the original point cloud, distorted point cloud data is generated. Parameter sharing means that the convolutional kernel applies the same weight parameters at different positions of the input. This enables the model to share the parameters of the convolutional kernel at different positions without having to independently learn new parameters for each position, thus greatly reducing the number of parameters in the model. 2D representation means representing the three-dimensional point cloud data in a two-dimensional plane through a certain method. Common methods include the Bird's Eye View (BEV) and projection. The purpose of the global information fusion module is to extract global information, that is, context information, by performing operations such as convolution and dilated convolution on the feature map to help the network understand the spatial relationships between objects and the environmental structure. The foreground prior module is used to predict the foreground region, that is, where target objects may exist in the image. This module uses a foreground prediction network to extract foreground information and generate a foreground prediction map. The region extraction head is a module in the object detection network, mainly used to generate region candidate boxes. The consistency loss is used to ensure the output consistency of the model on different input data (original and distorted), aiming to reduce the impact of data distortion on the model performance and maintain the stability of the model in various situations.

[0045] Point cloud voxel features are features generated by dividing the point cloud data into voxels (small three-dimensional cube units). Each voxel contains information about a certain area in the point cloud (such as position and reflection intensity). Shallow Bird's Eye View (BEV) features are image features extracted by projecting the point cloud data onto a two-dimensional plane (Bird's Eye View, BEV). Deep Bird's Eye View features are features obtained after multiple layers of processing and learning. Compared with shallow features, they contain more high-order information, such as the specific shape of objects, background information, and the relationships between objects. Region candidate features generate some candidate regions that may contain targets through a region candidate network. These regions are used for subsequent object classification and localization. Foreground prediction features are features output from the foreground prior module, showing the probability of each position in the image being a foreground region.

[0046] The global information fusion module extracts the first feature and the second feature of the shallow bird's-eye view features through a convolutional layer and a dilated convolutional layer respectively. Then, the first feature and the second feature are added together to obtain a feature value. Combining with the linear attention module, through two pointwise convolutional layers, the shallow bird's-eye view features processed by the global information fusion module are deformed to generate a query feature matrix and a key feature matrix. Through a fully connected layer, the query feature matrix and the key feature matrix are processed respectively, and a matrix multiplication operation is performed on the processed query feature matrix and key feature matrix to obtain an attention map. According to the attention map, an output attention map is generated through a pointwise convolution operation and a normalization operation. The feature value and the output attention map are multiplied matrix-wise to generate the global information output, and an addition operation is performed on the global information output, the feature value, and the shallow bird's-eye view features to generate the final output feature map.

[0047] The foreground prior module extracts the foreground prediction information of the shallow bird's-eye view features to generate a foreground prediction map. According to the ground truth box, a Gaussian foreground target map is generated. The foreground prediction map is normalized through a sigmoid function. According to the normalization result, combined with a consistency loss function, the loss value between the foreground prediction map and the Gaussian foreground target map is calculated. According to the loss value, the foreground prediction feature map is determined. The foreground prediction feature map and the final output feature map are concatenated and fused to generate the deep bird's-eye view features.

[0048] It should be noted that by training with distorted point cloud data (such as occlusions, sparse data), the model can still maintain stability and high accuracy when processing incomplete or imperfect input data. The global information fusion module is used to fuse local information with global information, adding a linear attention module on the basis of convolutional operations, enriching features by means of residual connections, and enhancing the spatial relationship of objects. The foreground prior module can generate foreground targets from the ground truth box. The foreground prediction module is added to output foreground predictions in the bird's-eye view by category to fit the foreground targets, and this foreground information is fused with the original features to obtain the foreground prior, effectively enhancing foreground retention.

[0049] S3: Construct a target detection model, where the target detection model includes a parallel first branch and a second branch. The first branch and the second branch both include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module.

[0050] Among them, the target detection model is a deep learning model designed to identify and locate specific targets (such as objects, people, vehicles, etc.) from input data. The first branch and the second branch refer to two independent but simultaneously running paths in the model.

[0051] Among them, the preprocessing module is a module for processing distorted point clouds and historical point clouds. The global information fusion module is an integral part of the object detection network, responsible for fusing the input features, extracting global context information, so as to better understand and analyze the spatial relationships and background information of objects.

[0052] Among them, the foreground prior module is used to predict the foreground information of the target object (for example, the probability of the target object appearing at a certain position). The detection head module is the last part of the model, used for classifying and regressing the target based on the processed feature map, that is, predicting the category (such as pedestrian, vehicle, cyclist) and position (such as bounding box coordinates) of each target.

[0053] It should be noted that, first, parallel processing enables the model to handle data of different qualities simultaneously, learn how to perform object detection under data with different distortion degrees, and improves the model's adaptability to incomplete and occluded data. Second, each branch of the model includes a global information fusion module, a foreground prior module, and a detection head module. These modules combine multi-scale information and foreground prediction, thus helping the model to perform object classification and localization more accurately.

[0054] In a possible implementation manner, the processing process of the preprocessing module specifically includes:

[0055] Through structured processing, voxels of the historical point cloud and the distorted point cloud are extracted respectively.

[0056] Among them, a voxel is a small cube in three-dimensional space, similar to a pixel in a two-dimensional image.

[0057] According to the size of the voxels, the three-dimensional space region is divided to determine point features.

[0058] The point features are averaged to generate voxel features.

[0059] According to the voxel features, combined with multi-layer sparse convolution downsampling and splicing techniques, shallow bird's-eye view features are generated.

[0060] Among them, voxel features are the feature representations of the points within each voxel in the point cloud, usually obtained by average pooling or other aggregation methods for the points within the voxel. Multi-layer sparse convolution downsampling refers to reducing the size or resolution of the data in a certain way. The splicing technique refers to connecting different feature maps in the channel dimension or the spatial dimension.

[0061] Among them, shallow bird's-eye view features are the features extracted in the BEV space after operations such as voxelization, convolution, and downsampling of the point cloud data. It is mainly used to describe the basic shape and position of objects, facilitating subsequent object detection.

[0062] In a possible implementation, the process of the global information fusion module processing the shallow bird's-eye view features specifically includes:

[0063] S401: Respectively, through a convolutional layer and a dilated convolutional layer, extract the first feature and the second feature of the shallow bird's-eye view features.

[0064] Among them, the convolutional layer is a neural network layer that performs a convolution operation on the input data through a convolution kernel (filter). The dilated convolutional layer is a technique for expanding the convolution kernel. By introducing holes (i.e., unconnected units) between the convolution kernels, the convolution operation can cover a larger receptive field and capture more context information.

[0065] S402: Add the first feature and the second feature to obtain a feature value:

[0066] v = Conv(I) + DConv(I)

[0067] Among them, v represents the feature value, Conv represents the ordinary convolution operation, I represents the shallow bird's-eye view features, and DConv represents the dilated convolution operation.

[0068] Specifically, the feature value refers to the fused features obtained through the convolution and dilated convolution operations. These feature values are the result of adding the first feature and the second feature of the shallow bird's-eye view features, representing the comprehensive information extracted from multiple features.

[0069] S403: In combination with the linear attention module, through two pointwise convolutional layers, deform the shallow bird's-eye view features processed by the global information fusion module to generate a query feature matrix and a key feature matrix.

[0070] Among them, the linear attention module calculates the similarity through the query and the key, thereby generating attention weights. The query feature matrix is one of the inputs in the linear attention module, representing the part that the network hopes to focus on in the current input features. The key feature matrix is one of the inputs paired with the query matrix, used to represent the storage or label of the features. The network calculates the attention value by comparing the similarity between the query and the key.

[0071] S404: Through a fully connected layer, process the query feature matrix and the key feature matrix respectively.

[0072] Among them, the fully connected layer is a neural network layer where each input unit is connected to each output unit. The fully connected layer is usually used for feature transformation, mapping the input features to a new feature space.

[0073] S405: Perform a matrix multiplication operation on the processed query feature matrix and key feature matrix to obtain an attention map.

[0074] Among them, the attention map is calculated through the attention mechanism and represents the importance of different regions in the input features. The value at each position indicates the degree of attention of that position to the model task.

[0075] S406: According to the attention map, through pointwise convolution operation and normalization operation, generate an output attention map:

[0076]

[0077] Among them, q represents the query feature matrix, k represents the key feature matrix, FC represents the fully connected layer, reshape represents the shape rearrangement operation, PW represents two pointwise convolution operations, attention represents the output attention map, and sotfmax represents the softmax function.

[0078] Specifically, the output attention map is obtained by performing pointwise convolution and normalization (such as softmax) on the attention map, and represents the weights that the network should assign to each region.

[0079] S407: Multiply the feature values and the output attention map matrix to generate a global information output.

[0080] Among them, the global information output contains weighted features of the global information. After passing through the attention mechanism, the network can focus on more important regions.

[0081] S408: Perform an addition operation on the global information output, the feature values, and the shallow bird's-eye view features to generate a final output feature map.

[0082] Among them, the final output feature map is obtained by performing an addition operation on the global information output, the feature values, and the shallow bird's-eye view features. These feature maps will be used for subsequent object detection tasks.

[0083] It should be noted that the main function of the global information fusion module is to process the shallow bird's-eye view features, extract context information and local features. In object detection tasks, deep learning models usually require multi-level features to improve detection accuracy. The task of this module is to extract features through convolution and dilated convolution.

[0084] In a possible implementation manner, the processing process of the foreground prior module specifically includes:

[0085] Extract the foreground prediction information of the shallow bird's-eye view features to generate a foreground prediction map.

[0086] Among them, the foreground prediction map is an image generated by the foreground prediction module, which usually contains the probability of the presence of an object in the image.

[0087] Generate a Gaussian foreground object map according to the ground truth box.

[0088] Among them, the ground truth box is a rectangular box used to mark the true position and size of the target object, usually generated by manual annotation or automatic annotation tools. The Gaussian foreground object map is a map generated based on the ground truth box, which is used to represent the probability distribution of the existence of the target object.

[0089] Normalize the foreground prediction map through the sigmoid function.

[0090] Among them, the sigmoid function is a commonly used activation function, and its output value is between 0 and 1, usually used for probability mapping.

[0091] According to the normalization result, combined with the consistency loss function, calculate the loss value between the foreground prediction map and the Gaussian foreground object map.

[0092] Among them, the consistency loss function is a loss function used to measure the difference between the model output and the target (ground truth). It is used to calculate the difference between the foreground prediction map and the Gaussian foreground object map, so as to optimize the model and make the prediction result closer to the real target.

[0093] Determine the foreground prediction feature map according to the loss value.

[0094] Among them, the foreground prediction feature map is a feature map generated after calculating the loss value of the foreground prediction map, which contains the prediction results of the model for the foreground area.

[0095] Stitch and fuse the foreground prediction feature map and the final output feature map to generate the deep bird's-eye view feature.

[0096] Among them, the deep bird's-eye view feature is the final feature map processed through multiple modules (such as global information fusion, foreground prior, etc.) starting from the shallow bird's-eye view feature.

[0097] It should be noted that the goal of the foreground prior module is to use the prior information of the target (such as category, position, size, etc.) to predict the foreground area of the object. It usually calibrates the position of the object in the image by generating a foreground prediction map in the model.

[0098] In a possible implementation manner, the generation process of the foreground prediction map specifically includes:

[0099] According to the foreground prediction information, combine the transposed convolution operation and the upsampling operation to generate a multi-scale foreground feature map.

[0100] Among them, the transposed convolution operation (also called upsampling convolution) is an operation used to restore the image resolution. The upsampling operation is the process of increasing the size of the low-resolution feature map to a higher resolution. The multi-scale foreground feature map is a set of feature maps extracted by processing input data of different scales (resolutions).

[0101] Sum the multi-scale foreground feature maps.

[0102] Based on the summed multi-scale foreground feature map, a convolution operation is performed to generate a foreground prediction map with the same number of categories and channels as the shallow bird's-eye view feature.

[0103] It should be noted that deconvolution and upsampling operations generate a multi-scale foreground feature map, which is then combined with convolution operations to generate a foreground prediction map with the same number of categories and channels. The purpose of these operations is to help the model better locate and classify target objects by fusing multi-scale features with foreground information, especially when dealing with complex environments (such as occlusion or sparse data), which can improve detection accuracy and robustness.

[0104] In a possible implementation, the calculation formula of the sigmoid function is specifically:

[0105]

[0106] Where S represents the sigmoid function, m represents the input value, and e represents the base of the natural logarithm.

[0107] In a possible implementation, the consistency loss function is specifically:

[0108]

[0109] Among them, L cons represents the consistency loss function, Represents the loss of the original point cloud in the region candidate and classification stages, Represents the loss of distorted point cloud in the region candidate and classification stages, Represents the loss of the original point cloud in the regression stage, Represents the loss of the distorted point cloud in the regression stage.

[0110] S4: Using the distorted point cloud and the historical point cloud as training sets, combined with the consistency loss function, the target detection model is trained with the first distortion degree and the second distortion degree respectively, wherein the first distortion degree is smaller than the second distortion degree.

[0111] The training set is the dataset used to train the model, consisting of both the original point cloud and the distorted point cloud. The consistency loss function measures the difference between the model output and the target. It is commonly used in tasks such as self-distillation to ensure that the model's output remains consistent when presented with data of varying quality (e.g., original data versus distorted data).

[0112] Among them, the first distortion degree and the second distortion degree refer to the degree of change in the quality of the point cloud data after operations such as sparsification or occlusion processing. The first distortion degree usually represents a relatively light distortion, and the second distortion degree represents a relatively severe distortion. Data with different distortion degrees can be used to train the model to ensure that the model can adapt to data inputs of various qualities.

[0113] It should be noted that by using the original point cloud and the distorted point cloud as the training set and combining with the consistency loss function, the model can maintain robustness and stability when facing data of different qualities. By training with data of different distortion degrees, the model can adapt to diverse inputs in the real world (such as occlusion, sparsity, etc.) and reduce the performance degradation caused by data quality fluctuations. This training method enables the network to effectively learn on data of different qualities, ensuring high accuracy and robustness in practical applications, especially in scenarios where the data is imperfect.

[0114] S5: Obtain the real-time point cloud of the target to be detected.

[0115] Among them, the real-time point cloud refers to the real-time data obtained by the system through a lidar or other sensors in the actual environment.

[0116] S6: Input the real-time point cloud into the trained target detection model and output the target detection result of the target to be detected.

[0117] Among them, the trained target detection model means that the model has learned how to accurately identify and locate target objects from point cloud data and optimized its adaptability to data of different qualities through the consistency loss function.

[0118] Among them, the target detection result is the result output by the model, usually including the recognized object category (such as pedestrians, vehicles, etc.) and the position of the object in the image or point cloud (usually represented by a bounding box).

[0119] It should be noted that, first of all, by inputting the real-time point cloud, the system can quickly identify and locate target objects in a dynamic environment, thus achieving instant perception. Secondly, since the model has been self-distilled trained on diverse data (including distorted data), it has strong robustness and adaptability and can still perform accurate detection when facing incomplete or occluded point cloud data. Finally, by using the real-time point cloud input, the system can quickly respond to changes in the surrounding environment, meet the requirements of high real-time and high-precision for tasks such as autonomous driving, and enhance safety and reliability.

[0120] In a possible implementation manner, the detection head is used to output the detection result of the target to be detected.

[0121] In the present invention, the global information fusion module aims to alleviate the lack of global information extraction ability in traditional convolutional modules, which can only obtain local information, easily leading to sensitivity to noise and affecting the detection performance. Through feature transformation of the bird's-eye view features using two convolutional modules, and at the same time using two dilated convolutional modules to obtain different receptive fields, and taking [specific values] as values to obtain more context. In order to obtain global information without increasing a large amount of computation, the attention weight is obtained by using a linear attention module. The specific operation is to first split the input bird's-eye view into two branches, use pointwise convolution to reduce the feature dimension to 1, then scale it to a 1D vector, and then obtain vectors with lengths equal to the length and width of the bird's-eye view through fully connected layers respectively. And perform dot product recovery to obtain the same size as the bird's-eye view, and then restore the number of channels through pointwise convolution, and normalize in the channel direction to obtain the attention weight, which is multiplied by the previously obtained value to obtain a feature map containing global information. Residual connection is performed on the features to increase the modeling ability of complex spatial relationships and enhance the scene-level semantic consistency. Figure 1 Connect to increase the modeling ability of complex spatial relationships and enhance the scene-level semantic consistency.

[0122] In the present invention, the foreground prior module adds a foreground prediction network. Under the bird's-eye view, according to the supervision information generated by the ground truth box, more object position and size information are obtained, and it is applied to subsequent 3D object detection. It includes a foreground generation and prediction module and a multi-source feature fusion module.

[0123] The beneficial effects brought by the technical solution provided by the embodiments of the present invention at least include:

[0124] In the embodiments of the present invention, by obtaining the historical point cloud of the target to be detected, sparsifying and occluding the historical point cloud to generate distorted point clouds with different distortion degrees, and then constructing a target detection model, where the target detection model includes parallel first and second branches, and both the first and second branches include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module. The target detection model can better learn the impact of distorted data on target detection, and use the distorted point cloud and the historical point cloud as the training set, combined with the consistency loss function, to train the target detection model with the first distortion degree and the second distortion degree respectively. Finally, obtain the real-time point cloud of the target to be detected, input the real-time point cloud into the trained target detection model, and output the target detection result of the target to be detected, thereby improving the detection accuracy, increasing the generalization ability of the model, and enhancing the model's recognition ability for occlusions and sparse point clouds.

[0125] Refer to the attached Figure 3 illustrates a schematic structural diagram of a point cloud object detection system based on self-distillation provided by the present invention.

[0126] The present invention also provides a point cloud object detection system 20 based on self-distillation, which is applied to the above-mentioned point cloud object detection method based on self-distillation, and includes:

[0127] Processor 201.

[0128] Memory 202, on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor 201, a point cloud object detection method based on self-distillation as in the method embodiment is implemented.

[0129] The point cloud object detection system 20 provided by the present invention can execute the above-mentioned point cloud object detection method based on self-distillation and achieve the same or similar technical effects. To avoid repetition, the present invention will not elaborate further.

[0130] The beneficial effects brought by the technical solution provided by the embodiments of the present invention at least include:

[0131] In the embodiments of the present invention, by obtaining the historical point cloud of the target to be detected, sparsifying and occluding the historical point cloud to generate distorted point clouds with different distortion degrees, and then constructing an object detection model. The object detection model includes parallel first and second branches. The first and second branches both include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module. The object detection model can better learn the influence of distorted data on object detection, and use the distorted point cloud and the historical point cloud as the training set. Combining with the consistency loss function, the object detection model is trained with the first distortion degree and the second distortion degree respectively. Finally, the real-time point cloud of the target to be detected is obtained, input into the trained object detection model, and the object detection result of the target to be detected is output, thereby improving the detection accuracy, increasing the generalization ability of the model, and improving the recognition ability of the model for occluded and sparse point clouds.

[0132] It should be understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0134] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, or a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0135] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood by referring to the context before and after.

[0136] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or a similar expression refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0137] It should be understood that in various embodiments of the present invention, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0138] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0139] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the devices, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0143] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0144] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the point cloud object detection method based on self-distillation as in the method embodiment.

[0145] The computer-readable storage medium provided by the present invention can implement the steps and effects of the point cloud object detection method based on self-distillation in the above method embodiment. To avoid repetition, the present invention will not elaborate further.

[0146] The beneficial effects brought by the technical solution provided by the embodiments of the present invention at least include:

[0147] In the embodiment of the present invention, by obtaining the historical point cloud of the target to be detected, sparsifying and occluding the historical point cloud to generate distorted point clouds with different degrees of distortion. Then, a target detection model is constructed. The target detection model includes a first branch and a second branch in parallel. The first branch and the second branch both include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module. The target detection model can better learn the influence of distorted data on object detection, and use the distorted point cloud and the historical point cloud as the training set. Combining with the consistency loss function, the target detection model is trained with the first distortion degree and the second distortion degree respectively. Finally, the real-time point cloud of the target to be detected is obtained, and the real-time point cloud is input into the trained target detection model to output the object detection result of the target to be detected, thereby improving the detection accuracy, increasing the generalization ability of the model, and enhancing the recognition ability of the model for occluded and sparse point clouds.

[0148] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

[0149] The following points need to be explained:

[0150] (1) The accompanying drawings of the embodiments of the present invention only relate to the structures involved in the embodiments of the present invention, and other structures can refer to the general designs.

[0151] (2) For clarity, in the accompanying drawings used to describe the embodiments of the present invention, the thickness of layers or regions is enlarged or reduced, that is, these drawings are not drawn to actual scale. It can be understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element can be "directly" on or under the other element or there can be intervening elements.

[0152] (3) Without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0153] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A point cloud object detection method based on self-distillation, characterized in that, Including: S1: Obtain the historical point cloud of the target to be detected; S2: Perform sparsification processing and occlusion processing on the historical point cloud to generate distorted point clouds with different degrees of distortion; S3: Construct a target detection model, where the target detection model includes a parallel first branch and a second branch, and both the first branch and the second branch include a preprocessing module, a global information fusion module, a foreground prior module, and a detection head module; S4: Use the distorted point cloud and the historical point cloud as a training set, and combine a consistency loss function to train the target detection model with a first distortion degree and a second distortion degree respectively, where the first distortion degree is less than the second distortion degree; S5: Obtain the real-time point cloud of the target to be detected; S6: Input the real-time point cloud into the trained target detection model and output the target detection result of the target to be detected.

2. The method for point cloud object detection based on self-distillation according to claim 1, wherein The processing process of the preprocessing module specifically includes: Extract the voxels of the historical point cloud and the distorted point cloud respectively through structured processing; Divide the three-dimensional space region according to the size of the voxels to determine point features; Perform average processing on the point features to generate voxel features; Generate a shallow bird's-eye view feature according to the voxel features, combining multi-layer sparse convolution downsampling and splicing techniques.

3. The method for point cloud object detection based on self-distillation according to claim 2, wherein The process of the global information fusion module processing the shallow bird's-eye view feature specifically includes: S401: Extract the first feature and the second feature of the shallow bird's-eye view feature through a convolutional layer and a dilated convolutional layer respectively; S402: Add the first feature and the second feature to obtain a feature value: v = Conv(I)+DConv(I); Where v represents the feature value, Conv represents a normal convolution operation, I represents the shallow bird's-eye view feature, and DConv represents a dilated convolution operation; S403: Combine a linear attention module and, through two pointwise convolutional layers, deform the shallow bird's-eye view feature processed by the global information fusion module to generate a query feature matrix and a key feature matrix; S404: Process the query feature matrix and the key feature matrix respectively through a fully connected layer; S405: Perform a matrix multiplication operation on the processed query feature matrix and key feature matrix to obtain an attention map; S406: Generate an output attention map according to the attention map through a pointwise convolution operation and a normalization operation: Where q represents the query feature matrix, k represents the key feature matrix, FC represents the fully connected layer, reshape represents a shape rearrangement operation, PW represents two pointwise convolutional operations, attention represents the output attention map, and sotfmax represents the softmax function; S407: Multiply the feature value and the output attention map in matrix form to generate a global information output; S408: Perform an addition operation on the global information output, the feature value, and the shallow bird's-eye view feature to generate a final output feature map.

4. The method for point cloud object detection based on self-distillation according to claim 1, wherein The processing process of the foreground prior module specifically includes: Extract the foreground prediction information of the shallow bird's-eye view feature to generate a foreground prediction map; Generate a Gaussian foreground target map according to the ground truth box; Normalize the foreground prediction map through the sigmoid function; According to the normalization result, combined with the consistency loss function, calculate the loss value between the foreground prediction map and the Gaussian foreground target map; Determine the foreground prediction feature map according to the loss value; Stitch and fuse the foreground prediction feature map and the final output feature map to generate a deep bird's-eye view feature.

5. The method for point cloud object detection based on self-distillation according to claim 4, wherein The generation process of the foreground prediction map specifically includes: According to the foreground prediction information, combined with the transposed convolution operation and the upsampling operation, generate a multi-scale foreground feature map; Sum the multi-scale foreground feature maps; Based on the summed multi-scale foreground feature map, through the convolution operation, generate a foreground prediction map with the same number of categories and channels as those in the shallow bird's-eye view feature.

6. The method for point cloud object detection based on self-distillation according to claim 4, wherein, The calculation formula of the sigmoid function is specifically: where S represents the sigmoid function, m represents the input value, and e represents the base of the natural logarithm.

7. The method for point cloud object detection based on self-distillation according to claim 4, wherein The consistency loss function is specifically: Among them, L cons represents the consistency loss function, represents the loss generated by the original point cloud in the region candidate and classification stage, represents the loss generated by the distorted point cloud in the region candidate and classification stage, represents the loss generated by the original point cloud in the regression stage, represents the loss generated by the distorted point cloud in the regression stage.

8. The method for point cloud object detection based on self-distillation according to claim 1, characterized in that The detection head is used to output the detection result of the target to be detected.

9. A point cloud object detection system based on self-distillation, characterized in that, Including: A processor; A memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method for point cloud object detection based on self-distillation according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the method for point cloud object detection based on self-distillation according to any one of claims 1 to 8 is implemented.

Images