A three-dimensional object detection method and system based on cross-modal decoupling knowledge transfer
Through cross-modal decoupling knowledge transfer, the knowledge in the LiDAR teacher network is migrated to the binocular detector, and the knowledge distillation of the deep and spatial BEV features is enhanced and decoupled classification heads is solved, and the low performance and three-dimensional binocular view detectors are solved. The problem of foreground and background imbalance in header tasks improves the accuracy of three-dimensional object detection.
Patent Information
- Application Number
- CN202411273178.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-12
AI Technical Summary
The binocular view detector has low performance in three-dimensional object detection, and the foreground and background are unbalanced in three-dimensional header tasks. The existing methods fail to effectively utilize Bird-Eye-View environment scene representation and background information.
Using a cross-modal decoupling knowledge transfer method, the knowledge in the high-performance LiDAR teacher network is transferred to the stereoscopic student network of binocular detectors, and the knowledge of distillation and decoupling of classification heads is maintained through depth and spatial BEV feature enhancement, structure and maintenance of distillation and decoupling of classification heads. Distillation improves the performance of binocular 3D object detection.
Through multi-dimensional feature enhancement and decoupling classification head knowledge distillation, the model pays more attention to prospective goals in classification tasks, and at the same time effectively utilizes background information, improving the accuracy and performance of target detection.
Smart Images

Figure CN118799665B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a three-dimensional target detection method and system based on cross-modal decoupling knowledge transfer. Background Art
[0002] In recent years, autonomous driving technology has attracted much attention. Accurately perceiving complex traffic scenes and ensuring the safety and reliability of autonomous driving vehicles largely rely on accurate three-dimensional positioning of target objects. Currently, mainstream 3D detection tasks can be divided into four different groups based on the difference in input: point cloud as input, stereo view as input, monocular image as input, and multi-modal input.
[0003] As a popular LiDAR detector in the industry, it can directly feedback the depth information of the surrounding environment and achieve a good balance between efficiency and accuracy. However, its high deployment cost, sparse point cloud and lack of semantic information limit its application. In contrast, the method based on stereo cameras has the advantages of low deployment cost while providing rich and dense pixel textures, and can easily detect and classify targets. It has a natural advantage in long-distance target detection and vision-based recognition of road elements such as traffic lights, traffic lines and sidewalks. However, the main challenge facing 3D object detectors that rely on stereo cameras is the inherent inaccuracy associated with depth estimation, which leads to a huge performance gap compared to LiDAR-based detectors. LiDAR point clouds can provide precise positioning and capture three-dimensional spatial information, making them a valuable reference for stereo cameras to perform detection tasks.
[0004] In this context, a number of excellent cross-knowledge distillation (KD) works have emerged, and they have begun to explore adding LiDAR point clouds to 3D object detection to improve stereo performance. Some works project LiDAR signals onto the image plane and use object-level and scene-level extraction to transfer knowledge from the LiDAR teacher detector to the baseline model; another study uses the x-component-kd method for regression analysis and the cross-anchor logit-kd method for response level classification to bridge the gap between stereo and LiDAR methods. Recent work has introduced a technique for knowledge extraction of modal transformations in 3D voxel space while maintaining the structure of each detector, but there are currently two technical difficulties in this regard: First, BEV (Bird-Eye-View), as a common environmental scene representation method, can clearly display the scale and position of the target object, but it has not been effectively applied in the above work; second, these methods all focus on extracting important information in the foreground area, while ignoring the important clues contained in the background. At this stage, a 3D target detection method and system based on cross-modal decoupled knowledge transfer is needed. Summary of the invention
[0005] In order to solve the problem of relatively low performance of binocular view detectors and imbalance of foreground and background in 3D head classification tasks, the present invention provides a 3D object detection method and system based on cross-modal decoupled knowledge transfer. The present invention transfers the knowledge in the high-performance LiDAR teacher network to the stereo student network based on the binocular detector through the knowledge distillation method to improve the performance of binocular 3D object detection.
[0006] In a first aspect, the present invention provides a three-dimensional object detection method based on cross-modal decoupling knowledge transfer, which adopts the following technical solution:
[0007] A three-dimensional object detection method based on cross-modal decoupling knowledge transfer, comprising:
[0008] Acquire original data, including stereo images and original point cloud data;
[0009] Feature extraction is performed based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps;
[0010] Constructing a feature enhancement network according to the divided feature graph, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network;
[0011] Using a feature enhancement network for structure-preserving extraction, including using the feature enhancement network to normalize instances of the teacher model and the student model respectively, and using the normalization to define a correlation distillation loss;
[0012] Decouple classification head knowledge distillation based on structure-preserving extraction, including decoupling the logit KD method into target classification distillation and non-target classification distillation, and calculating the target classification distillation loss and non-target classification distillation loss through KL divergence;
[0013] The loss function is defined based on the knowledge distillation of the disentangled classification head.
[0014] Furthermore, the construction of the upper branch and the lower branch includes compressing the spatial dimension of the feature map to obtain the statistic Z, reducing the feature dimension using a dimensionality reduction layer, then processing the reduced feature using a ReLU activation function, and finally restoring the reduced feature dimension using and through a dimensionality increase layer, processing the increased feature according to a Sigmoid activation function to obtain an attention weight, and obtaining the final output of the upper branch by element-by-element multiplication of the feature map and the attention weight.
[0015] Furthermore, the construction of the upper branch and the lower branch also includes performing maximum pooling and average pooling operations according to the obtained feature map, splicing the pooled results, further extracting features from the spliced results using a convolutional layer, and then calculating the attention map using a sigmoid function, and obtaining an output weighted feature map by performing element-by-element multiplication of the attention map and the input feature map. The formula for calculating the attention map is:
[0016] ,
[0017] in, represents the activation function, Represents the input feature map Perform the maximum pooling operation. Represents the input feature map Perform average pooling operation. Indicates concatenating the results of maximum pooling and average pooling. represents a convolutional layer with a 7×7 kernel.
[0018] Furthermore, the structure-preserving extraction using the feature enhancement network includes reshaping the instances of the teacher model and the student model respectively, converting the original three-dimensional tensor into a two-dimensional tensor, calculating the similarity matrix based on the reshaped instances, and normalizing each row vector of the matrix to unit length through L2 normalization.
[0019] Furthermore, the structure-preserving extraction using the feature enhancement network also includes defining a loss function based on the L2 normalization results of the teacher model and the student model, and the calculation formula of the loss function is:
[0020] ,
[0021] in, Expressed as a normalized coefficient, It is expressed as the difference between the similarity matrices of the teacher model and the student model, represents the Frobenius norm.
[0022] Furthermore, the decoupled classification head knowledge distillation based on structure-preserving extraction includes the number of all sample positions selected by the ATSS head, and the output probability map of the teacher model according to the number of samples. And the student model output probability map Reshape the dimension to get and , use the softmax function to reshape and Normalization is performed to obtain the confidence distribution of all anchor points of the same target object.
[0023] Furthermore, the loss function is defined according to the decoupled classification head knowledge distillation, including constructing an overall loss function according to the decoupled classification head knowledge distillation loss and the structure-preserving distillation loss, and continuously adjusting the Sim-DistNet model parameters in an end-to-end manner. The overall loss function expression is:
[0024] ,
[0025] in, Denoted as task loss, Expressed as the structure-preserving distillation loss, Decoupling the knowledge distillation loss of the classification head.
[0026] In the second aspect, a three-dimensional object detection system based on cross-modal decoupling knowledge transfer includes:
[0027] The data acquisition module is configured to: acquire original data, including acquiring stereo images and original point cloud data;
[0028] The feature extraction module is configured to: perform feature extraction based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps;
[0029] The feature enhancement module is configured to: construct a feature enhancement network according to the divided feature graph, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network;
[0030] The structure preserving module is configured to: perform structure preserving extraction using a feature enhancement network, including normalizing instances of the teacher model and the student model using the feature enhancement network, and defining a correlation distillation loss using the normalization;
[0031] The distillation module is configured to: perform decoupled classification head knowledge distillation according to structure-preserving extraction, including decoupling the logitKD method into target classification distillation and non-target classification distillation, and calculate the target classification distillation loss and the non-target classification distillation loss through KL divergence;
[0032] The loss module is configured to define the loss function based on the knowledge distillation of the decoupled classification head.
[0033] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a three-dimensional target detection method based on cross-modal decoupling knowledge transfer.
[0034] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, the processor being used to implement various instructions; the computer-readable storage medium being used to store multiple instructions, the instructions being suitable for being loaded by the processor and executing the three-dimensional target detection method based on cross-modal decoupling knowledge transfer.
[0035] In summary, the present invention has the following beneficial technical effects:
[0036] 1. The present invention uses the depth and space BEV feature enhancement (DSBFE) module to highlight the key features in the depth information using global information embedding and Excitation operations in the upper branch, and uses maximum pooling, average pooling and convolution operations to calculate the attention map in the lower branch to emphasize the importance of spatial position. This multi-dimensional feature enhancement can more comprehensively extract and highlight the target features in the 3D scene, so that the model pays more attention to the foreground target in the classification task while not ignoring the role of background information, thereby improving the accuracy of target detection.
[0037] 2. This paper adopts structure-preserving distillation (Loss_SPD) and decoupled classification head knowledge distillation (Loss_(DCH -KD)) to enable the student model to learn richer feature representation and classification knowledge from the teacher model. In particular, the decoupled classification head knowledge distillation considers target classification and non-target classification separately, which can more finely adjust the classification ability of the student model and further improve the accuracy of target detection.
[0038] 3. The present invention decouples the classification head knowledge distillation into target classification distillation (TCD) and non-target classification distillation (NCD). For TCD, it only provides predictions for the target class, which can more centrally learn the features of the target class; for NCD, it only considers the knowledge transfer between non-target classes, which helps to better understand the background information and can effectively balance the weights of foreground and background in classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a schematic diagram of the overall process of a three-dimensional target detection method based on cross-modal decoupling knowledge transfer according to an embodiment of the present invention.
[0040] Figure 2 It is a schematic diagram of the overall process structure of the cross-modal teacher-student paradigm Sim-DistNet framework of an embodiment of the present invention.
[0041] Figure 3 It is a schematic diagram of a network architecture for enhancing the depth and space BEV features according to an embodiment of the present invention.
[0042] Figure 4 It is a diagram of the classic knowledge distillation and a schematic diagram of the knowledge distillation of the decoupled classification head according to an embodiment of the present invention.
[0043] Figure 5 An additional visualization result diagram is provided on the KITTI dataset of an embodiment of the present invention. DETAILED DESCRIPTION
[0044] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0045] Example 1
[0046] Reference Figure 1 , A three-dimensional object detection method based on cross-modal decoupling knowledge transfer in this embodiment includes: acquiring original data, including acquiring a stereo image and original point cloud data;
[0047] Feature extraction is performed based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps;
[0048] Constructing a feature enhancement network according to the divided feature graph, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network;
[0049] Using a feature enhancement network for structure-preserving extraction, including using the feature enhancement network to normalize instances of the teacher model and the student model respectively, and using the normalization to define a correlation distillation loss;
[0050] Decouple classification head knowledge distillation based on structure-preserving extraction, including decoupling the logit KD method into target classification distillation and non-target classification distillation, and calculating the target classification distillation loss and non-target classification distillation loss through KL divergence;
[0051] The loss function is defined based on the knowledge distillation of the disentangled classification head.
[0052] Specifically, a three-dimensional object detection method based on cross-modal decoupling knowledge transfer includes the following steps:
[0053] like Figure 1 As shown in the figure, the present invention transfers the knowledge in the high-performance LiDAR teacher model to the stereo student model based on the stereo detector through the knowledge distillation method to improve the performance of stereo 3D target detection. Specifically, the present invention proposes a Sim-DistNet framework, which includes three main components: deep and spatial BEV feature enhancement, structure-preserving distillation, and decoupled distillation of the classification head.
[0054] First, to address the relatively low performance of the stereo view detector, this study adopted a deep and spatial BEV feature enhancement module to improve the quality of feature extraction by feeding stereo images and raw point cloud data into the network.
[0055] Secondly, in order to enable the student model in the stereo to learn similar information from the high activation areas of the teacher model of LiDAR, this study introduced a structure-preserving distillation method. By calculating the difference in the L2 normalized outer product of instance features between the teacher model and the student model, a structure-preserving distillation loss function is defined to guide the student model to learn relevant information in the teacher model.
[0056] Finally, in order to solve the problem of foreground and background imbalance in the 3D head classification task, this study proposed a decoupled distillation method for the classification head. By introducing the cross entropy loss function and hotspot labels, the weights of the foreground and background were balanced and the performance of the classification task was improved.
[0057] like Figure 2 As shown, S1, obtains stereo images and original point cloud data as input, including obtaining three-dimensional features through the LIGA-Stereo network backbone output ,Will Rescaled to 2D BEV feature map , and passed to the convolutional layer to divide it into two feature maps and ;
[0058] In 3D object detection, the performance of the stereo view detector is suboptimal compared to that of the LiDAR detector. The fundamental reason is that the feature extraction and depth information estimation of the stereo camera face the challenge of disparity calculation, because it needs to find matching feature points between the two views, which may be affected by many factors, such as perspective changes, low-texture areas, lighting changes, and occlusion, resulting in reduced quality and depth of image feature extraction. In contrast, LiDAR can directly obtain point cloud data without feature matching, so the quality of feature extraction is higher.
[0059] To solve this problem effectively, we avoid using a single spatial convolutional network directly. Instead, they are integrated into a new deep and spatial BEV feature enhancement network (DSBFE) as Figure 3 As shown in the figure, a feature enhancement network is constructed based on the divided feature maps. The network contains two levels of branches, and each part is sent to a special path to capture different types of contextual information. Figure 3 The detailed network architecture of deep and spatial BEV feature enhancement is given. Specifically, we are obtaining We then use the DSGN method to achieve this goal, transforming the 3D volume Rescaled to 2D BEV feature map , and the generated Pass it to two 1 × 1 convolutional layers to divide it into two parts and , as DSBFE:
[0060] ,
[0061] ,
[0062] Among them, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map. The convolution operation is performed with a convolution kernel of size. represents a 2D bird's eye view (BEV) feature map, and Through The two feature maps obtained by convolution operation have the number of channels in each part. , which is half of the original number of channels.
[0063] like Figure 3 As shown, Figure 3 fscale is expressed as element-by-element multiplication, S2, to obtain enhanced BEV features, using Perform global information embedding operation, then perform Excitation operation to obtain attention weight s, through Perform element-by-element multiplication with s to obtain the final output of the upper branch;
[0064] Establish the interdependence between feature channels and have the ability to adaptively learn channel features. Therefore, the compression and excitation module SE can better understand the spatial depth information. In the upper branch of DSBFE, we use the SE block to enhance the encoding of depth information. Simply put, it is to input The global information embedding operation is performed on the channel to compress the global spatial information into a channel descriptor. Formally, the statistic Z∈ Is It is generated by compressing its spatial dimension H×W (that is, compressing the kth channel), so the kth element of Z is expressed as:
[0065] , ,
[0066] in, Expressed as a compression function, Expressed as The kth channel of The value at , after the output of the above formula, can be interpreted as a set of local descriptors, the statistical information of which expresses the entire feature image. In order to utilize the information aggregated in the Squeeze operation, a second operation Excitation is performed, the purpose of which is to fully capture the attention of spatial depth information, using two fully connected layers (i.e., a dimensionality reduction layer and a dimensionality increase layer with a dimensionality reduction rate of r), and parameterizing the valve mechanism with ReLU and Sigmoid functions:
[0067] ,
[0068] in, Expressed as activation function, represents the sigmoid activation function, represents the ReLU activation function, Indicates the valve mechanism, and Represented as two different weight matrices, It plays a role in dimensionality reduction in the Excitation operation, which converts the input statistic Z from a higher dimension Mapping to lower dimensions , The function of the dimensionality increaser is to increase the intermediate features processed by the first fully connected layer and the ReLU activation function from the dimension Mapping back to dimensions , and then undergoes nonlinear transformation through the ReLU activation function. After obtaining the attention weight s, the final output of the upper branch is obtained by the following formula:
[0069] ,
[0070] in, Expressed as Element-wise multiplication with s.
[0071] S3, based on the obtained Perform maximum pooling and average pooling, use convolutional layers and sigmoid functions to calculate the attention map and weighted feature map to finally obtain the lower branch, and cascade the lower branch and the upper branch to enhance the BEV feature;
[0072] Spatial attention mainly emphasizes the spatial dimension of feature maps, aiming to adjust the importance of features at different spatial locations. By calculating the attention distribution of feature maps, a larger weight is assigned to important feature areas, thereby enhancing the expressiveness of the model. Spatial attention is combined with channel attention to fully understand the input data and produce a more refined feature representation than the previous single attention mechanism. Figure 3 As shown, we The maximum pooling and average pooling methods are used to effectively describe the features. Then a convolutional layer with a 7 × 7 kernel and a sigmoid function are used to calculate the attention map that describes richer location information. Attention Map It can be calculated as:
[0073] ,
[0074] in represents the activation function, Indicates concatenating the results of maximum pooling and average pooling. Representing the convolutional layer of the 7×7 kernel, the output weighted feature map can be expressed as:
[0075] ,
[0076] in, represents element-by-element multiplication, so that the salient features in the feature map will be highlighted and the redundant features will be suppressed. Finally, we will get the deep context from the upper branch And the spatial 2D context obtained by the lower branch By cascading together to get the output :
[0077] ,
[0078] Among them, Concat represents a connection operation, which concatenates the two feature maps in the channel dimension.
[0079] S4. Using the obtained enhanced BEV features to perform structure-preserving extraction, including using the obtained enhanced BEV features to perform L2 normalization processing on instances of the teacher model and the student model respectively, and defining a correlation distillation loss based on the L2 normalization processing;
[0080] LiDAR sensors can provide rich geometric structure information and precise location information. Therefore, the BEV features extracted by the LiDAR model have strong representation capabilities. In order to enable the binocular detector to learn enough similar information of high-activation areas from the LiDAR teacher model, the present invention uses a simulated dependency distillation to enable the student model to better learn similar information in the instance features of the teacher model. Based on the enhanced BEV features obtained by the above DSBFE module, in order to better guide students to understand the most relevant information induced by the teacher, we define a distillation loss that penalizes instances. and The difference between the L2 normalized outer products:
[0081] ,
[0082] ,
[0083] in, Represented as a teacher model instance The remodeling of is a B×B matrix, represents the matrix transpose, M and N represent the total number of non-null features in the Mth instance and the Nth instance, respectively. C represents the feature channel dimension, T and S represent the LiDAR teacher model and the stereo student model, respectively. Intuitively, the entry (i, j) in Encodes the similarity information of the teacher layer activation caused by the i-th and j-th instances in the mini-batch, and applies row L2 normalization to obtain , where the symbol [i,:] represents the i-th row in the matrix, then the same is true for the student layer:
[0084] ,
[0085] ,
[0086] in, for Remodeling, is a B×B matrix. We define the simulated correlation distillation loss as:
[0087] ,
[0088] in, represents the Frobenius norm, and the equation is The sum of the average element-wise differences between the matrices.
[0089] like Figure 4 As shown, S5, decouple the classification head knowledge distillation according to the structure-preserving extraction, including decoupling the logit KD method into target classification distillation and non-target classification distillation, calculating the target classification distillation loss and the non-target classification distillation loss through KL divergence, and completing the construction of the Sim-DistNet model;
[0090] In cross-modal knowledge distillation, the classification task of 3D heads also faces great challenges because the teacher and student models are network architectures from different fields. Compared with 2D object detection, the foreground-background imbalance problem in 3D regions is more serious. This is because LiDAR scans are inherently sparse compared to camera images, and most of the 3D space is empty. Even in occupied spaces dominated by background (such as buildings and vegetation), the size of objects to be detected (such as cars and pedestrians) varies. Inspired by the DKD network, even background regions contain crucial knowledge, which may be prominent "dark knowledge". Therefore, it is particularly important to balance the weight of each part.
[0091] This example decouples the classification head knowledge distillation: Figure 4 The difference between the work in part a is that the traditional logitKD method is decoupled into target classification distillation (TCD) and non-target classification distillation (NCD), such as Figure 4 As shown in b in . TCD transfers knowledge through binary logit distillation, that is, it only provides predictions for the target class, while the specific predictions for each non-target class are unknown. NCD only considers the knowledge transfer between non-target classes. Specifically, for target classification distillation, we convert the output probability map of the LiDAR model ∈ And the output probability map of the student model ∈ Re-expressed as ∈ and ∈ , where E= , Here, , , Respectively represent the number of all positive sample positions selected by the ATSS head, the detection category and the anchor point of each position. Figure 4 As shown, and Represents the confidence distribution of all anchor points and , respectively, for the reshaped and of Dimensionally apply the softmax function to further normalize and flatten the probability map and , thereby obtaining the confidence distribution of all anchor points of the same target object and :
[0092] ,
[0093] The same is true for non-target classification distillation, except that all negative samples are selected through the ATSS head. Therefore, the confidence distribution of non-target classes of the teacher model and the student model is obtained through the above operation. and Finally, the target classification distillation loss and non-target classification distillation can be calculated by KL divergence, the formula is as follows:
[0094] ,
[0095] Therefore, the overall loss of our disentangled classification head knowledge distillation is expressed as:
[0096] ,
[0097] Among them, a and b are expressed as hyperparameters. In addition, it is possible to balance the importance of TCD and NCD by adjusting the values of hyperparameters a and b. Decoupling NCD and TCD provides an efficient and flexible method for logit distillation.
[0098] S6. Train the constructed Sim-DistNet model, including defining the overall loss function and training the Sim-DistNet model in an end-to-end manner.
[0099] In the task learning phase, based on stereo distillation, we also use the regression loss of the 3D head. In the distillation phase, we train the 3D task model in an end-to-end manner. The overall distillation loss formula is:
[0100] ,
[0101] in, Denoted as task loss, Expressed as the structure-preserving distillation loss, Decoupling the knowledge distillation loss of the classification head,The hyperparameters are fixed in all our experiments.
[0102] In order to further verify the feasibility and effectiveness of the method of the present invention, relevant verification experiments were carried out.
[0103] The experimental environment was built using the Python-based deep learning library Pytorch1.8.1. We divided the training samples into train split (3712 samples) and val split (3769 samples). We conducted an ablation study on this basis and reported the test results for all 7481 samples trained on the KITTI benchmark. The labels were divided into three difficulty levels: easy, medium, and hard, based on object size (focusing only on Car, Ped, and Cyc categories), occlusion, and truncation level. We reported 40 recall-based position indicators on the KITTI official test server (R40) and 11 recall-based position indicators on the validation set (R11) for fair comparison with previous work. The rotation IoU thresholds for the three categories are 0.7, 0.5, and 0.5, respectively.
[0104] The network of the present invention is used to perform 3D target detection on complex scene images on the KITTI-3D public dataset, and the classification and regression images corresponding to each road scene image are obtained, such as Figure 5As shown, the augmented reality experience is more realistic, overcomes the limitations of traditional methods, provides efficient and accurate solutions, and has important support for intelligent security systems in the fields of computer vision and artificial intelligence.
[0105] Example 2
[0106] The difference between this embodiment and embodiment 1 is that this embodiment provides a three-dimensional object detection system based on cross-modal decoupling knowledge transfer;
[0107] The data acquisition module is configured to: acquire original data, including acquiring stereo images and original point cloud data;
[0108] The feature extraction module is configured to: perform feature extraction based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps;
[0109] The feature enhancement module is configured to: construct a feature enhancement network according to the divided feature graph, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network;
[0110] The structure preserving module is configured to: perform structure preserving extraction using a feature enhancement network, including normalizing instances of the teacher model and the student model using the feature enhancement network, and defining a correlation distillation loss using the normalization;
[0111] The distillation module is configured to: perform decoupled classification head knowledge distillation according to structure-preserving extraction, including decoupling the logitKD method into target classification distillation and non-target classification distillation, and calculate the target classification distillation loss and the non-target classification distillation loss through KL divergence;
[0112] The loss module is configured to define the loss function based on the knowledge distillation of the decoupled classification head.
[0113] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a three-dimensional target detection method based on cross-modal decoupling knowledge transfer.
[0114] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded and executed by the processor to implement a three-dimensional target detection method based on cross-modal decoupling knowledge transfer.
[0115] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A three-dimensional object detection method based on cross-modal decoupling knowledge transfer, characterized in that: include: Acquire original data, including stereo images and original point cloud data; Feature extraction is performed based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps; A feature enhancement network is constructed according to the divided feature map, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network. The construction of the upper branch and the lower branch includes compressing the spatial dimension of the feature map to obtain a statistic Z, reducing the feature dimension using a dimensionality reduction layer, and then processing the reduced-dimensional feature using a ReLU activation function. Finally, the reduced-dimensional feature dimension is restored using and through a dimensionality increase layer, and the increased-dimensional feature is processed according to a Sigmoid activation function to obtain an attention weight. The final output of the upper branch is obtained by element-by-element multiplication of the feature map and the attention weight. It also includes performing maximum pooling and average pooling operations according to the obtained feature map, splicing the pooled results, and further extracting features from the spliced results using a convolutional layer. Then, the sigmoid function is used to calculate the attention map, and the output weighted feature map is obtained by element-by-element multiplication of the attention map and the input feature map. The formula for calculating the attention map is: , in, represents the activation function, Represents the input feature map Perform the maximum pooling operation. Represents the input feature map Perform average pooling operation. Indicates concatenating the results of maximum pooling and average pooling. represents a convolutional layer with a 7×7 kernel; Using a feature enhancement network for structure-preserving extraction, including using the feature enhancement network to normalize instances of the teacher model and the student model respectively, and using the normalization to define a correlation distillation loss; Decouple classification head knowledge distillation based on structure-preserving extraction, including decoupling the logit KD method into target classification distillation and non-target classification distillation, and calculating the target classification distillation loss and non-target classification distillation loss through KL divergence; The loss function is defined based on the knowledge distillation of the disentangled classification head.
2. According to claim 1, a three-dimensional object detection method based on cross-modal decoupling knowledge transfer is characterized in that: The structure-preserving extraction using a feature enhancement network includes reshaping instances of a teacher model and a student model respectively, converting the original three-dimensional tensor into a two-dimensional tensor, calculating a similarity matrix based on the reshaped instances, and normalizing each row vector of the matrix to unit length through L2 normalization.
3. The three-dimensional object detection method based on cross-modal decoupling knowledge transfer according to claim 1 is characterized in that: The method of using a feature enhancement network to perform structure preservation extraction also includes defining a loss function based on L2 normalization results of a teacher model and a student model. The calculation formula of the loss function is: , in, Expressed as a normalized coefficient, It is expressed as the difference between the similarity matrices of the teacher model and the student model, represents the Frobenius norm.
4. The three-dimensional object detection method based on cross-modal decoupling knowledge transfer according to claim 1 is characterized in that: The decoupled classification head knowledge distillation based on structure-preserving extraction includes the number of all sample positions selected by the ATSS head, and the output probability map of the teacher model according to the number of samples. And the student model output probability map Reshape the dimension to get and , use the softmax function to reshape and Normalization is performed to obtain the confidence distribution of all anchor points of the same target object.
5. The three-dimensional object detection method based on cross-modal decoupling knowledge transfer according to claim 1 is characterized in that: The loss function is defined according to the decoupled classification head knowledge distillation, including constructing an overall loss function according to the decoupled classification head knowledge distillation loss and the structure preservation distillation loss, and continuously adjusting the Sim-DistNet model parameters in an end-to-end manner. The overall loss function expression is: , in, Denoted as task loss, Expressed as the structure-preserving distillation loss, Decoupling the knowledge distillation loss of the classification head.
6. A three-dimensional object detection system based on cross-modal decoupling knowledge transfer, executing the method as claimed in claim 1, characterized in that: include: The data acquisition module is configured to: acquire original data, including acquiring stereo images and original point cloud data; The feature extraction module is configured to: perform feature extraction based on the acquired raw data, including scaling and feature mapping the three-dimensional features of the raw data, and passing it to the convolution layer to divide it into two feature maps; The feature enhancement module is configured to: construct a feature enhancement network according to the divided feature graph, including constructing an upper branch and a lower branch, and cascading the upper and lower branches to complete the construction of the feature enhancement network; The structure preserving module is configured to: perform structure preserving extraction using a feature enhancement network, including normalizing instances of the teacher model and the student model using the feature enhancement network, and defining a correlation distillation loss using the normalization; The distillation module is configured to: perform decoupled classification head knowledge distillation according to structure-preserving extraction, including decoupling the logit KD method into target classification distillation and non-target classification distillation, and calculate the target classification distillation loss and the non-target classification distillation loss through KL divergence; The loss module is configured to define the loss function based on the knowledge distillation of the decoupled classification head.
7. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing a three-dimensional target detection method based on cross-modal decoupled knowledge transfer as described in claim 1.
8. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the three-dimensional target detection method based on cross-modal decoupling knowledge transfer as described in claim 1.
