Indoor three-dimensional target detection method and system based on knowledge distillation and voting refinement

Abstract

This invention discloses an indoor 3D target detection method and system based on knowledge distillation and voting refinement. The method includes the following steps: using VoteNet as the teacher model and a voting 3D detection network as the student model; inputting the original point cloud data from the training dataset into the teacher model and student model respectively to generate teacher votes and student votes; inputting the teacher votes into a refinement module to generate refined teacher votes; inputting the refined teacher votes into a foreground mask module to generate a foreground mask for the refined votes; inputting the student votes into a feature alignment module to generate aligned student votes; calculating the total loss, voting consistency distillation loss, and refinement loss of the trained student model; and using the trained student model to efficiently detect indoor 3D targets.

Description

Technical Field

[0001] This invention relates to an indoor 3D target detection method based on knowledge distillation and voting refinement, as well as a corresponding indoor 3D target detection system and an electronic device using the detection method, belonging to the field of computer vision technology. Background Technology

[0002] Object detection is a fundamental task in the field of computer vision, involving the detection and classification of multiple semantic objects. 3D object detection, as a subset of object detection, focuses on identifying and locating objects from 3D point clouds, and it is widely used in practical applications such as autonomous driving and robot assistance.

[0003] In existing technologies, VoteNet is an efficient and representative 3D object detection model that achieves end-to-end 3D object detection based on point cloud data using Hough voting. However, higher-performance detection methods often come with heavier computational burdens and detection costs. Therefore, deploying the high-performance VoteNet model to real-world application terminals is difficult due to the limitations of the terminal devices' limited computing and storage resources. On the other hand, knowledge distillation is an effective method for compressing models. It uses a pre-trained heavyweight teacher model to help train a lightweight student model to improve model performance and is widely used in developing efficient models for various 2D tasks. However, research and application of knowledge distillation methods in 3D indoor object detection have received little attention. Furthermore, if knowledge distillation methods are directly applied to the VoteNet model, problems such as negative voting and foreground / background class imbalance in indoor 3D object detection will affect the supervision process of the student model, leading to degradation and inefficiency of knowledge distillation in object detection. Therefore, how to avoid the above problems and provide a method that can improve the accuracy of target detection with low computational complexity and low cost to meet the requirements of practical terminal applications has always been a very important technical research topic in this field.

[0004] Chinese invention patent ZL 202210602131.1 discloses a method for training and using a 3D object detection model. The method includes the following steps: acquiring a 3D object detection model and a training set; using the student network in the 3D object detection model to detect monocular images, obtaining a first detection result, a depth loss function, and a ground truth loss function; using the teacher network to detect point cloud data, obtaining a second detection result; calculating the BEV feature distillation loss function, the 3D relation distillation loss function, and the response distillation loss function based on the first and second detection results; calculating the overall loss function of the 3D object detection model based on the above five loss functions; and training the 3D object detection model based on the overall loss function. Summary of the Invention

[0005] The primary technical problem to be solved by this invention is to provide an indoor three-dimensional target detection method based on knowledge distillation and voting refinement.

[0006] Another technical problem to be solved by the present invention is to provide an indoor three-dimensional target detection system based on knowledge distillation and voting refinement.

[0007] Another technical problem to be solved by the present invention is to provide an electronic device employing the method.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] According to a first aspect of the present invention, an indoor three-dimensional target detection method based on knowledge distillation and voting refinement is provided, comprising the following steps:

[0010] (1) VoteNet is used as the teacher model and the voting 3D detection network is used as the student model. The number of feature channels in each layer of the student model is selected in a lightweight manner according to the hardware resource configuration of the application terminal.

[0011] (2) Input the original point cloud data of the three-dimensional objects in the training dataset into the teacher model and the student model respectively to generate teacher votes and student votes;

[0012] (3) Input the teacher votes into the refinement module to generate a refined teacher vote;

[0013] (4) Input the refined teacher votes into the foreground mask module to generate a foreground mask for the refined votes;

[0014] (5) Input the student votes into the feature alignment module to generate aligned student votes;

[0015] (6) Calculate the total loss, voting consistency distillation loss and refinement loss for training the student model, and use them to update the training of the student network, the foreground mask module and the refinement module, respectively;

[0016] (7) Deploy the trained student model to the application terminal and use the student model to efficiently detect indoor three-dimensional targets.

[0017] Preferably, in step (3), the refinement module consists of multiple refinement layers, used to refine the teacher votes to correct inaccurate teacher votes in the supervision of the student model during knowledge distillation; wherein, the refinement process of each refinement layer includes the following sub-steps:

[0018] (31) Calculate the internal position information based on the coordinates of the teacher votes output by the previous layer;

[0019] (32) Calculate the masked multi-head self-attention based on the characteristics of the teacher votes output from the previous layer and the internal position information;

[0020] (33) Calculate the multi-head cross attention of the mask;

[0021] (34) Calculate the coordinates and features of the teacher votes output by this layer based on the masked multi-head self-attention and the masked multi-head cross-attention.

[0022] Preferably, in steps (31) and (32), the internal position information B of the i-th refinement layer i The calculation satisfies the following formula:

[0023]

[0024] Among them, X i-1 The coordinates of the teacher votes output from the previous layer; X represents i-1 The distance between any two points in the equation; Indicates masking operation;

[0025] The mask multi-head self-attention The calculation satisfies the following formula:

[0026]

[0027] Among them, F i-1 The characteristic of teachers voting for the output of the previous layer; It is a connection of H attention features from different attention heads, where H represents the number of attention heads; This indicates that the softmax operation is performed row by row; C is the number of channels in the query, key, and value. This represents the linear projection operation at the end of the multi-head attention mask; and Represent the linear projections of the h-th head of the query, key, and value, respectively; (F i-1 ) T For feature F i-1 The transpose of .

[0028] Preferably, in step (34), the coordinates X of the teacher's vote output by the i-th refinement layer are... i and feature F i The calculations satisfy the following formulas:

[0029]

[0030]

[0031] in, For masked multi-head self-attention, Ψ(·) is the corresponding Multilayer perceptron; X i-1 The coordinates of the teacher votes output from the previous layer; For masking multi-head cross attention.

[0032] Preferably, in step (4), the foreground mask... i The calculation satisfies the following formula:

[0033]

[0034]

[0035] in, g represents the coordinates of the i-th refined vote in the refined teacher voting coordinate system. j Let s be the true coordinates of the center of the j-th object, where the j-th object is the true object closest to the i-th refined vote; j ∈R 3 σ is the length of the j-th true bounding box along the x, y, and z axes; σ is a non-zero positive value used to avoid the denominator of this calculation being zero. Let Frobenius norm be denoted; arg min is the arg min function; M is the number of votes.

[0036] Preferably, in step (5), the feature alignment module is implemented by a series of multilayer perceptron layers for dimensional alignment between the teacher votes and the student votes; the aligned student votes R s The calculation satisfies the following formula:

[0037] R s =Align(V s )

[0038] Among them, V s Voting for students; Align(·) is a projection function.

[0039] Preferably, in step (6), the voting consensus distillation loss L dis The calculation satisfies the following formula:

[0040]

[0041]

[0042] in, For the i-th refined feature of teacher voting; The coordinates for the i-th refined teacher vote; Mask i For the i-th refined teacher voting prospectus; Input the aligned features obtained from the student network into the j-th object; Input the aligned coordinates obtained from the student network for the j-th object; the j-th object is the real object that is closest to the i-th refined vote; Denotes the Frobenius norm; M is the number of votes;

[0043] The total loss L for training the student model is calculated according to the following formula:

[0044] L = L dis +λL reg +ηL cls

[0045] Among them, L reg For the regression loss in VoteNet, L cls For classification loss in VoteNet, λ and η are two hyperparameters used to balance different loss terms.

[0046] Preferably, in step (6), the refinement loss L ref The calculation satisfies the following formula:

[0047]

[0048] in, Let g be the coordinate of the i-th refined teacher vote; i s represents the ground real center of the i-th voting object; i It generates detailed votes. The corresponding seed; This indicates an indicator function used to indicate the seed s. i Whether it is on an object; M is the number of votes.

[0049] According to a second aspect of the present invention, an indoor 3D target detection system based on knowledge distillation and voting refinement is provided, comprising a teacher model, a student model, a refinement module, a foreground mask module, and a feature alignment module; wherein,

[0050] The teacher model uses the VoteNet model, the student model uses a voting-based 3D detection network, and the number of feature channels in each layer of the student model is set in a lightweight manner according to the hardware resource configuration of the application terminal.

[0051] The refinement module is used to refine teacher votes to correct inaccurate teacher votes in the supervision of the student model during knowledge distillation. The refinement module consists of multiple refinement layers, each of which includes a masked multi-head self-attention layer, a masked multi-head cross-attention layer, a self-distance calculation module, and a cross-distance calculation module.

[0052] The foreground masking module is used to mask the refined teacher votes so that background information can be used to focus more on the votes within the target bounding box during the distillation process.

[0053] The feature alignment module is used to perform alignment operations on student votes, realizing dimensional alignment between student votes and teacher votes; the feature alignment module is implemented by a series of multilayer perceptron layers.

[0054] The teacher model, in collaboration with the refinement module, the foreground mask module, and the feature alignment module, employs knowledge distillation and voting refinement methods to help train the student model and improve indoor 3D object detection performance.

[0055] According to a third aspect of the present invention, an electronic device is provided, the electronic device including a processor and a memory, the processor and the memory being coupled together; wherein,

[0056] The memory is used to store computer programs;

[0057] The processor is used to run a computer program stored in the memory to execute the above-described indoor 3D target detection method based on knowledge distillation and voting refinement.

[0058] Compared with existing technologies, the indoor 3D target detection method based on knowledge distillation and voting refinement provided by this invention uses the VoteNet teacher model to train the voting-based student model, resulting in a lightweight network model for indoor 3D target detection. This network model features relatively low computational complexity, relatively small data storage requirements, and high detection performance, making it easy to deploy on hardware-restricted applications for efficient indoor 3D target detection. Therefore, the indoor 3D target detection method based on knowledge distillation and voting refinement provided by this invention has the advantages of low hardware resource consumption, low detection cost, and high detection performance. Attached Figure Description

[0059] Figure 1 A flowchart of an indoor 3D target detection method based on knowledge distillation and voting refinement provided by the present invention;

[0060] Figure 2This is a schematic diagram of the voting generated by the VoteNet teacher model during indoor 3D object detection in an embodiment of the present invention;

[0061] Figure 3 This invention provides a schematic diagram of the overall framework structure of an indoor three-dimensional target detection system based on knowledge distillation and voting refinement;

[0062] Figure 4 This is a schematic diagram of the structure of the refinement module in an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical content of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 As shown in the figure, an indoor 3D target detection method based on knowledge distillation and voting refinement provided by an embodiment of the present invention includes at least the following steps:

[0066] S1: VoteNet is used as the teacher model, and a voting 3D detection network is used as the student model; the number of feature channels in each layer of the student model is selected to be lightweight based on the hardware resource configuration of the application terminal.

[0067] To achieve optimal indoor 3D object detection performance by using a pre-trained large teacher model to assist the training of a lightweight student model, the number of feature channels per layer in the student model needs to be selected based on the hardware resource configuration of the actual application terminal. For example, the number of feature channels per layer in the student model can be chosen to be half or a quarter of the number of feature channels per layer in the teacher model. Reducing the number of feature channels per layer in the student model can reduce computational complexity and the amount of data stored, thus adapting to the hardware resource configuration of the actual application terminal.

[0068] Voting 3D detection networks refer to three-stage architecture networks that use raw point cloud data as input. They utilize PointNet to group point features, perform object center voting through group clustering, and finally make predictions based on the voting features. Examples include MLCVNet, HGNet, and 3DSSD networks.

[0069] S2: Obtain the original point cloud P∈R of the 3D objects in the training dataset. N×3 Input the teacher model and student model respectively, and generate teacher seed S. t Teacher Votet and student seeds s Student Vote s .

[0070] The training dataset consists of raw point cloud data of collected 3D objects, where the raw point cloud P∈R N×3 This indicates that the point cloud has N points, each with a three-dimensional (x, y, z) coordinate.

[0071] S3: Teacher voting V t The input refinement module generates refined teacher votes in R. t .

[0072] like Figure 2 As shown, the VoteNet teacher model typically generates images that are far from the true center of the object when detecting 3D objects indoors, for example... Figure 2 The voting points, represented by black dots, are called negative votes. The presence of negative votes during the distillation process affects the supervision of the student model, leading to inaccurate and inefficient distillation knowledge. Therefore, this invention proposes using a refinement module to generate refined teacher votes R. t The refined voting process is as follows: Figure 2 The white dots represent voting points, thus avoiding the negative impact of negative votes on the student model.

[0073] S4: Refine the teacher voting R t The input foreground mask module generates a refined foreground mask for voting. i .

[0074] Since the imbalance between foreground and background categories can lead to the degradation of knowledge distillation in object detection, this invention proposes to use a foreground masking module to perform foreground masking operations. This allows the background information to be used more to focus on the votes within the target bounding box during the distillation process, thereby enhancing the distillation of 3D object detection.

[0075] S5: Vulnerability of Student Voting s The input feature alignment module generates aligned student votes in R. s .

[0076] Since the number of feature channels in each layer of the student model is selected in a lightweight manner according to the hardware resource configuration of the application terminal in step S1, and the student model has different feature channels than the teacher model, the student votes need to be aligned before calculating the voting loss.

[0077] S6: Calculate the total loss L for training the student model and the voting consensus distillation loss L. dis and refinement loss L ref These are used to update the training of the student network, the foreground mask module, and the refinement module, respectively.

[0078] S7: Deploy the trained student model to the actual application terminal and use the student model to efficiently detect indoor 3D targets.

[0079] like Figure 3 As shown in the figure, the overall framework structure of an indoor 3D target detection system based on knowledge distillation and voting refinement (referred to as the VRDistill framework) provided by this embodiment of the invention includes a teacher model, a student model, a refinement module, a foreground mask module, and a feature alignment module. The teacher model uses VoteNet, which consists of three main parts: a backbone network, a voting generation module, and a detection head. The backbone network is used to refine the input original point cloud P∈R. N×3 Generate seed S t The vote generation module is used to generate votes (V) based on the seed. t ∈R M×(3+C) Where M is the number of votes, 3 represents a three-dimensional coordinate for each vote, and C represents a feature for each vote; the detection head is used to obtain high-quality 3D bounding boxes by employing 3D nonmaximum suppression.

[0080] The following sections provide a detailed description of the specific steps and VRDistill framework of an indoor 3D target detection method based on knowledge distillation and voting refinement provided in the embodiments of the present invention.

[0081] In step S3, the refinement module is used to refine the teacher votes V t The process is refined to generate a more detailed teacher voting result (R). t This corrects the supervision mismatch problem caused by inaccurate teacher voting (i.e., negative voting) in the knowledge distillation process, affecting the student model. For example... Figure 4 As shown, the refinement module consists of multiple refinement layers, and its input data is teacher votes V. t Each vote includes coordinates X and features F. The structure and operation of each refinement layer are illustrated below using the i-th refinement layer as an example.

[0082] Each refinement layer includes a Masked Multi-Head Self-Attention layer, a Masked Multi-Head Cross-Attention layer, a Self-Distance Calculation Module, and a Cross-Distance Calculation Module.

[0083] The i-th refinement layer takes the output coordinates X of the previous layer. i-1 and feature F i-1 As input, and to produce the output coordinate X of this layer. i and output features F i The specific working process of the i-th refinement layer includes the following sub-steps:

[0084] S31: Based on the coordinate X output from the previous layer i-1 ∈R M×3 Calculate internal location information B i .

[0085] First, based on the coordinate X output from the previous layer i-1 ∈R M×3 X is calculated using the self-distance calculation module. i-1 Find the distance between any two points in the matrix and generate the internal position matrix D. i ∈R M×M , where M is the number of votes.

[0086] Secondly, maintain the internal position matrix D i The smallest K values ​​in each row remain unchanged (K is a positive integer), and the remaining items are masked by adding a very large negative number to them so that they are masked in the attention.

[0087] Internal location information B i Calculate using the following formula:

[0088]

[0089] in, X represents i-1 The distance between any two points in the equation; This represents a masking operation, which involves adding a very large negative number to all terms except the K minimum values; X i-1 This represents the coordinates output from the previous layer.

[0090] S32: Based on the feature F output from the previous layer i-1 and internal location information B iCalculate the multi-head self-attention mask

[0091] Based on the feature F output from the previous layer i-1 and internal location information B i Masked multi-head self-attention layer is used to compute masked multi-head self-attention. The calculation formula is as follows:

[0092]

[0093] in, It is a connection of H attention features from different attention heads, where H represents the number of attention heads; This indicates that the softmax (normalized exponential function) operation is performed row by row; C is the number of channels in the query, key, and value. This represents the linear projection operation at the end of the multi-head attention mask; and Represent the linear projections of the h-th head of the query, key, and value, respectively; (F i-1 ) T For the features F of the previous layer i-1 The transpose of .

[0094] S33: Calculate the multi-head cross-attention of the mask

[0095] Calculate masked multi-head cross-attention using a masked multi-head cross-attention layer. The calculation method is similar to the mask multi-head self-attention calculation in step S32, only the input data is different, so it will not be described in detail here.

[0096] S34: Calculate the X coordinates output by this layer. i and output features F i .

[0097] Based on the multi-head self-attention of the mask Multi-head cross-attention with mask Calculate the feature F of the output of this layer i as follows:

[0098]

[0099] Calculate the X coordinates output by this layer. i as follows:

[0100]

[0101] in, For masked multi-head self-attention, Ψ(·) is the corresponding Multilayer perceptron.

[0102] The structure and working process of the i-th refinement layer have been explained in detail above. After refinement by multiple refinement layers, the refinement module can finally obtain the refined teacher vote R. t =[X t F t ], where X t and F t These are the coordinates and features for refining the voting process.

[0103] In step S4, to address the degradation problem in object detection caused by the imbalance between foreground and background categories, a foreground mask module is introduced to refine the teacher vote R. t Performing foreground masking allows the distillation process to focus more on votes within the target bounding box using background information, thereby improving the distillation performance of 3D object detection. It's important to note that unlike 2D object detection, 3D object detection operates in a sparser space; using hard masks, as in 2D object detection distillation, might result in the inclusion of much low-quality information.

[0104] For the refined teacher voting coordinate X t , represented as in These are the coordinates of the i-th refined vote. The foreground mask for the i-th refined vote is... i The calculation is as follows:

[0105]

[0106]

[0107] Where σ is a non-zero positive value used to avoid the denominator of the calculation being zero. Usually, σ is taken as 1 to adjust Maski∈[0,1]. Let g be the coordinate of the i-th refined vote; j Let be the true coordinates of the center of the j-th object, where the j-th object is the true object closest to the i-th object in the vote; s j ∈R 3 It is the length of the j-th true bounding box along the x, y, and z axes; denoted as Frobenius norm; arg min is the arg min function; M is the number of votes.

[0108] As can be seen from formulas (5) and (6), as the distance between the vote and the nearest real object center increases, the mask value of the corresponding refinement vote decreases, and a smaller mask value indicates less attention is allocated to this refinement vote. By adjusting the importance of different point pairs in the foreground and background regions, a balance is achieved between foreground and background information during the distillation process using the foreground mask, thereby achieving more effective knowledge transfer while retaining valuable background knowledge.

[0109] In step S5, the feature alignment module is implemented by a series of multilayer perceptron (MLP) layers to achieve dimensional alignment between teacher votes and student votes. The student votes V... s The input feature alignment module generates aligned student votes in R. s Its expression is as follows:

[0110] R s =Align(V s (7)

[0111] Align(·) is a projection function.

[0112] Aligned student votes R s , represented as R s =[X s F s ], where X s and F s These are the coordinates and features of the aligned student votes, respectively.

[0113] In step S6, firstly, the total loss L for training the student model is calculated as follows:

[0114] L = L dis +λL reg +ηL cls (8)

[0115] Among them, L dis Distillation loss for voting consistency between the teacher and student models; L reg For the regression loss in VoteNet, L cls L is the classification loss in VoteNet. reg and L cls Used to provide accurate supervision for training student models. λ and η are two hyperparameters used to balance different loss terms.

[0116] Since 3D point clouds are unordered, teacher refinement voting is used in knowledge distillation. t =[X t F t The closest vote in the poll is used as the supervisor. Therefore, the voting consensus distillation loss Ldis The calculation is as follows:

[0117]

[0118]

[0119] in, For the i-th refined feature of teacher voting; Input the aligned features obtained from the student network for the j-th object; Mask i For the i-th refined teacher voting prospectus; Input the aligned coordinates obtained from the student network for the j-th object; Let i be the coordinates of the i-th refined teacher vote; Let Frobenius norm be denoted; M is the number of votes.

[0120] The vote of the teacher model output that is closest to the F-norm of the j-th object output by the student model can be obtained by formula (10). Then, the vote consistency distillation loss L can be obtained by formula (9). dis The training of the foreground mask module is updated. Finally, the total loss L is calculated according to formula (8) and used to update the training of the student network.

[0121] Secondly, since the refinement module is randomly initialized during the distillation process, it is necessary to calculate the refinement loss L. ref To provide supervision and achieve better refinement performance. Refinement loss L ref The calculation is as follows:

[0122]

[0123] in, Let g be the coordinate of the i-th refined vote in the refined teacher vote; i s represents the ground real center of the i-th voting object; i It generates detailed votes. The corresponding seeds, This indicates an indicator function used to indicate the seed s. i Whether it is on an object; M is the number of votes.

[0124] The refinement loss L can be obtained by calculation using formula (11). ref This is used for training more detailed modules.

[0125] Through the training in steps S2 to S6 above, a trained student model for indoor 3D target detection is finally obtained. This student model has the characteristics of relatively low computational complexity, relatively small amount of data storage, but high detection performance. It is deployed to a real application terminal and used to efficiently detect indoor 3D targets.

[0126] The above provides a detailed description of the specific steps and corresponding VRDistill framework for an indoor 3D target detection method based on knowledge distillation and voting refinement provided by embodiments of the present invention. To evaluate the effectiveness of the VRDistill framework, the inventors conducted comprehensive comparative detection experiments and ablation experiments on the indoor 3D target detection method using the VRDistill framework and detection models using other methods in the prior art. The detection process and evaluation results of each specific experiment are described below.

[0127] The experimental datasets used were ScanNet and SUN-RGBD to evaluate the performance of the detection models. ScanNet is an RGB-D video dataset consisting of indoor scenes, specifically designed for tasks such as 3D object detection, 3D instance segmentation, and semantic segmentation. The SUN-RGBD dataset is a single-view RGB-D dataset commonly used for 3D scene understanding.

[0128] First, a VoteNet teacher model is pre-trained. Then, the VRDistill framework is used to transfer knowledge from the teacher model to the student model. The teacher model's architecture follows the VoteNet network model, while the number of channels in the student model is halved or reduced to one-quarter compared to the teacher model. The alignment module uses a single-layer perceptron to align feature channels between the student and teacher models. When processing the ScanNet dataset, the original number of points N is set to 40,000; when processing the SUNRGBD dataset, the original number of points N is set to 20,000. In the voting generation module, 256 votes are generated. In the refinement module, based on experience, the number of refinement layers is set to 4, and K, representing the nearest point, is set to 5. During knowledge distillation, both the student model and the refinement module are trained from scratch. For the ScanNet dataset, the AdamW optimizer is used with parameters β1 = 0.9, β2 = 0.98, and ∈ = 10. -7 The default weight decay for most components is set to 0.00001, except for the refinement module, which is adjusted to 0.01 to prevent overfitting in the student model. A cosine learning rate scheduler is used to better approximate the global optimum. The batch size is 8, and the student model is trained for 180 epochs. Except for the initial learning rate, the settings on the SUNRGBD dataset are the same as those on the ScanNet dataset.

[0129] To compare the experimental results with those of indoor 3D object detection methods using the VRDistill framework, several existing 2D object detection distillation methods, including FGD, MGD, and PGD, were selected as baselines in the detection experiments. In the experimental results report below, "student (1 / 2)" or "student (1 / 4)" indicates that the student network has the same VoteNet network structure as the teacher network, but the number of channels in each layer is halved or reduced to one-quarter, and the student model is trained from scratch without using distillation. The number of feature channels of the detection model provided by this invention using the VRDistill framework, which was tested simultaneously with the student (1 / 2) or student (1 / 4) network, was also halved or reduced to one-quarter.

[0130] The detection results on the ScanNet dataset are shown in Table 1.

[0131] Table 1

[0132]

[0133] As shown in Table 1, firstly, compared to other distillation methods such as FGD, MGD, and PGD, the student (1 / 2) or student (1 / 4) models using knowledge distillation outperformed those trained from scratch, indicating the effectiveness of knowledge distillation for indoor 3D object detection. Secondly, the detection model using the VRDistill framework outperforms other distillation baseline methods, demonstrating the effectiveness and high performance of the VRDistill framework. For example, in a quarter-channel setting, the detection model using the VRDistill framework improves performance by 6.1% compared to the baseline method FGD. Thirdly, the detection model using the VRDistill framework even outperforms the teacher model. This is because the refinement module can correct inaccurate votes in the teacher model and pass this correction information to the student model.

[0134] The detection results on the SUNRGBD dataset are shown in Table 2.

[0135] Table 2

[0136]

[0137] As shown in Table 2, the detection model using the VRDistill framework exhibits better performance compared to other baseline methods. However, the experiments also revealed that the performance improvement on the SUNRGBD dataset is not as significant as on the ScanNet dataset. This may be because the scenes in the SUNRGBD dataset are more complete than those in ScanNet. In ScanNet, the ground truth bounding boxes are aligned to the axis without angles, while in the SUNRGBD dataset, each ground truth detection box is annotated with an orientation angle, making further performance improvements more difficult.

[0138] In addition, to evaluate the contribution of the thinning module and the foreground mask module to the performance improvement of VRDistill, the inventors conducted ablation experiments on the thinning module and the foreground mask module respectively. The experimental results are shown in Table 3.

[0139] Table 3

[0140]

[0141] As shown in Table 3, removing the refinement module resulted in a 1.2% mAP@0.25 accuracy loss for the model. This indicates that the teacher voting results were improved within the refinement module, making them closer to the object center and thus enhancing detection performance. Removing the foreground mask module resulted in a 0.5% mAP@0.25 accuracy loss for the model. Distillation using the foreground mask performed better than distillation without it, demonstrating the effectiveness of incorporating the foreground mask into the distillation process.

[0142] Building upon the aforementioned indoor 3D target detection method based on knowledge distillation and voting refinement, this invention further provides an indoor 3D target detection system based on knowledge distillation and voting refinement. For example... Figure 3 As shown, this indoor 3D object detection system includes a teacher model, a student model, a refinement module, a foreground masking module, and a feature alignment module. The teacher model uses the VoteNet model, while the student model uses a voting-based 3D detection network. Furthermore, the number of feature channels per layer in the student model is set to a lightweight configuration based on the hardware resources of the application terminal. The teacher model collaborative refinement module, foreground masking module, and feature alignment module employ knowledge distillation and voting refinement methods to help train the student model and achieve optimal indoor 3D object detection performance.

[0143] The VoteNet teacher model consists of a backbone network, a voting generation module, and a detection head. The backbone network generates teacher seeds based on the input raw point cloud. The voting generation module generates teacher votes based on the teacher seeds. The detection head uses 3D nonmaximum suppression to obtain high-quality 3D bounding boxes.

[0144] The refinement module is used to refine teacher votes, generating refined teacher votes to correct inaccurate teacher votes during the knowledge distillation process and supervise the student model. The refinement module consists of multiple refinement layers, each including a masked multi-head self-attention layer, a masked multi-head cross-attention layer, a self-distance calculation module, and a cross-distance calculation module.

[0145] The foreground masking module is used to mask the refined teacher votes, so that the background information can be used to focus more on the votes within the target bounding box during the distillation process, thereby improving the distillation effect of 3D target detection.

[0146] The feature alignment module aligns student votes, achieving dimensional alignment between student and teacher votes. Implemented by a series of multilayer perceptron layers, the module generates aligned student votes after the student votes are input.

[0147] This indoor 3D target detection system trains the student model multiple times using the original point cloud in the training dataset, ultimately obtaining a trained student model for indoor 3D target detection. This student model has the characteristics of relatively low computational complexity, relatively small amount of data storage, but high detection performance. It is deployed to a real-world application terminal and used to efficiently detect indoor 3D targets.

[0148] Furthermore, embodiments of the present invention also provide an electronic device. For example... Figure 5 As shown, the electronic device includes one or more processors and a memory. The memory is coupled to the processors and is used to store one or more computer programs. When these programs are executed by the processors, they enable the processors to implement the indoor 3D target detection method based on knowledge distillation and voting refinement as described in the above embodiments.

[0149] The processor controls the overall operation of the electronic device to complete all or part of the steps of the indoor 3D target detection method based on knowledge distillation and voting refinement. This processor module can be a central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), digital signal processing (DSP) chip, etc. The memory stores various types of data to support the operation of the electronic device. This data may include, for example, instructions for any application or method used to operate the electronic device, as well as application-related data. This memory module can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, etc.

[0150] In summary, compared with existing technologies, the indoor 3D target detection method based on knowledge distillation and voting refinement provided by this invention uses the VoteNet teacher model to train the voting-based student model, resulting in a lightweight network model for indoor 3D target detection. This network model features relatively low computational complexity, relatively small data storage requirements, and high detection performance, making it easily deployable on hardware-restricted applications for efficient indoor 3D target detection. Therefore, the indoor 3D target detection method based on knowledge distillation and voting refinement provided by this invention offers advantages such as lower hardware resource consumption, lower detection costs, and higher detection performance.

[0151] It should be noted that in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0152] The above provides a detailed description of the indoor 3D target detection method and system based on knowledge distillation and voting refinement provided by this invention. Any obvious modifications made by those skilled in the art without departing from the essential content of this invention will constitute an infringement of the patent rights of this invention and will incur corresponding legal liability.

Claims

1. An indoor 3D target detection method based on knowledge distillation and voting refinement, characterized in that... Includes the following steps: (1) VoteNet is used as the teacher model and the voting 3D detection network is used as the student model. The number of feature channels in each layer of the student model is selected in a lightweight manner according to the hardware resource configuration of the application terminal. (2) Input the original point cloud data of the three-dimensional objects in the training dataset into the teacher model and the student model respectively to generate teacher votes and student votes; (3) Input the teacher votes into the refinement module to generate refined teacher votes; the refinement module consists of multiple refinement layers, used to refine the teacher votes to correct inaccurate teacher votes in the supervision of the student model during knowledge distillation; wherein, the refinement process of each refinement layer includes the following sub-steps: (31) Calculate the internal position information based on the coordinates of the teacher votes output by the previous layer; (32) Calculate masked multi-head self-attention based on the characteristics of the teacher votes output from the previous layer and the internal position information; (33) Calculate masked multi-head cross-attention using a masked multi-head cross-attention layer; (34) Calculate the coordinates and features of the teacher votes output by this layer based on the masked multi-head self-attention and the masked multi-head cross-attention; In steps (31) and (32), the first The internal location information of each refinement layer The calculation satisfies the following formula: in, The coordinates of the teacher votes output from the previous layer; (·)express The distance between any two points in the equation; (·) represents a masking operation, which adds a very large negative number to all terms except the K minimum values, where K is a positive integer; The mask multi-head self-attention The calculation satisfies the following formula: in, The characteristic of teachers voting for the output of the previous layer; It comes from different points of attention. The connection of attention features, where H represents the number of attention heads; This indicates that the softmax operation is performed row by row; It is the number of channels in the query, key, and value; (·) represents the linear projection operation at the end of the mask multi-head attention; , (·)and (·) represent the first digits of the query, key, and value, respectively. Linear projection of the head; Features The transpose of the matrix; In step (34), the first The coordinates of the teacher votes output by each refinement layer. and characteristics The calculations satisfy the following formulas: in, To mask multi-head self-attention, For corresponding Multilayer perceptron; The coordinates of the teacher votes output from the previous layer; For masking multiple-head cross-attention; (4) Input the refined teacher votes into the foreground mask module to generate a foreground mask for the refined votes; (5) The student votes are input into the feature alignment module to generate aligned student votes; (6) Calculate the total loss for training the student model. 、 Voting consistency distillation loss and refinement loss are used to update the training of the student model, the foreground mask module, and the refinement module, respectively. (7) Deploy the trained student model to the application terminal and use the student model to efficiently detect indoor three-dimensional targets.

2. The indoor 3D target detection method based on knowledge distillation and voting refinement as described in claim 1, characterized in that... In step (4), the foreground mask The calculation satisfies the following formula: in, The first in the refined teacher voting coordinates A more detailed coordinate system for voting; For the first The true coordinates of the center of the i-th object, the th The object is related to the first The voting process involves refining the representation of the most closely related real-world object. It is the first The lengths of a true bounding box along the x, y, and z axes; It is a non-zero positive value, used to avoid the denominator of the calculation being zero; Represents the Frobenius norm; for function; M This represents the number of votes cast.

3. The indoor 3D target detection method based on knowledge distillation and voting refinement as described in claim 1, characterized in that... In step (5), the feature alignment module is implemented by a series of multilayer perceptron layers for dimensional alignment between the teacher votes and the student votes; Aligned student votes The calculation satisfies the following formula: in, Vote for students; It is a projection function.

4. The indoor 3D target detection method based on knowledge distillation and voting refinement as described in claim 1, characterized in that... In step (6), the voting consensus distillation loss The calculation satisfies the following formula: in, For the first A detailed feature of teacher voting; For the first A more detailed framework for teacher voting; For the first A detailed prospect mask for teacher voting; For the first Aligned features obtained by inputting an object into the student model; For the first The aligned coordinates of the first object input into the student model; the... The object is related to the first The most realistic object that best matches the detailed voting result; Represents the Frobenius norm; M The number of votes; The total loss for training the student model L The calculation satisfies the following formula: in, For the regression loss in VoteNet, For classification loss in VoteNet, λ and η are two hyperparameters used to balance different loss terms.

5. The indoor 3D target detection method based on knowledge distillation and voting refinement as described in claim 1, characterized in that... In step (6), the refinement loss The calculation satisfies the following formula: in, For the first A more detailed framework for teacher voting; For the first The ground-based real center of the voting object; It generates detailed voting results. The corresponding seed; Indicates a pointer function used to indicate the seed. Is it on an object? M This represents the number of votes cast.

6. An indoor three-dimensional target detection system based on knowledge distillation and voting refinement, used to implement the indoor three-dimensional target detection method according to any one of claims 1 to 5, characterized in that... It includes a teacher model, a student model, a refinement module, a foreground mask module, and a feature alignment module; among them, The teacher model uses the VoteNet model, and the student model uses a voting-based 3D detection network. Furthermore, the number of feature channels in each layer of the student model is set in a lightweight manner according to the hardware resource configuration of the application terminal. The refinement module is used to refine teacher votes to correct inaccurate teacher votes in the supervision of the student model during knowledge distillation. The refinement module consists of multiple refinement layers, each of which includes a masked multi-head self-attention layer, a masked multi-head cross-attention layer, a self-distance calculation module, and a cross-distance calculation module. The foreground masking module is used to mask the refined teacher votes so that background information can be used to focus more on the votes within the target bounding box during the distillation process. The feature alignment module is used to perform alignment operations on student votes, realizing dimensional alignment between student votes and teacher votes; the feature alignment module is implemented by a series of multilayer perceptron layers. The teacher model, in collaboration with the refinement module, the foreground mask module, and the feature alignment module, employs knowledge distillation and voting refinement methods to help train the student model and improve indoor 3D object detection performance.

7. An electronic device, characterized in that... It includes a processor and a memory, the processor and the memory being coupled together; wherein the memory is used to store computer programs; The processor is used to run a computer program stored in the memory to execute the indoor three-dimensional target detection method based on knowledge distillation and voting refinement as described in any one of claims 1 to 5.

Images