A lightweight three-dimensional semantic scene completion method and system based on triple mask semantic distillation
Patent Information
- Application Number
- CN202611145480.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-30
- Publication Date
- 2026-10-20
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the fields of autonomous driving environment perception, camera-based 3D semantic scene completion, knowledge distillation, and lightweight deep neural network technology. In particular, it relates to a lightweight 3D semantic scene completion method and system based on triple mask semantic distillation. Specifically, it relates to a method and system for predicting the occupancy state and semantic category label of each voxel after dividing the 3D space around the vehicle into a voxel grid. Background Technology
[0002] Autonomous driving systems need to understand the three-dimensional spatial environment surrounding the vehicle. Two-dimensional object detection or two-dimensional semantic segmentation can only provide the object category and location on the image plane, making it difficult to directly represent occluded areas, passable spaces, and the distribution of objects in three-dimensional space. The three-dimensional semantic scene completion task divides the space around the vehicle into a voxel grid and predicts the occupancy state and semantic category for each voxel, thus providing a more complete spatial representation for autonomous driving environment perception, path planning, and risk assessment.
[0003] Existing camera-input-based 3D semantic scene completion models typically employ large 2D image backbone networks and 3D Transformer spatial propagation modules. While these models offer good spatial modeling capabilities, their high parameter count, floating-point computation cost, and inference latency limit their application in edge devices or in-vehicle computing platforms. To reduce deployment overhead, the original model can be lightweighted. However, directly compressing the number of channels in the 3D Transformer or reducing the number of spatial propagation layers can easily degrade the model's ability to complete occluded and invisible regions.
[0004] Knowledge distillation can use a high-performance teacher model to guide the training of a lightweight student model, while general distillation methods typically teach the student model directly from the soft labels or intermediate features output by the teacher model. However, in 3D semantic scene completion tasks, the teacher model is not reliable across all voxels and categories. Some voxels may correspond to unknown regions or invalid labels, the teacher predictions for some voxels may have low confidence, and the quality of teacher predictions for some long-tail categories may be unstable. If the student model unconditionally learns all the outputs of the teacher model, then erroneous teacher predictions for low-confidence voxels, invalid voxels, and low-reliability categories may be passed to the student model, thus affecting the semantic completion performance of the lightweight student model.
[0005] Therefore, existing technologies still require a reliable distillation method suitable for lightweight 3D semantic scene completion models. This method should be able to filter reliable voxel-level semantic information in the teacher model output while maintaining the 3D spatial propagation capability, reduce the interference of low-quality teacher supervision on the student model, and improve the 3D semantic completion effect of lightweight student models. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a lightweight 3D semantic scene completion method based on triple mask semantic distillation.
[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0008] In a first aspect, the present invention provides a lightweight 3D semantic scene completion method based on triple mask semantic distillation, characterized by comprising the following steps: S1: Acquire image data captured by the vehicle-mounted camera and the corresponding three-dimensional voxel semantic labels; S2: Input the image data into the teacher model and obtain the voxel-level semantic Logit output by the teacher model; S3: Perform softmax processing on the voxel-level semantic Logit output by the teacher model to obtain the class probability distribution of the teacher model at each voxel position, and obtain the maximum class probability of the teacher and the predicted class of the teacher at each voxel position based on the class probability distribution. S4: Input the image data into the student model and obtain the voxel-level semantic Logit output by the student model; S5: Construct an effective voxel mask based on the three-dimensional voxel semantic labels, construct a teacher confidence mask based on the teacher's maximum class probability, and construct a class whitelist mask based on the class reliability index corresponding to the teacher's predicted class. S6: Construct a joint distillation mask based on the effective voxel mask, the teacher confidence mask, and the category whitelist mask, and determine the voxel positions participating in semantic distillation based on the joint distillation mask; S7: Apply temperature-controlled softmax processing to the voxel-level semantic Logit outputs of the teacher model and the student model respectively to obtain the teacher semantic probability distribution and the student semantic probability distribution; calculate the semantic distillation loss based on the teacher semantic probability distribution and the student semantic probability distribution at the voxel positions involved in semantic distillation. S8: Calculate the supervised loss of the student model based on the three-dimensional voxel semantic labels, and add the class balance Focal loss for auxiliary supervision; S9: Enable the corresponding loss terms in semantic distillation loss, supervision loss, and class balance Focal loss according to different training stages, and update the parameters of the student model; after training is completed, only the trained student model is used for 3D semantic scene completion prediction during the inference stage.
[0009] Optionally, in step S2, the teacher model adopts a three-dimensional semantic scene completion network, whose two-dimensional backbone network is ResNet-50, with 1024 FPN input channels and 128 FPN output channels; in step S4, the student model replaces ResNet-50 in the teacher model with ResNet-18, and reduces the FPN input channels from 1024 to 256.
[0010] Optionally, the voxel-level semantic Logit output by the teacher model and the voxel-level semantic Logit output by the student model are represented by a five-dimensional tensor, the formula of which is:
[0011] in, Indicates batch size. Represents the number of semantic categories. H , W and D These represent the dimensions of the voxel mesh in the vertical, horizontal, and vertical heights, respectively.
[0012] Optionally, in step S5, the effective voxel mask is generated based on voxels whose tag value is not 255 in the three-dimensional voxel semantic tags.
[0013] Optionally, in step S5, constructing the teacher confidence mask based on the teacher's maximum class probability specifically includes: Calculate the maximum class probability for teachers:
[0014] in, voxel position i The maximum class probability of teachers; For the teacher model at voxel positions The above belongs to the category The class probability; C Indicates the total number of semantic categories; c Indicates the current semantic category; The voxels constituting the teacher confidence mask are selected by comparing the maximum class probability with a pre-set confidence threshold. The selection rules are as follows:
[0015] in, For the teacher model at voxel positions The prediction confidence level; voxel position i It meets the confidence mask filtering requirements; Indicates voxel position i The confidence mask filtering requirements are not met. To pre-set the reliability threshold; Finally, all voxels that meet the screening requirements are used to construct a teacher confidence mask. .
[0016] Optionally, in step S5, constructing the category whitelist mask based on the category reliability index corresponding to the teacher's predicted category specifically includes: Based on the teacher model at voxel locations Based on the category probability distribution, determine the teacher's predicted category for each voxel:
[0017] in, Indicates the teacher model at voxel positions The predicted semantic category Indicates the teacher model at voxel positions The above belongs to the category c The class probability; C Indicates the total number of semantic categories; c Indicates the current semantic category; Using teacher prediction categories as the index, query the teacher model category-level history. IoU The table yields the corresponding category reliability index, where the category reliability index is the category-level historical intersection-union ratio, expressed as:
[0018] The historical intersection-union ratios at the category level are compared with preset category thresholds to filter voxels that constitute the category whitelist mask. The filtering rules are as follows:
[0019] in, =1 indicates that the teacher model is at the voxel position. i Meets the filtering requirements of the category whitelist mask; Indicates voxel position i The filtering requirements for the category whitelist mask are not met; Preset category threshold; Finally, a whitelist mask of all voxel composition categories that meet the screening requirements will be created. .
[0020] Optionally, in step S6, the joint distillation mask Represented as:
[0021] in, For effective voxel masks; For teacher confidence mask; This is the category whitelist mask.
[0022] Optionally, in step S7, the semantic distillation loss is the KL divergence loss after triple masking, expressed as:
[0023] in, i Indicates the position of a voxel; Indicates voxel position i Whether through a combined distillation mask, Indicates KL divergence; Indicates voxel position i Teacher semantic probability distribution; Indicates voxel position i The semantic probability distribution of students; Indicates the distillation temperature.
[0024] Optionally, in step S9, the training phase includes a first training phase and a second training phase, wherein the first training phase trains the student model based on supervised loss; and the second training phase trains the student model based on supervised loss, triple mask semantic distillation loss, and class balance Focal loss.
[0025] Secondly, the present invention provides a lightweight 3D semantic scene completion system based on triple mask semantic distillation, characterized in that it includes: The image acquisition module is used to acquire image data collected by the vehicle-mounted camera and the corresponding three-dimensional voxel semantic labels; The teacher prediction module is used to input image data into the teacher model and obtain the voxel-level semantic Logit output by the teacher model; the voxel-level semantic Logit output by the teacher model is processed by softmax to obtain the class probability distribution of the teacher model at each voxel position, and based on the class probability distribution, the maximum class probability of the teacher at each voxel position and the predicted class of the teacher are obtained. The student prediction module is used to input image data into the student model and obtain the voxel-level semantic Logit output by the student model. The triple mask construction module is used to construct an effective voxel mask based on the three-dimensional voxel semantic labels, a teacher confidence mask based on the teacher's maximum class probability, and a class whitelist mask based on the class reliability index corresponding to the teacher's predicted class. It also constructs a joint distillation mask based on the effective voxel mask, the teacher confidence mask, and the class whitelist mask, and determines the voxel positions participating in semantic distillation based on the joint distillation mask. The distillation loss calculation module is used to perform temperature-controlled softmax processing on the voxel-level semantic Logit output by the teacher model and the student model respectively to obtain the semantic probability distribution of the teacher and the semantic probability distribution of the student; at the voxel positions involved in semantic distillation, the semantic distillation loss is calculated based on the semantic probability distribution of the teacher and the semantic probability distribution of the student. The supervised loss calculation module is used to calculate the supervised loss of the student model based on the semantic labels of the three-dimensional voxels, and to add the class balance Focal loss for auxiliary supervision. The training optimization module is used to enable the corresponding loss terms in semantic distillation loss, supervision loss, and class balance Focal loss according to different training stages, and update the parameters of the student model. The inference output module is used to perform 3D semantic scene completion prediction using only the trained student model during the inference phase after training is completed.
[0026] The beneficial effects of this invention are: 1. This invention filters teacher supervision signals through a triple mask, wherein the effective voxel mask reduces the impact of invalid voxels on distillation loss; the teacher confidence mask reduces the misleading effect of low-confidence teacher predictions on student models; and the category whitelist mask reduces the transmission of erroneous knowledge on low-reliability categories; thereby improving the reliability of voxel-level semantic Logit distillation.
[0027] 2. This invention applies triple mask semantic distillation to a lightweight 3D semantic scene completion student model. While reducing the computational overhead of the 2D image, the student model can still learn category relationships through the filtered teacher semantic distribution, thereby mitigating the decline in semantic prediction performance caused by lightweighting.
[0028] 3. This invention employs cosine annealing semantic Logit distillation, which allows the student model to learn smoother class relationships in the early stages of distillation and obtain more explicit class boundary constraints in the later stages of distillation, thereby improving the stability of distillation training.
[0029] 4. This invention alleviates the problem of insufficient long-tail category learning in the 3D semantic scene completion task by using category-balanced Focal-assisted loss, and controls the number of candidate voxels input to the 3D spatial propagation module by using Top-K candidate voxels, thereby limiting the computational scale under abnormal conditions and improving the stability of the model training and inference process.
[0030] 5. The triple mask semantic distillation of the present invention can enable the student model to learn a more reliable voxel-level semantic distribution while filtering out low-quality teacher supervision. As a result, the lightweight student model can still maintain results close to those of the teacher model in multi-class semantic voxel prediction after compressing the two-dimensional backbone network and feedforward mapping channel, thus maintaining the prediction ability of the main semantic categories while reducing computational overhead. Attached Figure Description
[0031] Figure 1 This is an overall flowchart of an embodiment of the present invention; Figure 2 This is a schematic diagram comparing the structures of the teacher model and the lightweight student model in an embodiment of the present invention; Figure 3 This is a schematic diagram of the triple-masked semantic Logit distillation process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the phased training and temperature annealing scheduling in an embodiment of the present invention; Figure 5 This is a schematic diagram of the lightweight 3D semantic scene completion system according to an embodiment of the present invention. Detailed Implementation
[0032] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.
[0033] Example 1 This embodiment provides a lightweight 3D semantic scene completion method based on triple mask semantic distillation, the specific process of which is as follows: Figure 1 As shown, it includes the following steps: S101: Acquire image data collected by the vehicle-mounted camera and the corresponding 3D voxel semantic labels, and obtain the category reliability index table of the teacher model. The 3D voxel semantic labels are used to represent the occupancy state and semantic category of each voxel in the voxel space; the category reliability index table is used to represent the historical prediction reliability of the teacher model in each semantic category, where the category reliability index is the category-level historical intersection-union ratio of the teacher model on the validation set, or a pre-set category reliability table can be used.
[0034] S102, input the image data into the frozen teacher model to obtain the voxel-level semantic Logit output by the teacher model. The teacher model is a pre-trained (original) 3D semantic scene completion model, and its parameters remain fixed when training the student model.
[0035] S103, perform softmax processing on the voxel-level semantic Logit output by the teacher model to obtain the class probability distribution of the teacher model at each voxel position; based on the class probability distribution, obtain the maximum class probability of the teacher and the predicted class of the teacher at each voxel position. The maximum class probability of the teacher is used to construct the teacher confidence mask, and the predicted class of the teacher is used to query the class reliability index table.
[0036] S104: Input the image data into the lightweight student model to obtain the voxel-level semantic Logit output by the student model. The lightweight student model is constructed by compressing the two-dimensional image feature extraction module and the feedforward mapping module, while retaining the three-dimensional spatial propagation module.
[0037] S105. Construct an effective voxel mask based on the three-dimensional voxel semantic labels, construct a teacher confidence mask based on the teacher's maximum class probability, and construct a class whitelist mask based on the class reliability index corresponding to the teacher's predicted class.
[0038] like Figure 2 As shown, in this embodiment, the teacher model's 2D backbone network is ResNet-50, with 1024 FPN input channels and 128 FPN output channels. The 3D spatial propagation module includes 3 Cross Transformer layers and 2 Self Transformer layers. The student model adopts an asymmetric lightweight structure. The student model replaces the ResNet-50 in the teacher model with ResNet-18, reduces the FPN input channels from 1024 to 256, and maintains the FPN output channels at 128. In addition, the student model retains the 128-dimensional Transformer embedding, 3 Cross Transformer layers, and 2 Self Transformer layers, and compresses the feedforward mapping channels from 256 to 64. This structure reduces the computational cost at the 2D image end and the feedforward mapping end while preserving voxel spatial propagation capabilities.
[0039] The lightweighting method in this embodiment is asymmetric lightweighting. This method prioritizes compressing the two-dimensional image feature extraction module and the feedforward mapping module, while retaining the main structure of the three-dimensional spatial propagation module. This setting can avoid the decrease in spatial completion capability caused by directly compressing the number of three-dimensional Transformer layers or the embedded dimension.
[0040] In this embodiment, all voxel-level semantic Logit are represented as five-dimensional tensors, denoted as:
[0041] in, Indicates batch size. Represents the number of semantic categories. The dimensions of the voxel mesh in the vehicle coordinate system are defined in three spatial directions (vertical, horizontal, and vertical height). Each voxel position corresponds to a length of... The unnormalized class prediction vector. In this embodiment... The voxel mesh size is .
[0042] Effective voxel mask Generated based on 3D voxel semantic tags, this system is used to exclude "ignored" or "unknown" voxels, ensuring that distillation loss is not calculated in the invalid tag region. When a voxel tag belongs to an invalid, ignored, or unknown tag, the corresponding voxel does not participate in semantic distillation; when a voxel tag belongs to a valid semantic category, the corresponding voxel can be used as a candidate for distillation. A tag value of 255 in the 3D voxel semantic tag indicates an "ignored" or "unknown" voxel. The system... Generate an effective voxel mask. Voxels with a label value of 255 are not included in the semantic distillation and class equalization auxiliary loss calculation.
[0043] Teacher confidence mask The system generates class probability distributions based on the output of the teacher model. It inputs images from the vehicle-mounted camera into the frozen teacher model, obtaining the voxel-level semantic Logit output by the teacher model. Subsequently, it performs temperature-controlled softmax processing on this voxel-level semantic Logit to obtain the class probability distributions of the teacher model at each voxel location, such as... Figure 3 As shown.
[0044] Let the voxel-level semantic Logit output by the teacher model be... ,in Indicates the teacher model at voxel positions Top category The unnormalized prediction score. For After softmax processing, the teacher model at the voxel position The above belongs to the category The class probability is expressed as:
[0045] in, Represents the total number of semantic categories. This indicates a category index.
[0046] The system further calculates the voxel positions. The maximum class probability of teachers:
[0047] in, Indicates the teacher model at voxel positions The maximum value among all class probabilities.
[0048] Teacher confidence mask It can be represented as:
[0049] in, This indicates a pre-set confidence threshold. ;when At that time, it indicates that the teacher model is at the voxel position. The prediction confidence level on the above meets the distillation requirements; when At that time, the voxel position does not participate in semantic distillation.
[0050] In other words, the teacher confidence mask is used to retain voxels whose maximum class probability is greater than a preset confidence threshold. In the current embodiment, the preset confidence threshold is 0.85. If the teacher model's maximum class prediction probability for a voxel is not greater than 0.85, then that voxel is not included in the distillation.
[0051] Category whitelist mask This is generated based on a table of teacher prediction categories and category reliability indicators. Specifically, the system generates the model based on the teacher's prediction at voxel locations. The predicted teacher category is determined by the category probability distribution, and the predicted teacher category is used as an index to query the category reliability index, thereby obtaining the historical reliability index corresponding to the category.
[0052] Teacher model at voxel position The prediction categories are represented as follows:
[0053] in, Indicates the teacher model at voxel positions The predicted semantic category Indicates the teacher model at voxel positions The above belongs to the category The class probability.
[0054] The system is based on Using the index, query the historical IoU table for teacher models at the category level to obtain the historical intersection-union ratio corresponding to the predicted category of that teacher:
[0055] in, This represents the query function in the historical IoU table of the teacher model category level.
[0056] The category reliability metric can be the category-level historical IoU of the teacher model on the validation set, or it can be a category reliability value from a pre-defined category reliability table. In one implementation, the category-level historical IoU uses a pre-defined SemanticKITTI 20-class teacher category IoU list. For the teacher model at voxel positions... Predicted categories If the historical intersection-union ratio of this category satisfies:
[0057] If the voxel is selected, it is filtered through the category whitelist; otherwise, it is not included in semantic distillation. In one implementation, a preset category threshold is represented. Then the category whitelist mask. It can be represented as:
[0058] Among them, The value of the category whitelist mask at voxel position i indicates whether the teacher-predicted category of the voxel belongs to the historically reliable category. =1 indicates that the teacher model is at the voxel position. i Meets the filtering requirements of the category whitelist mask; Indicates voxel position i The filtering requirements for the category whitelist mask are not met; Preset category threshold; In other words, the category whitelist mask is used to retain voxels whose historical IoU of predicted categories is greater than a preset category threshold. In the current embodiment, the category-level historical IoU uses a preset SemanticKITTI 20-category teacher category IoU list, or it can be obtained from the category-level evaluation results of the teacher model on the validation set. In the current embodiment, the preset category threshold is 0.15. If the historical IoU of the teacher model's predicted category is not greater than 0.15, then the voxel corresponding to that category will not participate in distillation.
[0059] S106. Construct a joint distillation mask based on the effective voxel mask, the teacher confidence mask, and the category whitelist mask. The joint distillation mask is used to determine the voxel positions participating in semantic distillation.
[0060] In step S107, the voxel-level semantic Logit output by the teacher model and the student model are subjected to temperature-controlled softmax processing to obtain the teacher's semantic probability distribution and the student's semantic probability distribution for calculating the distillation loss. At the voxel positions after joint distillation masking, the semantic distillation loss is calculated based on the teacher's semantic probability distribution and the student's semantic probability distribution. In other words, the teacher model and the student model output voxel-level semantic Logit respectively, the system performs temperature-controlled softmax processing on the teacher's Logit and the student's Logit, and calculates the KL divergence loss at the voxel positions after joint distillation masking.
[0061] S108 calculates the supervision loss of the student model based on the semantic labels of the 3D voxels. Furthermore, the class-balanced Focal loss can be combined to provide auxiliary supervision for long-tailed classes and difficult voxels.
[0062] In this embodiment, the triple mask (joint mask) includes an effective voxel mask. Teacher confidence mask and category whitelist mask Combined distillation mask Represented as:
[0063] in, Used to filter out ignore voxels or unknown voxels; Voxels used to filter teacher models whose maximum class probability is greater than a preset reliability threshold; Used to filter voxels whose category reliability index is greater than a preset category threshold.
[0064] The effective voxel mask, teacher confidence mask, and category whitelist mask together constitute the joint distillation mask. Only voxels that simultaneously meet all three conditions participate in semantic Logit distillation. This mechanism can reduce the interference of invalid voxels, low-confidence voxels, and low-reliability categories on student model training.
[0065] For teacher model output voxel-level semantics Student model outputs voxel-level semantics The softmax processing with temperature is represented as:
[0066]
[0067] in, Indicates the distillation temperature. Represents the semantic probability distribution of teachers. Represents the semantic probability distribution of students. This indicates that the frozen teacher model outputs a voxel-level semantic Logit for the input vehicle camera image. This represents the voxel-level semantic Logit of the lightweight student model outputting the same input vehicle camera image.
[0068] The semantic distillation loss is the KL divergence loss after triple masking, expressed as:
[0069] in, Indicates the position of a voxel; Indicates voxel position Whether to use a combined distillation mask; Indicates KL divergence; Indicates voxel position i Teacher semantic probability distribution; Indicates voxel position i The semantic probability distribution of students; Indicates the distillation temperature.
[0070] In this embodiment, the distillation temperature is scheduled using cosine annealing. The distillation temperature can be calculated based on the current training round. The temperature scheduler calculates the temperature based on the current training round and updates it at the beginning of each training round. The maximum temperature is 3.0, and the minimum temperature is 1.0. The temperature scheduling can be expressed as:
[0071] in, Indicates the current training round; Indicates the maximum temperature; Indicates the minimum temperature; This indicates the current training progress.
[0072] In this embodiment, the training loss of the student model includes a supervision loss, a triple mask semantic distillation loss, and a class balance Focal auxiliary loss. The supervision loss is calculated from the student model's prediction results and the true voxel semantic labels; the triple mask semantic distillation loss is calculated from the voxel positions of the teacher model and the student model after joint masking; and the class balance Focal auxiliary loss is used to increase the training weights of long-tailed categories and difficult voxels.
[0073] The total loss function is expressed as:
[0074] in, This represents the supervised loss based on real voxel semantic labels. This represents the semantic distillation loss of the triple mask. This represents the class balance Focal auxiliary loss. and These represent the corresponding loss weights. In this embodiment, the semantic distillation loss weight is 1.0, and the category equalization Focal loss weight is 0.5.
[0075] S109: Activate the appropriate loss term based on the current training phase and update the parameters of the lightweight student model. In training epochs 1-5, the student model is trained solely based on supervised loss; in training epochs 6-15, the student model is trained using supervised loss, triple-mask semantic distillation loss, and class-balanced Focal-assisted loss. In other words, the training process includes a first training phase and a second training phase. The first training phase trains the student model solely based on 3D voxel semantic labels; the second training phase trains the student model using both 3D voxel semantic labels and the teacher model's voxel-level logit. The first training phase consists of training epochs 1-5, and the second training phase consists of training epochs 6-15. Figure 4 As shown.
[0076] After training, the inference phase uses only the trained lightweight student model for 3D semantic scene completion prediction. The system inputs images from the vehicle-mounted camera into the student model, which outputs the occupancy state and semantic category in 3D voxel space. The teacher model does not participate in computation during the inference phase, thus reducing the computational overhead of the inference phase.
[0077] In addition, this embodiment performs Top-K filtering on the candidate voxel set to ensure that the number of candidate voxels input to the 3D spatial propagation module does not exceed a preset upper limit. The Top-K candidate voxel filtering is based on the proposal score in the sample metadata. The system first filters candidate voxels with a proposal value greater than 0; when the number of candidate voxels exceeds the preset upper limit, the top K candidate voxels are selected as input to the 3D spatial propagation module according to their proposal scores from highest to lowest. In this embodiment, the candidate voxel K is set to 20000 during the training phase and 40000 during the inference phase. This setting controls the computational scale of the 3D spatial propagation module under abnormal candidate voxel counts, reduces the risk of memory fluctuations, and improves the stability of the training and inference processes.
[0078] The results of this embodiment are shown in Table 1. The final distilled student model achieved 12.97% mIoU and 43.62% IoU. Compared to the teacher model's 13.72% mIoU, the student model retained approximately 94.53% of the semantic completion performance. The number of 2D backbone parameters in the student model decreased by approximately 52.36%, 2D backbone FLOPs decreased by approximately 55.57%, and the total number of parameters decreased by approximately 21.77%. Under the unified forward inference velocimetry script, the forward inference speed of the student model improved from 0.1 FPS in the teacher model to 0.7 FPS. The category-level IoU results are shown in Table 2, where the differences represent the student model IoU minus the teacher model IoU.
[0079] Table 1 Comparison of Core Validation Results
[0080] Table 2 Comparison of Category-Level Results
[0081] Table 2 shows that the lightweight student model, after compressing the 2D backbone network and feedforward mapping channels, still maintains results close to the teacher model in multi-class semantic voxel prediction. The differences in categories such as road, vegetation, fence, terrain, and pole are relatively small, while categories such as person, bicyclist, trunk, terrain, pole, and traffic-sign show some improvement compared to the teacher model. These results indicate that triple-mask semantic distillation can filter out low-quality teacher supervision, enabling the student model to learn more reliable voxel-level semantic distributions, and maintaining the predictive ability of major semantic categories while reducing computational overhead.
[0082] In summary, this embodiment provides a lightweight 3D semantic scene completion method based on triple mask semantic distillation. This method uses a triple mask to filter reliable teacher supervision signals and trains a lightweight student model to complete voxel-level 3D semantic scene completion. Specifically, it includes: first, acquiring images from an onboard camera and 3D voxel semantic labels; the teacher model remains frozen and outputs a voxel-level semantic logit; the student model receives the same image input and outputs a voxel-level semantic logit; the system constructs a triple mask (joint mask) based on the real voxel labels and the teacher output; the triple mask includes an effective voxel mask, a teacher confidence mask, and a category whitelist mask; the system calculates the semantic distillation loss in the voxel regions filtered by the joint mask, and simultaneously calculates the supervision loss and category balance Focal auxiliary loss based on the real labels. The model updates the student model parameters based on the total loss; after training, the system only calls the student model for 3D semantic scene completion prediction.
[0083] Example 2 like Figure 5 As shown, this embodiment proposes a lightweight 3D semantic scene completion system based on triple mask semantic distillation, which is used to implement a lightweight 3D semantic scene completion method based on triple mask semantic distillation in the aforementioned embodiment. The system includes: an image acquisition module, a teacher prediction module, a student prediction module, a triple mask construction module, a distillation loss calculation module, a supervised loss calculation module, a training optimization module, and an inference output module.
[0084] The system comprises the following modules: an image acquisition module for acquiring images from an in-vehicle camera and corresponding 3D voxel semantic labels; a teacher prediction module for inputting the in-vehicle camera image into a frozen teacher model to obtain the voxel-level semantic Logit output by the teacher model; a softmax processing module for the voxel-level semantic Logit to obtain the class probability distribution of the teacher model at each voxel position; and a student prediction module for inputting the in-vehicle camera image into a lightweight student model to obtain the voxel-level semantic Logit output by the student model. The `git` module is used to construct a triple mask, a teacher confidence mask, and a category whitelist mask, and generate a joint distillation mask based on these three. The distillation loss calculation module is used to calculate the semantic distillation loss of the triple mask based on the voxel-level semantic logit of the teacher model and the student model. The supervision loss calculation module is used to calculate the supervision loss based on the 3D voxel semantic labels, and can also calculate the category balance Focal auxiliary loss. The training optimization module is used to update the parameters of the lightweight student model based on the total loss. The inference output module is used to output the 3D semantic scene completion result by calling only the lightweight student model after training.
[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0086] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A lightweight 3D semantic scene completion method based on triple mask semantic distillation, characterized in that, Includes the following steps: S1: Acquire image data captured by the vehicle-mounted camera and the corresponding three-dimensional voxel semantic labels; S2: Input the image data into the teacher model and obtain the voxel-level semantic Logit output by the teacher model; S3: Perform softmax processing on the voxel-level semantic Logit output by the teacher model to obtain the class probability distribution of the teacher model at each voxel position, and obtain the maximum class probability of the teacher and the predicted class of the teacher at each voxel position based on the class probability distribution. S4: Input the image data into the student model and obtain the voxel-level semantic Logit output by the student model; S5: Construct an effective voxel mask based on the three-dimensional voxel semantic labels, construct a teacher confidence mask based on the teacher's maximum class probability, and construct a class whitelist mask based on the class reliability index corresponding to the teacher's predicted class. S6: Construct a joint distillation mask based on the effective voxel mask, the teacher confidence mask, and the category whitelist mask, and determine the voxel positions participating in semantic distillation based on the joint distillation mask; S7: Apply temperature-controlled softmax processing to the voxel-level semantic Logit outputs of the teacher model and the student model respectively to obtain the teacher semantic probability distribution and the student semantic probability distribution; calculate the semantic distillation loss based on the teacher semantic probability distribution and the student semantic probability distribution at the voxel positions involved in semantic distillation. S8: Calculate the supervised loss of the student model based on the three-dimensional voxel semantic labels, and add the class balance Focal loss for auxiliary supervision; S9: Enable the corresponding loss terms in semantic distillation loss, supervision loss, and class balance Focal loss according to different training stages, and update the parameters of the student model; after training is completed, only the trained student model is used for 3D semantic scene completion prediction during the inference stage.
2. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S2, the teacher model uses a three-dimensional semantic scene completion network, with a two-dimensional backbone network of ResNet-50, 1024 FPN input channels, and 128 FPN output channels. In step S4, the student model replaces the ResNet-50 in the teacher model with ResNet-18 and reduces the FPN input channels from 1024 to 256.
3. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: The voxel-level semantic Logit output by the teacher model and the voxel-level semantic Logit output by the student model are represented by a five-dimensional tensor, and its formula is as follows: in, Indicates batch size. Represents the number of semantic categories. H , W and D These represent the dimensions of the voxel mesh in the vertical, horizontal, and vertical heights, respectively.
4. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S5, the effective voxel mask is generated based on voxels whose tag value is not 255 in the three-dimensional voxel semantic tags.
5. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S5, constructing the teacher confidence mask based on the teacher's maximum class probability specifically includes: Calculate the maximum class probability for teachers: in, voxel position i The maximum class probability of teachers; For the teacher model at voxel positions The above belongs to the category The class probability; C Indicates the total number of semantic categories; c Indicates the current semantic category; The voxels constituting the teacher confidence mask are selected by comparing the maximum class probability with a pre-set confidence threshold. The selection rules are as follows: in, For the teacher model at voxel positions The prediction confidence level; voxel position i It meets the confidence mask filtering requirements; Indicates voxel position i The confidence mask filtering requirements are not met. To pre-set the reliability threshold; Finally, all voxels that meet the screening requirements are used to construct a teacher confidence mask. .
6. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S5, constructing the category whitelist mask based on the category reliability index corresponding to the teacher's predicted category specifically includes: Based on the teacher model at voxel locations Based on the category probability distribution, determine the teacher's predicted category for each voxel: in, Indicates the teacher model at voxel positions The predicted semantic category Indicates the teacher model at voxel positions The above belongs to the category c The class probability; C Indicates the total number of semantic categories; c Indicates the current semantic category; Using teacher prediction categories as the index, query the teacher model category-level history. IoU The table yields the corresponding category reliability index, where the category reliability index is the category-level historical intersection-union ratio, expressed as: The historical intersection-union ratios at the category level are compared with preset category thresholds to filter voxels that constitute the category whitelist mask. The filtering rules are as follows: in, =1 indicates that the teacher model is at the voxel position. i Meets the filtering requirements of the category whitelist mask; Indicates voxel position i The filtering requirements for the category whitelist mask are not met; Preset category threshold; Finally, a whitelist mask of all voxel composition categories that meet the screening requirements will be created. .
7. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S6, the combined distillation mask Represented as: in, For effective voxel masks; For teacher confidence mask; This is the category whitelist mask.
8. The lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S7, the semantic distillation loss is the KL divergence loss after triple masking, expressed as: in, i Indicates the position of a voxel; Indicates voxel position i Whether through a combined distillation mask, Indicates KL divergence; Indicates voxel position i Teacher semantic probability distribution; Indicates voxel position i The semantic probability distribution of students; Indicates the distillation temperature.
9. A lightweight 3D semantic scene completion method based on triple mask semantic distillation according to claim 1, characterized in that: In step S9, the training phase includes a first training phase and a second training phase, wherein the first training phase trains the student model based on supervised loss; and the second training phase trains the student model based on supervised loss, triple mask semantic distillation loss, and class balance Focal loss.
10. A lightweight 3D semantic scene completion system based on triple mask semantic distillation, characterized in that, include: The image acquisition module is used to acquire image data collected by the vehicle-mounted camera and the corresponding three-dimensional voxel semantic labels; The teacher prediction module is used to input image data into the teacher model and obtain the voxel-level semantic Logit output by the teacher model. The voxel-level semantic Logit output by the teacher model is processed by softmax to obtain the class probability distribution of the teacher model at each voxel position. Based on the class probability distribution, the maximum class probability of the teacher at each voxel position and the predicted class of the teacher are obtained. The student prediction module is used to input image data into the student model and obtain the voxel-level semantic Logit output by the student model. The triple mask construction module is used to construct an effective voxel mask based on the three-dimensional voxel semantic labels, construct a teacher confidence mask based on the teacher's maximum class probability, and construct a category whitelist mask based on the category reliability index corresponding to the teacher's predicted category. A joint distillation mask is constructed based on the effective voxel mask, the teacher confidence mask, and the category whitelist mask. The voxel positions participating in semantic distillation are determined based on the joint distillation mask. The distillation loss calculation module is used to perform temperature-controlled softmax processing on the voxel-level semantic Logit output by the teacher model and the student model respectively to obtain the semantic probability distribution of the teacher and the semantic probability distribution of the student; at the voxel positions involved in semantic distillation, the semantic distillation loss is calculated based on the semantic probability distribution of the teacher and the semantic probability distribution of the student. The supervised loss calculation module is used to calculate the supervised loss of the student model based on the semantic labels of the three-dimensional voxels, and to add the class balance Focal loss for auxiliary supervision. The training optimization module is used to enable the corresponding loss terms in semantic distillation loss, supervision loss, and class balance Focal loss according to different training stages, and update the parameters of the student model. The inference output module is used to perform 3D semantic scene completion prediction using only the trained student model during the inference phase after training is completed.