A conditional dropout training method and system for RGB-D semantic segmentation

Abstract

The application discloses a kind of RGB-D semantic segmentation conditional discard training method and system, belong to computer vision depth learning multimodal perception technical field, using two-stage continuous training paradigm, first stage loads pre-training RGB-D semantic segmentation model and freezes original encoder and decoder;Second stage replicates original encoder as auxiliary encoder;Randomly construct complete, RGB missing and depth missing input during training, extract multi-layer features by double encoder, generate compensation features by zero initialization 1×1 convolution and residual fusion with original features, send into frozen decoder supervised training, only update auxiliary encoder and convolution injection layer;Inference stage adapts complete, RGB missing and depth missing three kinds of scenes, other modal input scenes can also be adapted subsequently.The application does not need to modify original network architecture, does not need to train from scratch, can greatly improve the segmentation robustness of model in single modal missing scene, training cost is low, and universality is strong.

Description

Technical Field

[0001] This invention relates to the field of computer vision deep learning multimodal perception technology, and in particular to a conditional dropout training method and system for RGB-D semantic segmentation. Background Technology

[0002] RGB-D semantic segmentation is an important task in the field of computer vision, aiming to assign semantic category labels to each pixel in an image using data from both RGB images and depth maps. With the widespread use of 3D sensors, RGB-D data is becoming increasingly readily available, and RGB-D semantic segmentation has broad application prospects in fields such as autonomous driving, robot navigation, augmented reality, and scene understanding.

[0003] Currently, mainstream RGB-D semantic segmentation models generally adopt a dual-branch fusion architecture, where RGB images and depth maps extract features through independent encoders, and then feature fusion is performed at multiple levels. Representative works include: DFormer, which uses a differential convolutional attention mechanism to encode RGB and depth data through a unified encoder; and Sigma, which uses a Siamese Mamba network structure and leverages a selectively structured state-space model to achieve multimodal semantic segmentation. These models have all achieved excellent segmentation performance under full modality input conditions.

[0004] Furthermore, existing RGB-D semantic segmentation models generally employ an end-to-end training paradigm, i.e., end-to-end supervised training is performed on complete RGB-D paired data to optimize the cross-entropy loss or panoptic quality loss function, driving the model parameters to converge towards the direction of optimal segmentation accuracy. Existing technologies suffer from the following core shortcomings: (1) Performance drops drastically when modalities are missing: Existing RGB-D models are only trained on complete modal data. When RGB or deep modalities are missing, the performance drops sharply. Experiments show that DFormer-B's mIoU drops from 55.6% to 25.5% when RGB is missing on the NYUv2 dataset, a decrease of 25.3 percentage points. (2) Inability to effectively utilize residual modalities: Even if a single modal (RGB or depth) contains sufficient semantic information, existing models cannot effectively extract and utilize this information, resulting in segmentation results that are far worse than those of specialized models that only use residual modalities; (3) Lack of robust design: Existing research mainly focuses on the optimization of fusion mechanism under complete modality, and there is insufficient research on the problem of missing modality, which limits the deployment of RGB-D model in safety-critical or resource-constrained environments; (4) High cost of retraining: If a model that supports the missing modalities needs to be trained from scratch, the architecture needs to be redesigned and a complete training process needs to be carried out, which is very costly in terms of both computation and time. Summary of the Invention

[0005] The purpose of this invention is to provide a conditional dropout training method and system for RGB-D semantic segmentation. The core of this method is to propose a two-stage continuous training paradigm that enhances the modality missing robustness of pre-trained RGB-D semantic segmentation models through a Conditional Dropout (ConD) module. Starting from an existing pre-trained RGB-D semantic segmentation model, this approach, without redesigning the original backbone network or retraining the entire model from scratch, utilizes modality conditional sampling, encoder copy freezing, zero-initialization convolutional injection, and multi-conditional supervised loss to enable the model to simultaneously adapt to complete RGB-D input, RGB missing input, and deeply missing input.

[0006] To achieve the above objectives, this invention provides a conditional dropout training method for RGB-D semantic segmentation, comprising a two-stage continuous training process and an inference deployment process: Step S1: First-stage model loading: Obtain the pre-trained original RGB-D semantic segmentation base model, which includes the original encoder. With the original decoder Freeze the original encoder Original decoder All network parameters; Step S2: Second-stage conditional dropout continuous training: Copy the original encoder to obtain the auxiliary encoder. During training iterations, modal input conditions are randomly sampled from a preset set of conditions. Based on the sampled conditions, corresponding inputs are constructed and fed into the original encoder and the auxiliary encoder for multi-layer feature extraction. The output features of each layer of the auxiliary encoder are processed by a corresponding zero-initialized 1×1 convolutional injection layer to generate auxiliary injection features. These features are then fused with the corresponding layer features of the original encoder using residual fusion to obtain multi-layer fused features. The multi-layer fused features are fed into the frozen original decoder to calculate the segmentation task loss. Only backpropagation is used to update the parameters of the auxiliary encoder and each convolutional injection layer. Step S3: Inference stage: Construct corresponding conditional inputs based on the actual available modalities, and feed the conditional inputs into the frozen original encoder and the trained auxiliary encoder respectively; the output of the auxiliary encoder is processed by a zero-initialized 1×1 convolutional injection layer to generate auxiliary injection features, and fused with the residual features of the corresponding layer of the original encoder; the fused features are fed into the frozen original decoder to output pixel-level semantic segmentation results.

[0007] Preferably, the original RGB-D semantic segmentation base model in the first stage is a two-branch fusion architecture model, including pre-trained models in DFormer or Sigma Siamese Mamba networks.

[0008] Preferably, in step S2, modal condition variables are set. ,in ; This indicates that the RGB mode is valid. This indicates that the RGB mode is missing; This indicates that the depth mode is valid. This indicates a lack of deep modes; The preset condition set is as follows: ; These correspond to complete RGB-D input, missing RGB input, and depth-missing input, respectively; each training iteration starts from the condition set. A medium probability is used to sample an input condition, and the sampling probability satisfies: .

[0009] Preferably, in step S2, based on the sampled conditions Construct the actual input tensor: ; ; in, , These are the original RGB image and the depth map, respectively. A zero tensor with the same dimensions as the RGB image. A zero tensor with the same dimensions as the depth map; , These are the RGB input and depth input after conditional discarding, respectively.

[0010] Preferably, the auxiliary encoder Network structure and original encoder Completely identical; all parameters of the original encoder are copied during initialization, as shown in the following expression: ; Original encoder parameters during training Original decoder parameters Keep them fixed, only auxiliary encoder parameters Parameters of each zero-initialized 1×1 convolutional injection layer Participate in gradient updates.

[0011] Preferably, in step S2, the specific method for multi-layer feature extraction and residual fusion is as follows: For those with An encoder with multiple feature levels, where the original encoder and the auxiliary encoder each perform multi-level feature extraction on the conditional input, to obtain... and ,in For the original encoder Layer output features, For the auxiliary encoder Layer output features; For each layer of auxiliary features Set the corresponding 1×1 convolutional injection layer Convolutional layer weights With bias Initialized to zero, the generated auxiliary injection features are: ; in, This represents a 1×1 convolution operation; The auxiliary injected features are residually fused with the corresponding layer features of the original encoder to obtain the fused features: ; in, For the first Layer injection strength coefficient; at the beginning of training , .

[0012] Preferably, the method for calculating the loss of the segmentation task is as follows: The fused features from each layer are input into the original decoder, and the pixel-level class probability distribution is obtained through the Softmax layer. The pixel-level cross-entropy loss is used as the task loss. ; in, For the set of valid pixels, For indicator functions, The total number of semantic categories. For pixels The true category, For pixels Belongs to the The predicted probability of a class; The overall optimization objective of the second stage is to minimize the task loss under the conditional sampling distribution. The optimization process only updates the parameters of the auxiliary encoder and the convolutional injection layer.

[0013] Preferably, in step S3, the specific processing method of the reasoning stage is as follows: During inference, the corresponding conditions are selected based on the available modalities: when both RGB and depth are available, a complete RGB-D input is used; when RGB is missing, an all-zero tensor is used to replace the RGB input; when depth is missing, an all-zero tensor is used to replace the depth input. The conditional inputs are fed into the frozen original encoder and the trained auxiliary encoder, respectively. The outputs of each layer of the auxiliary encoder are processed by the corresponding zero-initialized 1×1 convolutional injection layer to generate auxiliary injection features, which are then fused with the residual features of the corresponding layers of the original encoder. The fused features are fed into the frozen original decoder to output pixel-level semantic segmentation results.

[0014] A conditional dropout training system for RGB-D semantic segmentation includes: The model loading module is used to load the pre-trained original RGB-D segmentation model and freeze all parameters of the original encoder and decoder; The auxiliary encoding module is used to replicate the original encoder structure and parameters to generate a trainable auxiliary encoder, and to configure zero-initialized 1×1 convolutional injection layers for each feature level. The modal random sampling module is used to sample modal input conditions according to a preset probability during training iterations and construct corresponding conditional input tensors. The feature fusion module is used to perform multi-layer forward coding of the original encoder and the auxiliary encoder in parallel, calculate the compensation features of each level and fuse them with the original feature residuals; The loss training module is used to calculate the segmentation task loss. Backpropagation only updates the parameters of the auxiliary encoder and each convolutional injection layer. The inference output module is used to construct corresponding conditional inputs based on the actual input modality, call the original encoder, auxiliary encoder, zero convolutional injection layer and original decoder, complete feature fusion and output semantic segmentation results.

[0015] Therefore, the present invention employs the above-described conditional dropout training method and system for RGB-D semantic segmentation, which has the following advantages: (1) Significantly improved robustness to modal missing features: On the NYUv2 dataset, the mIoU of the DFormer-B model with RGB missing features increased from 25.5% to 43.5%, an improvement of 18.0 percentage points; the mIoU of the DFormer-B model with deep missing features increased from 35.2% to 48.3%, an improvement of 13.1 percentage points; the mIoU of the Sigma-S model with RGB missing features increased from 13.0% to 41.5%, an improvement of 28.5 percentage points; and the mIoU of the Sigma-S model with deep missing features increased from 50.3% to 52.9%, an improvement of 2.6 percentage points. (2) Maintaining complete modal performance: Under complete RGB-D input conditions, the mIoU of DFormer-B increased slightly from 55.6% to 55.7%, and the mIoU of Sigma-S increased from 56.6% to 57.2%, proving that the present invention can basically maintain complete modal performance while enhancing robustness; (3) Significantly reduced performance degradation: The average performance degradation of DFormer-B decreased from -25.3 percentage points to -9.8 percentage points, an improvement of 15.5 percentage points; the average performance degradation of Sigma-S decreased from -25.0 percentage points to -10.0 percentage points, an improvement of 15.0 percentage points; (4) Low training cost and strong versatility: Only the second stage of fine-tuning is required, without the need for complete training from scratch, which greatly reduces the computational cost and training time; it is compatible with mainstream RGB-D segmentation models such as DFormer and Sigma, and has been verified to be effective on the NYUv2 and SUN RGB-D dual standard datasets.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of a conditional dropout training method for RGB-D semantic segmentation in an embodiment of the present invention; Figure 2 This is a schematic diagram of the real-world modality loss problem in an embodiment of the present invention and a comparison diagram of the effects of the present invention; Figure 3 This is a flowchart of the two-stage ConD conditional drop-out training paradigm in an embodiment of the present invention; Figure 4 This is a qualitative comparison diagram of segmentation results under multimodal conditions in embodiments of the present invention; Figure 5 This is a comparative visualization of feature heatmaps in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] Example 1 like Figures 1-5 As shown in the figure, this embodiment proposes a conditional dropout training method for RGB-D semantic segmentation. The specific implementation steps are as follows: 1. Two-stage training architecture: Phase 1: Use the existing pre-trained RGB-D semantic segmentation model as initialization, directly using its published checkpoints without additional training.

[0021] Phase Two: Starting from the checkpoint of Phase One, continue training. Duplicate the original encoder to obtain the auxiliary encoder. and freeze the original encoder. and the original decoder All parameters are used to train only the auxiliary encoder and the zero-initialized convolutional injection layer. This process can be represented as: ; During the training, , Keep fixed. , These are trainable parameters.

[0022] in, Indicates the original encoder parameters. Indicates the original decoder parameters. Indicates the auxiliary encoder parameters, This indicates zero-initialization of the 1×1 convolutional injection layer parameters. Through these parameter constraints, the complete modal representations already learned by the original model are preserved, and the second stage only learns compensation information for missing modal conditions.

[0023] 2. Modal random input strategy: Define modal condition variables ,in ; This indicates that the RGB mode is valid. This indicates that the RGB mode is missing; This indicates that the depth mode is valid. This indicates a missing deep modality. The second phase of training uses the following set of conditions: ; in, Corresponds to complete RGB-D input, The corresponding RGB values ​​are missing, but the depth is valid. The corresponding RGB values ​​are valid but depth is missing. Each training iteration starts from the condition set. Randomly sample one input condition, preferably using equal probability sampling: ; Based on the sampled conditions Construct the actual input: ; ; in, Represents a zero tensor with the same dimensions as the RGB image. This represents a zero tensor with the same size as the depth map. and These are the RGB input and depth input after conditional discarding, respectively. In this method, modal loss does not require additional sensor data; instead, it is simulated during the training phase using zero-tensor substitution.

[0024] 3. Multi-layer feature extraction and auxiliary feature generation mechanism: For those with The encoders at each feature level, the original frozen encoder, and the auxiliary encoder each extract features from the conditional input: ; ; in, Indicates the original frozen encoder at the 1st Features of layer output, Indicates the auxiliary encoder at the 1st Features of the layer output. The auxiliary encoder progressively learns to extract compensatory semantic information from the remaining available modalities during the second stage of training.

[0025] 4. Zero-initialization convolutional feature injection and residual fusion mechanism: For each layer of auxiliary features Set up a 1×1 convolutional injection layer Its weights and biases are initialized to zero at the start of training. The auxiliary injected features are: ; Initialization satisfies , .

[0026] in, This represents a 1×1 convolution operation. and They represent the first Weights and biases of layer zero convolutions.

[0027] The auxiliary injected features are then residually fused with the original frozen encoder features: ; in, For the first The layer injection strength coefficient is preferably set to 1; in another embodiment, wherein... It can also be set as a learnable coefficient or a manually set coefficient.

[0028] Since the convolution parameters are zero at the beginning of training, the following conditions are met at the beginning of training: ; This feature ensures that the model output remains consistent with the original pre-trained model at the start of the second stage of training, preventing the auxiliary branch from disrupting the complete modal performance of the original model during the initialization phase; as training progresses, Gradually learn and inject effective semantic compensation information from partial modal inputs.

[0029] 5. Segmentation prediction mechanism: The fused features from each layer are input into the original frozen decoder to obtain the probability distribution of each pixel belonging to each semantic category: ; ; in, Indicates the first The pixel belongs to the first Predicted probability of class Indicates the total number of semantic categories. Indicates the first The predicted category for each pixel.

[0030] 6. Multi-condition supervised training objective function: For semantic segmentation tasks, pixel-level cross-entropy loss is the preferred method for task loss: ; in, This represents the set of valid pixels used in the loss calculation. This indicates the indicator function. If the original model uses panoramic quality loss, Dice loss, Lovasz loss, or other task-specific losses, the above cross-entropy loss can be replaced or combined with the corresponding task loss.

[0031] The overall optimization objective of the second stage is to minimize the task loss under the conditional sampling distribution: ; in, Represents the training dataset. This represents the aforementioned conditional sampling distribution. This represents the model after adding conditional dropout and zero convolution injection. This represents the cross-entropy loss or the task-specific loss used in the original model. During the optimization process... and Keep it fixed, update only and .

[0032] Optional Implementation: To further constrain output stability under full modal input, a full modality preservation loss can be introduced. Let the prediction distribution of the original pre-trained model under full RGB-D input be... The predicted distribution of the ConD model under the complete condition is as follows: The loss for preserving the complete mode is then defined as: ; When this optional hold constraint is introduced, the total loss is: ; in, express divergence, To maintain the loss weights. When this optional constraint is not used, If the value is set to 0, the present invention can still achieve stable training by freezing the original network and injecting zero-initialized residuals.

[0033] 7. Reasoning stage processing method: During the inference phase, conditional inputs are constructed based on the actual available modalities. When both RGB and depth are available, the complete modal conditions are used; when RGB is missing, the RGB missing conditions are used, and the RGB input is replaced with a zero tensor; when depth is missing, the depth missing conditions are used, and the depth input is replaced with a zero tensor. Subsequently, feature extraction, residual injection, feature fusion, and segmentation prediction are performed sequentially.

[0034] Example 2: Implementation of the ConD conditional dropout training method based on DFormer-B: This invention uses a unified hardware environment: two NVIDIA RTX3090 GPUs; input images are uniformly scaled to 480×640 resolution; the optimizer, learning rate scheduling strategy, and data augmentation scheme are completely based on the original configuration of the corresponding basic model in the original paper; the training batch size is BatchSize=8, and the number of iterations is Epoch=100; data augmentation includes random horizontal flipping and RGB color jitter; the dataset is divided into: NYUv2 training set with 795 images and test set with 654 images; the SUN RGB-D dataset contains 10335 images, of which 5285 are in the training set and 5050 are in the test set; the depth map is normalized to the [0,1] value range.

[0035] (1) Model selection: DFormer-B was used as the base model. This model has been pre-trained on the NYUv2 dataset and the full modal mIoU is 55.6%.

[0036] (2) Two-stage training process: The two-stage training paradigm adopted in this embodiment is as follows: Figure 3 As shown, the upper part of the figure corresponds to the first stage: directly using the checkpoints of the pre-trained RGB-D model, with all parameters of the encoder and decoder frozen; the lower part of the figure corresponds to the second stage: freezing the original network parameters, training a copy of the encoder, and randomly simulating three input conditions (complete, missing RGB, and missing depth); the auxiliary encoder and zero convolutional layer are labeled with trainable markers, and the original backbone network is labeled with frozen markers.

[0037] Phase 1: Load the officially released pre-trained checkpoints and freeze the original encoder. With the original decoder All parameters; Second stage: Copying the original encoder to obtain the auxiliary encoder For each feature level of the encoder, a 1×1 zero-initialized convolutional injection layer is set, with an injection intensity coefficient of [missing value]. Set to 1; in each training iteration, select the full RGB-D, RGB missing, or depth missing configuration with a 1 / 3 probability, based on the sampled conditions. The actual input is constructed; the dual encoders perform multi-layer feature extraction in parallel to complete residual fusion; the fused features are fed into the frozen decoder to calculate the cross-entropy loss, and only backpropagation is used to update the parameters of the auxiliary encoder and each convolutional injection layer.

[0038] (3) Training results: ① Complete mode mIoU: 55.7% (+0.1%); ② RGB missing mIoU: 43.5% (+18.0%); ③Deeply missing mIoU: 48.3% (+12.9%).

[0039] The improvement effect of this embodiment on the real-world modality missing problem is as follows: Figure 2 As shown. Figure 2 This demonstrates the performance degradation caused by modal missing issues (such as sensor failure, low light, and glare) in real-world RGB-D systems, and the effect of introducing the ConD module to maintain stable performance under different input conditions (RGB only, depth only, and full RGB-D). On the left, the original model can only work normally under full modality, and outputs serious errors when a single modality is missing. On the right, after introducing ConD, stable segmentation results can be output under all conditions.

[0040] Qualitative comparison of segmentation results under different modal conditions, for example Figure 4 As shown. Figure 4 The segmentation results under different modal conditions were qualitatively compared, showing the differences in segmentation quality between the baseline model (DFormer, Sigma) and the ConD augmented model under three conditions: complete modality, RGB missing, and depth missing. It can be seen that the baseline model has large-area class errors and contour breaks when the modality is missing. The segmentation mask output by this embodiment is highly consistent with the ground truth.

[0041] Example 3: ConD Enhancement Based on Sigma-S: (1) Model selection: Sigma-S was used as the basic model, with a complete modal mIoU of 56.6%; (2) Training settings: exactly the same as in Example 2, with zero-initialized convolutional injection layers configured for each level of the Sigma-S encoder, and the same three equal-probability modal sampling strategy and loss function used; (3) Training results: ① Complete mode mIoU: 57.2% (+0.6%); ② RGB missing mIoU: 41.5% (+28.5%); ③Deeply missing mIoU: 52.9% (+2.6%).

[0042] The qualitative segmentation result in this embodiment also corresponds to... Figure 4 The Sigma baseline and Sigma-ConD columns in the table show that, in scenarios where RGB modes are missing, this method significantly improves the segmentation accuracy of the Sigma model, effectively correcting the severe performance degradation of the baseline model when RGB modes are lost.

[0043] Example 4: Ablation Control Experiment (Verifying the necessity of the three core components and ensuring complete variable control): The unified baseline model is DFormer-B. The complementarity of the four core components—modal dropout, encoder duplication, backbone freezing, and zero convolution injection—was verified sequentially by controlling variables. The settings and results for each group are as follows: (1) Control group A (pure original baseline, no improvement): intact modality mIoU=51.2%, RGB missing mIoU=13.6%, depth missing mIoU=41.7%; (2) Control group B (only modal random sampling, no auxiliary encoder branch): the complete modal mIoU dropped significantly to 47.5%, and the pre-trained backbone weights were destroyed; (3) Control group C (copying the encoder but not freezing the original backbone network): complete mode mIoU=48.8%, the original accuracy of the complete mode is damaged; (4) Experimental group D (complete scheme of the present invention: random modal discarding + encoder copying + backbone freezing + zero convolutional residual injection): complete modality mIoU=51.4%, RGB missing mIoU=38.1%, depth missing mIoU=45.3%; Ablation experiment conclusion: Modal random sampling, encoder replication, backbone global freezing, and zero convolution injection are all indispensable components. Only by using them in combination can we simultaneously ensure the accuracy of complete modalities and the robustness to modal loss.

[0044] The characteristic action mechanism of each component can be obtained through Figure 5 The characteristic heatmap provides intuitive verification. Figure 5 The graph is displayed in three columns: the feature heatmap of the frozen original encoder output, the supplementary feature heatmap generated by the auxiliary encoder, and the total fused feature heatmap after adding the residuals. Rows correspond to three scenarios: complete modality, RGB missing, and depth missing. As can be seen from the graph, in the modality missing scenario, the feature responses of the original encoder contain many weak responses or blurry regions. The auxiliary encoder can specifically supplement the semantic responses of these regions, and the fused total features can completely cover all object regions in the scene, verifying the effectiveness of the residual injection mechanism.

[0045] Example 5: Implementation of a Conditional Dropout Training System for RGB-D Semantic Segmentation: The system is deployed on an industrial vision server equipped with dual RTX 3090 graphics cards and includes six functional modules: Model loading module: Reads the local pre-trained DFormer / Sigma weight file and calls the deep learning framework to freeze all parameters of the original encoder and decoder; Auxiliary encoding module: replicates the encoder network structure and weights to generate a trainable auxiliary encoder, and configures zero-initialized 1×1 convolutional injection layers for each feature level; Modal random sampling module: Reads the RGB-D paired dataset, samples modal conditions with equal probability of 1 / 3 during iterative training, and constructs the corresponding conditional input tensor; Feature fusion module: Performs multi-layer forward coding of the original encoder and auxiliary encoder in parallel, calculates the compensation features of each level and fuses them with the original feature residuals; Loss training module: feeds the fused features into the frozen decoder to generate a segmentation prediction map, calculates pixel-level cross-entropy loss, and backpropagation only updates the parameters of the auxiliary encoder and each convolutional injection layer. Inference output module: It is used to construct corresponding conditional inputs based on the actual input modality, call the original encoder, auxiliary encoder, zero convolutional injection layer and original decoder, complete feature fusion and output semantic segmentation results.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A conditional dropout training method for RGB-D semantic segmentation, characterized in that, This includes a two-stage continuous training process and an inference deployment process: Step S1: First-stage model loading: Obtain the pre-trained original RGB-D semantic segmentation base model, which includes the original encoder. With the original decoder Freeze the original encoder Original decoder All network parameters; Step S2: Second-stage conditional dropout continuous training: Copy the original encoder to obtain the auxiliary encoder. During training iterations, modal input conditions are randomly sampled from a preset set of conditions. Based on the sampled conditions, corresponding inputs are constructed and fed into the original encoder and the auxiliary encoder respectively for multi-layer feature extraction. The output features of each level of the auxiliary encoder are processed by the corresponding zero-initialized 1×1 convolutional injection layer to generate auxiliary injection features, which are then residually fused with the corresponding level features of the original encoder to obtain multi-layer fused features. The multi-layer fused features are fed into the frozen original decoder to calculate the segmentation task loss, and only the parameters of the auxiliary encoder and each convolutional injection layer are updated by backpropagation. Step S3: Inference phase: Construct corresponding conditional inputs based on the actual available modalities, and send the conditional inputs to the frozen original encoder and the trained auxiliary encoder respectively; The output of the auxiliary encoder is processed by a zero-initialized 1×1 convolutional injection layer to generate auxiliary injected features, which are then fused with the residual features of the corresponding layer of the original encoder. The fused features are fed into the frozen original decoder, which outputs pixel-level semantic segmentation results.

2. The conditional dropout training method for RGB-D semantic segmentation according to claim 1, characterized in that: The original RGB-D semantic segmentation base model in the first stage is a dual-branch fusion architecture model, including pre-trained models in DFormer or SigmaSiamese Mamba networks.

3. The conditional dropout training method for RGB-D semantic segmentation according to claim 1, characterized in that: In step S2, the modal condition variables are set. ,in ; This indicates that the RGB mode is valid. This indicates that the RGB mode is missing; This indicates that the depth mode is valid. This indicates a lack of deep modes; The preset condition set is as follows: ； These correspond to complete RGB-D input, missing RGB input, and depth-missing input, respectively; each training iteration starts from the condition set. A medium probability is used to sample an input condition, and the sampling probability satisfies: 。 4. The conditional dropout training method for RGB-D semantic segmentation according to claim 3, characterized in that: In step S2, based on the sampled conditions Construct the actual input tensor: ； ； in, , These are the original RGB image and the depth map, respectively. A zero tensor with the same dimensions as the RGB image. A zero tensor with the same dimensions as the depth map; , These are the RGB input and depth input after conditional discarding, respectively.

5. The conditional dropout training method for RGB-D semantic segmentation according to claim 1, characterized in that: Auxiliary encoder Network structure and original encoder Completely identical; all parameters of the original encoder are copied during initialization, as shown in the following expression: ； Original encoder parameters during training Original decoder parameters Keep them fixed, only auxiliary encoder parameters Parameters of each zero-initialized 1×1 convolutional injection layer Participate in gradient updates.

6. The conditional dropout training method for RGB-D semantic segmentation according to claim 1, characterized in that: In step S2, the specific method for multi-layer feature extraction and residual fusion is as follows: For those with An encoder with multiple feature levels, where the original encoder and the auxiliary encoder each perform multi-level feature extraction on the conditional input, to obtain... and ,in For the original encoder Layer output features, For the auxiliary encoder Layer output features; For each layer of auxiliary features Set the corresponding 1×1 convolutional injection layer Convolutional layer weights With bias Initialized to zero, the generated auxiliary injection features are: ； in, This represents a 1×1 convolution operation; The auxiliary injected features are residually fused with the corresponding layer features of the original encoder to obtain the fused features: ； in, For the first Layer injection strength coefficient; at the beginning of training , .

7. The conditional dropout training method for RGB-D semantic segmentation according to claim 1, characterized in that: The method for calculating the loss of the segmentation task is as follows: The fused features from each layer are input into the original decoder, and after passing through the Softmax layer, a pixel-level class probability distribution is obtained. Pixel-level cross-entropy loss is used as the task loss. ； in, For the set of valid pixels, For indicator functions, The total number of semantic categories. For pixels The true category, For pixels Belongs to the The predicted probability of a class; The overall optimization objective of the second stage is to minimize the task loss under the conditional sampling distribution. The optimization process only updates the parameters of the auxiliary encoder and the convolutional injection layer.

8. The conditional dropout training method for RGB-D semantic segmentation according to claim 7, characterized in that: In step S3, the specific processing method for the reasoning stage is as follows: During inference, the corresponding conditions are selected based on the actual available modalities: when both RGB and depth are available, a complete RGB-D input is used; when RGB is missing, an all-zero tensor is used to replace the RGB input; when depth is missing, an all-zero tensor is used to replace the depth input. The conditional inputs are fed into the frozen original encoder and the trained auxiliary encoder, respectively. The outputs of each layer of the auxiliary encoder are processed by the corresponding zero-initialized 1×1 convolutional injection layer to generate auxiliary injection features, which are then fused with the feature residuals of the corresponding layers of the original encoder. The fused features are fed into the frozen original decoder, which outputs pixel-level semantic segmentation results.

9. A conditional dropout training system for RGB-D semantic segmentation, used to implement the conditional dropout training method for RGB-D semantic segmentation according to any one of claims 1-8, characterized in that, include: The model loading module is used to load the pre-trained original RGB-D segmentation model and freeze all parameters of the original encoder and decoder; The auxiliary encoding module is used to replicate the original encoder structure and parameters to generate a trainable auxiliary encoder, and to configure zero-initialized 1×1 convolutional injection layers for each feature level. The modal random sampling module is used to sample modal input conditions according to a preset probability during training iterations and construct corresponding conditional input tensors. The feature fusion module is used to perform multi-layer forward coding of the original encoder and the auxiliary encoder in parallel, calculate the compensation features of each level and fuse them with the original feature residuals; The loss training module is used to calculate the segmentation task loss. Backpropagation only updates the parameters of the auxiliary encoder and each convolutional injection layer. The inference output module is used to construct corresponding conditional inputs based on the actual input modality, call the original encoder, auxiliary encoder, zero convolutional injection layer and original decoder, complete feature fusion and output semantic segmentation results.

Images