A method for constructing a multi-modal ViT visual model structure considering a modal missing situation

Abstract

This invention discloses a method for constructing a multimodal ViT visual model structure that considers modality missing scenarios. The method addresses the limitations of traditional ViT multimodal visual models in simulating modality missing scenarios through data augmentation during model training, and the lack of consideration for modality missing scenarios in model structure design. It proposes a multimodal ViT visual model structure that considers modality missing scenarios, with the following construction steps: Step 1, Single-modal branch structure design; Step 2, Missing modality detection; Step 3, Projection of missing modalities; Step 4, Multimodal fusion ViT. This invention effectively handles modality missing scenarios, maintaining high performance and stable output even under modality missing conditions, improving model accuracy and robustness, and reducing potential security risks caused by modality missing.

Description

Technical Field

[0001] This invention belongs to the field of multimodal computer vision technology, and relates to a multimodal ViT vision model, specifically a method for constructing a multimodal ViT vision model structure that considers modality-deficient scenarios. Background Technology

[0002] With the rapid development of computer technology, computer vision technology has reached a relatively mature stage and is widely used in various tasks. Among them, multimodal vision models can obtain richer and more comprehensive information from data of different modalities compared to single-modal models, and utilize the complementary features between different modalities to make the output results have higher accuracy and robustness.

[0003] In traditional multimodal ViT vision models, the ability of data augmentation to simulate modality loss during model training is limited, and the model structure design has not yet taken into account the scenario of modality loss. Once a certain modality is completely lost due to situations such as sensor offline or data transmission errors, the accuracy and robustness of the model will drop sharply. In tasks with high safety requirements, such as autonomous driving, this may lead to serious safety hazards. Summary of the Invention

[0004] To address the issue that traditional multimodal ViT vision models do not adequately consider modality loss scenarios in their structural design, leading to a significant decrease in model accuracy and robustness under modality loss conditions, and potentially causing security risks, this invention provides a method for constructing a multimodal ViT vision model structure that considers modality loss scenarios. The model constructed using this method can maintain high performance even in modality loss scenarios, which is of great significance for ensuring the model's accuracy and robustness.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for constructing a multimodal ViT visual model structure that considers modality-deficient scenarios includes the following steps:

[0007] Step 1: Single-modal branch structure design:

[0008] Step 11: CameraViT Structural Design

[0009] CameraViT is used to segment images into patches for learning. The specific steps are as follows:

[0010] (1) Each sheet of length H I Width is W I Image Divide into N C The length is νcH Width is ν cW The patch, i.e.

[0011] (2) Expand each patch to a length of ν cH ×ν cW The 3×1 one-dimensional feature is then input into the MLP:

[0012]

[0013] in, These are the features of the input MLP. This is the output of the MLP, and the length D of both is... c Maintain consistency;

[0014] (3) Add a learnable embedding E to the feature sequence c and location embedding E that preserves location information posc The generated embedded sequence feature Camera Token is represented as:

[0015]

[0016] Where N is the number of patches, x class These are trainable parameters used for feature classification;

[0017] (4) Z0 is used as the original feature and input into the CameraViT encoder. The final output image feature H c Represented as:

[0018]

[0019] Where LN represents the LayerNormalization operation, L c Indicates the depth of the Transformer model;

[0020] Steps 1 and 2: LidarViT structural design:

[0021] LidarViT is used to process point cloud input from LiDAR. It processes the input by dividing the raw point cloud in 3D space into voxels. The specific steps are as follows:

[0022] (1) Passing through a length, width and height of ν lH ×ν lW ×ν lD The cube has length, width, and height H l ×W l ×D l The point cloud space is divided into N lX is a set of features containing t voxels. p Represented as:

[0023]

[0024] in, x represents the laser point within the voxel. i ,y i ,z i R represents the 3D coordinates of the i-th laser point. 3 Represents three-dimensional Euclidean space;

[0025] (2) Calculate the centroid (c) of all points in the voxel using the mean. x ,c y ,c z (and add the offset of each point relative to the centroid as a supplement, voxel feature set) Updated to:

[0026]

[0027] in, R represents the set of 3D coordinates of the i-th laser point and its offset complement. 6 Represents six-dimensional Euclidean space;

[0028] (3) Transform the updated feature set into a length of ν lH ×ν lW ×ν lD For a one-dimensional vector, then for each encoded feature i:

[0029]

[0030] The point-level feature labels are transformed into the feature space f by FCN. i In the middle, for f i Max pooling is performed element-wise on the data to obtain local fusion features. via f i and The connection obtains the connection characteristics.

[0031]

[0032] (4) Add a learnable embedding E to the feature sequence l and position embedding E posl For a length of D l The embedded sequence feature LidarToken is represented as:

[0033]

[0034] Where M is the number of voxels;

[0035] (5) Y0 is used as the original feature and input into the LidarViT encoder. The final output point cloud feature H l Represented as:

[0036]

[0037] Among them, L l The depth of the Transformer model;

[0038] Step 2: Detection of missing modes:

[0039] To detect missing modes, it is necessary to evaluate the importance of the feature sequence output by the encoder, and filter out features with zero or little information using a scoring function:

[0040]

[0041] Here, MSA (Multi-head self-attention) represents a multi-head self-attention layer, and MLP (Multilayer Perceptron) represents a multilayer perceptron. This represents the feature output of the l-th layer in the m-th modal encoder. express The rating, This represents the output of the MSA layer; the scoring function dynamically represents the importance of the feature sequence of the m-th modality at layer l. Once the importance score is lower than the threshold, intermodal projection will be performed to replace the corresponding modal features.

[0042] Step 3: Projection of missing modes:

[0043] Step 31: Projection of missing modes:

[0044] (1) Define a single feature projection as:

[0045]

[0046] Where, x m [n] represents the nth layer feature of the mth modality being projected, x m' [n] represents the n′ layer feature of the m′ modality being projected, and h is a shallow multilayer perceptron;

[0047] (2) If all features in a mode are considered, then the modal projection can be represented as a concatenation of feature sequence projections:

[0048]

[0049] (3) For features The pruning in the feature sequence dynamically detects low importance scores from all features, replacing unimportant features with projections of other embeddings. This dynamic process occurs before each encoding layer, therefore... Updated to:

[0050]

[0051] Where Ⅱ is the condition indicator, and its output is a mask vector. θ is the threshold; the operator ⊙ indicates element-wise multiplication;

[0052] The loss function is expressed as:

[0053]

[0054] Among them, L m This represents the feature loss of the m-th mode. The first norm is used to indicate whether intermodal projection substitution has been performed, and λ is a hyperparameter for evaluating the loss of different modes.

[0055] Step 3.2: Align the projected positions.

[0056] Since feature substitution destroys the positional information in the original features, in order to mitigate this problem, the positional embedding of the original feature sequence is preserved and the positional embedding in subsequent layers is frozen. Finally, the positional embedding is substituted before the task header. Regardless of whether the original feature sequence has been substituted, the model still retains the original positional information.

[0057] Step 4: Multimodal Fusion ViT

[0058] Step 4: 1. Use an MLP to connect image features H c and point cloud features H l To obtain N m A fusion feature encoding X m :

[0059] X m =MLP(H c ∪H L )

[0060] Step 42, in a length of D m Add learnable embedding E to the feature sequence m and position embedding E posm The fused feature embedding sequence feature FusionToken is represented as:

[0061]

[0062] Step 4.3: Final Output Feature H m Lth of the Transformer encoder m Layer output:

[0063]

[0064] Steps four and five: By adding different task heads, complete multimodal instance segmentation, object detection, and object tracking visual tasks.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] This invention addresses the limitations of traditional ViT multimodal vision models in simulating modality loss during training through data augmentation, and their failure to consider modality loss scenarios in their model structure design. It proposes a multimodal ViT vision model structure that incorporates modality loss scenarios. This invention effectively handles modality loss scenarios, maintaining high performance and stable output even under these conditions, improving model accuracy and robustness, and reducing potential security risks caused by modality loss. Attached Figure Description

[0067] Figure 1 A flowchart illustrating the overall process of constructing a multimodal ViT visual model structure that considers modality-deficient scenarios.

[0068] Figure 2 The structural design diagram of the multimodal ViT visual model considering modality loss scenarios. Detailed Implementation

[0069] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0070] This invention provides a method for constructing a multimodal ViT visual model structure that considers modality missing scenarios. The method learns the features of objects in different modalities and the projection relationships between these features in the branches, introduces a feature scoring mechanism to filter out missing modalities, and finally performs feature projection transformations between modalities to replace the features of objects in the missing modalities, thereby improving the model's accuracy and robustness. Figure 1 and Figure 2 As shown, the specific steps are as follows:

[0071] Step 1: Single-modal branch structure design:

[0072] Step 11: CameraViT Model Structure Design

[0073] The CameraViT model is used to segment images into patches for learning. The specific steps are as follows:

[0074] (1) Each sheet of length H I Width is W I Image Divide into N C The length is ν cH Width is ν cW The patch, i.e.

[0075] (2) Expand each patch to a length of ν cH ×ν cW The 3×1 one-dimensional feature is then input into the MLP:

[0076]

[0077] in, These are the features of the input MLP. This is the output of the MLP, and the length D of both is... c Maintain consistency.

[0078] (3) Add a learnable embedding E to the feature sequence c and location embedding E that preserves location information posc The generated embedded sequence feature Camera Token is represented as:

[0079]

[0080] Where N is the number of patches, x class These are trainable parameters used for feature classification.

[0081] (4) Z0 is used as the original feature and input into the CameraViT encoder. The final output image feature H c It can be represented as:

[0082]

[0083] Where LN represents the LayerNormalization operation, L c This indicates the depth of the Transformer model.

[0084] The CameraViT encoder is a Transformer encoder consisting of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block. Image features H cIt can accomplish 2D vision tasks by adding different task heads and has the ability to connect with other features from different modalities to perform multimodal tasks.

[0085] Steps 1 and 2: LidarViT structural design:

[0086] LidarViT is used to process point cloud input from LiDAR, processing the input by dividing the raw point cloud in 3D space into voxels. Due to the sparsity of point clouds, voxels that do not contain points are first removed to reduce computational burden; the number of non-empty voxels is used as the input length of the model. From voxels containing more than N points, N points are randomly retained, reducing computational burden while mitigating potential bias caused by inconsistent point cloud density. The specific steps are as follows:

[0087] (1) Passing through a length, width and height of ν lH ×ν lW ×ν lD The cube has length, width, and height H l ×W l ×D l The point cloud space is divided into N l The feature set of *t* voxels is represented as:

[0088]

[0089] Where, x i ,y i ,z i This represents the 3D coordinates of each point.

[0090] (2) Calculate the centroid (c) of all points in the voxel using the mean. x ,c y ,c z The feature set of voxel is updated by adding the offset of each point relative to the centroid as a supplement.

[0091]

[0092] (3) Transform the updated feature set into a length of ν lH ×ν lW ×ν lD For a one-dimensional vector, then for each encoded feature i:

[0093]

[0094] The point-level feature labels are transformed into the feature space f by FCN. iIn this model, FCN consists of a linear layer, a batch normalization layer, and a sigmoid linear units layer. For f... i Max pooling is performed element-wise on the data to obtain local fusion features. via f i and The connection can yield connection characteristics.

[0095]

[0096] (4) Add a learnable embedding E to the feature sequence l and position embedding E posl For a length of D l The embedded sequence feature LidarToken can be represented as:

[0097]

[0098] Where M is the number of voxels.

[0099] (5) Y0 is used as the original feature and input into the LidarViT encoder. The final output point cloud feature H l Represented as:

[0100]

[0101] Among them, L l This represents the depth of the Transformer model.

[0102] The LidarViT encoder is a Transformer encoder consisting of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block. Point cloud features H l 3D vision tasks can be accomplished by adding different task heads, and it has the ability to connect with other features from different modalities to perform multimodal tasks.

[0103] Step 2: Detection of missing modes:

[0104] To detect missing modes, it is necessary to evaluate the importance of the feature sequence output by the encoder, and filter out features with zero or little information using a scoring function:

[0105]

[0106] in, This represents the feature output of the l-th layer in the m-th modal encoder.

[0107] The scoring function dynamically represents the importance of the feature sequence of the m-th modality at layer l. Once the importance score is lower than the threshold (usually set to 0.01), intermodal projection will be performed to replace the corresponding modal features.

[0108] Step 3: Projection of missing modes:

[0109] Step 31: Projection of missing modes:

[0110] (1) Define a single feature projection as:

[0111]

[0112] Where, x m [n] represents the nth layer feature of the mth modality being projected, x m' [n] represents the n′ layer feature of the m′ modality being projected, and h is a shallow multilayer perceptron.

[0113] (2) If all features in a mode are considered, the mode projection can be represented as a concatenation of feature sequence projections:

[0114]

[0115] (3) For features The pruning in the feature sequence dynamically detects low importance scores from all features, replacing unimportant features with projections of other embeddings. This dynamic process occurs before each encoding layer, therefore... Updated to:

[0116]

[0117] Where Ⅱ is the condition indicator, and its output is a mask vector. θ is a threshold typically set to 0.01; the operator ⊙ indicates element-wise multiplication;

[0118] The loss function can be expressed as:

[0119]

[0120] Among them, L m This represents the feature loss of the m-th mode. The first norm is used to indicate whether intermodal projection substitution has been performed, and λ is a hyperparameter for evaluating the loss of different modes.

[0121] Step 3.2: Align the projected positions.

[0122] Since feature substitution destroys the positional information in the original features, this method preserves the positional embedding of the original feature sequence and freezes the positional embedding in subsequent layers to mitigate this problem. Finally, the positional embedding is substituted before the task header. Regardless of whether the original feature sequence has been substituted, the model still retains the original positional information.

[0123] Step 4: Multimodal Fusion ViT

[0124] Step 4: 1. Use an MLP to connect image features H c and point cloud features H l To obtain N m A fusion feature encoding X m :

[0125] X m =MLP(H c ∪H L )

[0126] Step 42, in a length of D m Add learnable embedding E to the fused feature sequence m and position embedding E posm The fused feature embedding sequence feature FusionToken can be represented as:

[0127]

[0128] Step 4.3: Final Output Feature H m The Lth modality of the ViT encoder is derived from multimodal fusion. m Layer output:

[0129]

[0130] The multimodal fusion ViT encoder is a Transformer encoder consisting of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block.

[0131] Steps four and five: By adding different task heads, multimodal visual tasks such as instance segmentation, object detection, and object tracking can be completed. The task head, as the output layer, is used to obtain the output of a specific task, and its general structure consists of FCN and SoftMax. Different hyperparameters are trained according to specific tasks such as classification, detection, and segmentation, and specific application scenarios. In addition, to improve model performance and reduce training costs, CameraViT and LidarViT are pre-trained, and then the fusion model is fine-tuned according to the specific task.

[0132] Example 1:

[0133] This embodiment verifies the method of the present invention using an object detection task head on the multimodal public dataset KITTI. Three common label classes—Car, Pedestrian, and Cyclist—were selected for the experiment, totaling 7518 frames. In this embodiment, a certain modality's data was randomly removed from all data at a percentage, and mAP was used as the evaluation metric. The implementation results are shown in Table 1.

[0134] Table 1 shows the implementation results in the KITTI dataset.

[0135]

[0136] The results show that the present invention has significantly higher accuracy than traditional multimodal target detection when modality is missing. As the proportion of missing modality data increases, the accuracy advantage of the present invention becomes more obvious, proving that the structural design of the present invention is effective in modality missing scenarios.

[0137] Example 2:

[0138] This embodiment verifies the method of the present invention using a semantic segmentation task head on the multimodal public dataset Nuscenes. Four common label classes—Car, Pedestrian, Bicycle, and Motorcycle—were selected for the experiment, totaling 404 frames. In this embodiment, a certain modality's data was randomly removed from all data at a percentage, and mIoU was used as the evaluation metric. The implementation results are shown in Table 2.

[0139] Table 2 shows the implementation results in the Nuscenes dataset.

[0140]

[0141] The results show that as the proportion of missing modalities increases, the advantages of the present invention compared with traditional multimodal semantic segmentation methods gradually increase, proving that the structural design of the present invention is effective for modality-missing scenarios.

[0142] As can be seen from the above embodiments, the method of the present invention can maintain high performance and stable output in the multimodal computer vision field, even in the case of modality loss, compared with the traditional multimodal ViT vision model, thus improving the accuracy and robustness of the model.

Claims

1. A method for constructing a multimodal ViT visual model structure considering modality loss scenarios, characterized in that... The method includes the following steps: Step 1: Single-modal branch structure design: Step 11: CameraViT Structural Design CameraViT is used to segment images into patches for learning; Steps 1 and 2: LidarViT structural design: LidarViT is used to process point cloud input from LiDAR by dividing the raw point cloud in 3D space into voxels for input processing. Step 2: Detection of missing modes: To detect missing modes, it is necessary to evaluate the importance of the feature sequence output by the encoder, and filter out features with zero or little information using a scoring function: Here, MSA represents a multi-head self-attention layer, and MLP represents a multilayer perceptron. This represents the feature output of the l-th layer in the m-th modal encoder. express The rating, This represents the output of the MSA layer; the scoring function dynamically represents the importance of the feature sequence of the m-th modality at layer l. Once the importance score is lower than the threshold, intermodal projection will be performed to replace the corresponding modal features. Step 3: Projection of missing modes: Step 31: Projection of missing modes: (1) Define a single feature projection as: Where, x m [n] represents the nth layer feature of the mth modality being projected, x m' [n] represents the n′ layer feature of the m′ modality being projected, and h is a shallow multilayer perceptron; (2) If all features in a mode are taken into account, then the mode projection can be represented as a concatenation of feature sequence projections: (3) For features The pruning in the feature sequence dynamically detects low importance scores from all features, replacing unimportant features with projections of other embeddings. This dynamic process occurs before each encoding layer, therefore... Updated to: Where Ⅱ is the condition indicator, and its output is a mask vector. θ is the threshold; the operator ⊙ indicates element-wise multiplication; The loss function is expressed as: Among them, L m This represents the feature loss of the m-th mode. The first norm is used to indicate whether intermodal projection substitution has been performed, and λ is a hyperparameter for evaluating the loss of different modes. Step 3.2: Align the projected positions. Since feature substitution destroys the positional information in the original features, in order to mitigate this problem, the positional embedding of the original feature sequence is preserved and the positional embedding in subsequent layers is frozen. Finally, the positional embedding is substituted before the task header. Regardless of whether the original feature sequence has been substituted, the model still retains the original positional information. Step 4: Multimodal Fusion ViT Step 4:

1. Use an MLP to connect image features H c and point cloud features H l To obtain N m A fusion feature encoding X m : X m =MLP(H c ∪H L ) Step 42, in a length of D m Add learnable embedding E to the feature sequence m and position embedding E posm The fused feature embedding sequence feature Fusion Token is represented as: Step 4.3: Final Output Feature H m Lth of the Transformer encoder m Layer output: Steps four and five: By adding different task heads, complete multimodal instance segmentation, object detection, and object tracking visual tasks.

2. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 1, characterized in that... The specific steps of each step are as follows: (1) Each sheet of length H I Width is W I Image Divide into N C The length is ν cH Width is ν cW The patch, i.e. (2) Expand each patch to a length of ν cH ×ν cW The 3×1 one-dimensional feature is then input into the MLP: in, These are the features of the input MLP. This is the output of the MLP, and the length D of both is... c Maintain consistency; (3) Add a learnable embedding E to the feature sequence c and location embedding E that preserves location information posc The generated embedded sequence feature Camera Token is represented as: Where N is the number of patches, x class These are trainable parameters used for feature classification; (4) Z0 is used as the original feature and input into the CameraViT encoder. The final output image feature H c Represented as: Hc=LN(Z Lc ) Where LN represents the LayerNormalization operation, L c This indicates the depth of the Transformer model.

3. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 1, characterized in that... The CameraViT encoder is a Transformer encoder consisting of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block.

4. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 1, characterized in that... The specific steps of steps one and two are as follows: (1) Passing through a length, width and height of ν lH ×ν lW ×ν lD The cube has length, width, and height H l ×W l ×D l The point cloud space is divided into N l X is a set of features containing t voxels. p Represented as: in, x represents the laser point within the voxel. i ,y i ,z i R represents the 3D coordinates of the i-th laser point. 3 Represents three-dimensional Euclidean space; (2) Calculate the centroid (c) of all points in the voxel using the mean. x ,c y ,c z (and add the offset of each point relative to the centroid as a supplement, voxel feature set) Updated to: in, R represents the set of 3D coordinates of the i-th laser point and its offset complement. 6 Represents six-dimensional Euclidean space; (3) Transform the updated feature set into a length of ν lH ×ν lW ×ν lD For a one-dimensional vector, then for each encoded feature i: The point-level feature labels are transformed into the feature space f by FCN. i In the middle, for f i Max pooling is performed element by element to obtain local fusion features. via f i and The connection obtains the connection characteristics. (4) Add a learnable embedding E to the feature sequence l and position embedding E posl For a length of D l The embedded sequence feature LidarToken is represented as: Where M is the number of voxels; (5) Y0 is used as the original feature and input into the LidarViT encoder. The final output point cloud feature H l Represented as: Among them, L l This represents the depth of the Transformer model.

5. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 4, characterized in that... The LidarViT encoder is a Transformer encoder consisting of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block.

6. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 4, characterized in that... The FCN consists of a linear layer, a BatchNormalization layer, and a Sigmoid LinearUnits layer.

7. The method for constructing a multimodal ViT visual model structure considering modality loss scenarios according to claim 1, characterized in that... The multimodal fusion ViT encoder is a Transformer encoder composed of alternating MSA layers and MLP blocks. Its hyperparameters are obtained through pre-training. Layer normalization is applied before each MLP block, and residual connections are applied after each MLP block.

Images