A multimodal fusion method and system based on a full-modal teacher model and a missing-modal student model

Abstract

The application discloses a kind of multi-modal fusion method and system based on full mode teacher model and missing mode student model.The method includes converting point cloud and residual perspective image into 3D point feature and category semantic feature corresponding to point cloud, 2D point feature and category semantic feature corresponding to residual perspective image;Point cloud category-point feature is aggregated into 3D point feature and category semantic feature corresponding to point cloud;Missing perspective image is cut into left, middle and right three parts according to the set proportion, left and right are completed by intra-modal information, middle is completed by inter-modal information;After the image feature of the left and right parts of the completed missing perspective image is converted into 2D point feature, it is connected with the 2D point feature of middle part to obtain the 2D point feature of missing perspective image, and the 2D point feature of missing perspective image and the 2D point feature of residual perspective image constitute the 2D point feature after completion.The robustness of multi-modal fusion model in missing scene is improved.

Description

Technical Field

[0001] This invention relates to the field of image signal processing technology, and in particular to a multimodal fusion method and system based on a full-modal teacher model and a missing-modal student model. Background Technology

[0002] In autonomous driving, semantic segmentation is a key task for vehicles to understand and make decisions about their environment. Compared with single-modal information, multimodal information can improve the perception accuracy of the model through complementary advantages. However, in real-world road scenarios, some modal data may inevitably be missing due to various factors. Therefore, how to handle the problem of incomplete modal data to improve the robustness of multimodal fusion segmentation models has become an urgent research challenge.

[0003] Currently, the handling of missing data in the field of multimodal fusion mainly falls into two categories. One category involves labeling the missing data, constructing a missing modality network model using a neural network, and then inputting the data into the model for training to obtain the optimal missing model. The other category utilizes multimodal fusion features to complete the missing data and performs adaptive dynamic fusion based on the completed features.

[0004] However, the former passively relies on the remaining modal features by training the missing modality model, without addressing the completion of missing features. The latter ignores common information between modalities and complementary information within modalities, resulting in limited accuracy in feature completion and thus affecting the robustness of the multimodal fusion model in missing scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion method and system based on a full-modal teacher model and a missing-modal student model, to solve the technical problems of low accuracy of feature completion and poor robustness of multimodal fusion caused by the lack of feature completion in existing technologies. The various technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention provides a multimodal fusion method based on a full-modal teacher model and a missing-modal student model. The steps based on the student model include:

[0008] The point cloud and residual view image are transformed into 3D point features and category semantic features corresponding to the point cloud, and 2D point features and category semantic features corresponding to the residual view image.

[0009] Aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features;

[0010] The missing viewpoint image is cropped into three parts: left, middle, and right, according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. Specifically, the image features of adjacent viewpoints of the missing viewpoint image are used as intramodal information. The predicted value of point cloud category-point feature is used as intermodal information and multiplied with the corresponding 3D point features of the middle part. The product is used as the 2D point features of the middle part.

[0011] After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the 2D point features corresponding to the remaining viewpoint images constitute the completed 2D point features.

[0012] In one or more embodiments, the left and right portions are completed using intramodal information, including:

[0013] For the left side, referencing the features of the adjacent left-view image of the missing view image, the left side of the missing view image is completed by cropping the features of the right half of the left view image.

[0014] For the right side, referencing the features of the adjacent right-view image of the missing view image, the right side of the missing view image is completed by cropping the features of the left half of the right view image.

[0015] In one or more embodiments, the point cloud is converted into its corresponding 3D point features and category semantic features, including:

[0016] Extract voxel features from the point cloud, and project the voxel features to obtain the corresponding 3D point features of the point cloud; obtain the semantic category probability distribution of the voxel features, and multiply the probability distribution with the voxel features to obtain the category semantic features of the point cloud.

[0017] In one or more embodiments, the remaining viewpoint image is converted into its corresponding 2D point features and category semantic features, including:

[0018] Extract image features from the remaining viewpoint image. Project the image features to obtain the corresponding 2D point features of the remaining viewpoint image. Obtain the semantic category probability distribution of the image features. Multiply the probability distribution with the image features to obtain the category semantic features corresponding to the remaining viewpoint image.

[0019] In one or more embodiments, the method further includes multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features, and segmentation of the fused features to obtain the student model segmentation prediction result.

[0020] In one or more embodiments, the method further includes obtaining fusion features corresponding to all modalities based on the teacher model, the steps of which include:

[0021] Voxel features are extracted from the point cloud. The voxel features are then projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained. The probability distribution is then multiplied with the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality.

[0022] Image features are extracted from multi-view images, and the image features are projected to obtain the 2D point features of the image corresponding to the full modality; the semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality;

[0023] Multimodal fusion is performed on the 3D point features of the point cloud, the semantic features of the point cloud category, the 2D point features of the image, and the semantic features of the image category corresponding to the full modality. The fused features are then segmented to obtain the segmentation prediction results of the teacher model.

[0024] In one or more embodiments, after obtaining the teacher model segmentation prediction result, the method further includes:

[0025] The point cloud 3D point features and point cloud category semantic features corresponding to the full modality are aggregated into point cloud category-point features; the image 2D point features and image category semantic features corresponding to the full modality are aggregated into image category-point features; and the common information ground value between the point cloud category-point features and the image category-point features corresponding to the full modality is captured.

[0026] In one or more embodiments, it further includes:

[0027] The distillation relationship between the teacher and student models is determined by the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​between the teacher model and the predicted values ​​of the intermodal information obtained from the student model.

[0028] Another aspect of the present invention provides a multimodal fusion system based on a full-modal teacher model and a missing-modal student model, comprising:

[0029] The first LiDAR point cloud branch is used to convert the point cloud into corresponding 3D point features and category semantic features.

[0030] The residual view branch is used to transform the residual view image into 2D point features and category semantic features corresponding to the residual view image.

[0031] The first intermodal information extraction module is used to aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features, and use the predicted value of the point cloud category-point features as the intermodal information of the missing modality.

[0032] The missing modality completion module is used to crop the missing viewpoint image into three parts: left, middle, and right, according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the corresponding 2D point features of the remaining viewpoint images constitute the completed 2D point features.

[0033] The first fusion module is used to perform multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features. The fused features are then segmented to obtain the segmentation prediction results of the student model.

[0034] In one or more embodiments, it further includes:

[0035] The second LiDAR point cloud branch is used to extract voxel features from the point cloud. The voxel features are projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained, and the probability distribution is multiplied by the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality.

[0036] The multi-view image branch is used to extract image features from multi-view images. The image features are projected to obtain the 2D point features of the image corresponding to the full modality. The semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality.

[0037] The second intermodal information extraction module is used to aggregate the 3D point features and point cloud category semantic features corresponding to the full modality into point cloud category-point features; aggregate the 2D point features and image category semantic features corresponding to the full modality into image category-point features; and capture the common information truth values ​​between the point cloud category-point features and image category-point features corresponding to the full modality.

[0038] The second fusion module is used to perform multimodal fusion of the point cloud 3D point features, point cloud category semantic features, image 2D point features, and image category semantic features corresponding to the full modality, and to segment the fused features to obtain the teacher model segmentation prediction results.

[0039] The distillation module is used to determine the distillation relationship between the teacher and student models by using the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​of the inter-modal models obtained from the teacher model and the predicted values ​​of the inter-modal information obtained from the student model.

[0040] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects:

[0041] This invention improves the perceptual performance of a multimodal fusion model in missing scenarios by introducing a contrastive distillation mechanism between the full-modal teacher model and the missing modality student model. It effectively addresses the problems of insufficient supervision signals and inadequate accuracy of feature completion in missing models, thereby enhancing the robustness of the multimodal fusion model in missing scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0043] Figure 1 This is a flowchart of a multimodal fusion method based on a full-modal teacher model and a missing modality student model according to an embodiment of the present invention;

[0044] Figure 2 This is a structural block diagram of a multimodal fusion system based on a full-modal teacher model and a missing-modal student model, according to an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0046] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0047] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0048] Example 1:

[0049] like Figure 1 As shown, this invention provides a multimodal fusion method based on a full-modal teacher model and a missing-modal student model. The steps based on the student model include:

[0050] S100. Convert the point cloud and the residual view image into 3D point features and category semantic features corresponding to the point cloud, and 2D point features and category semantic features corresponding to the residual view image.

[0051] The aforementioned point cloud refers to the point cloud acquired by LiDAR (Light Detection and Ranging, airborne LiDAR), while the remaining view images are images acquired by the missing view camera, including both remaining view images and missing view images. In this embodiment, the images acquired by the missing view camera are completed using a student model.

[0052] Based on the above embodiments, the point cloud is transformed into 3D point features and category semantic features corresponding to the point cloud, including:

[0053] Extract voxel features from the point cloud, and project the voxel features to obtain the corresponding 3D point features of the point cloud; obtain the semantic category probability distribution of the voxel features, and multiply the probability distribution with the voxel features to obtain the category semantic features of the point cloud.

[0054] In a specific implementation, firstly, voxel features of the point cloud are extracted through a 3D backbone. The voxel features are then projected to obtain 3D point features. The semantic category probability distribution is obtained through a classifier based on MLP (Multilayer Perceptron). This probability distribution is then multiplied with the voxel features to obtain the category semantic features corresponding to the point cloud.

[0055] Furthermore, the remaining viewpoint image is transformed into 2D point features and category semantic features corresponding to the remaining viewpoint image, including:

[0056] Extract image features from the remaining viewpoint image. Project the image features to obtain the corresponding 2D point features of the remaining viewpoint image. Obtain the semantic category probability distribution of the image features. Multiply the probability distribution with the image features to obtain the category semantic features corresponding to the remaining viewpoint image.

[0057] In a specific implementation, firstly, the remaining viewpoint image features are extracted using a 2D backbone. Then, the image features are projected to obtain 2D point features. A semantic category probability distribution is obtained using an MLP-based classifier. This probability distribution is then multiplied by the image features to obtain the category semantic features corresponding to the remaining viewpoint image.

[0058] S200. Aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features.

[0059] Aggregation can be handled through a self-attention mechanism, where the semantic category feature is Q, which serves as the query (Query, Q); and the point features are K and V, which serve as the key (Key, K) and value (Value, V), respectively.

[0060] S300. The missing viewpoint image is cropped into three parts—left, middle, and right—according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. Specifically, the image features of adjacent viewpoints of the missing viewpoint image are used as intramodal information; the predicted value of the point cloud category-point feature is used as intermodal information and multiplied with the corresponding 3D point features of the middle part, and the product is used as the 2D point features of the middle part.

[0061] Based on the above embodiments, the left and right portions are completed using intramodal information, including:

[0062] For the left side, referencing the features of the adjacent left-view image of the missing view image, the left side of the missing view image is completed by cropping the features of the right half of the left view image.

[0063] For the right side, referencing the features of the adjacent right-view image of the missing view image, the right side of the missing view image is completed by cropping the features of the left half of the right view image.

[0064] In a specific implementation, the predicted value of point cloud category-point feature is calculated in a network consisting of a Transformer decoder and linear layers.

[0065] S400. After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the 2D point features corresponding to the remaining viewpoint images constitute the completed 2D point features.

[0066] The method for converting the features of the left and right parts of the completed missing viewpoint image into 2D point features is the same as the steps for converting the remaining viewpoint image into its corresponding 2D point features, and will not be repeated here.

[0067] Based on the above embodiments, the method further includes multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features, and segmentation of the fused features to obtain the student model segmentation prediction results.

[0068] The aforementioned multimodal fusion can be achieved through a cross-attention mechanism and a feedforward neural network, and the final segmentation result can be obtained through an MLP-based segmentation head.

[0069] Based on the above embodiments, the method also includes obtaining fusion features corresponding to all modalities based on the teacher model, including:

[0070] Voxel features are extracted from the point cloud. The voxel features are then projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained. The probability distribution is then multiplied with the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality.

[0071] Image features are extracted from multi-view images, and the image features are projected to obtain the 2D point features of the image corresponding to the full modality; the semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality;

[0072] Multimodal fusion is performed on the 3D point features of the point cloud, the semantic features of the point cloud category, the 2D point features of the image, and the semantic features of the image category corresponding to the full modality. The fused features are then segmented to obtain the segmentation prediction results of the teacher model.

[0073] In this step, multi-view images are acquired by a full camera.

[0074] Based on the above embodiments, after obtaining the teacher model segmentation prediction results, the method further includes:

[0075] The 3D point features and point cloud category semantic features corresponding to all modalities are aggregated into point cloud category-point features;

[0076] The 2D point features and semantic features of the image corresponding to the full modality are aggregated into image category-point features;

[0077] Capture the ground truth of common information between point cloud category-point features and image category-point features corresponding to all modalities.

[0078] In a specific implementation, the similarity between point cloud category-point feature and image category-point feature corresponding to the full modality is measured by the optimal transmission theory, thereby capturing the true value of common information between modalities.

[0079] Based on the above embodiments, it also includes:

[0080] The distillation relationship between the teacher and student models is determined by the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​of the inter-modal information obtained from the teacher model and the predicted values ​​of the inter-modal information obtained from the student model. This enables the teacher model to guide the student model and deliver valuable supervisory signals.

[0081] In summary, this embodiment provides a knowledge distillation framework from a full-modal teacher model to a missing modal student model. It transmits valuable supervisory signals through inter-modal information distillation of the teacher and student models, and fusion feature distillation of the teacher and student models. Further learning of inter-modal commonalities and intra-modal complementary information effectively fills in missing modal features, improving the robustness of the multimodal fusion model in missing modal scenarios.

[0082] Example 2:

[0083] This embodiment provides a multimodal fusion system based on a full-modal teacher model and a missing modality student model, including:

[0084] The first LiDAR point cloud branch is used to transform the point cloud into corresponding 3D point features and category semantic features through the student model.

[0085] The residual view branch is used to transform the residual view image into 2D point features and category semantic features corresponding to the residual view image through the student model.

[0086] The first intermodal information extraction module is used to aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features, and use the predicted value of the point cloud category-point features as the intermodal information of the missing modality.

[0087] The missing modality completion module is used to crop the missing viewpoint image into three parts: left, middle, and right, according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the corresponding 2D point features of the remaining viewpoint images constitute the completed 2D point features.

[0088] The first fusion module is used to perform multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features. The fused features are then segmented to obtain the segmentation prediction results of the student model.

[0089] Based on the above embodiments, it also includes:

[0090] The second LiDAR point cloud branch is used to extract voxel features from the point cloud. The voxel features are projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained, and the probability distribution is multiplied by the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality.

[0091] The multi-view image branch is used to extract image features from multi-view images. The image features are projected to obtain the 2D point features of the image corresponding to the full modality. The semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality.

[0092] The second intermodal information extraction module is used to aggregate the 3D point features and point cloud category semantic features corresponding to the full modality into point cloud category-point features; aggregate the 2D point features and image category semantic features corresponding to the full modality into image category-point features; and capture the common information truth values ​​between the point cloud category-point features and image category-point features corresponding to the full modality.

[0093] The second fusion module is used to perform multimodal fusion of the point cloud 3D point features, point cloud category semantic features, image 2D point features, and image category semantic features corresponding to the full modality, and to segment the fused features to obtain the teacher model segmentation prediction results.

[0094] The distillation module is used to determine the distillation relationship between the teacher and student models by using the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​of the inter-modal models obtained from the teacher model and the predicted values ​​of the inter-modal information obtained from the student model.

[0095] Furthermore, the first lidar point cloud branch, the remaining view branch, the first intermodal information extraction module, the missing modality completion module, and the first fusion module form a missing modality student model, while the second lidar point cloud branch, the multi-view image branch, the second intermodal information extraction module, and the second fusion module form a full-modal teacher model.

[0096] It should be noted that the steps and related configurations of the methods corresponding to each branch and module are consistent with those described in Embodiment 1, and can be found in Embodiment 1.

[0097] It should be understood that the above embodiments are only special cases and do not indicate that the present invention is implemented in such a way.

[0098] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0099] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A multimodal fusion method based on a full-modal teacher model and a missing-modal student model, characterized in that, The steps based on the student model include: The point cloud and residual view image are transformed into 3D point features and category semantic features corresponding to the point cloud, and 2D point features and category semantic features corresponding to the residual view image. Aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features; The missing viewpoint image is cropped into three parts: left, middle, and right, according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. Specifically, the image features of adjacent viewpoints of the missing viewpoint image are used as intramodal information. The predicted value of point cloud category-point feature is used as intermodal information and multiplied with the corresponding 3D point features of the middle part. The product is used as the 2D point features of the middle part. After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the 2D point features corresponding to the remaining viewpoint images constitute the completed 2D point features.

2. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 1, characterized in that, The left and right sides are completed using intramodal information, including: For the left side, referencing the features of the adjacent left-view image of the missing view image, the left side of the missing view image is completed by cropping the features of the right half of the left view image. For the right side, referencing the features of the adjacent right-view image of the missing view image, the right side of the missing view image is completed by cropping the features of the left half of the right view image.

3. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 1, characterized in that, The point cloud is transformed into its corresponding 3D point features and category semantic features, including: Extract voxel features from the point cloud, and project the voxel features to obtain the corresponding 3D point features of the point cloud; obtain the semantic category probability distribution of the voxel features, and multiply the probability distribution with the voxel features to obtain the category semantic features of the point cloud.

4. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 1, characterized in that, The remaining viewpoint images are transformed into their corresponding 2D point features and category semantic features, including: Extract image features from the remaining viewpoint image. Project the image features to obtain the corresponding 2D point features of the remaining viewpoint image. Obtain the semantic category probability distribution of the image features. Multiply the probability distribution with the image features to obtain the category semantic features corresponding to the remaining viewpoint image.

5. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 1, characterized in that, It also includes multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features, and segmentation of the fused features to obtain the student model segmentation prediction results.

6. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 5, characterized in that, It also includes obtaining fusion features corresponding to all modalities based on the teacher model, the steps of which include: Voxel features are extracted from the point cloud. The voxel features are then projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained. The probability distribution is then multiplied with the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality. Image features are extracted from multi-view images, and the image features are projected to obtain the 2D point features of the image corresponding to the full modality; the semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality; Multimodal fusion is performed on the 3D point features of the point cloud, the semantic features of the point cloud category, the 2D point features of the image, and the semantic features of the image category corresponding to the full modality. The fused features are then segmented to obtain the segmentation prediction results of the teacher model.

7. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 6, characterized in that, After obtaining the teacher model segmentation prediction results, the following is also included: The 3D point features and point cloud category semantic features corresponding to all modalities are aggregated into point cloud category-point features; The 2D point features and semantic features of the image corresponding to the full modality are aggregated into image category-point features; Capture the ground truth of common information between point cloud category-point features and image category-point features corresponding to all modalities.

8. The multimodal fusion method based on a full-modal teacher model and a missing-modal student model according to claim 7, characterized in that, Also includes: The distillation relationship between the teacher and student models is determined by the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​between the teacher model and the predicted values ​​of the intermodal information obtained from the student model.

9. A multimodal fusion system based on a full-modal teacher model and a missing-modal student model, characterized in that, include: The first LiDAR point cloud branch is used to convert the point cloud into 3D point features and category semantic features corresponding to the point cloud. The residual view branch is used to transform the residual view image into 2D point features and category semantic features corresponding to the residual view image. The first intermodal information extraction module is used to aggregate the 3D point features and category semantic features corresponding to the point cloud into point cloud category-point features, and use the predicted value of the point cloud category-point features as the intermodal information of the missing modality. The missing modality completion module is used to crop the missing viewpoint image into three parts: left, middle, and right, according to a set ratio. The left and right parts are completed using intramodal information, while the middle part is completed using intermodal information. After converting the image features of the left and right parts of the completed missing viewpoint image into 2D point features, they are connected with the 2D point features of the middle part to obtain the 2D point features of the missing viewpoint image. The 2D point features of the missing viewpoint image and the 2D point features corresponding to the remaining viewpoint images constitute the completed 2D point features. The first fusion module is used to perform multimodal fusion of the 3D point features and category semantic features corresponding to the point cloud, the completed 2D point features, and the image category semantic features. The fused features are then segmented to obtain the segmentation prediction results of the student model.

10. A multimodal fusion system based on a full-modal teacher model and a missing-modal student model according to claim 9, characterized in that, Also includes: The second LiDAR point cloud branch is used to extract voxel features from the point cloud. The voxel features are projected to obtain the 3D point features of the point cloud corresponding to the full modality. The semantic category probability distribution of the voxel features is obtained, and the probability distribution is multiplied by the voxel features to obtain the semantic features of the point cloud category corresponding to the full modality. The multi-view image branch is used to extract image features from multi-view images. The image features are projected to obtain the 2D point features of the image corresponding to the full modality. The semantic category probability distribution of the image features is obtained, and the probability distribution is multiplied with the image features to obtain the semantic features of the image category corresponding to the full modality. The second intermodal information extraction module is used to aggregate the 3D point features and point cloud category semantic features corresponding to the full modality into point cloud category-point features; and to aggregate the 2D point features and image category semantic features corresponding to the full modality into image category-point features. Capture the ground truth of common information between point cloud category-point features and image category-point features corresponding to all modalities; The second fusion module is used to perform multimodal fusion of the point cloud 3D point features, point cloud category semantic features, image 2D point features, and image category semantic features corresponding to the full modality, and to segment the fused features to obtain the teacher model segmentation prediction results. The distillation module is used to determine the distillation relationship between the teacher and student models by using the KL divergence between the fusion features obtained from the teacher model and the fusion features obtained from the student model, as well as the mean squared error loss between the ground truth values ​​of the inter-modal models obtained from the teacher model and the predicted values ​​of the inter-modal information obtained from the student model.

Images