Multi-modal feature fusion method and device
By determining weights for multimodal features and explicitly separating modality weight allocation from deep feature extraction, the problem of lack of interpretability in multimodal feature fusion is solved, enabling traceability and quantification of modality contribution, and improving the accuracy and efficiency of multimodal medical image segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 孙雨飞
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-12
AI Technical Summary
Existing multimodal feature fusion methods lack interpretability. Modal weight allocation is coupled with deep feature extraction, making it impossible to explicitly reflect the contribution of each modality to the final segmentation result, which affects clinical application and optimization.
By assigning a corresponding weight to each modal feature, expanding the weight dimension to be the same as the modal feature, determining the weighted modal feature based on the expanded weight, and determining the fusion of multimodal features through feature fusion, modal weight allocation and deep feature extraction are explicitly separated, and a lightweight modal weighting layer is designed to achieve interpretability.
It achieves traceability and quantification of modal feature contributions, enhances the interpretability of multimodal feature fusion, improves the accuracy and efficiency of segmentation tasks, and is applicable to multimodal medical image segmentation.
Smart Images
Figure CN122020536A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multimodal feature fusion, and more specifically, to a method and apparatus for multimodal feature fusion. Background Technology
[0002] Multimodal feature fusion is widely used in medical image segmentation. Existing nnU-Net models and mainstream multimodal segmentation networks primarily employ implicit fusion, with the core approach being to directly merge input data from different modalities through tensor concatenation or simple addition, before feeding the entire dataset into a convolutional module for deep feature extraction. This type of method requires no separate fusion logic design, is simple to implement, and is currently the mainstream choice in both industry and academia.
[0003] For existing implicit fusion techniques, modality weight allocation is coupled with deep feature extraction. The network cannot explicitly reflect the contribution of each modality to the final segmentation result, lacks interpretability, and is not conducive to clinical application and subsequent optimization.
[0004] Therefore, how to enhance the interpretability of multimodal feature fusion has become a technical problem that needs to be solved in this field. Summary of the Invention
[0005] In view of this, this application proposes a method and apparatus for fusing multimodal features to enhance the interpretability of multimodal feature fusion.
[0006] In a first aspect, this application provides a method for fusing multimodal features, the method comprising: determining a corresponding weight for each modal feature in the multimodal features; expanding the dimension of the corresponding weight to be the same as the dimension of the modal feature for each modal feature in the multimodal features to determine the expanded weight; determining a corresponding weighted modal feature for each modal feature in the multimodal features based on its corresponding expanded weight; and determining a fused feature of the multimodal features based on the determined weighted modal feature.
[0007] Optionally, for each modal feature in the multimodal features, the corresponding weight is determined, including: performing a global average pooling operation on the modal feature to determine the global feature vector corresponding to the modal feature; and determining the weight corresponding to the modal feature based on the determined global feature vector and the Sigmoid function.
[0008] Optionally, for each modal feature in the multimodal features, a corresponding weight is determined, including: determining the weight corresponding to the modal feature based on the correspondence between the modal features and the weights.
[0009] Optionally, for each modal feature in the multimodal features, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight, including: for the modal feature, based on the tensor broadcasting mechanism, expanding the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the expanded weight.
[0010] Secondly, this application also provides a multimodal feature fusion apparatus, which includes: a weight determination module, used to determine a corresponding weight for each modal feature in the multimodal features; an extended weight determination module, used to extend the dimension of the corresponding weight for each modal feature in the multimodal features to the same dimension as the modal feature, so as to determine the extended weight; a weighted modal feature determination module, used to determine a corresponding weighted modal feature for each modal feature in the multimodal features based on its corresponding extended weight; and a fusion feature determination module, used to determine the fusion feature of the multimodal features based on the determined weighted modal features.
[0011] Optionally, the weight determination module determines the corresponding weight for each modal feature in the multimodal features, including: performing a global average pooling operation on the modal feature to determine the global feature vector corresponding to the modal feature; and determining the weight corresponding to the modal feature based on the determined global feature vector and the Sigmoid function.
[0012] Optionally, the weight determination module determines the corresponding weight for each modal feature in the multimodal features, including: determining the weight corresponding to the modal feature based on the correspondence between the modal features and the weights.
[0013] Optionally, the extended weight determination module expands the dimension of the corresponding weight to be the same as the dimension of the modal feature for each modal feature in the multimodal features to determine the extended weight, including: for the modal feature, based on the tensor broadcast mechanism, expanding the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the extended weight.
[0014] Thirdly, this application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described fusion method.
[0015] Fourthly, this application also provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the above-described fusion method.
[0016] According to the technical solution of this application, a corresponding weight is determined for each modal feature in the multimodal features. The dimension of the corresponding weight for each modal feature is expanded to be the same as the dimension of the modal feature to determine the expanded weight. A corresponding weighted modal feature is determined for each modal feature based on its corresponding expanded weight. The fusion feature of the multimodal features is determined based on the determined weighted modal features. Thus, the fusion of multimodal features is achieved. Furthermore, in the process of multimodal feature fusion, the weight of each modal feature is independently determined, and the modal weight allocation is explicitly implemented. This makes the contribution of each modal feature traceable and quantifiable, and explicitly reflects the contribution of each modal feature to the final segmentation result, enhancing the interpretability of multimodal feature fusion and facilitating clinical application and subsequent optimization.
[0017] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description
[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application, and the illustrative embodiments and descriptions thereof are used to explain this application. In the drawings: Figure 1 This is a flowchart of a multimodal feature fusion method according to a preferred embodiment of this application; Figure 2 This is a structural block diagram of a multimodal feature fusion device according to a preferred embodiment of this application. Detailed Implementation
[0019] The technical solution of this application will now be described in detail with reference to the accompanying drawings and embodiments.
[0020] The real-time technical solution provided in this application can be applied to medical image segmentation, focusing on multimodal feature fusion optimization based on deep learning. nnU-Net (no-new-UNet), as a classic network in this field, is widely used in multimodal medical image segmentation tasks such as CT and MRI due to its adaptive data network configuration and strong feature extraction capabilities. Its core logic is to extract multimodal features and output segmentation results through an encoder-decoder structure.
[0021] In a first aspect, embodiments of this application provide a method for fusing multimodal features.
[0022] Figure 1 This is a flowchart of a multimodal feature fusion method according to a preferred embodiment of this application. Figure 1 As shown, the fusion method includes the following: The number of modal features included in the multimodal features can be determined according to specific circumstances.
[0023] In step S10, for each modal feature in the multimodal features, the corresponding weight is determined.
[0024] In step S11, for each modal feature in the multimodal features, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight.
[0025] In step S12, for each modal feature in the multimodal features, the corresponding weighted modal feature is determined based on its corresponding extended weight.
[0026] In step S13, based on the determined weighted modal features, the fusion features of the multimodal features are determined.
[0027] Optionally, in this embodiment of the application, determining the corresponding weight for each modal feature in the multimodal features may include the following:
[0028] Perform global average pooling on the modal features to determine the global feature vector corresponding to the modal features.
[0029] For example, if the original modal features are three-dimensional, a global average pooling operation is performed on each original modal feature to compress the original three-dimensional spatial features into a global feature vector of dimension [B, C], thereby capturing the overall contribution information of each modal feature. Here, B represents the batch size, and C represents the number of feature channels.
[0030] Based on the determined global feature vector, the weights corresponding to the modal features are determined by combining the Sigmoid function.
[0031] The determined global feature vector is input into a single neuron with the activation function Sigmoid. The Sigmoid function maps each element in the global feature vector to the interval [0,1] to determine the dynamic weight value corresponding to each modality feature. The dimension of the weight value is [B, C].
[0032] The technical solution provided by the embodiments of this application can adaptively adjust the importance of each modal feature according to the actual input modal features, thereby realizing dynamic weight calculation.
[0033] Optionally, in this embodiment, determining the corresponding weight for each modal feature in the multimodal features may include the following: determining the weight corresponding to the modal feature based on the correspondence between modal features and weights.
[0034] Specifically, the correspondence between modal features and weights is predetermined. When it is necessary to determine the weights, the correspondence is invoked to determine the weights.
[0035] The correspondence between modal features and weights can be determined based on offline statistical methods.
[0036] Specifically, before model training, fixed weight values are determined by offline statistical analysis of the average contribution of each modality feature in the target task dataset. For example, nnU-Net models for single modality features are trained separately through cross-validation experiments, and the segmentation accuracy of the nnU-Net models trained for each single modality feature is recorded. For each modality feature, the corresponding weight is determined based on the proportion of its segmentation accuracy in the corresponding nnU-Net model. This establishes the correspondence between modality features and weights. The correspondence between modality features and weights is pre-stored and directly called upon during use to determine the weights of the modality features. For example, the correspondence between modality features and weights can be pre-stored in the modality weighting layer. In this way, during model inference, there is no need to dynamically calculate weights through global average pooling and single-neuron mapping; the pre-set fixed weight values are directly called.
[0037] The method of determining weights by pre-determining the correspondence between modal features and weights still explicitly separates modal feature weight allocation from deep feature extraction. Only the weight update method changes from dynamic adjustment based on adaptive input data to static pre-setting based on dataset statistics. It does not require changing the overall structure of the ModalWeightedLayer or the integration method with the nnU-Net model. It is suitable for scenarios where the quality of modal features is stable, the data distribution is uniform, and the contribution of each modal feature is small in different samples. Its advantage is that it simplifies the calculation process of the modal weighted layer and further reduces the inference latency of the model.
[0038] Optionally, in the embodiments of this application, the corresponding weight is determined for each modal feature in the multimodal features, which can be determined based on gradient sensitivity analysis.
[0039] Specifically, weights can be determined based on the following. This method is primarily used during the model training or offline analysis phase. For a pre-trained base segmentation model, its network parameters are fixed, multimodal feature input data is fed into the model, and the final loss function L is calculated relative to each modal feature X. i gradient L. The magnitude of this gradient reflects the sensitivity of the modal feature to the model output, i.e., its contribution. By performing the above calculation on all samples on a representative validation set and taking their statistical average, the average gradient sensitivity score s for each modal feature can be obtained. i Finally, for each modal feature, the score is normalized to obtain the static weight w used in the inference phase. i That is, to determine the weights corresponding to the modal features.
[0040] Optionally, in this embodiment of the application, for each modal feature in the multimodal features, expanding the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the expanded weight may include the following: For modal features, based on a tensor broadcasting mechanism, expanding the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the expanded weight.
[0041] To enable the weights to be computed with the original modal features, the dimensions of the weights can be expanded to match the dimensions of the original modal features. For example, in the above implementation, the original modal features are three-dimensional, and the determined weights have dimensions [B, C]. Based on the tensor broadcasting mechanism, these weights are expanded to [B, C, H, W], perfectly matching the dimensions of the original modal features. Here, B represents the batch size, C represents the number of feature channels, H represents the height, and W represents the width. This allows for operation without changing the size of the original features or introducing additional computational overhead.
[0042] Optionally, in this embodiment of the application, determining the corresponding weighted modal feature based on its corresponding extended weight for each modal feature in the multimodal features may include the following:
[0043] For each modal feature, it is multiplied by its corresponding extended weights to obtain a weighted modal feature. Specifically, for each modal feature, the modal feature is multiplied element-wise with its corresponding extended weights to obtain a weighted single modal feature, i.e., a weighted modal feature. For example, if the dimensions of the extended weights are [B, C, H, W], and the weights of the original modal features are also [B, C, H, W], then the weights of the weighted modal features will be [B, C, H, W].
[0044] Optionally, determining the fusion feature of the multimodal features based on the determined weighted modal features may include the following: The determined weighted modal features are added element-wise to obtain the fusion feature. For example, referring to the above embodiments, the dimensions of the weighted modal features are [B, C, H, W], and the weighted modal features of all modal features together constitute a unified weighted fusion feature, the dimensions of which remain [B, C_total, H, W]. The unified weighted fusion feature is achieved by adding the weighted modal features of each modality (all with dimensions [B, C, H, W]) element-wise at the same spatial location (H, W) and channel dimension (C). Furthermore, C_total is the sum of the number of channels of each modal feature, which directly determines the expansion scale of the fusion feature in the channel dimension, ensuring dimensional compatibility with traditional multimodal stitching methods and facilitating seamless integration of subsequent network modules. The technical solution provided in this application ensures that the obtained fusion features have the same dimensions as the fusion features obtained by direct splicing of traditional multimodal features, thus ensuring the compatibility of subsequent network modules.
[0045] Based on the technical solution provided in this application, a lightweight explicit modal weighting layer can be added to the classic nnU-Net model, and a decoupled multimodal feature fusion logic is adopted to fully realize the coordinated optimization of interpretability, segmentation accuracy, and model efficiency. Specifically, a simple modal weighting layer is designed, which consists of three parts: a weight determination module, a tensor broadcasting module, and an element-wise multiplication module. It avoids complex convolutional or fully connected layers throughout, ensuring lightweight characteristics. Specifically, the weight determination module determines the weights of the modal features; the tensor broadcasting module expands the dimensions based on the tensor broadcasting mechanism; and the element-wise multiplication module determines the weighted fusion features through element-wise multiplication and determines the fusion features based on the determined weighted fusion features.
[0046] Regarding integration with the nnU-Net model, the fusion features of multimodal features can be determined before reaching the front end of the nnU-Net encoder. That is, the aforementioned modality weighting layer can be directly embedded into the front end of the nnU-Net encoder. The specific data flow is as follows: the original multimodal features are first fed into the modality weighting layer, and after a series of processes including weight determination, tensor broadcasting, and element-wise multiplication, a unified fusion feature is output. This fusion feature is then directly fed into the original nnU-Net encoder structure for deep feature extraction. Subsequent modules such as the decoder, skip connections, and output layer all maintain the original design of nnU-Net without any structural modifications or parameter adjustments. Regarding the lightweight feature, the core computation of the modal weighted layer only includes global average pooling, single neuron mapping, tensor broadcasting, and element-wise multiplication. The number of parameters comes only from the weight parameters of a single neuron, and its number is C×1 (C is the number of channels in a single modality). Compared with the original nnU-Net's millions of parameters, the number of new parameters accounts for less than 1%. In terms of computation, the computational complexity of element-wise multiplication is low, which is basically equivalent to the computational complexity of traditional multimodal direct concatenation. Therefore, it will not significantly increase the overall computational burden of the nnU-Net model.
[0047] Furthermore, regarding integration with the nnU-Net model, the fusion features can be determined in the middle of the nnU-Net encoder. Specifically, the embedding position of the modality-weighted layer can be adjusted from the front end of the nnU-Net encoder to a middle layer. The specific implementation involves preserving the entire internal structure of the modality-weighted layer and only changing its integration position within the nnU-Net model. For example, after embedding the modality-weighted layer into the third convolutional block of the encoder, the data flow is adjusted so that the original multimodal features are first fed into the first three convolutional layers of the nnU-Net encoder to extract basic shallow features; then, the shallow features of each modality are fed into the modality-weighted layer for weight calculation and weighted fusion to generate fusion features; finally, these fusion features are fed into subsequent convolutional modules of the encoder for further deep feature extraction. Subsequent modules such as the decoder, skip connections, and output layer retain the original design of nnU-Net. The core logic of this scheme is to first extract basic features and then perform modality weighting. By filtering out some noise in the original data through the first few convolutional layers, modality weights are then assigned to the more representative shallow features. This still achieves decoupling between fusion and extraction. It is suitable for task scenarios where modal differences are not fully reflected in shallow features and the contribution of each modality can only be determined after basic feature extraction, such as multimodal segmentation tasks where some modal original data has a lot of noise and shallow feature recognition is low.
[0048] The core inventive objectives of the technical solutions provided in this application include the following aspects: 1) Providing a highly interpretable multimodal feature fusion scheme that explicitly implements modal weight allocation, making the contribution of each modality traceable and quantifiable. 2) Decoupling modal weight allocation from deep feature extraction, allowing the network to learn based on high-quality weighted features, thereby improving the accuracy of segmentation tasks. 3) Designing a lightweight modal fusion structure that, while ensuring fusion effectiveness, does not significantly increase the network's computational burden and deployment cost, balancing fusion performance and operational efficiency.
[0049] The technical solutions provided by the embodiments of this application can realize explicit allocation of modal weights, decoupled modal fusion and deep feature extraction, improve the interpretability and accuracy of multimodal segmentation tasks, and at the same time take into account fusion performance and operating efficiency.
[0050] Secondly, embodiments of this application also provide a multimodal feature fusion apparatus.
[0051] Figure 2 This is a structural block diagram of a multimodal feature fusion apparatus according to a preferred embodiment of this application. Figure 2 The fusion device shown includes a weight determination module 10, an extended weight determination module 20, a weighted modal feature determination module 30, and a fusion feature determination module 40.
[0052] Specifically, the weight determination module 10 is used to determine the corresponding weight for each modal feature in the multimodal features; the extended weight determination module 20 is used to extend the dimension of the corresponding weight to the same dimension as the modal feature for each modal feature in the multimodal features to determine the extended weight; the weighted modal feature determination module 30 is used to determine the corresponding weighted modal feature for each modal feature in the multimodal features based on its corresponding extended weight; and the fusion feature determination module 40 is used to determine the fusion feature of the multimodal features based on the determined weighted modal features.
[0053] Optionally, the weight determination module 10 determines the corresponding weight for each modal feature in the multimodal features, including: performing a global average pooling operation on the modal features to determine the global feature vector corresponding to the modal features; and determining the weight corresponding to the modal features based on the determined global feature vector and the Sigmoid function.
[0054] Optionally, the weight determination module 10 determines the corresponding weight for each modal feature in the multimodal features, including: determining the weight corresponding to the modal feature based on the correspondence between the modal features and the weights.
[0055] Optionally, the extended weight determination module 20 extends the dimension of the corresponding weight to be the same as the dimension of the modal feature for each modal feature in the multimodal features to determine the extended weight, including: for the modal feature, based on the tensor broadcasting mechanism, extending the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the extended weight.
[0056] The specific working principle and benefits of the fusion device provided in this application are similar to those of the fusion method provided in this application, and will not be repeated here.
[0057] Thirdly, embodiments of this application also provide a machine-readable storage medium storing instructions that cause a machine to execute the above-described fusion method.
[0058] Fourthly, embodiments of this application also provide an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the executable instructions to implement the above-described fusion method.
[0059] The preferred embodiments of this application have been described in detail above. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solution of this application, and these simple modifications all fall within the protection scope of this application.
[0060] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this application will not describe the various possible combinations separately.
[0061] Furthermore, various different implementations of this application can be combined in any way, as long as they do not violate the spirit of this application, they should also be regarded as the content disclosed in this application.
Claims
1. A method for fusing multimodal features, characterized in that, The fusion method includes: For each modal feature in the multimodal features, determine the corresponding weight; For each modal feature in the multimodal features, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight; For each modal feature in the multimodal features, the corresponding weighted modal feature is determined based on its corresponding extended weight; Based on the determined weighted modal features, the fusion features of the multimodal features are determined.
2. The fusion method according to claim 1, characterized in that, For each modal feature in the multimodal features, determine the corresponding weight, including: Perform a global average pooling operation on the modal features to determine the global feature vector corresponding to the modal features; Based on the determined global feature vector, the weights corresponding to the modal features are determined using the Sigmoid function.
3. The fusion method according to claim 1, characterized in that, For each modal feature in the multimodal features, determine the corresponding weight, including: Based on the correspondence between modal features and weights, the weights corresponding to the modal features are determined.
4. The fusion method according to claim 1, characterized in that, For each modal feature in the multimodal features, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight, including: For the modal feature, based on the tensor broadcasting mechanism, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight.
5. A multimodal feature fusion device, characterized in that, The fusion device includes: The weight determination module is used to determine the corresponding weight for each modality feature in the multimodal features; An extended weight determination module is used to extend the dimension of the corresponding weight for each modal feature in the multimodal features to the same dimension as the modal feature, so as to determine the extended weight; The weighted modal feature determination module is used to determine the corresponding weighted modal feature for each modal feature in the multimodal features based on its corresponding extended weight; The fusion feature determination module is used to determine the fusion features of the multimodal features based on the determined weighted modal features.
6. The fusion apparatus according to claim 5, characterized in that, The weight determination module determines the corresponding weight for each modal feature in the multimodal features, including: Perform a global average pooling operation on the modal features to determine the global feature vector corresponding to the modal features; Based on the determined global feature vector, the weights corresponding to the modal features are determined using the Sigmoid function.
7. The fusion apparatus according to claim 5, characterized in that, The weight determination module determines the corresponding weight for each modal feature in the multimodal features, including: Based on the correspondence between modal features and weights, the weights corresponding to the modal features are determined.
8. The fusion apparatus according to claim 5, characterized in that, The extended weight determination module, for each modal feature in the multimodal features, extends the dimension of the corresponding weight to be the same as the dimension of the modal feature to determine the extended weight, including: For the modal feature, based on the tensor broadcasting mechanism, the dimension of the corresponding weight is expanded to be the same as the dimension of the modal feature to determine the expanded weight.
9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the fusion method according to any one of claims 1-4.
10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the fusion method according to any one of claims 1-4.