A multimodal medical image classification method and system based on adaptive gradient modulation

Abstract

The present invention discloses a multimodal medical image classification method and system with adaptive gradient modulation. Based on a modality alignment and fusion module, two original image modalities are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, which is then input into an adaptive gradient adjustment module. Based on the adaptive gradient adjustment module, SV attribution technology is used to introduce a single-modality net marginal benefit and a full-modality joint incremental benefit. Two indicators are established to measure the competitive intensity of a modality in a multimodal model and the intensity of the incremental benefit generated by the full-modality joint. During gradient backpropagation, the two indicators are used to adjust the balance between modalities during model training by controlling the gradient magnitudes of the two modalities, thereby maximizing the total benefit of all modalities and obtaining a trained multimodal model. The trained multimodal model is then used to output the classification category of the medical image. The present invention improves the efficiency and accuracy of medical image interpretation and processing.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal medical image classification method, system, terminal and computer-readable storage medium for adaptive gradient modulation. Background Art

[0002] With the rapid development of medical imaging technology, the role of medical images in clinical diagnosis is becoming increasingly prominent. However, due to differences in imaging equipment and their principles, single-modality medical images cannot fully reflect a patient's health status, limiting clinicians' ability to accurately diagnose diseases and formulate treatment plans. Multimodal medical images can provide more comprehensive information for complex diseases. However, the interpretation and diagnosis of medical images is a labor-intensive task that requires manual annotation of target areas in medical images, placing a heavy burden on clinicians' daily work.

[0003] With the advancement of deep learning technology, numerous computer-assisted recognition methods for lesion image recognition have emerged. Previous approaches typically focus on refining supervised learning of a single modality or extracting effective cross-modal information to achieve more reliable feature learning. However, these approaches often overlook the imbalance in feature learning across modalities in multimodal tasks. Recent studies have highlighted the challenges associated with uneven contributions and varying convergence rates between modalities. Ignoring these imbalances in joint learning models can lead to inefficient information utilization and compromise overall performance. During training, the modality containing more salient semantic information often dominates the training, receiving more optimization attention, resulting in inferior performance of other modalities. This phenomenon has been attributed to "modal competition," and some studies have attempted to improve the performance of joint learning models by adjusting the training process. While these methods have achieved success in late-stage fusion models, they fail to support inter-modal interaction, and the underlying mechanisms of modal competition remain underexplored.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a multimodal medical image classification method, system, terminal and computer-readable storage medium with adaptive gradient modulation, aiming to solve the problems of low efficiency and poor accuracy in medical image interpretation and processing in the prior art.

[0006] To achieve the above object, the present invention provides a multimodal medical image classification method based on adaptive gradient modulation, the multimodal medical image classification method based on adaptive gradient modulation comprising the following steps:

[0007] Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module;

[0008] Based on the adaptive gradient adjustment module, SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the non-competitive modal state and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During gradient backpropagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model.

[0009] The trained multimodal model is used to identify the medical image and output the classification category of the medical image.

[0010] In addition, to achieve the above-mentioned object, the present invention further provides a multimodal medical image classification system with adaptive gradient modulation, wherein the multimodal medical image classification system with adaptive gradient modulation includes:

[0011] a modality alignment fusion module, configured to project the first original image modality and the second original image modality into a shared semantic space for cross-modal alignment, obtain a fused embedding representation, and input the fused embedding representation into the adaptive gradient adjustment module;

[0012] The adaptive gradient adjustment module uses SV attribution technology to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which represent the non-competitive modal state and the incremental benefit generated by the cooperation between modalities in the multimodal model, respectively. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal cooperation. During gradient backpropagation, the balance between the modalities in the model training process is adjusted based on these two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model.

[0013] The medical image classification module is used to identify medical images using the trained multimodal model and output the classification category of the medical images.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an adaptive gradient modulated multimodal medical image classification program stored in the memory and runnable on the processor, wherein the adaptive gradient modulated multimodal medical image classification program, when executed by the processor, implements the steps of the adaptive gradient modulated multimodal medical image classification method described above.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method as described above are implemented.

[0016] In the present invention, based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module; based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model, and establish two indicators to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During gradient backpropagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained; the trained multimodal model is used to recognize medical images and output the classification category of the medical image. The present invention can adaptively modulate the competition intensity between modalities without reducing the incremental benefits of all modalities. At the same time, the modalities can interact freely without being restricted by late fusion, thereby improving the accuracy of the model in identifying lesion images and improving the efficiency and accuracy of medical image interpretation and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of a preferred embodiment of the adaptive gradient modulated multimodal medical image classification method of the present invention;

[0018] Figure 2 2 is a schematic diagram of a modal balance framework based on Shapley values ​​in a preferred embodiment of the multimodal medical image classification method with adaptive gradient modulation of the present invention;

[0019] Figure 3 2 is a schematic diagram of a modality alignment and fusion module in a preferred embodiment of the multimodal medical image classification method with adaptive gradient modulation according to the present invention;

[0020] Figure 4 2 is a schematic diagram of an adaptive gradient adjustment module in a preferred embodiment of the multimodal medical image classification method with adaptive gradient modulation of the present invention;

[0021] Figure 5 1 is a structural diagram of a preferred embodiment of the multimodal medical image classification system with adaptive gradient modulation according to the present invention;

[0022] Figure 6 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] Deep multimodal learning has made significant progress in recent years. However, effectively integrating information from multiple modalities remains a major challenge. To improve model performance, researchers have proposed a variety of complex fusion methods, primarily categorized as early fusion, late fusion, and hybrid fusion. In early fusion, inputs from different modalities are directly concatenated to form a unified embedding representation. The advantage of this strategy is that the entire prediction process can be completed with a single-path model, making the training process relatively simple. In contrast, late fusion processes features from each modality separately through multiple independent paths and fuses them at the output of each path. Although late fusion ignores low-level feature interactions and struggles to learn a joint representation, it can leverage pre-trained models from each path, making it more flexible in practice than early fusion. Hybrid fusion, on the other hand, has no fixed rules and allows for the integration of modalities at different stages, feature levels, or modules, effectively promoting inter-modal collaboration. The optimal hybrid fusion strategy often depends on the specific task. However, in some multimodal tasks, the introduction of additional modalities only offers limited performance improvements. For example, in the CMU-MOSEI dataset, the accuracy of a text-based unimodal model was only 1% lower than that of a multimodal model that leveraged both text and audio. Similar results have been verified on other multimodal datasets.

[0025] Compared to unimodal learning, multimodal learning can integrate richer information from multiple perspectives. However, the differences between modalities make information extraction and fusion more complex and challenging. Specifically, the amount and complexity of information contained in different modalities vary, leading to varying degrees of difficulty in training networks for each modality and causing optimization asymmetry. During training, the dominant modality, which contains significant information, often receives more optimization attention and exhibits better performance, inhibiting the optimization progress of other modalities. Recent research has theoretically analyzed the training process of late-fusion models and proposed the concept of "modal competition" to explain the emergence of the dominant modality. Furthermore, some research has focused on developing methods to regulate the training of multimodal models to balance the learning of different modalities and achieve better performance. However, these regulation methods are mostly limited to late-fusion models, significantly limiting their application scenarios. More importantly, the mechanisms underlying their effectiveness have not been thoroughly explored. Research in this area, for example, has exploited uncertainty to obtain noise-free multi-view representations through dynamic uncertainty-aware networks. Others have designed gradient blending techniques to alleviate overfitting in multimodal models by fusing the best modalities. For example, balancing learning speed based on the conditional usage of different modalities. For example, some emphasize the heterogeneity of different network branches in joint training and recommend using modality-specific early stopping mechanisms to avoid overfitting. Others have proposed using modality-specific learning rates for different branches in multimodal models to fully tap the capabilities of the network structure. Recently, some have dynamically adjusted the gradients of each modality based on its output amplitude, assuming that in an ideal multimodal model, the output amplitudes of each modality should tend to be balanced. Therefore, the gradients of modalities with larger outputs are appropriately weakened in each training iteration. Some have also introduced a robust multimodal learning method that mitigates the negative impact of low-quality or noisy modalities by estimating the energy score of each modality.

[0026] Most existing methods focus on optimizing fusion methods, focusing on refining the supervision of each modality or extracting effective cross-modal information to achieve more reliable feature learning. However, these methods often ignore the imbalance in feature learning between different modalities in the task, while increasing model complexity, resulting in a large number of parameters and difficulty in convergence.

[0027] Existing multimodal balancing methods are mostly limited to late-stage fusion models, often requiring the design of multiple modules to simultaneously optimize data from different modalities, which complicates the training process. Furthermore, these methods often overlook interactions between modalities, limiting their practical application. More importantly, the mechanisms by which these methods improve the performance of multimodal models have not been thoroughly explored.

[0028] Existing modal balancing methods based on Shapley Value (SV) focus solely on the balance between modalities, but fail to fully consider the overall benefits of all modalities. These methods force modal learning to achieve balance, ignoring information asymmetry between modalities. For example, some modalities naturally contain more information and perform better in classification tasks. Sometimes, the combined synergy of modalities is suppressed in order to maintain modal balance, resulting in a decrease in overall model performance.

[0029] In the current field of artificial intelligence, research and application of simultaneous inter-modal balance and interaction optimization are relatively limited, which limits the performance of machine learning systems in processing multimodal data tasks. To fill this research gap and improve model performance in diverse data environments, we propose a new approach to address the challenges of modal balance and interaction.

[0030] This invention aims to fully utilize the unique and shared features of multimodality to achieve modal balance while promoting full interaction between modalities, thereby maximizing model performance. Through SV, the present invention proposes an adaptive gradient modulation method. This method effectively regulates modal competition while maintaining modal balance, allowing the model to move towards maximum all-modal benefit. It also allows free interaction between modalities, freeing it from the constraints of late-stage fusion strategies, ultimately improving the model's accuracy in lesion image recognition tasks.

[0031] The multimodal medical image classification method with adaptive gradient modulation described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the adaptive gradient modulated multimodal medical image classification method includes the following steps:

[0032] Step S10: Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module.

[0033] Specifically, a modality alignment fusion module is used to project two different original image modalities into a shared semantic space for cross-modal alignment, while retaining the original representations of different modalities. In order to align the modalities to learn complementary features, the present invention proposes modeling the two modalities in a shared semantic space while retaining the original representations of different modalities. Figure 3 As shown in the figure, after the first original image modality and the second original image modality pass through a convolutional layer of the Resnet network to obtain features, the two features are sent to the shared self-attention layer for semantic alignment, and feature fusion is performed through their respective cross-attention layers. The features added in at least keep the original features from being lost. The process is shown in the following formula:

[0034]

[0035] Among them, W i Represents the first feature of the first original image modality obtained by the i-th convolutional layer, N i Represents the second feature of the second original image modality obtained by the i-th convolutional layer, represents the feature of the first feature after the shared self-attention layer, Represents the feature of the second feature after the shared self-attention layer, and Represents the features after inter-modal interaction, SA represents the self-attention layer, CA represents the cross-attention layer, and FFN represents the feedforward layer;

[0036] After four layers of convolutional layers and four modality alignment fusion modules, the features after the interaction between the modalities are and Send to the adaptive gradient adjustment module.

[0037] Step S20: Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During the gradient backpropagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained.

[0038] Specifically, the fused embedding representation obtained from the modal fusion module in the previous step is input into the adaptive gradient adjustment module. The SV attribution technology is used to introduce the concepts of single-modal net marginal benefit and full-modal joint incremental benefit, which are used to represent the modal state without competition and the incremental benefit generated by cooperation between modalities in the multimodal model, respectively. Based on these two concepts, indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the cooperation of all modalities. During gradient backpropagation, the balance between modalities in the model training process is adjusted based on these two indicators by controlling the gradient size of the two modalities, while maximizing the total benefit of all modalities. These two indicators are the basis for studying modal competition behavior and adaptive gradient adjustment balance in the entire invention.

[0039] One way to study the mechanism behind multimodal models is to quantify the degree to which the modes in the model influence each other. Due to the complexity and nonlinearity of neural network models, it is not feasible to separate the calculation of the part that considers competition. Based on SV attribution technology, the present invention can obtain information about each mode in the absence of competition, so that these models can accurately reflect the competition intensity of the mode. SV was originally proposed in coalition game theory to fairly distribute expenditures among participants based on their contribution to the total expenditure. Next, we will introduce its application in model balance, with a visual example as follows: Figure 4 As shown, in order to maintain generality, if There are k modalities, x represents a total set of images of paired modalities, represents the image of the first mode of x, represents the image of the kth mode of x, m k represents the kth mode, M={m1,…,m k} is the set of all modes, and are the outputs of the multimodal model and the multimodal model, respectively. For any subset {i} represents a set with only mode i, mode m i Net contribution to subset C (marginal contribution) for:

[0040]

[0041] In order to evaluate the mode m i The net contribution to the total mode requires the Take the average value as shown below:

[0042]

[0043] in, represents the net marginal benefit;

[0044] Because when a full-modal zero tensor is input into the multimodal model, the value obtained is 0, that is, So the output of the multimodal model is:

[0045]

[0046] in, represents the net marginal benefit of mode i in the total mode;

[0047] So for a total of two modalities, the net marginal benefit of the first original image modality m1 in the total modality is for:

[0048]

[0049] Wherein, m1 represents the first original image modality, m2 represents the second original image modality, Indicates filling the m1 branch in the multimodal model with all zero tensors, Indicates filling the m2 branch in the multimodal model with all zero tensors, represents the team benefits of m1 and m2, express and the team benefits of m2, represents m1 and team benefits.

[0050] The present invention uses cross entropy to measure the distance between the net contribution and the label for any modality m, which is defined as the confidence level, namely:

[0051]

[0052] Among them, y represents the one-hot vector, y[i] represents the true value, and the trust degree ρ m Indicates the distance between mode m and the label, ρ m The larger it is, the greater its contribution to the multimodal model. N represents the total number of a batch of data. It represents the net marginal benefit of mode m in the total modes.

[0053] In the process of maintaining modal balance, forcibly suppressing the synergy between modalities often leads to a decline in the overall performance of the model. This is because some modalities naturally contain more information and therefore learn relatively faster. Based on this, the present invention uses Shapley Value to define a new indicator to measure the overall strength of the synergy between all modalities to optimize model performance. For the first original image modality and the second original image modality, the incremental benefit of the full modality joint is:

[0054]

[0055] in, represents the joint incremental benefit of all modalities, ρ a represents the trustworthiness of the whole modality, represents the net marginal benefit of all modes.

[0056] The participation level of individual modalities is adjusted by adjusting the contribution of each modality during back propagation of each modality and the overall benefit of all modalities, which can be expressed by the following formula:

[0057]

[0058] Among them, t represents the current specific number of iterations of training, θ represents the trainable network parameters, and η represents the learning rate. represents the cross entropy loss function, φ represents the efficiency of the team, φ m represents the net marginal benefit of the mth mode in the team, represents the coefficient, It is used to control the size of the update signal of mode m at iteration t. Intuitively, if the mode is too strong (weak), we want to suppress (amplify) its update signal while ensuring that the benefits of all modes are not reduced.

[0059] The strength of one modality is measured relative to the strength of another modality and the incremental benefit of all modalities. For the first original image modality m1:

[0060]

[0061] in, represents the strength of m1 relative to m2 at the tth iteration, represents the trust level of m1 at the tth iteration, represents the trust of the full modality at the tth iteration, represents the trust level of m2 at the tth iteration.

[0062] At the same time, the average moving index is used to record the historical data of trust and historical references to modal strength regulation

[0063]

[0064] in, represents the trust of m at the tth iteration, α represents the hyperparameter, and Represents the historical data of three variables respectively;

[0065] Finally got The expression is:

[0066]

[0067] in, represents the coefficient, Used to control the size of the update signal of mode m1 at iteration t.

[0068] Step S30: Use the trained multimodal model to identify the medical image and output the classification category of the medical image.

[0069] After training, the model can accurately output image classification categories. By introducing a modality alignment fusion module and an adaptive gradient adjustment module, this approach provides an efficient and reliable solution for medical image recognition. This invention adaptively modulates the intensity of competition between modalities while maintaining the incremental benefits of all modalities. Furthermore, the modalities can interact freely without being restricted by post-fusion, thereby improving the accuracy of the model in identifying lesion images.

[0070] The modality alignment fusion module projects different modalities into a shared semantic space, achieving cross-modal alignment while preserving the original representations of each modality. The adaptive gradient adjustment module balances the weights of the various modalities during model training while ensuring that the incremental benefits of all modalities are not reduced. Figure 2 It is the overall flow chart of the present invention, Figure 2 The masking of M1 / M2 in the model refers to the absence of the modal M1 / M2 branch, which should be filled with some predefined baseline value. When masking a single mode or all modes, the present invention replaces the masked mode with an all-zero tensor. The proposed method is not only applicable to two modes but can also be extended to multiple modes by modifying the expression of the modal marginal contribution.

[0071] This invention aims to fully leverage the unique and shared features of multimodality, achieving modal balance while promoting full interaction between modalities, thereby maximizing model performance. Through SV, an adaptive gradient modulation method is developed. This method effectively regulates modal competition while maintaining modal balance, aligning the model with the goal of maximizing all-modal benefits. It also allows for free interaction between modalities, freeing them from the constraints of late-stage fusion strategies. This ultimately improves the model's accuracy in lesion image recognition tasks.

[0072] Beneficial effects of the present invention:

[0073] (1) This paper proposes a modality alignment fusion method based on shared self-attention and interactive cross-attention. This method aligns the two modalities in a shared semantic space, not only narrowing the differences between the different modalities but also achieving deep interaction between the modalities through multi-layer cross-attention, allowing the information of each modality to be fully integrated. Finally, the original representation of each modality is preserved through residual connections, significantly improving the accuracy of the recognition task.

[0074] (2) This paper proposes an adaptive gradient modulation method to improve the performance of jointly trained multimodal models. Based on SV, this method designs a metric based on the concepts of single-modal net contribution and full-modal incremental benefit. It is used to adaptively modulate the competition intensity in the multimodal model while ensuring that the overall benefit does not decrease. Our modulation method has broad applicability and can be applied to any fusion strategy model.

[0075] The method proposed in this invention can adapt to any multimodal fusion method. It can be flexibly applied to multiple modalities to achieve cross-modal feature extraction and integration. It can be combined with domain generalization, domain adaptation and other technologies to cope with more complex application scenarios.

[0076] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal medical image classification method of adaptive gradient modulation, the present invention also provides a multimodal medical image classification system of adaptive gradient modulation, wherein the multimodal medical image classification system of adaptive gradient modulation includes:

[0077] A modality alignment fusion module 51 is configured to project the first original image modality and the second original image modality into a shared semantic space for cross-modal alignment, obtain a fused embedding representation, and input the fused embedding representation into the adaptive gradient adjustment module;

[0078] Adaptive gradient adjustment module 52 is used to use SV attribution technology to introduce single-modal net marginal benefit and all-modal joint incremental benefit, which respectively represent the non-competitive modal state and the incremental benefit generated by cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefit generated by the cooperation of all modalities. During gradient backpropagation, based on these two indicators, the balance between modalities in the model training process is adjusted by controlling the gradient size of the two modalities to maximize the total benefit of all modalities and obtain a trained multimodal model.

[0079] The medical image classification module 53 is configured to use the trained multimodal model to identify the medical image and output the classification category of the medical image.

[0080] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal medical image classification method and system of adaptive gradient modulation, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0081] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores an adaptive gradient modulated multimodal medical image classification program 40, which can be executed by the processor 10, thereby realizing the adaptive gradient modulated multimodal medical image classification method of the present application.

[0082] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program code or process data stored in the memory 20, such as executing the adaptive gradient modulated multimodal medical image classification method.

[0083] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0084] In one embodiment, when the processor 10 executes the adaptive gradient modulated multimodal medical image classification program 40 in the memory 20 , the steps of the adaptive gradient modulated multimodal medical image classification method described above are implemented.

[0085] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method as described above are implemented.

[0086] In summary, the present invention provides a multimodal medical image classification method, system, terminal and storage medium with adaptive gradient modulation. The method includes: based on a modality alignment and fusion module, projecting a first original image modality and a second original image modality into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and inputting the fused embedding representation into an adaptive gradient adjustment module; based on the adaptive gradient adjustment module, using SV attribution technology, introducing a single-modal net marginal benefit and a full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by cooperation between modalities in the multimodal model, establishing two indicators to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint; during gradient backpropagation, based on the two indicators, by controlling the gradient size of the two modalities, the balance between the modalities of the model training process is adjusted to maximize the total benefit of the full modality, thereby obtaining a trained multimodal model; using the trained multimodal model to recognize the medical image, and outputting the classification category of the medical image. The present invention can adaptively modulate the competition intensity between modalities without reducing the incremental benefits of all modalities. At the same time, the modalities can interact freely without being restricted by late fusion, thereby improving the accuracy of the model in identifying lesion images and improving the efficiency and accuracy of medical image interpretation and processing.

[0087] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0088] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0089] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal medical image classification method with adaptive gradient modulation, characterized in that: The adaptive gradient-modulated multimodal medical image classification method comprises: Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module; Based on the adaptive gradient adjustment module, SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the non-competitive modal state and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During gradient backpropagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model. The trained multimodal model is used to identify the medical image and output the classification category of the medical image.

2. The multimodal medical image classification method based on adaptive gradient modulation according to claim 1, characterized in that: The modality alignment fusion module projects the first original image modality and the second original image modality into a shared semantic space for cross-modality alignment to obtain a fused embedding representation, and inputs the fused embedding representation into the adaptive gradient adjustment module, specifically including: After the first and second original image modalities pass through a convolutional layer of the ResNet network to obtain features, the two features are sent to the shared self-attention layer for semantic alignment. The features are fused through their respective cross-attention layers, and the features added in at least keep the original features intact: ； ； ； ； in, Indicates that the first original image modality has been The first feature obtained by the convolutional layer is Indicates that the second original image modality has been The second feature obtained by the convolution layer is represents the feature of the first feature after the shared self-attention layer, Represents the feature of the second feature after the shared self-attention layer, and Represents the features after interaction between modalities, represents the self-attention layer, represents the cross attention layer, represents the feedforward layer; After four layers of convolutional layers and four modality alignment fusion modules, the features after the interaction between the modalities are and Send to the adaptive gradient adjustment module.

3. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the incremental benefits generated by the non-competitive modal state and the cooperation between modalities in the multimodal model, specifically including: like have modal, represents a total set of paired modal images, express The image of the first modality, express No. An image of a modality, Indicates the modal, is the set of all modes, and are the outputs of the multimodal model and the multimodal model, respectively. For any subset , Indicates only A collection of modals, modals Subset Net contribution for: ； To evaluate the modal The net contribution to the total mode requires the Take the average: ； in, represents the net marginal benefit; The output of the multimodal model is: ； in, Indicates modality Net marginal benefit in the total modality; For a total of two modalities, the first original image modality Net marginal benefit in the total mode for: ； in, represents the first original image modality, represents the second original image modality, Representation into the multimodal model The branch is filled with all-zero tensors, Representation into multimodal model The branch is filled with all-zero tensors, express and Team benefits, express and Team benefits, express and Team benefits; Use cross entropy to measure the The net contribution of and the distance between the label is defined as the trust: ； in, represents a one-hot vector, Indicates truth value, confidence Indicates modality The distance from the label, The larger it is, the greater its contribution to the multimodal model. Indicates the total number of a batch of data, Indicates modality Net marginal benefit in the total modality.

4. The multimodal medical image classification method based on adaptive gradient modulation according to claim 3, characterized in that: The two indicators are used to measure the competitive strength of a certain mode in the multimodal model and the strength of the incremental benefits generated by the combination of all modes. Specifically, they include: A new metric is defined using Shapley Value to measure the overall strength of collaboration among all modalities to optimize model performance. For the first and second original image modalities, the incremental benefit of all-modal collaboration is: ； ； in, represents the joint incremental benefit of all modalities, Indicates the trust level of all modalities; The participation level of individual modalities is adjusted by adjusting the contribution of each modality during backpropagation of each modality and the overall benefit of all modalities: ； in, Indicates the current specific iteration number of training, represents the trainable network parameters, represents the learning rate, represents the cross entropy loss function, Indicates the effectiveness of the team, Indicates the The net marginal benefit of a modality in a team, represents the coefficient, To control the mode In iteration Updates the size of the signal.

5. The multimodal medical image classification method based on adaptive gradient modulation according to claim 4, characterized in that: During the gradient back propagation, the balance between the modes in the model training process is adjusted by controlling the gradient size of the two modes based on the two indicators, so as to maximize the total benefit of all modes and obtain a trained multimodal model. Specifically, the following steps are performed: The strength of one modality is measured relative to the strength of another modality and the incremental benefit of all modalities. For the first original image modality : ； in, Indicates in At the iteration Relative to The strength of Indicates in At the iteration Trust, Indicates in The trust of all modalities at the iteration, Indicates in At the iteration Trustworthiness; Use the moving average to record historical data on trust and historical references to modal strength regulation : ； ； in, Indicates in At the iteration Trust, represents the hyperparameter, 、 and Represents the historical data of three variables respectively; get The expression is: ； in, represents the coefficient, To control the mode In iteration Updates the size of the signal.

6. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: The modality alignment fusion module is further configured to preserve the original representations of the first original image modality and the second original image modality.

7. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: The adaptive gradient adjustment module is used to balance the weights between the modalities during the model training process while ensuring that the incremental benefits of all modalities are not reduced.

8. A multimodal medical image classification system with adaptive gradient modulation, characterized in that: The adaptive gradient modulated multimodal medical image classification system comprises: a modality alignment fusion module, configured to project the first original image modality and the second original image modality into a shared semantic space for cross-modal alignment, obtain a fused embedding representation, and input the fused embedding representation into the adaptive gradient adjustment module; The adaptive gradient adjustment module uses SV attribution technology to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which represent the non-competitive modal state and the incremental benefit generated by the cooperation between modalities in the multimodal model, respectively. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal cooperation. During gradient backpropagation, the balance between the modalities in the model training process is adjusted based on these two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model. The medical image classification module is used to identify medical images using the trained multimodal model and output the classification category of the medical images.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an adaptive gradient modulated multimodal medical image classification program stored in the memory and executable on the processor. When the adaptive gradient modulated multimodal medical image classification program is executed by the processor, the steps of the adaptive gradient modulated multimodal medical image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method according to any one of claims 1 to 7 are implemented.

Images