Multi-modal medical image classification method and system based on adaptive gradient modulation

Abstract

The invention discloses a multi-modal medical image classification method and system based on adaptive gradient modulation, and the method comprises the steps: carrying out the modal projection of two original images to a shared semantic space for cross-modal alignment based on a modal alignment fusion module, obtaining a fusion embedding expression, and inputting the fusion embedding expression into an adaptive gradient adjustment module; based on an adaptive gradient adjustment module, a single-mode net marginal benefit and a full-mode joint increment benefit are introduced by using an SV attribution technology, and two indexes are established to measure the competitive strength of a certain mode in a multi-mode model and the strength of the increment benefit generated by full-mode joint. On the basis of the two indexes, the balance condition between modes in the model training process is adjusted by controlling the gradient sizes of the two modes, so that the total benefit of the full mode is maximized, and a trained multi-mode model is obtained; and outputting the classification category of the medical image by using the trained multi-modal model. According to the invention, the efficiency and accuracy of medical image interpretation and processing are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multi-modal medical image classification method, system, terminal and computer-readable storage medium with adaptive gradient modulation. Background Art

[0002] With the rapid development of medical imaging technology, the role of medical images in clinical diagnosis has become increasingly prominent. However, due to the differences in imaging equipment and their principles, single-modality medical images are difficult to fully reflect the patient's health status, limiting clinicians' accurate diagnosis of diseases and formulation of treatment plans. When faced with complex diseases, multimodal medical images can provide more comprehensive information. However, the interpretation and diagnosis of medical images is a labor-intensive task that requires manual annotation of target areas in medical images, which places a heavy burden on clinicians' daily work.

[0003] With the development of deep learning technology, many computer-aided recognition methods for identifying lesion images have emerged. Previous methods usually focus on refining the supervised learning of a single modality or extracting effective cross-modal information to achieve more reliable feature learning. However, these methods often ignore the imbalance problem of feature learning of different modalities in multimodal tasks. Some recent studies have highlighted the challenges associated with the imbalanced contributions and different convergence rates between modalities. In joint learning models, ignoring these imbalances may lead to inefficient information utilization and impair overall performance. During training, the modality containing more significant semantic information usually dominates the training and receives more optimization attention, resulting in inferior performance of other modalities. This phenomenon is attributed to "modal competition", and some studies have attempted to improve the performance of joint learning models by adjusting the training process. Although these methods have achieved results in late fusion models, they cannot support interactions between modalities, and the deep mechanism of modal competition has not been fully explored.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal medical image classification method, system, terminal and computer-readable storage medium with adaptive gradient modulation, aiming to solve the problems of low efficiency and poor accuracy in medical image interpretation and processing in the prior art.

[0006] To achieve the above object, the present invention provides a multi-modal medical image classification method with adaptive gradient modulation, and the multi-modal medical image classification method with adaptive gradient modulation comprises the following steps:

[0007] Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modality alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module;

[0008] Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During the gradient back propagation, the balance between the modalities in the model training process is adjusted by controlling the gradient size of the two modalities based on the two indicators, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained;

[0009] The trained multimodal model is used to identify the medical image and output the classification category of the medical image.

[0010] In addition, to achieve the above-mentioned purpose, the present invention also provides a multi-modal medical image classification system with adaptive gradient modulation, wherein the multi-modal medical image classification system with adaptive gradient modulation comprises:

[0011] A modality alignment fusion module, used for projecting the first original image modality and the second original image modality into a shared semantic space for cross-modality alignment, obtaining a fused embedding representation, and inputting the fused embedding representation into an adaptive gradient adjustment module;

[0012] Adaptive gradient adjustment module, which uses SV attribution technology to introduce single-modal net marginal benefits and all-modal joint incremental benefits, which represent the non-competitive modal state and the incremental benefits generated by cooperation between modalities in the multimodal model, respectively. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefits generated by the joint use of all modalities. During gradient backpropagation, the balance between modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model.

[0013] The medical image classification module is used to identify medical images using the trained multimodal model and output the classification category of the medical images.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an adaptive gradient modulated multimodal medical image classification program stored in the memory and executable on the processor, wherein the adaptive gradient modulated multimodal medical image classification program, when executed by the processor, implements the steps of the adaptive gradient modulated multimodal medical image classification method as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method as described above are implemented.

[0016] In the present invention, based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modal alignment to obtain a fused embedding representation, and the fused embedding representation is input into an adaptive gradient adjustment module; based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce a single-modality net marginal benefit and a full-modality joint incremental benefit, which respectively represent the modality state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model, and establish two indicators to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modality joint. During gradient back propagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained; the trained multimodal model is used to recognize the medical image, and the classification category of the medical image is output. The present invention can adaptively modulate the competition intensity between modalities without reducing the incremental benefits of all modalities. At the same time, the modalities can interact freely without being restricted by late fusion, thereby improving the accuracy of the model in identifying lesion images and improving the efficiency and accuracy of medical image interpretation and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a preferred embodiment of the multi-modal medical image classification method of adaptive gradient modulation of the present invention;

[0018] Figure 2 It is a schematic diagram of a modality balance framework based on Shapley value in a preferred embodiment of the multi-modality medical image classification method of adaptive gradient modulation of the present invention;

[0019] Figure 3 is a schematic diagram of a modality alignment and fusion module in a preferred embodiment of the multi-modality medical image classification method with adaptive gradient modulation of the present invention;

[0020] Figure 4 Schematic diagram of an adaptive gradient adjustment module in a preferred embodiment of the adaptive gradient modulated multimodal medical image classification method of the present invention;

[0021] Figure 5 It is a structural diagram of a preferred embodiment of the multi-modal medical image classification system of the present invention with adaptive gradient modulation;

[0022] Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] In recent years, deep multimodal learning has made significant progress. However, how to effectively integrate information from multiple modalities remains a major challenge. In order to improve model performance, researchers have proposed a variety of complex fusion methods, which are mainly divided into early fusion, late fusion and hybrid fusion. In early fusion, inputs from different modalities are directly spliced ​​together to form a unified embedding representation. The advantage of this strategy is that the entire prediction process can be completed with a single-path model, making the training process relatively simple. In contrast, late fusion processes the features of each modality separately through multiple independent paths and fuses them at the output stage of each path. Although late fusion ignores the interaction of low-level features and has difficulty learning joint representations, it can utilize pre-trained models of each path, so it is more flexible than early fusion in practice. Hybrid fusion has no fixed rules and allows modalities to be integrated at different stages, feature levels or modules, which can effectively promote collaboration between modalities. The best hybrid fusion strategy usually depends on the specific task. However, in some multimodal tasks, the introduction of additional modalities can only bring limited performance improvements. For example, in the CMU-MOSEI dataset, the accuracy of the text-based unimodal model is only 1% lower than that of the multimodal model that uses both text and audio modalities. Similar phenomena have also been verified on other multimodal datasets.

[0025] Compared with unimodal learning, multimodal learning can integrate richer information from multiple perspectives. However, the differences between different modalities make the extraction and fusion of information more complex and challenging. Specifically, the amount and complexity of information contained in different modalities are different, which leads to different difficulties in network training for each modality, thus causing the problem of optimization inconsistency. During the training process, the dominant modality containing significant information often receives more optimization attention and shows better performance, which inhibits the optimization progress of other modalities. Recent studies have theoretically analyzed the training process of late fusion models and proposed the concept of "modal competition" to explain the generation of dominant modalities. In addition, some studies are committed to developing methods to regulate the training of multimodal models to balance the learning of different modalities to achieve better performance. However, most of these regulation methods are limited to late fusion models, which greatly limits their application scenarios. More importantly, the effectiveness mechanism behind these methods has not been deeply explored. In the research in this field, for example, some people use uncertainty to obtain noise-free multi-view representations through dynamic uncertainty-aware networks. Others have designed gradient mixing techniques to alleviate the overfitting problem in multimodal models by fusing the best modalities. For example, balancing the learning speed according to the conditional usage rate of different modalities. For example, some emphasize the heterogeneity of different network branches in joint training and suggest using modality-specific early stopping mechanisms to avoid overfitting. Others have proposed using modality-specific learning rates for different branches in multimodal models to fully tap the capabilities of the network structure. Recently, some have dynamically adjusted the gradients of each modality according to its output amplitude, assuming that in an ideal multimodal model, the output amplitudes of each modality should tend to be balanced. Therefore, the modal gradient with a larger output will be appropriately weakened in each training iteration. Someone has also introduced a robust multimodal learning method that reduces the negative impact of low-quality or noisy modalities by estimating the energy score of each modality.

[0026] Most existing methods focus on optimizing the fusion method, focusing on refining the supervision of each modality, or extracting effective cross-modal information to obtain more reliable feature learning. However, these methods often ignore the imbalance problem of feature learning between different modalities in the task, and at the same time increase the complexity of the model, resulting in a large number of parameters and difficulty in convergence.

[0027] Most existing multimodal balancing methods are limited to late fusion models, which usually require the design of multiple modules to optimize data from different modalities simultaneously, which makes the training process more complicated. In addition, these methods often ignore the interaction between modalities, which limits their practical applications. More importantly, the mechanisms of these methods in improving the performance of multimodal models have not been deeply explored.

[0028] Existing modal balance methods based on Shapley Value (SV) only focus on the balance between modalities, but do not fully consider the overall benefits of all modalities. This method forcibly requires modal synchronous learning to achieve balance, ignoring the information asymmetry between modalities. For example, some modalities naturally contain more information and are better in classification tasks. Sometimes, in order to maintain modal balance, the joint synergy of modalities is suppressed, which in turn leads to a decrease in the overall performance of the model.

[0029] In the current field of artificial intelligence, there are relatively few studies and applications on simultaneous inter-modal balance and interaction optimization, which limits the performance of machine learning systems in processing multimodal data tasks. In order to fill this research gap and improve the performance of models in diverse data environments, we propose a new method to address the challenges of modal balance and interaction.

[0030] The present invention aims to make full use of the unique features and shared features in multimodality, achieve modal balance, and promote full interaction between modalities, thereby maximizing model performance. Through SV, the present invention proposes an adaptive gradient modulation method, which effectively regulates modal competition while maintaining modal balance, so that the model moves in the direction of maximizing the benefits of all modalities, and allows free interaction between modalities, no longer restricted by the late fusion strategy, and ultimately improves the accuracy of the model in lesion image recognition tasks.

[0031] The multimodal medical image classification method with adaptive gradient modulation described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the adaptive gradient modulated multimodal medical image classification method comprises the following steps:

[0032] Step S10: Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modality alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module.

[0033] Specifically, a modality alignment fusion module is used to project two different original image modalities into a shared semantic space for cross-modal alignment, while retaining the original representations of different modalities; in order to align the modalities to learn complementary features, the present invention proposes modeling the two modalities in a shared semantic space while retaining the original representations of different modalities. Figure 3 As shown in the figure, after the first original image modality and the second original image modality pass through a convolutional layer of the Resnet network to obtain features, the two features are sent to the shared self-attention layer for semantic alignment, and feature fusion is performed through their respective cross-attention layers. The features added in at least keep the original features from being lost. The process is shown in the following formula:

[0034]

[0035] Among them, W i represents the first feature of the first original image modality obtained by the i-th convolutional layer, N i Represents the second feature of the second original image modality obtained by the i-th convolutional layer, represents the feature of the first feature after the shared self-attention layer, represents the feature of the second feature after the shared self-attention layer, and Represents the features after inter-modal interaction, SA represents the self-attention layer, CA represents the cross-attention layer, and FFN represents the feed-forward layer;

[0036] After four convolutional layers and four modality alignment fusion modules, the features after the interaction between the modalities are and Send to the adaptive gradient adjustment module.

[0037] Step S20: Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During the gradient back propagation, the balance between the modalities in the model training process is adjusted by controlling the gradient size of the two modalities based on the two indicators, so that the total benefit of all modalities is maximized, and a trained multimodal model is obtained.

[0038] Specifically, the fused embedding representation obtained from the modal fusion module in the previous step is input into the adaptive gradient adjustment module. The SV attribution technology is used to introduce the concepts of single-modal net marginal benefit and all-modal joint incremental benefit, which are used to represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model, respectively. Based on these two concepts, indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the cooperation of all modalities. During gradient backpropagation, the balance between modalities in the model training process is adjusted based on these two indicators by controlling the gradient size of the two modalities, while maximizing the total benefit of all modalities. These two indicators are the basis for studying modal competition behavior and adaptive gradient adjustment balance in the entire invention.

[0039] One way to study the mechanism behind multimodal models is to quantify the degree to which the modes in the model influence each other. Due to the complexity and nonlinearity of neural network models, it is not feasible to separate the calculations that consider competition. Based on SV attribution technology, the present invention can obtain information about the performance of each mode in the absence of competition, so that these models can truly reflect the competition intensity of the mode. SV was originally proposed in coalition game theory to fairly distribute expenditures among participants based on their contribution to total expenditures. Next, we will introduce its application in model balance, with a visual example such as Figure 4 As shown, without loss of generality, if There are k modalities, x represents a total set of images of paired modalities, represents the image of the first mode of x, represents the image of the kth mode of x, m k represents the kth mode, M={m1,…,m k} is the set of all modes, and are the outputs of the multimodal model and the multimodal model, respectively. For any subset {i} represents a set with only mode i, mode m i Net contribution to subset C (marginal contribution) for:

[0040]

[0041] In order to evaluate the mode m i The net contribution to the total mode requires the addition of Take the average value as shown below:

[0042]

[0043] in, represents the net marginal benefit;

[0044] Because when a full-modal zero tensor is input into the multimodal model, the value obtained is 0, that is, So the output of the multimodal model is:

[0045]

[0046] in, represents the net marginal benefit of mode i in the total mode;

[0047] So for a total of two modalities, the net marginal benefit of the first original image modality m1 in the total modality is for:

[0048]

[0049] Wherein, m1 represents the first original image modality, m2 represents the second original image modality, Indicates filling the m1 branch in the multimodal model with all zero tensors. Indicates filling the m2 branch in the multimodal model with all zero tensors, represents the team benefits of m1 and m2, express and the team benefits of m2, represents m1 and team benefits.

[0050] The present invention uses cross entropy to measure the distance between the net contribution and the label for any modality m, which is defined as the confidence, namely:

[0051]

[0052] Among them, y represents the one-hot vector, y[i] represents the true value, and the confidence ρ m Indicates the distance between mode m and the label, ρ m The larger the value, the greater its contribution to the multimodal model. N represents the total number of a batch of data. Represents the net marginal benefit of mode m in the total modes.

[0053] In the process of maintaining modal balance, forcibly suppressing the synergy between modalities often leads to a decrease in the overall performance of the model. This is because some modalities naturally contain more information and therefore learn relatively faster. Based on this, the present invention uses Shapley Value to define a new indicator to measure the overall strength of the synergy between all modalities to optimize model performance. For the first original image modality and the second original image modality, the incremental benefit of all modal joint is:

[0054]

[0055] in, represents the joint incremental benefit of all modalities, ρ a represents the trust of the whole modality, represents the net marginal benefit of all modes.

[0056] The participation level of individual modes is adjusted by adjusting the contribution of each mode during the back propagation of each mode and the overall benefit of all modes, which can be expressed by the following formula, namely:

[0057]

[0058] Among them, t represents the current specific iteration number of training, θ represents the trainable network parameter, η represents the learning rate, represents the cross entropy loss function, φ represents the benefit in the team, φ m represents the net marginal benefit of the mth mode in the team, represents the coefficient, It is used to control the size of the update signal of mode m at iteration t. Intuitively speaking, if the mode is too strong (weak), it is hoped to suppress (amplify) its update signal while ensuring that the benefits of all modes are not reduced.

[0059] The strength of one modality is measured relative to the strength of another modality and the incremental benefit of all modalities. For the first raw image modality m1:

[0060]

[0061] in, represents the strength of m1 relative to m2 at the tth iteration, represents the trust of m1 at the tth iteration, represents the trust of the whole modality at the tth iteration, represents the trust level of m2 at the tth iteration.

[0062] At the same time, the average moving index is used to record the historical data of trust and historical references to modal strength regulation

[0063]

[0064] in, represents the trust of m at the tth iteration, α represents the hyperparameter, and Represents the historical data of three variables respectively;

[0065] Finally got The expression is:

[0066]

[0067] in, represents the coefficient, Used to control the size of the update signal of mode m1 at iteration t.

[0068] Step S30: Use the trained multimodal model to identify the medical image and output the classification category of the medical image.

[0069] After training, the model can accurately output the classification category of the image. By introducing the strategy of modality alignment fusion module and adaptive gradient adjustment module, an efficient and reliable solution is provided for the field of medical image recognition. The present invention can adaptively modulate the competition intensity between modalities without reducing the incremental benefit of all modalities, and the modalities can interact freely without being restricted by late fusion, thereby improving the accuracy of the model in identifying lesion images.

[0070] Through the modality alignment fusion module, different modalities are projected into a shared semantic space to achieve cross-modal alignment while retaining the original representation of each modality. Through the adaptive gradient adjustment module, the weights between the modalities during the model training process are balanced while ensuring that the incremental benefits of all modalities are not reduced. Figure 2 It is the overall flow chart of the present invention, Figure 2 The shielding M1 / M2 in the above refers to the missing branch of the mode M1 / M2 in the model, which should be filled with some predefined baseline values. When shielding one mode or shielding all modes, the present invention replaces the shielded mode with a full zero tensor. The method proposed by the present invention is not only applicable to two modes, but can also be extended to multiple modes by modifying the expression of the marginal contribution of the mode.

[0071] The present invention aims to make full use of the unique and shared features in multimodality, achieve modal balance, and promote full interaction between modalities, thereby maximizing model performance. Through SV, an adaptive gradient modulation method is proposed. This method effectively regulates modal competition while maintaining modal balance, so that the model moves in the direction of maximizing the benefits of all modalities, and allows free interaction between modalities, no longer restricted by the late fusion strategy, and ultimately improves the accuracy of the model in lesion image recognition tasks.

[0072] Beneficial effects of the present invention:

[0073] (1) This paper proposes a modality alignment fusion method based on shared self-attention and interactive cross-attention. This method aligns two modalities in a shared semantic space, which not only reduces the difference between different modalities, but also realizes deep interaction between modalities through multi-layer cross-attention, so that the information of each modality can be fully integrated. Finally, the original representation of each modality is retained through residual connection, which significantly improves the accuracy of the recognition task.

[0074] (2) This invention proposes an adaptive gradient modulation method to improve the performance of the multimodal model of joint training. This method designs a metric based on the concepts of single-modal net contribution and full-modal incremental benefit based on SV, which is used to adaptively modulate the competition intensity in the multimodal model while ensuring that the overall benefit does not decrease. Our modulation method has wide applicability and can be applied to any fusion strategy model.

[0075] The method proposed in the present invention can adapt to any multimodal fusion method. It can be flexibly applied to multiple modalities to achieve cross-modal feature extraction and integration. It can be combined with domain generalization, domain adaptation and other technologies to cope with more complex application scenarios.

[0076] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal medical image classification method of adaptive gradient modulation, the present invention also provides a multimodal medical image classification system of adaptive gradient modulation, wherein the multimodal medical image classification system of adaptive gradient modulation includes:

[0077] A modality alignment fusion module 51 is used to project the first original image modality and the second original image modality into a shared semantic space for cross-modality alignment, obtain a fused embedding representation, and input the fused embedding representation into an adaptive gradient adjustment module;

[0078] The adaptive gradient adjustment module 52 is used to use the SV attribution technology to introduce the single-modal net marginal benefit and the all-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between the modalities in the multimodal model, and establish two indicators to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the joint use of all modalities. During the gradient back propagation, the balance between the modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model;

[0079] The medical image classification module 53 is used to identify the medical image using the trained multimodal model and output the classification category of the medical image.

[0080] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal medical image classification method and system of adaptive gradient modulation, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0081] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores an adaptive gradient modulated multimodal medical image classification program 40, which can be executed by the processor 10, thereby realizing the adaptive gradient modulated multimodal medical image classification method in the present application.

[0082] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the adaptive gradient modulated multimodal medical image classification method.

[0083] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0084] In one embodiment, when the processor 10 executes the adaptive gradient modulated multi-modal medical image classification program 40 in the memory 20 , the steps of the adaptive gradient modulated multi-modal medical image classification method described above are implemented.

[0085] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method as described above are implemented.

[0086] In summary, the present invention provides a multimodal medical image classification method, system, terminal and storage medium for adaptive gradient modulation, the method comprising: based on a modality alignment fusion module, projecting a first original image modality and a second original image modality into a shared semantic space for cross-modal alignment, obtaining a fused embedding representation, and inputting the fused embedding representation into an adaptive gradient adjustment module; based on the adaptive gradient adjustment module, using SV attribution technology, introducing a single-modality net marginal benefit and a full-modality joint incremental benefit, respectively representing the modality state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model, establishing two indicators to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modality joint, and during gradient back propagation, adjusting the balance between modalities in the model training process based on the two indicators by controlling the gradient size of the two modalities, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained; using the trained multimodal model to recognize the medical image, and outputting the classification category of the medical image. The present invention can adaptively modulate the competition intensity between modalities without reducing the incremental benefits of all modalities. At the same time, the modalities can interact freely without being restricted by late fusion, thereby improving the accuracy of the model in identifying lesion images and improving the efficiency and accuracy of medical image interpretation and processing.

[0087] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0088] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0089] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal medical image classification method with adaptive gradient modulation, characterized in that: The adaptive gradient modulated multimodal medical image classification method comprises: Based on the modality alignment fusion module, the first original image modality and the second original image modality are projected into a shared semantic space for cross-modality alignment to obtain a fused embedding representation, and the fused embedding representation is input into the adaptive gradient adjustment module; Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the modal state without competition and the incremental benefit generated by the cooperation between modalities in the multimodal model. Two indicators are established to measure the competition intensity of a certain modality in the multimodal model and the intensity of the incremental benefit generated by the full-modal joint. During the gradient back propagation, the balance between the modalities in the model training process is adjusted by controlling the gradient size of the two modalities based on the two indicators, so that the total benefit of the full modality is maximized, and a trained multimodal model is obtained; The trained multimodal model is used to identify the medical image and output the classification category of the medical image.

2. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: The modality alignment fusion module projects the first original image modality and the second original image modality into a shared semantic space for cross-modality alignment to obtain a fused embedding representation, and inputs the fused embedding representation into the adaptive gradient adjustment module, specifically including: After the first original image modality and the second original image modality are passed through a convolutional layer of the Resnet network to obtain features, the two features are sent to the shared self-attention layer for semantic alignment, and feature fusion is performed through their respective cross-attention layers. The features added in at least keep the original features from being lost: Among them, W i represents the first feature of the first original image modality obtained by the i-th convolutional layer, N i Represents the second feature of the second original image modality obtained by the i-th convolutional layer, represents the feature of the first feature after the shared self-attention layer, represents the feature of the second feature after the shared self-attention layer, and Represents the features after inter-modal interaction, SA represents the self-attention layer, CA represents the cross-attention layer, and FFN represents the feed-forward layer; After four convolutional layers and four modality alignment fusion modules, the features after the interaction between the modalities are and Send to the adaptive gradient adjustment module.

3. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: Based on the adaptive gradient adjustment module, the SV attribution technology is used to introduce the single-modal net marginal benefit and the full-modal joint incremental benefit, which respectively represent the incremental benefits generated by the non-competitive modal state and the cooperation between modalities in the multimodal model, specifically including: like There are k modalities, x represents a total set of images of paired modalities, represents the image of the first mode of x, represents the image of the kth mode of x, m k represents the kth mode, M={m1,…,m k } is the set of all modes, and are the outputs of the multimodal model and the multimodal model, respectively. For any subset {i} represents a set with only mode i, mode m i Net contribution to subset C for: In order to evaluate the mode m i The net contribution to the total mode requires the addition of Take the average: in, represents the net marginal benefit; The output of the multimodal model is: in, represents the net marginal benefit of mode i in the total mode; For a total of two modalities, the net marginal benefit of the first original image modality m1 in the total modality is for: Wherein, m1 represents the first original image modality, m2 represents the second original image modality, Indicates filling the m1 branch in the multimodal model with all zero tensors. Indicates filling the m2 branch in the multimodal model with all zero tensors, represents the team benefits of m1 and m2, express and the team benefits of m2, represents m1 and Team benefits; The cross entropy is used to measure the distance between the net contribution and the label for any modality m, which is defined as the confidence: Among them, y represents the one-hot vector, y[i] represents the true value, and the confidence ρ m Indicates the distance between mode m and the label, ρ m The larger the value, the greater its contribution to the multimodal model. N represents the total number of a batch of data. Represents the net marginal benefit of mode m in the total modes.

4. The multimodal medical image classification method with adaptive gradient modulation according to claim 3, characterized in that: The two indicators are used to measure the competition intensity of a certain mode in the multimodal model and the intensity of the incremental benefits generated by the combination of all modes, including: A new indicator is defined using Shapley Value to measure the overall strength of collaboration among all modalities to optimize model performance. For the first original image modality and the second original image modality, the incremental benefit of all modalities is: in, represents the joint incremental benefit of all modalities, ρ a represents the trust of the whole modality, represents the net marginal benefit of all modes; The participation level of individual modalities is adjusted by adjusting the contribution of each modality during back propagation of each modality and the overall benefit of all modalities: Among them, t represents the current specific iteration number of training, θ represents the trainable network parameter, η represents the learning rate, represents the cross entropy loss function, φ represents the benefit in the team, φ m represents the net marginal benefit of the mth mode in the team, represents the coefficient, Used to control the size of the update signal of mode m at iteration t.

5. The multimodal medical image classification method with adaptive gradient modulation according to claim 4, characterized in that: During the gradient back propagation, the balance between the modes in the model training process is adjusted by controlling the gradient size of the two modes based on the two indicators, so as to maximize the total benefit of all modes and obtain a trained multimodal model, which specifically includes: The strength of one modality is measured relative to the strength of another modality and the incremental benefit of all modalities. For the first raw image modality m1: in, represents the strength of m1 relative to m2 at the tth iteration, represents the trust of m1 at the tth iteration, represents the trust of the whole modality at the tth iteration, represents the trust of m2 at the tth iteration; Use the moving average index to record historical data on trust and historical references to modal strength regulation in, represents the trust of m at the tth iteration, α represents the hyperparameter, and Represents the historical data of three variables respectively; get The expression is: in, represents the coefficient, Used to control the size of the update signal of mode m1 at iteration t.

6. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: The modality alignment fusion module is also used to preserve the original representations of the first original image modality and the second original image modality.

7. The multimodal medical image classification method with adaptive gradient modulation according to claim 1, characterized in that: The adaptive gradient adjustment module is used to balance the weights between the modes during the model training process while ensuring that the incremental benefits of all modes are not reduced.

8. An adaptive gradient modulated multimodal medical image classification system, characterized in that: The adaptive gradient modulated multimodal medical image classification system comprises: A modality alignment fusion module, used for projecting the first original image modality and the second original image modality into a shared semantic space for cross-modality alignment, obtaining a fused embedding representation, and inputting the fused embedding representation into an adaptive gradient adjustment module; Adaptive gradient adjustment module, which uses SV attribution technology to introduce single-modal net marginal benefits and all-modal joint incremental benefits, which represent the non-competitive modal state and the incremental benefits generated by cooperation between modalities in the multimodal model, respectively. Two indicators are established to measure the competition intensity of a modality in the multimodal model and the intensity of the incremental benefits generated by the joint use of all modalities. During gradient backpropagation, the balance between modalities in the model training process is adjusted based on the two indicators by controlling the gradient size of the two modalities, so as to maximize the total benefit of all modalities and obtain a trained multimodal model. The medical image classification module is used to identify medical images using the trained multimodal model and output the classification category of the medical images.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an adaptive gradient modulated multimodal medical image classification program stored in the memory and executable on the processor. When the adaptive gradient modulated multimodal medical image classification program is executed by the processor, the steps of the adaptive gradient modulated multimodal medical image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an adaptive gradient modulated multimodal medical image classification program, and when the adaptive gradient modulated multimodal medical image classification program is executed by a processor, the steps of the adaptive gradient modulated multimodal medical image classification method according to any one of claims 1 to 7 are implemented.

Images