A method and device for alleviating multi-modal remote sensing image model imbalance learning
Patent Information
- Application Number
- CN202610548354.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-23
- Publication Date
- 2026-08-07
AI Technical Summary
第一个问题表现为不同单模态特征提取器间梯度幅度的差异，导致不同模态的编码器表现出不同的更新速率，可能出现某一模态主导学习过程，而另一模态学习不足的“懒惰”现象
1、本发明的方法针对每个模态，对所述第一梯度分量依次进行自适应杠杆调节和梯度正交化处理，通过自适应杠杆调节，基于预测不确定性动态平衡各模态的学习速度，避免模态"懒惰"现象；进一步通过梯度正交化处理，消除模态间梯度冲突，提升特征融合效果；该方法从梯度调制角度平衡各模态的学习过程，同时考虑梯度幅度和方向的双重调节机制，兼顾了模态内表示能力与模态间兼容性，增强了各模态之间的融合，解决了多模态数据融合中的模态不平衡问题；
Smart Images
Figure CN122530742A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimodal machine learning technology, specifically relating to a method and apparatus for mitigating imbalanced learning in multimodal remote sensing image models. Background Technology
[0002] In recent years, multimodal remote sensing learning has demonstrated great potential in various fields such as land cover classification, change detection, target recognition, and environmental monitoring. Multimodal remote sensing image processing, by integrating remote sensing data from multiple sensors or different bands (such as visible light, infrared, hyperspectral, and synthetic aperture radar), aims to achieve more comprehensive and accurate perception and recognition of land targets. To fully utilize pre-trained single-modal feature extractors, a common multimodal remote sensing learning framework constructs a "multi-encoder" architecture, which is jointly optimized through modality-specific loss and modality-shared loss.
[0003] However, the inherent heterogeneity of multimodal remote sensing data poses a significant challenge to effectively utilizing its correlation and complementarity, a phenomenon known as the modal imbalance problem. For example, images acquired by different sensors differ in resolution, signal-to-noise ratio, and information content; different spectral bands may exhibit drastically different response characteristics to the same ground feature. In such cases, multimodal methods often fail to achieve the desired results and may even perform worse than the best single-modal method.
[0004] Recent research attributes modality imbalance to two key gradient factors associated with unimodal encoders. The first problem manifests as differences in gradient magnitudes between different unimodal feature extractors, leading to varying update rates across different modalities. This can result in one modality dominating the learning process while another under-learns, exhibiting a "lazy" phenomenon. The second problem arises when gradients from the competing objectives of modality-specific loss and modality fusion loss conflict, causing inefficient encoder updates and hindering model convergence to the optimal solution, potentially worsening the results.
[0005] While existing methods have made some progress, they mainly focus on enhancing representation capabilities within a single modality, neglecting the crucial factor of compatibility between cross-modal features. In remote sensing image processing, if features from different modalities cannot be reasonably fused and complemented, such feature fusion may exhibit performance degradation, limiting the effectiveness of multimodal fusion in complex remote sensing scenarios. Summary of the Invention
[0006] To address the aforementioned problems in the existing technology, this invention provides a method and apparatus for mitigating imbalanced learning in multimodal remote sensing image models. This invention provides a method for mitigating imbalanced learning in multimodal remote sensing image models, comprising the following steps: Input data containing several modalities is fed into a multi-encoder multi-modal learning model, which outputs several modal-specific prediction results and fused prediction results. Based on the modality-specific prediction results, the fusion prediction results, and the true labels, the total training loss for each modality is calculated, wherein the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term; The total gradient of the total training loss with respect to the model parameters for each modality is calculated by backpropagation, and the total gradient for each modality is decomposed into a first gradient component derived from the modality fusion loss term and a second gradient component derived from the modality-specific loss term. For each mode, the first gradient component is sequentially subjected to adaptive lever adjustment and gradient orthogonalization to obtain the modulated first gradient component; Using the modulated first gradient component and second gradient component of each modality, the parameters of the multi-encoder multimodal learning model are updated by gradient descent, and the process is iterated until the model converges to obtain the trained multi-encoder multimodal learning model.
[0007] In one embodiment of the present invention, the multi-encoder multimodal learning model includes several modality-specific encoders, a multimodal fusion module, a fusion classifier, and several modality-specific classifiers, wherein, The modality-specific encoder is used to extract modality-specific features from the input data of the corresponding modality; The multimodal fusion module is used to fuse the modality-specific features of several modalities to obtain fused multimodal features; The fusion classifier is used to classify the fused multimodal features to obtain the fusion prediction result; The modality-specific classifiers are used to classify the modality-specific features of the corresponding modalities to obtain modality-specific prediction results.
[0008] In one embodiment of the present invention, the total training loss for each modality is:
[0009] in, The total training loss for each modality, For modal fusion loss term, This is a modal-specific loss term.
[0010] In one embodiment of the present invention, the adaptive lever adjustment step, for each mode, includes: For each modality, an uncertainty metric is calculated based on the corresponding modality-specific prediction results; wherein, for classification tasks, the uncertainty metric includes normalized Shannon entropy based on the prediction probability distribution; The uncertainty measure is converted into a prediction confidence level; Based on the predicted confidence of all samples within a batch, gradient magnitude adjustment weights are calculated for each modality; Multiply the first gradient component of each mode by its corresponding gradient magnitude adjustment weight to obtain the magnitude-adjusted first gradient component.
[0011] In one embodiment of the present invention, the formula for calculating the normalized Shannon entropy is:
[0012] in, It is the first A measure of uncertainty in a modality. It is the first The modality prediction belongs to the first... The probability of each category It is the total number of categories. It is the first The first sample The predicted probability distribution of each modality , Indicates the first One sample.
[0013] In one embodiment of the present invention, the prediction confidence level is:
[0014] in, For the first The sample at the th Confidence level on each modality.
[0015] In one embodiment of the present invention, the formula for calculating the gradient magnitude adjustment weight is as follows:
[0016] in, It is the first The weights of each modality It is the first The sample at the th Confidence level on each modality It refers to the batch size. It's a temperature over-parameter. This represents the total number of modes.
[0017] In one embodiment of the present invention, the gradient orthogonalization process includes: For any two different modes and Calculate its first gradient component after adaptive lever adjustment. and The dot product; If the dot product is negative, then an orthogonal projection operation is performed from the gradient components. Subtract its conflict gradient component from the middle Projected components in the direction; Within a training batch, the above operation is performed iteratively on all conflicting modal gradient pairs.
[0018] In one embodiment of the present invention, the calculation formula for the orthogonalization projection operation is:
[0019] in, The orthogonalized mode gradient, It is a constant.
[0020] Another embodiment of the present invention provides an apparatus for mitigating imbalanced learning in multimodal remote sensing image models, comprising: The prediction result generation module is used to input input data containing several modalities into a multi-encoder multimodal learning model and output several modal-specific prediction results and fused prediction results. The loss calculation module is used to calculate the total training loss for each modality based on the modality-specific prediction results, the fusion prediction results, and the real labels, wherein the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term; The gradient calculation module is used to calculate the total gradient of the total training loss of each modality with respect to the model parameters through backpropagation, and to decompose the total gradient of each modality into a first gradient component derived from the modality fusion loss term and a second gradient component derived from the modality-specific loss term.
[0021] The gradient modulation module is used to perform adaptive lever adjustment and gradient orthogonalization processing on the first gradient component for each mode to obtain the modulated first gradient component. The parameter update module is used to update the parameters of the multi-encoder multimodal learning model by using the modulated first gradient component and the second gradient component of each modality through gradient descent, and iterates until the model converges to obtain the trained multi-encoder multimodal learning model.
[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The method of the present invention performs adaptive lever adjustment and gradient orthogonalization processing on the first gradient component for each modality. Through adaptive lever adjustment, the learning speed of each modality is dynamically balanced based on prediction uncertainty to avoid the "lazy" phenomenon of modality. Furthermore, through gradient orthogonalization processing, gradient conflicts between modalities are eliminated to improve the feature fusion effect. This method balances the learning process of each modality from the perspective of gradient modulation, and considers the dual adjustment mechanism of gradient magnitude and direction. It takes into account the representation capability within the modality and the compatibility between modalities, enhances the fusion between modalities, and solves the modality imbalance problem in multimodal data fusion. 2. The method of this invention significantly outperforms existing best methods on multiple multimodal datasets, and does not require the introduction of additional parameters. It can be widely used as a plug-and-play module in various multimodal learning frameworks. Attached Figure Description
[0023] Figure 1 This is a flowchart illustrating a method for mitigating imbalanced learning in multimodal remote sensing image models, provided by an embodiment of the present invention. Figure 2 A schematic diagram illustrating the framework of a method for mitigating imbalanced learning in multimodal remote sensing image models, provided by an embodiment of the present invention. Figures 3a-3c This is a schematic diagram illustrating the motivation behind the present invention; Figure 4 This diagram illustrates the effectiveness verification of the method for mitigating imbalanced learning in multimodal remote sensing image models according to the present invention. Detailed Implementation
[0024] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0025] Example 1 To achieve more effective multimodal learning and overcome the modality imbalance problem, this embodiment provides a method to alleviate the imbalance learning of multimodal remote sensing image models. This method directly balances the learning process of each modality from the perspective of gradient modulation, thereby enhancing the fusion between modalities. Gradient modulation mainly presents two problems: i) how to dynamically adjust the gradient amplitude of different modalities; ii) how to eliminate gradient direction conflicts between different modalities.
[0026] To address the aforementioned issues, this embodiment provides a gradient modulation method that includes adaptive gradient lever adjustment and gradient orthogonalization. This method directly modifies the gradient during backpropagation to resolve these problems. Specifically, the adaptive lever component dynamically reweights the gradient magnitude using the prediction entropy as a variance estimator. Furthermore, the gradient orthogonalization component ensures the acquisition of compatible multimodal features through the projection of conflicting gradients.
[0027] Please see Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a method for mitigating imbalanced learning in multimodal remote sensing image models according to an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the framework of a method for mitigating imbalanced learning in multimodal remote sensing image models, provided by an embodiment of the present invention.
[0028] The method for mitigating imbalanced learning in multimodal remote sensing image models in this embodiment includes the following steps: S1. Input the input data containing several modalities into the multi-encoder multi-modal learning model, and output several modal-specific prediction results and fusion prediction results.
[0029] like Figure 2 As shown, the multi-encoder multimodal learning model includes several modality-specific encoders E. (1) ,…,E (M) Multimodal fusion module, fusion classifier H fus And several modality-specific classifiers H1, ..., HM. The number of modality-specific encoders and classifiers corresponds one-to-one with the number of modalities; that is, if there are M modalities, the number of modality-specific encoders and classifiers corresponds to M. For example, if the modalities include audio, visible light images, and SAR images, then there are 3 modality-specific encoders and 3 modality-specific classifiers.
[0030] Specifically, a modal encoder is used to extract input data from the corresponding mode. x (m) Extract modality-specific features f (m) ,in, m =1,…,M; The multimodal fusion module is used to combine modality-specific features of several modalities. f (1) ,…, f (M) The features are fused to obtain the fused multimodal features. f fus The fusion classifier is used to process the fused multimodal features. f fus Classification is performed to obtain fusion prediction results; modality-specific classifiers are used to classify the modality-specific features of the corresponding modalities. f (m) Classification is performed to obtain modality-specific prediction results.
[0031] S2. Based on several modality-specific prediction results, fusion prediction results and real labels, calculate the total training loss for each modality, where the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term.
[0032] Specifically, the total training loss for each modality is:
[0033] in, The total training loss for each modality, For modal fusion loss term, This is a modal-specific loss term.
[0034] It is understandable that for modality 1, the total training loss is... For modality M, the total training loss is .
[0035] S3. Calculate the total gradient of the total training loss with respect to the model parameters for each modality through backpropagation, and decompose the total gradient for each modality into the first gradient component originating from the modality fusion loss term. and the second gradient component derived from the mode-specific loss term ,in, m =1,…,M.
[0036] Specifically, for mode 1, the total gradient includes the gradient derived from the mode fusion loss term. and the second gradient component derived from the mode-specific loss term For mode M, the total gradient includes the gradient derived from the mode fusion loss term. and the second gradient component derived from the mode-specific loss term .
[0037] S4. For each mode, the first gradient component Adaptive lever adjustment (AL) and gradient orthogonalization (GO) are performed sequentially to obtain the modulated first gradient component. The adaptive lever adjustment (AL) and gradient orthogonalization (GO) form the GOAL module.
[0038] Step S4 specifically includes: S41. For each mode, the first gradient component is first adaptively adjusted by lever to obtain the first gradient component after adaptive lever adjustment.
[0039] S41a. Uncertainty Calculation: For each modality, an uncertainty metric is calculated based on the corresponding modality-specific prediction results; where, for classification tasks, the uncertainty metric includes the normalized Shannon entropy based on the predicted probability distribution, calculated using the following formula:
[0040] in, It is the first A measure of uncertainty in a modality. It is the first The modality prediction belongs to the first... The probability of each category It is the total number of categories. It is the first The first sample The predicted probability distribution of each modality , Indicates the first One sample.
[0041] S41b, Confidence Calculation: Converting uncertainty measures into prediction confidence levels: ,in, For the first The sample at the th Confidence level on each modality.
[0042] S41c, Weight Calculation: Based on the prediction confidence of all samples within a batch, calculate the gradient magnitude adjustment weights for each modality. The calculation formula is:
[0043] in, It is the first The weights of each modality It is the first The sample at the th Confidence level on each modality It refers to the batch size. It's a temperature over-parameter. This represents the total number of modes.
[0044] S41c, the first gradient component of each mode Multiply by its corresponding gradient magnitude adjustment weight The first gradient component after amplitude adjustment is obtained: The amplitude adjustment is then completed. The first gradient component after amplitude adjustment is the first gradient component after adaptive lever adjustment.
[0045] S42. Perform gradient orthogonalization on the first gradient component after adaptive lever adjustment to obtain the modulated first gradient component.
[0046] S42a, For any two different modes and Calculate its first gradient component after adaptive lever adjustment. and The dot product.
[0047] S42b. If the dot product is negative, then perform an orthogonal projection operation from the gradient components. Subtract its conflict gradient component from the middle The projection component in the direction is calculated using the following formula:
[0048] in, The orthogonalized mode gradient, It is a very small constant added to ensure numerical stability.
[0049] S42c, Iterative processing: Within a training batch, the above operation is performed iteratively on all conflicting modal gradient pairs.
[0050] S5. Using the modulated first and second gradient components of each modality, update the parameters of the multi-encoder multimodal learning model through gradient descent and iterate until the model converges to obtain the trained multi-encoder multimodal learning model.
[0051] Specifically, the modulated first gradient component of each mode Second gradient component The parameters of all modality-specific encoders, the multimodal fusion module, and each classifier are updated using the gradient descent algorithm. This process is repeated iteratively until the model converges, completing the training of the multimodal learning model. After training, the multimodal data to be tested is input into the trained model to output the prediction results.
[0052] Based on the same inventive concept, embodiments of the present invention also provide a device for mitigating imbalanced learning in multimodal remote sensing image models, comprising: The prediction result generation module is used to input input data containing several modalities into a multi-encoder multimodal learning model and output several modal-specific prediction results and fused prediction results. The loss calculation module is used to calculate the total training loss for each modality based on the modality-specific prediction results, the fusion prediction results, and the real labels, wherein the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term; The gradient calculation module is used to calculate the total gradient of the total training loss of each modality with respect to the model parameters through backpropagation, and to decompose the total gradient of each modality into a first gradient component derived from the modality fusion loss term and a second gradient component derived from the modality-specific loss term.
[0053] The gradient modulation module is used to perform adaptive lever adjustment and gradient orthogonalization processing on the first gradient component for each mode to obtain the modulated first gradient component. The parameter update module is used to update the parameters of the multi-encoder multimodal learning model by using the modulated first gradient component and the second gradient component of each modality through gradient descent, and iterates until the model converges to obtain the trained multi-encoder multimodal learning model.
[0054] For details on the specific execution process of each of the above modules and their excellent effects, please refer to the method implementation examples, which will not be repeated here.
[0055] This embodiment further verifies the effectiveness of the method of the present invention through simulation experiments.
[0056] 1. Simulation conditions: All experiments were conducted on a single NVIDIA GeForce 5090 GPU. This embodiment uses ResNet-18 as the backbone network for both audio and visual modalities, while the text modal uses a pre-trained BERT-based model for feature extraction. For audio input, log-Melogram spectrograms were extracted as input features; for video input, two frames were sampled for each segment from the CREMA-D, CMU-MOSI, MELD, and IEMOCAP datasets; and three frames were sampled from the Kinetics-Sounds and AVE datasets. Feature fusion was performed using feature connections along the channel dimension, and an SGD optimizer with a momentum value of 0.9 and a weight decay of 1×10⁻⁶ was applied. 4 The learning rate is set to 4×10. 4 The batch size is 32, and classification accuracy and macro F1 score are used as evaluation metrics. Temperature parameter. Set to 1.0, numerical stability constant Set to 1×10 8 .
[0057] 2. Simulation content: Simulation 1: A comparative experiment was conducted on the Bigearth dataset using the method of this invention (GOAL) and the basic method. The results are shown in Table 1.
[0058] Table 1 Comparison of results between the method of the present invention and the basic method.
[0059] As can be clearly seen from Table 1, the method of the present invention has achieved optimal performance in all evaluation metrics. The accuracy on this dataset is significantly improved compared with the baseline method, which proves the effectiveness of the present invention in solving the modal imbalance problem.
[0060] To better illustrate this, this example uses the CREMA-D dataset, which includes both audio and visual modalities. Please refer to [link / reference]. Figures 3a-3c , Figures 3a-3c This is a schematic diagram illustrating the motivation behind the present invention, showing the gradient conflict problem in the baseline method and the feature compatibility improvement achieved by the GOAL method. Specifically, Figure 3a A histogram of the cosine similarity between g(1) and g(2) in a baseline method without a balancing strategy is presented, showing a clear conflict between them. Furthermore, in Figure 3b In this study, canonical correlation analysis (CCA) was applied to track the evolution of the feature correlation coefficients between f(1) and f(2) during training. The correlations of all four methods showed a consistent increase, indicating a gradual improvement in multimodal feature compatibility. Notably, the GOAL module in this embodiment achieved the highest final coefficient, validating its greater focus on optimization. Furthermore, as... Figure 3c As shown, by incorporating the GO component into existing gradient magnitude-based methods, the performance of this approach is continuously improved. This further demonstrates the general effectiveness and motivation of the method in this embodiment.
[0061] Please see Figure 4 , Figure 4 This diagram illustrates the effectiveness verification of the method for mitigating imbalanced learning in multimodal remote sensing image models according to the present invention. To further demonstrate the effectiveness of GOAL in mitigating the imbalanced multimodal learning problem, this embodiment visualizes the fusion features of the baseline method (basic stitching operation) and the model of the present invention on the CREMA-D dataset using t-SNE. Figure 4 It can be observed that, compared with the basic splicing operation, the model of this invention generates more compact and well-separated clusters, thereby producing higher classification performance.
[0062] The method of this invention performs adaptive lever adjustment and gradient orthogonalization on the first gradient component for each modality. Through adaptive lever adjustment, the learning speed of each modality is dynamically balanced based on prediction uncertainty to avoid the "lazy" phenomenon of modality. Furthermore, through gradient orthogonalization, gradient conflicts between modalities are eliminated, improving the feature fusion effect. This method balances the learning process of each modality from the perspective of gradient modulation, and considers the dual adjustment mechanism of gradient magnitude and direction. It takes into account the representation capability within the modality and the compatibility between modalities, enhances the fusion between modalities, and solves the modality imbalance problem in multimodal data fusion.
[0063] Experiments on multiple public datasets demonstrate that this invention significantly outperforms existing state-of-the-art methods in both classification and regression tasks. Furthermore, the method of this invention requires no additional parameters and can be widely applied as a plug-and-play module to various multimodal learning frameworks.
[0064] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.
[0065] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.
[0066] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A method for mitigating imbalanced learning in multimodal remote sensing image models, characterized in that, Including the following steps: Input data containing several modalities is fed into a multi-encoder multi-modal learning model, which outputs several modal-specific prediction results and fused prediction results. Based on the modality-specific prediction results, the fusion prediction results, and the true labels, the total training loss for each modality is calculated, wherein the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term; The total gradient of the total training loss with respect to the model parameters for each modality is calculated by backpropagation, and the total gradient for each modality is decomposed into a first gradient component derived from the modality fusion loss term and a second gradient component derived from the modality-specific loss term. For each mode, the first gradient component is sequentially subjected to adaptive lever adjustment and gradient orthogonalization to obtain the modulated first gradient component; Using the modulated first gradient component and second gradient component of each modality, the parameters of the multi-encoder multimodal learning model are updated by gradient descent, and the process is iterated until the model converges to obtain the trained multi-encoder multimodal learning model.
2. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 1, characterized in that, The multi-encoder multimodal learning model includes several modality-specific encoders, a multimodal fusion module, a fusion classifier, and several modality-specific classifiers, wherein... The modality-specific encoder is used to extract modality-specific features from the input data of the corresponding modality; The multimodal fusion module is used to fuse the modality-specific features of several modalities to obtain fused multimodal features; The fusion classifier is used to classify the fused multimodal features to obtain the fusion prediction result; The modality-specific classifiers are used to classify the modality-specific features of the corresponding modalities to obtain modality-specific prediction results.
3. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 1, characterized in that, The total training loss for each modality is: in, The total training loss for each modality, For modal fusion loss term, This is a modal-specific loss term.
4. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 1, characterized in that, For each mode, the adaptive lever adjustment steps include: For each modality, an uncertainty metric is calculated based on the corresponding modality-specific prediction results; wherein, for classification tasks, the uncertainty metric includes normalized Shannon entropy based on the prediction probability distribution; The uncertainty measure is converted into a prediction confidence level; Based on the predicted confidence of all samples within a batch, gradient magnitude adjustment weights are calculated for each modality; Multiply the first gradient component of each mode by its corresponding gradient magnitude adjustment weight to obtain the magnitude-adjusted first gradient component.
5. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 4, characterized in that, The formula for calculating the normalized Shannon entropy is: in, It is the first A measure of uncertainty in a modality. It is the first The modality prediction belongs to the first... The probability of each category It is the total number of categories. It is the first The first sample The predicted probability distribution of each modality , Indicates the first One sample.
6. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 4, characterized in that, The prediction confidence level is: in, For the first The sample at the th Confidence level on each modality.
7. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 4, characterized in that, The formula for calculating the gradient magnitude adjustment weight is as follows: in, It is the first The weights of each modality It is the first The sample at the th Confidence level on each modality It refers to the batch size. It's a temperature over-parameter. This represents the total number of modes.
8. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 1, characterized in that, The steps of gradient orthogonalization include: For any two different modes and Calculate its first gradient component after adaptive lever adjustment. and The dot product; If the dot product is negative, then an orthogonal projection operation is performed from the gradient components. Subtract its conflict gradient component from the middle Projected components in the direction; Within a training batch, the above operation is performed iteratively on all conflicting modal gradient pairs.
9. The method for mitigating imbalanced learning in multimodal remote sensing image models according to claim 8, characterized in that, The calculation formula for the orthogonalization projection operation is as follows: in, The orthogonalized mode gradient, It is a constant.
10. A device for mitigating imbalanced learning in multimodal remote sensing image models, characterized in that, include: The prediction result generation module is used to input input data containing several modalities into a multi-encoder multimodal learning model and output several modal-specific prediction results and fused prediction results. The loss calculation module is used to calculate the total training loss for each modality based on the modality-specific prediction results, the fusion prediction results, and the real labels, wherein the total training loss for each modality includes a modality fusion loss term and a modality-specific loss term; The gradient calculation module is used to calculate the total gradient of the total training loss with respect to the model parameters for each modality through backpropagation, and to decompose the total gradient for each modality into a first gradient component derived from the modality fusion loss term and a second gradient component derived from the modality-specific loss term. The gradient modulation module is used to perform adaptive lever adjustment and gradient orthogonalization processing on the first gradient component for each mode to obtain the modulated first gradient component. The parameter update module is used to update the parameters of the multi-encoder multimodal learning model by using the modulated first gradient component and the second gradient component of each modality through gradient descent, and iterates until the model converges to obtain the trained multi-encoder multimodal learning model.