An arbitrary modal missing dynamic learning method based on low-rank fine-tuning

Abstract

This disclosure presents a dynamic learning method for arbitrary modality missing data based on low-rank fine-tuning, comprising: acquiring multimodal input data and uniformly imputing missing modalities using placeholder input representations; extracting features and jointly modeling each modality data; embedding a dynamic hybrid expert low-rank adaptation module; constructing a missing-aware routing mechanism and selecting dynamic low-rank experts to participate in the computation; extracting category feature representations from complete modality samples and constructing a category semantic prototype library based on category labels; projecting the feature representations of missing modality samples onto a shared available subspace and distilling and aligning them with the semantic prototypes of the corresponding categories; optimization and training; testing and classification. This embodiment can handle different modality combinations of missing data in a single model; utilizes a dynamic hybrid expert low-rank adaptation module to improve the model's adaptability to arbitrary modality missing scenarios; and utilizes a category semantic prototype distillation mechanism to suppress category distribution drift, thereby improving the model's discriminative stability and classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of multimodal intelligent information processing technology, and in particular to a dynamic learning method for arbitrary modality missing based on low-rank fine-tuning. Background Technology

[0002] With the development of artificial intelligence technology, multimodal learning methods that integrate multiple information sources such as images, text, voice, sensor signals, and time-series data have been widely applied in scenarios such as intelligent sensing, medical analysis, industrial inspection, security identification, and human-computer interaction. By jointly modeling the complementary information between different modalities, multimodal classification models can usually achieve stronger representation capabilities and higher classification accuracy than single-modal models.

[0003] However, in practical applications, multimodal data is often difficult to guarantee in its completeness. On the one hand, due to factors such as acquisition equipment failure, signal obstruction, transmission interruption, asynchronous sampling, limited acquisition costs, and privacy restrictions, one or more modalities are often missing. On the other hand, the modal missing patterns of different samples are usually not consistent, and may manifest as any combination of missing modalities. This modal incompleteness leads to a significant decline in the performance of traditional multimodal models, because most existing methods assume that complete modal inputs are available during the training and testing phases, making it difficult to adapt to complex and ever-changing real-world application environments.

[0004] Existing research on the problem of missing modalities can generally be categorized as follows: modality completion-based methods, incomplete multimodal learning-based methods, parameter-efficient fitting-based methods, and knowledge distillation-based methods. Modality completion-based methods reconstruct missing modalities through generative models or cross-modal mappings, but these methods are typically computationally expensive and prone to introducing reconstruction errors when modal differences are significant. Missing modality learning-based methods usually design separate models or training processes for specific missing modalities, resulting in insufficient model scalability and difficulty in uniformly handling arbitrary modality loss. While parameter-efficient fitting-based methods can reduce training costs, they often only focus on input adaptation or shallow conditional adjustment, lacking structured modeling of knowledge adapted to missing modalities. Knowledge distillation-based methods can alleviate feature shift problems to some extent, but most rely on sample-level pairing supervision, making it difficult to stably constrain the decision space in complex missing modality scenarios.

[0005] Furthermore, when training with a mixture of complete and missing modality samples, samples of the same class often exhibit significant distributional differences in the feature space due to the different available modalities. This can easily cause class decision boundary drift, thereby reducing the robustness and generalization ability of the classifier. Existing methods typically lack stable class-level semantic anchors, making it difficult to maintain a consistent discrimination structure under multiple modality missing conditions. Therefore, how to construct a multimodal classification method that can uniformly handle arbitrary modality missingness, balance parameter efficiency and robustness, and stabilize the decision space has become an urgent technical problem to be solved.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0007] The purpose of this invention is to provide a dynamic learning method for arbitrary modality loss based on low-rank fine-tuning, thereby overcoming, to at least to some extent, one or more problems caused by the limitations and defects of related technologies.

[0008] This invention first provides a dynamic learning method for arbitrary modality loss based on low-rank fine-tuning, comprising: S101: Acquire multimodal input data, construct a modality availability indicator vector for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; S102: Input each modal data into the corresponding modal encoder, extract modal feature representations, and send each modal feature into the shared backbone network for joint modeling to obtain multimodal intermediate representations; S103: A dynamic hybrid expert low-rank adaptation module is introduced into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modal missing modes. S104: Construct a missing-aware routing mechanism, input the intermediate representation of the current sample and the modality availability indicator vector into the routing network, generate the gating weights of each dynamic low-rank expert, and select at least one dynamic low-rank expert to participate in the calculation based on the gating weights, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. S105: Extract category feature representations from complete modal samples and construct a category semantic prototype library based on category labels; continuously update the category semantic prototype library using a sliding update strategy to form stable category-level semantic anchors; S106: For missing modality samples, project their feature representations processed by the dynamic hybrid expert low-rank adaptation module onto the shared available subspace, and align them with the semantic prototypes of the corresponding categories by distillation. S107: Joint optimization and training are performed using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario; S108: Apply the trained model to the testing phase, and process the input multimodal data under arbitrary modality missing conditions to obtain multimodal classification results.

[0009] In this invention, S102 specifically includes the following steps: Each modal data is input into its corresponding modal encoder to extract modal feature representations; The features of each modality are fused to obtain the initial input representation of the backbone network; By jointly modeling the initial input representations with the shared backbone network, we obtain the intermediate representations of multimodal samples in a unified feature space, i.e., the multimodal intermediate representations.

[0010] In this invention, in S103, a dynamic hybrid expert low-rank adaptation module is inserted into one or more feature transformation layers of the shared backbone network.

[0011] In this invention, S104 specifically includes the following steps: Embedding mapping is performed on the modality availability indicator vector to obtain the missing mode representation; The multimodal intermediate representations are then concatenated with the missing pattern representations, and the routing features are obtained through a projection function. The initial weights of each dynamic low-rank expert are calculated using the routing matrix; Select the top K dynamic low-rank experts with the largest initial weights to participate in the calculation, and define the gating weights.

[0012] In this invention, step S105 specifically includes the following steps: Extract category feature representations from complete modality samples and construct a category semantic prototype library based on category labels; For any sample, if all its modalities exist, then its final feature representation after passing through the backbone network and the dynamic hybrid expert low-rank adaptation module is normalized; The semantic prototypes in the semantic prototype library are updated using an exponential moving average method.

[0013] In this invention, in S106, the prototype distillation loss of the missing modal sample is defined as the cosine alignment loss.

[0014] The present invention further provides an arbitrary modality missing dynamic learning system based on low-rank fine-tuning, comprising: The data acquisition and preprocessing module is used to acquire multimodal input data, construct modality availability indicator vectors for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; The modeling module is used to input the modal data of each mode into the corresponding modal encoder, extract the modal feature representation, and send the modal features into the shared backbone network for joint modeling to obtain multimodal intermediate representations; An expert embedding module is used to introduce a dynamic hybrid expert low-rank adaptation module into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modality missing modes. The expert selection module is used to construct a missing-aware routing mechanism. It inputs the intermediate representation of the current sample and the modality availability indicator vector into the routing network to generate the gating weights of each dynamic low-rank expert. Based on the gating weights, at least one dynamic low-rank expert is selected to participate in the calculation, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. The category semantic prototype library construction and update module is used to extract category feature representations from complete modality samples and construct a category semantic prototype library based on category labels; a sliding update strategy is used to continuously update the category semantic prototype library to form a stable category-level semantic anchor point; The distillation alignment module is used to project the feature representation of missing modality samples, which has been processed by the dynamic hybrid expert low-rank adaptation module, onto a shared available subspace and distill and align it with the semantic prototype of the corresponding category. The training module is used to jointly optimize and train the network using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario. The classification module is used to apply the trained model to the testing phase, and processes the input multimodal data to obtain multimodal classification results under arbitrary modality missing conditions.

[0015] The technical solution provided by this invention may include the following beneficial effects: This invention designs a unified multimodal classification framework for arbitrary modality missing scenarios. This framework can handle different modality combinations of missing data in a single model without requiring separate training of independent models for different missing modes, thereby improving the method's versatility and deployment efficiency. This invention also designs a dynamic hybrid expert low-rank adaptation module. This module dynamically routes data by combining missing mode information and sample semantic information, adaptively selecting experts from multiple dynamic low-rank experts that match the current input to participate in feature transformation, thereby improving the model's adaptability to arbitrary modality missing scenarios while balancing parameter efficiency and classification performance. Finally, this invention designs a category semantic prototype distillation mechanism. This mechanism constructs a category prototype library using complete modality samples and performs category-level distillation constraints on missing modality samples in a shared available subspace, thereby suppressing category distribution drift caused by modality inconsistency and improving the model's discrimination stability and classification accuracy in complex missing scenarios. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0017] Figure 1 A flowchart illustrating an arbitrary modality missing dynamic learning method based on low-rank fine-tuning in an exemplary embodiment of this disclosure is shown. Figure 2 The diagram illustrates the structure of an arbitrary modality missing dynamic learning system based on low-rank fine-tuning in an exemplary embodiment of this disclosure, demonstrating the overall structure of multimodal feature extraction, dynamic low-rank expert selection, shared backbone network processing, and category prototype distillation. Figure 3 A schematic diagram of a dynamic low-rank expert adapter structure in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0019] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0020] This example implementation first provides a dynamic learning method for arbitrary modality loss based on low-rank fine-tuning. Please refer to [reference needed]. Figure 1 and Figure 2 The method may include: S101-108, as follows: S101: Acquire multimodal input data, construct a modality availability indicator vector for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; S102: Input each modal data into the corresponding modal encoder, extract modal feature representations, and send each modal feature into the shared backbone network for joint modeling to obtain multimodal intermediate representations; S103: A dynamic hybrid expert low-rank adaptation module is introduced into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modal missing modes. S104: Construct a missing-aware routing mechanism, input the intermediate representation of the current sample and the modality availability indicator vector into the routing network, generate the gating weights of each dynamic low-rank expert, and select at least one dynamic low-rank expert to participate in the calculation based on the gating weights, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. S105: Extract category feature representations from complete modal samples and construct a category semantic prototype library based on category labels; continuously update the category semantic prototype library using a sliding update strategy to form stable category-level semantic anchors; S106: For missing modality samples, project their feature representations processed by the dynamic hybrid expert low-rank adaptation module onto the shared available subspace, and align them with the semantic prototypes of the corresponding categories by distillation. S107: Joint optimization and training are performed using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario; S108: Apply the trained model to the testing phase, and process the input multimodal data under arbitrary modality missing conditions to obtain multimodal classification results.

[0021] This invention designs a unified multimodal classification framework for arbitrary modality missing scenarios. This framework can handle different modality combinations of missing data in a single model without requiring separate training of independent models for different missing modes, thereby improving the method's versatility and deployment efficiency. This invention also designs a dynamic hybrid expert low-rank adaptation module. This module dynamically routes data by combining missing mode information and sample semantic information, adaptively selecting experts from multiple dynamic low-rank experts that match the current input to participate in feature transformation, thereby improving the model's adaptability to arbitrary modality missing scenarios while balancing parameter efficiency and classification performance. Finally, this invention designs a category semantic prototype distillation mechanism. This mechanism constructs a category prototype library using complete modality samples and performs category-level distillation constraints on missing modality samples in a shared available subspace, thereby suppressing category distribution drift caused by modality inconsistency and improving the model's discrimination stability and classification accuracy in complex missing scenarios.

[0022] The specific process of each step in the above embodiments will be described below.

[0023] S1, in the input construction stage, provides a unified representation for multimodal input samples and introduces modality availability labeling and modality placeholder mechanisms so that samples with different modality missing modes can be input into the subsequent network in a consistent form.

[0024] (1) First, define the multimodal input samples and their class labels. Suppose the input sample consists of M modalities, then the sample and its class label are represented as follows:

[0025] Where, x (i) Let represent the data of the i-th modality, y represent the sample category label, and Y represent the category set.

[0026] Then, a modality availability indicator vector is constructed to characterize the existence state of each modality in the current sample, which is represented as follows:

[0027] Wherein, when the i-th mode exists, =1; when the i-th mode is missing =0. The modality availability indicator vector described above can uniformly describe the input state in any modality-missing scenario.

[0028] (2) Construct modal placeholder inputs for missing modalities, retain the original inputs for existing modalities, and use preset placeholder inputs for missing modalities. To achieve a unified input format, the following steps are performed:

[0029] The unified multimodal input sample can be represented as:

[0030] in, This represents the input of the i-th mode after modal placeholder processing.

[0031] By using the padding method, samples with different missing modalities can be uniformly mapped to the same input space without changing the overall network input structure, providing a consistent data representation for subsequent modality coding and shared backbone network processing.

[0032] S2, in the feature extraction stage, the unified multimodal inputs are fed into the corresponding modal encoders and joint representation learning is completed in the shared backbone network to obtain intermediate representations suitable for subsequent dynamic expert adaptation.

[0033] (1) Input the unified modal data into the corresponding modal encoders to extract modal feature representations, as shown below:

[0034] in, This represents the encoder corresponding to the i-th mode. This represents the feature representation of the i-th modality. Through this step, the basic semantic information of different modalities can be extracted separately.

[0035] (2) The features of each modality are fused to obtain the initial input representation of the backbone network, as shown below:

[0036] Wherein, Φ( ) represents the modal fusion function.

[0037] (3) The initial input represents the input shared backbone network and is jointly modeled to obtain the intermediate representation of multimodal samples in a unified feature space, namely the multimodal intermediate representation.

[0038] Specifically, the fused features (initial input representation) are fed into a shared backbone network, and after multiple layers of feature transformation, an intermediate representation of the sample is obtained. The layer output is represented as:

[0039] in, Indicates the first Each feature transformation layer.

[0040] Through the above process, intermediate representations of multimodal samples in a unified feature space can be obtained.

[0041] S3. In the feature adaptation stage, a dynamic hybrid expert low-rank adaptation module is introduced into the shared backbone network to enhance the model's ability to adapt to different modal missing modes.

[0042] (1) Multiple dynamic low-rank experts are constructed in the feature transformation layer of the shared backbone network. Each dynamic low-rank expert performs low-rank mapping on the input features, which is expressed as:

[0043] in,

[0044] Let r and d represent the dimensionality reduction and increase matrices of the k-th dynamic low-rank expert, respectively, where r represents the low-rank dimension and d represents the feature dimension. Through low-rank mapping, effective feature adaptation can be achieved with relatively small parameter overhead.

[0045] (2) The outputs of multiple dynamic low-rank experts are weighted and aggregated to obtain the adaptation residual of the current layer, which is expressed as:

[0046] in, Indicates the gating weight, z For input features.

[0047] (3) The adaptation residual is added to the original output of the shared backbone network to obtain the enhanced layer output, which is expressed as:

[0048] This step allows for supplementary adjustments to the features based on the different states of the input samples.

[0049] S4, in the expert selection phase, a missing-aware routing mechanism is constructed to adaptively select suitable dynamic low-rank experts to participate in the computation based on the feature information of the current sample and the modality availability. Please refer to [reference needed]. Figure 3 , Figure 3 In the diagram, Mode 1 experts represent the full modal input scenario, while Mode 2 experts represent the missing modal input scenario.

[0050] (1) Embedding and mapping the modal availability vectors yields the missing mode representation:

[0051] The input feature z is then concatenated with the missing pattern representation ψ(m), and then processed by the projection function. ( Obtain routing characteristics:

[0052] in,[ ; [z;ψ(m)] represents the vector concatenation operation, ψ(m) represents the embedding representation of the modality availability vector, and [z;ψ(m)] represents the feature concatenation operation. ( ) represents the projection function. By introducing modality missing information, the routing process can take into account both semantic features and modal states.

[0053] (2) The weight distribution of each dynamic low-rank expert is generated using the routing matrix, which is expressed as:

[0054] Among them, W ( ) Indicates the first The routing matrix of the layer.

[0055] (3) To improve computational efficiency, only the highest-weighted dynamic low-rank experts are retained for computation, and their gating weights are expressed as follows:

[0056] in express A set of K dynamic low-rank expert indices is provided at the beginning of each layer. This missing-aware routing mechanism allows for the dynamic selection of the most suitable expert path for different input samples.

[0057] S5. In the category modeling stage, a category semantic prototype library is built, and the prototypes of each category are continuously updated using complete modal samples to form a stable category center representation.

[0058] (1) Establish a category semantic prototype library, which is represented as follows:

[0059] in, This represents the semantic prototype of the c-th class sample. This prototype library is used to store stable semantic centers for each class under full modality conditions.

[0060] (2) Determine whether the current sample is a complete modality sample. When the sample satisfies that all modalities are available, that is:

[0061] The feature representation of this sample is normalized and expressed as follows:

[0062] (3) Based on the sample category, the corresponding category prototype. The update will be performed in the following form:

[0063] in, This represents the sliding coefficient for prototype updates. By continuously updating the category prototype using complete modality samples, a stable category semantic center can be gradually formed.

[0064] S6. In the missing sample constraint stage, perform category semantic prototype distillation on samples with missing modalities to reduce the category distribution offset between complete modal samples and missing modal samples.

[0065] (1) Determine whether the current sample is a missing modality sample. When the sample has a missing modality, that is:

[0066] The sample features and their corresponding class prototypes are mapped to a shared available subspace. The representations of the sample features and their corresponding class prototypes in the shared available subspace are as follows:

[0067] in, For the final feature representation, For category labels.

[0068] in, This indicates that shared subspace mapping operations are available.

[0069] (2) Calculate the alignment loss between the missing modality sample and the corresponding class prototype in the shared available subspace, which is expressed as:

[0070] This distillation constraint allows missing modality samples to converge toward the stable semantic center of their respective categories (i.e., toward the center of the complete modality category) within the scope of shared available information, reducing decision boundary drift caused by modality missingness and thus improving the classification stability of the model under arbitrary modality missingness conditions.

[0071] S7, during the training phase, jointly optimizes the classification objective and the prototype distillation objective to obtain network parameters applicable to any modality-deficient scenario.

[0072] (1) Input the final features of the sample into the classification head to obtain the predicted probability of the corresponding category, which is expressed as:

[0073] in These represent the weight matrix and bias term of the classification head, respectively. This represents the final feature representation of the sample.

[0074] (2) Construct the classification loss, which is expressed as:

[0075] Where N represents the number of samples in the current batch.

[0076] (3) The prototype distillation loss of all missing modal samples in the current batch is averaged to obtain:

[0077] in, This represents the set of indices of the missing modality samples in the current training batch. This indicates the number of samples.

[0078] (4) Construct a joint optimization objective, which is expressed as:

[0079] in, The weighting coefficient represents the loss from the prototype distillation.

[0080] By jointly optimizing the classification loss and the prototype distillation loss, the classification performance and class distribution stability of the model can be guaranteed simultaneously.

[0081] S8, during the inference phase, directly classifies and predicts input samples with missing modalities without needing to retrain the model for specific missing patterns.

[0082] (1) The test samples are processed in the same way as the training phase, including input construction, modality encoding, shared backbone feature extraction and dynamic expert adaptation, to obtain the final feature representation.

[0083] (2) The predicted probabilities of each category are output using the classification head, which are expressed as follows:

[0084] Where c represents the candidate category index.

[0085] (3) Finally, the category with the highest predicted probability is taken as the output result, which is expressed as:

[0086] During the training phase, the dynamic expert selection strategy and category semantic constraint relationship under different modal missing modes have been learned. Therefore, during the testing phase, there is no need to retrain the model for specific missing modes. Robust classification under arbitrary modal missing conditions can be achieved, which improves the actual deployment efficiency and application value of the method.

[0087] The beneficial effects of this invention include: First, this invention combines the multimodal classification problem with arbitrary modal missing features with dynamic expert adaptation and category prototype distillation mechanism, avoiding the dependence of traditional multimodal models on complete modal data and the rigid training mode of "learning all categories at once", making it more suitable for long-term deployment and multi-scenario applications.

[0088] This invention designs a dynamic low-rank expert adaptation framework for scenarios with arbitrary modalities missing. This framework can dynamically select appropriate experts for feature transformation based on sample features and modality availability, while maintaining the semantic prototype stability of samples with complete categories, thereby enhancing the complementarity of information from different modalities and decision consistency.

[0089] This invention designs a category semantic prototype distillation mechanism. By constructing a category prototype library using complete modal samples and distilling and aligning missing modal samples in a shared available subspace, the feature representation of missing samples is effectively constrained to move closer to the semantic center of their respective categories, thereby improving the robustness and generalization ability of the model to any missing modal input.

[0090] Secondly, the combination of dynamic low-rank expert adaptation and category prototype distillation in this invention enables the multimodal classification model to continuously learn new categories without forgetting old ones, adapting to changing environments with missing modality combinations, saving computational costs for model retraining and historical data storage space, and meeting the classification task requirements in complex dynamic multimodal scenarios. This method is widely applicable to heterogeneous multimodal data such as images, text, speech, and sensors, and maintains stable classification performance and decision consistency in different application scenarios.

[0091] This disclosure also provides an arbitrary modality missing dynamic learning system based on low-rank fine-tuning, including: The data acquisition and preprocessing module is used to acquire multimodal input data, construct modality availability indicator vectors for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; The modeling module is used to input the modal data of each mode into the corresponding modal encoder, extract the modal feature representation, and send the modal features into the shared backbone network for joint modeling to obtain multimodal intermediate representations; An expert embedding module is used to introduce a dynamic hybrid expert low-rank adaptation module into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modality missing modes. The expert selection module is used to construct a missing-aware routing mechanism. It inputs the intermediate representation of the current sample and the modality availability indicator vector into the routing network to generate the gating weights of each dynamic low-rank expert. Based on the gating weights, at least one dynamic low-rank expert is selected to participate in the calculation, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. The category semantic prototype library construction and update module is used to extract category feature representations from complete modality samples and construct a category semantic prototype library based on category labels; a sliding update strategy is used to continuously update the category semantic prototype library to form a stable category-level semantic anchor point; The distillation alignment module is used to project the feature representation of missing modality samples, which has been processed by the dynamic hybrid expert low-rank adaptation module, onto a shared available subspace and distill and align it with the semantic prototype of the corresponding category. The training module is used to jointly optimize and train the network using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario. The classification module is used to apply the trained model to the testing phase, and processes the input multimodal data to obtain multimodal classification results under arbitrary modality missing conditions.

[0092] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0093] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0094] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A dynamic learning method for arbitrary modality loss based on low-rank fine-tuning, characterized in that, include: S101: Acquire multimodal input data, construct a modality availability indicator vector for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; S102: Input each modal data into the corresponding modal encoder, extract modal feature representations, and send each modal feature into the shared backbone network for joint modeling to obtain multimodal intermediate representations; S103: A dynamic hybrid expert low-rank adaptation module is introduced into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modal missing modes. S104: Construct a missing-aware routing mechanism, input the intermediate representation of the current sample and the modality availability indicator vector into the routing network, generate the gating weights of each dynamic low-rank expert, and select at least one dynamic low-rank expert to participate in the calculation based on the gating weights, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. S105: Extract category feature representations from complete modal samples and construct a category semantic prototype library based on category labels; continuously update the category semantic prototype library using a sliding update strategy to form stable category-level semantic anchors; S106: For missing modality samples, project their feature representations processed by the dynamic hybrid expert low-rank adaptation module onto the shared available subspace, and align them with the semantic prototypes of the corresponding categories by distillation. S107: Joint optimization and training are performed using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario; S108: Apply the trained model to the testing phase, and process the input multimodal data under arbitrary modality missing conditions to obtain multimodal classification results.

2. The arbitrary modality missing dynamic learning method based on low-rank fine-tuning according to claim 1, characterized in that, S102 specifically includes the following steps: Each modal data is input into its corresponding modal encoder to extract modal feature representations; The features of each modality are fused to obtain the initial input representation of the backbone network; By jointly modeling the initial input representations with the shared backbone network, we obtain the intermediate representations of multimodal samples in a unified feature space, i.e., the multimodal intermediate representations.

3. The arbitrary modality missing dynamic learning method based on low-rank fine-tuning according to claim 1, characterized in that, In S103, a dynamic hybrid expert low-rank adaptation module is inserted into one or more feature transformation layers of the shared backbone network.

4. The arbitrary modality missing dynamic learning method based on low-rank fine-tuning according to claim 1, characterized in that, S104 specifically includes the following steps: Embedding mapping is performed on the modality availability indicator vector to obtain the missing mode representation; The multimodal intermediate representations are then concatenated with the missing pattern representations, and the routing features are obtained through a projection function. The initial weights of each dynamic low-rank expert are calculated using the routing matrix; Select the top K dynamic low-rank experts with the largest initial weights to participate in the calculation, and define the gating weights.

5. The arbitrary modality missing dynamic learning method based on low-rank fine-tuning according to claim 1, characterized in that, S105 specifically includes the following steps: Extract category feature representations from complete modality samples and construct a category semantic prototype library based on category labels; For any sample, if all its modalities exist, then its final feature representation after passing through the backbone network and the dynamic hybrid expert low-rank adaptation module is normalized; The semantic prototypes in the semantic prototype library are updated using an exponential moving average method.

6. The arbitrary modality missing dynamic learning method based on low-rank fine-tuning according to claim 1, characterized in that, In S106, the prototype distillation loss for missing modal samples is defined as the cosine alignment loss.

7. A dynamic learning system for arbitrary modal missing features based on low-rank fine-tuning, characterized in that, The system includes: The data acquisition and preprocessing module is used to acquire multimodal input data, construct modality availability indicator vectors for the multimodal input data, and uniformly fill in missing modalities using placeholder input representations; The modeling module is used to input the modal data of each mode into the corresponding modal encoder, extract the modal feature representation, and send the modal features into the shared backbone network for joint modeling to obtain multimodal intermediate representations; An expert embedding module is used to introduce a dynamic hybrid expert low-rank adaptation module into the shared backbone network. The dynamic hybrid expert low-rank adaptation module includes multiple dynamically low-rank experts set in parallel. Each dynamic low-rank expert is used to perform low-rank incremental mapping on the input features. The multiple dynamic low-rank experts participate in the feature adaptation process of the current sample through dynamic routing selection under different modality missing modes. The expert selection module is used to construct a missing-aware routing mechanism. It inputs the intermediate representation of the current sample and the modality availability indicator vector into the routing network to generate the gating weights of each dynamic low-rank expert. Based on the gating weights, at least one dynamic low-rank expert is selected to participate in the calculation, thereby realizing conditional expert activation and adaptive feature modulation for different modality missing modes. The category semantic prototype library construction and update module is used to extract category feature representations from complete modality samples and construct a category semantic prototype library based on category labels; a sliding update strategy is used to continuously update the category semantic prototype library to form a stable category-level semantic anchor point; The distillation alignment module is used to project the feature representation of missing modality samples, which has been processed by the dynamic hybrid expert low-rank adaptation module, onto a shared available subspace and distill and align it with the semantic prototype of the corresponding category. The training module is used to jointly optimize and train the network using classification loss and category semantic prototype distillation loss to obtain network parameters for a multimodal classification model applicable to any modality missing scenario. The classification module is used to apply the trained model to the testing phase, and processes the input multimodal data to obtain multimodal classification results under arbitrary modality missing conditions.

Images