A multi-modal large model continuous learning method for visual question answering task

Abstract

This invention discloses a continuous learning method for multimodal large models for visual question answering tasks. Addressing the cross-task knowledge interference problem caused by differences in image visual distribution and semantic shifts in question instructions when multimodal large models sequentially adapt to various visual question answering tasks such as open-domain question answering, knowledge reasoning question answering, text recognition question answering, image classification, and visual localization, this method incorporates the novel visual-linguistic knowledge captured by the residual stream from various visual question answering tasks into the corresponding global base pool. This results in an expanded global base pool, ensuring the memory stability of the multimodal large model in subsequent visual question answering tasks, reducing spatial redundancy, improving parameter utilization efficiency, and freezing the parameters of the input and output global base pools during LoRA model training, thus achieving zero catastrophic forgetting at the physical level.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and natural language processing, and specifically relates to a multimodal large model continuous learning method for visual question answering tasks. Background Technology

[0002] Multimodal Large Language Models (MLLMs) have been widely used in visual question answering tasks due to their superior vision-language reasoning capabilities. Visual question answering tasks include, but are not limited to, open-domain image question answering, knowledge-based reasoning question answering, text recognition question answering, visual reasoning question answering, image classification, and visual localization. Each task takes an image and a natural language question as input and a text answer as output. However, the data flow in real-world deployment scenarios typically exhibits dynamic evolution, requiring models to sequentially adapt to the various visual question answering tasks. The visual distribution of images differs significantly across tasks; for example, natural scene images (open-domain question answering), text-dense images (text recognition question answering), and scientific charts (knowledge-based reasoning question answering) differ greatly in their low-level visual features. Furthermore, the semantic structure of the question commands also exhibits cross-domain shifts; for instance, "What animal is in the picture?" and "Which location in the image corresponds to this region?" have drastically different intents. During continuous learning for new tasks, parameter updates can easily overwrite visual-language alignment knowledge learned from older tasks, leading to a sharp decline in the model's ability to answer previously learned visual question-answering tasks—a phenomenon known as catastrophic forgetting. How to enable the model to effectively maintain its ability to answer previously learned tasks while sequentially adapting to multiple types of visual question-answering tasks—that is, achieving an effective balance between memory stability and learning plasticity—has become a core technical challenge that urgently needs to be addressed in the field of continuous learning for visual question answering.

[0003] Currently, continuous learning techniques based on Low-Rank Adaptation (LoRA) mainly fall into two categories. The first is based on regularization, which attempts to preserve historical knowledge by imposing constraints on weight updates. This strong constraint mechanism often weakens the model's ability to learn new tasks, making it difficult for the model to maintain stability while simultaneously adapting to new tasks. The second is based on parameter isolation, typically employing a hybrid expert architecture. While fixed-architecture methods attempt to improve parameter efficiency by reusing shared LoRA experts, updating shared experts easily leads to knowledge interference between tasks. Dynamic-architecture methods, while completely avoiding interference by allocating independent LoRA modules for each new task, result in significant storage overhead. Furthermore, due to complete parameter isolation, dynamic-architecture methods cannot effectively utilize common knowledge across tasks.

[0004] To address the aforementioned issues, optimizing continuous learning efficiency by leveraging the geometric properties of the model parameter space has become a new research direction. Existing parameter isolation strategies are mostly limited to coarse-grained expert selection, i.e., routing between discrete expert modules. This invention, through in-depth geometric analysis, reveals that despite the semantic differences between different tasks, there is significant subspace redundancy between the update matrices learned by different tasks within the LoRA parameter space. This means that the knowledge of different tasks can essentially be reconstructed through a linear combination of a shared set of low-dimensional orthogonal basis bases, without needing to store completely independent parameters for each task. Therefore, there is an urgent need to propose a multimodal continuous learning method based on LoRA orthogonal basis sharing and mixing, aiming to alleviate the stability and plasticity dilemma in continuous learning and significantly improve parameter utilization efficiency while ensuring an extremely low forgetting rate. Summary of the Invention

[0005] This invention provides a multimodal large model continuous learning method for visual question answering tasks, including: A LoRA model comprising a multiplexed stream and a residual stream is constructed. The multiplexed stream includes an initial mixing matrix, an input global basis pooling matrix, and an output global basis pooling matrix. The residual stream and the initial mixing matrix are updated with gradients based on the training sample set for each visual question answering task. The training samples include images, question instructions, and corresponding text answers. The input low-rank matrix and output low-rank matrix of the residual stream after gradient update are decomposed to obtain multiple residual vector pairs constructed from input vectors and corresponding output vectors. The input vectors and corresponding output vectors are projected onto the corresponding input or output global basis pooling, and the input or output vectors with fidelity below a threshold are selected. The selected input or output vectors are transformed and added to the corresponding input or output global basis pooling to obtain the expanded global basis pooling. The mixture matrix of the current visual question answering task is obtained based on the expanded global basis pooling, residual vector pairs and gradient-updated mixture matrix. The final expanded global basis pooling is obtained after training on multiple stored visual question answering tasks. When applied, the task that is semantically closest to the input visual question answering task is indexed from multiple stored visual question answering tasks. The weight update amount corresponding to the current task is obtained based on the mixing matrix corresponding to the found visual question answering task and the final expanded global base pool.

[0006] Preferably, the method for obtaining the initial mixing matrix includes: Based on the image instance and text instruction of the current visual question answering task, the visual feature centroid and instruction feature centroid of the current task are obtained through the visual encoder and instruction encoder. The mixing coefficients of historical tasks are obtained by normalizing the weighted cosine similarity of the visual feature centroids and instruction feature centroids of the current visual question answering task and the historical visual question answering task. The initial mixture matrix is ​​obtained by aligning the mixture matrices of each historical task to the dimension of the current input global base pool or output global base pool, multiplying them by the corresponding mixture coefficients, and then summing them.

[0007] Preferably, the visual feature centroid and instruction feature centroid of the current task are obtained based on the image instance and text instruction of the current task through a visual encoder and an instruction encoder, including: We obtain multiple image instances and text instructions for the current visual question answering task, and then calculate the average value of each image instance to obtain the visual feature centroid. The centroid of the instruction feature is obtained by averaging the values ​​of each text instruction after encoding it using an instruction encoder.

[0008] Preferably, the mixing coefficients for the historical tasks are obtained by normalizing the weighted cosine similarity of the visual feature centroids and instruction feature centroids of the current visual question answering task and the historical visual question answering task, including: The visual feature centroid similarity value is obtained by comparing the visual feature centroids of the current visual question answering task and the visual feature centroids of the historical visual question answering tasks using cosine similarity. The similarity value of the instruction centroids is obtained by comparing the cosine similarity between the centroids of the instruction features in the current visual question answering task and the centroids of the instruction features in the historical tasks. The weighted sum of the centroid similarity values ​​of visual features and command features is then normalized using the Softmax function with a temperature coefficient to obtain the mixing coefficient of the current historical task.

[0009] Preferably, the task that is semantically closest to the input visual question answering task is indexed from multiple stored visual question answering tasks, including: Construct a unique task key for the current task based on the visual feature centroid and instruction feature centroid of each visual question answering task, and store the unique task keys for the multiple visual question answering tasks. The image instances and text instructions of the input visual question answering task are semantically matched with the unique task key of each stored visual question answering task to find the visual question answering task with the closest semantics.

[0010] Preferably, the decomposition of the input low-rank matrix and output low-rank matrix of the updated residual stream includes: The input low-rank matrix is ​​decomposed column by column to obtain multiple input vectors, and the output low-rank matrix is ​​decomposed row by row to obtain multiple output vectors corresponding to the input vectors. Each input vector and its corresponding output vector form a residual vector pair.

[0011] Preferably, after projecting the input vector and the corresponding output vector onto the corresponding input or output global basis pooling, input or output vectors with fidelity below a threshold are selected, including: Project the input vector and the corresponding output vector of each residual vector pair onto the corresponding input or output global basis pooling to obtain the best reconstruction approximation on the input or output global basis pooling. The input vector and its corresponding output vector are compared with the best reconstruction approximation on the input or output global basis pooling to obtain the fidelity of the input vector and its corresponding output vector. If the fidelity is lower than the threshold, the corresponding input or output vector is orthogonalized and normalized and then added to the corresponding input or output global base pool to obtain an expanded global base pool. The expanded global base pool includes an input expanded global base pool or an output expanded global base pool. If the fidelity is greater than or equal to the threshold, the corresponding input or output vector is pruned.

[0012] Preferably, the mixture matrix for the current task is obtained based on the expanded global base pooling, residual vector pairs, and the mixture matrix updated by the gradient. k Mixing matrix for: , , ,in, To update the blending matrix after applying the gradients of the current task The dimensions are unified to either the input global base pool or the output global base pool. This is the scaling factor for the residual flow, used to adjust the intensity of learning new features; is the rank of the residual matrix, used to control the order of magnitude of the parameters in the residual flow. and The first i Input vector of residual vector pairs Input expand global base pool The coordinate projection on the coordinate plane and the corresponding output vector Expanding the global base pool in the output Coordinate projection on.

[0013] Preferably, the final expanded global base pool is obtained after training on multiple tasks based on storage, including: The expanded global basis pooling includes an input expanded global basis pooling or an output expanded global basis pooling. The input expanded global basis pooling or the output expanded global basis pooling corresponding to the previous task is used as the input global basis pooling and the output global basis pooling of the LoRA model for the current task. After training the LoRA model on multiple tasks, the final expanded global basis pooling is obtained by supplementing the input global basis pooling and the output global basis pooling with corresponding input vectors and output vectors.

[0014] Preferably, gradient updates are performed on the residual stream and the initial mixing matrix based on the training sample set for each visual question answering task, including: Obtain the training sample set corresponding to each task, and train the LoRA model using the training sample set corresponding to each visual question answering task. The LoRA model also includes pre-trained weights. During the training process, freeze the input global base pooling, the output global base pooling, and the pre-trained weights, and update the gradient of the residual flow and the initial mixing matrix using the cross loss function constructed from the predicted values ​​and the ground values.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention preserves the original global base pooling when training the residual stream and initial mixture matrix of the LoRA model for each visual response task. This ensures that the updated mixture matrix retains the original information, thereby suppressing catastrophic forgetting in visual response tasks. During training, this invention introduces a large amount of new information by updating the residual stream. After decomposing the updated residual stream, it projects it into the global base pooling and compares it with fidelity. Information with high repetition with previous information is pruned, while information with low repetition, i.e., fresher information, is retained. This achieves plasticity, i.e., learning new information while improving parameter utilization efficiency and computational efficiency. Attached Figure Description

[0016] Figure 1 The flowchart illustrates a multimodal large-model continuous learning method for visual question answering tasks, as provided in a specific embodiment of the present invention. Detailed Implementation

[0017] This invention addresses the catastrophic forgetting problem caused by cross-task knowledge interference when multimodal large models sequentially adapt to multiple visual question-answering tasks. It provides a continuous learning method for multimodal large models oriented towards visual question-answering tasks. Starting from the geometric redundancy characteristics of the low-rank adaptation parameter space, the method abandons the traditional discrete expert selection paradigm and proposes a decoupled design that uses a shared orthogonal basis and a task-specific mixing matrix to represent knowledge from multiple visual question-answering tasks.

[0018] This invention addresses the catastrophic forgetting problem caused by cross-task knowledge interference when multimodal large models sequentially adapt to multiple visual question-answering tasks. It provides a continuous learning method for multimodal large models oriented towards visual question-answering tasks. Starting from the geometric redundancy characteristics of the low-rank adaptation parameter space, the method abandons the traditional discrete expert selection paradigm and proposes a decoupled design that uses a shared orthogonal basis and a task-specific mixing matrix to represent knowledge from multiple visual question-answering tasks.

[0019] This invention incorporates the novel visual-linguistic knowledge captured by the residual stream from different tasks into the corresponding input or output global base pool by including the low-fidelity input or output vectors obtained from decomposing the residual stream. This results in an expanded global base pool, which ensures the memory stability of the multimodal large model in subsequent applications, reduces spatial redundancy, and improves parameter utilization efficiency. Furthermore, when training the LoRA model, the parameters of the input and output global base pools are frozen, thus achieving zero catastrophic forgetting at the physical level. This effectively suppresses cross-task knowledge interference caused by differences in image visual distribution and semantic shifts in question instructions, ensuring that the model's ability to answer learned visual question-answering tasks is not overwritten by new tasks. This invention constructs a dedicated mixing matrix for each type of visual question answering task, integrating the weights of the residual stream into the mixing matrix. This results in different tasks having corresponding mixing matrices. In subsequent applications, it can automatically route to the most matching historical task parameters based on the semantic features of the input image and the question instruction. It indexes the task with the closest semantics to the input task from multiple stored tasks, and obtains the weight update amount for the current task based on the mixing matrix corresponding to the found task and the final expanded global base pool. While retaining the answering ability of learned visual question answering tasks, it continuously acquires visual-language alignment knowledge for new tasks, thereby achieving an effective balance between memory stability and learning plasticity, and significantly improving parameter utilization efficiency and computational efficiency.

[0020] This invention mitigates cross-task knowledge interference and suppresses catastrophic forgetting at the parameter isolation level through a collaborative design of incremental expansion of the basal pool and independent storage of hybrid matrix tasks. The weight update amount during model learning of the i-th visual question-answering task is decomposed into a shared global basal pool (…). , ) and task-specific hybrid matrix The product form: When learning new tasks, this scheme ensures interference-free isolation of knowledge from old tasks through both the basal pool and the blending matrix. Regarding the basal pool... , Freeze during training and update incrementally only in the progressive subspace augmentation strategy if and only if there exists a fidelity in the residual stream below a threshold. Only when a novel component is obtained is that component orthogonalized and appended as a new basis vector; the original basis vectors remain unchanged. After the basis pool is expanded, the mixture matrix corresponding to historical task k is... Through zero filling Alignment is performed to the new base pool dimension, meaning the blending coefficients corresponding to the new dimension are initialized to zero, thus ensuring the expanded globally shared base pool. and Product with historical mixing matrix Numerically, it is completely equivalent to the original version, and the weight updates for historical tasks are unaffected by the base pool expansion. Regarding the blending matrix, each task k has its own independent blending matrix. Learning new tasks At that time, gradient updates only apply to The old task k corresponds to Without participating in any optimization process, its stored value is fully preserved. Combining the above two points, this invention ensures that the historical subspace is not tampered with at the basal pool level through incremental expansion and zero-filling, and completely avoids parameter overwriting at the hybrid matrix level through task-independent storage. Thus, at the parameter isolation level, it cuts off the propagation path of cross-task knowledge interference, thereby achieving the suppression of catastrophic forgetting from a mechanistic perspective.

[0021] This invention achieves decoupling of stability and flexibility at the architectural level through a two-stream structure. When learning the k-th visual question-answering task, the reuse stream is based on a frozen global base pool, optimizing only the task-specific blending matrix. Since the global base pooling parameters do not participate in gradient updates, the weight update amounts corresponding to historical tasks are... Throughout the optimization process of the reuse stream, the numerical values ​​remain constant, thus ensuring the model's stable memorization of learned tasks. Simultaneously, a task-aware initialization strategy is used to weight the historical mixing matrix with the mixing coefficients of historical tasks, forming the mixing matrix for the new task k. Semantic warm-start is provided, enabling the reuse stream to effectively utilize historical priors during the initial training phase, further promoting the positive transfer of historical knowledge. The residual stream introduces a trainable low-rank matrix. , Its optimization objective is to capture novel semantic components in the current task, specifically fitting visual-linguistic features that cannot be linearly represented by the existing basis subspace. This provides the model with sufficient parameter freedom to adapt to the distribution shift unique to the new task, ensuring learning plasticity. The gradient update objectives of the two sub-streams are completely separated; the loss gradient of the reused stream is only propagated to... The loss gradient of the residual flow is only propagated to , The two are not coupled in the parameter space, thus fundamentally ensuring the simultaneous realization of stability and flexibility at the architectural level.

[0022] This invention improves parameter efficiency through a fidelity filtering mechanism. Traditional dynamic architecture methods allocate a complete, independent LoRA module to each visual question-answering task, with the number of parameters increasing strictly linearly with the number of tasks K. The parameter growth pattern of this invention is determined by the fidelity filtering mechanism, which optimizes the parameter selection process for fidelity. The components, of which It is a fidelity threshold; if it is determined that the vector can be fully expressed by the existing basis subspace, it is directly pruned without adding a new vector to the basis pool; only fidelity is considered. The novel components are orthogonalized before being added to the basis pool. Due to the widespread cross-task subspace redundancy between visual-language semantics of different tasks, a large number of components are determined to be reconstructable from the existing basis and are pruned during the projection test. This results in the parameter growth rate of this invention being much lower than the linear upper bound of traditional dynamic architecture methods, and the parameter efficiency continuously improving with the increase of the number of tasks.

[0023] This invention provides a multimodal large model continuous learning method for visual question answering tasks, such as... Figure 1 As shown, it includes: S1. Construct a task-aware, two-stream trained model, which includes a pre-trained multimodal large model and a task-aware, two-stream trained LoRA model. The LoRA model includes a reuse stream, a residual stream, and pre-trained weights. The reuse stream includes an initial mixing matrix. Wmix Input global base pool A pool and output global base pool B pool The residual stream includes the input low-rank matrix. A res and output low-rank matrix B res Gradient updates are performed on the residual flow and initial mixing matrix of the LoRA model based on the training sample set for each task, while the input global basis pooling is frozen during training. A pool Output global base pool B pool With regard to pre-trained weights, it can be understood that the multimodal large model provided in the specific embodiments of the present invention is a pre-trained large model with a certain semantic recognition capability.

[0024] In one specific embodiment, this embodiment performs gradient updates on the residual flow and the initial mixing matrix based on the training sample set for each task, including: A training sample set is obtained for each visual question answering task. Each sample in the training sample set contains an image, a question instruction, and a corresponding text answer. The LoRA model is trained using the training sample set for each task. During the training process, the input global base pooling, the output global base pooling, and the pre-trained weights are frozen. The gradient of the residual flow and the initial mixing matrix is ​​updated using the cross loss function constructed from the predicted values ​​and ground truth values ​​of the multimodal large model output.

[0025] In one specific embodiment, in order to accelerate the convergence of the reuse stream and promote the positive transfer of knowledge, this invention proposes a task-aware initialization strategy based on semantic prototypes, namely, a method for obtaining an initial mixing matrix, comprising: obtaining the visual feature centroid and instruction feature centroid of the current task based on the image instance and text instruction of the current visual question answering task through a visual encoder and an instruction encoder; normalizing the visual feature centroid and instruction feature centroid of the current visual question answering task and the historical visual question answering task through weighted cosine similarity to obtain the mixing coefficients of the historical tasks; aligning the mixing matrices of each historical visual question answering task to the dimension of the current input global base pool or output global base pool, multiplying them with the corresponding mixing coefficients, and summing them to obtain the initial mixing matrix.

[0026] In one specific embodiment, this embodiment obtains the visual feature centroid and instruction feature centroid of the current task based on the image instance and text instruction of the current visual question answering task through a visual encoder and an instruction encoder, including: The system obtains multiple image instances and text instructions for the current task, encodes each image instance using a visual encoder, and calculates the mean value to obtain the visual feature centroid. The centroid of the instruction feature is obtained by averaging the values ​​of each text instruction after encoding it using an instruction encoder.

[0027] In one specific embodiment, this embodiment normalizes the weighted cosine similarity of the visual feature centroids and instruction feature centroids of the current visual question answering task and the historical visual question answering task to obtain the mixing coefficients of the historical task, including: The similarity value of the visual feature centroids of the current visual question answering task and the historical visual question answering tasks is obtained by comparing the cosine similarity. The similarity value of the instruction feature centroids of the current task and the historical instruction feature centroids of the visual question answering tasks is obtained by comparing the cosine similarity. The similarity values ​​of the visual feature centroids and the instruction feature centroids are then weighted, summed, and normalized using the Softmax function with a temperature coefficient to obtain the mixing coefficients of the current and historical tasks.

[0028] Specifically, for the first Each task, the system randomly samples Each instance utilizes a visual encoder. and instruction encoder The modality-specific eigencentroids are calculated using the following formula: ,in, and These represent the first and second parts of the task. One image instance and one text instruction instance, The total number of sampled instances. The resulting task key. This characterizes the semantic center of the task. Subsequently, to quantify the current task versus historical tasks... The correlation between them is determined by weighted cosine similarity in this invention. To perform the measurement, by introducing weighting coefficients. and To balance the contributions of visual and textual modalities: ， The calculated similarity scores are normalized using a Softmax function with a temperature coefficient to obtain the mixing coefficient. Finally, the mixing matrix of historical tasks is weighted and summed using this coefficient to obtain the initial mixing matrix for the current task. Initialization parameters: ,in The operation is used to fill and align the historical smaller-dimensional matrices to the dimensions of the current base pooling. This initialization method allows the model to perform a warm start based on similar historical experience, significantly improving training efficiency and performance.

[0029] S2. Decompose the input low-rank matrix and output low-rank matrix of the residual stream after gradient update to obtain multiple residual vector pairs constructed from input vectors and corresponding output vectors. Project the input vectors and corresponding output vectors to the corresponding input or output global basis pooling, and then filter out the input or output vectors with fidelity below the threshold. Transform the filtered input or output vectors and add them to the corresponding input or output global basis pooling to obtain the expanded global basis pooling. Based on the expanded global basis pooling, residual vector pairs and gradient-updated mixing matrix, obtain the mixing matrix of the current task. After training multiple tasks, the final expanded global basis pooling is obtained.

[0030] Specifically, to efficiently integrate novel knowledge captured in the residual stream into a shared basis pool while strictly controlling the growth of parameter size, this invention proposes a progressive subspace augmentation strategy based on projection fidelity. After the training phase of the current task is completed, the low-rank matrix in the residual stream is first... and Decomposed into For rank-1 vectors ,in and These represent the input projection direction and the output reconstruction direction, respectively. Matrix A is first multiplied by the input x, which is equivalent to projecting the input feature vector onto the space of matrix A. The intermediate result is then multiplied by matrix B to obtain the output, meaning matrix B plays the role of reconstructing the output. Subsequently, the system quantifies the novelty of each component relative to the current base pooling. Using the input vector... For example, calculate its position in the current basis space. Projection fidelity in : , here This represents the best approximation of the reconstruction of the vector in the current low-dimensional subspace, while The value reflects the explanatory power of the existing basis for this new feature. A specific embodiment of the present invention introduces a preset fidelity threshold. This dynamically determines whether to extend the substrate. The calculated fidelity... This indicates that the component contains novel semantic information that cannot be expressed by the existing basis vectors. The system will then perform Gram-Schmidt orthogonalization on the vector and normalize it into a new basis vector. Join In the middle. Conversely, if If the component is deemed redundant, it is pruned to prevent unnecessary parameter inflation. This process is also applied to... The update will enhance it. or enhanced That is, the expanded global shared base pool and .

[0031] After updating and expanding the basis pool, to maintain the consistency of the model architecture during the inference phase, this invention further performs a lossless merging operation, updating the weights of the residual streams into the mixing matrix. Specifically, the coordinate projection of the residual vectors in the expanded basis space is first calculated. and , These refer to the expanded globally shared base pool, which is then updated according to the following formula to obtain the current blending matrix for the visual question answering task. : , , ,in, To update the blending matrix after applying the gradients of the current task The dimensions are unified to either the input global base pool or the output global base pool. This is the scaling factor for the residual flow, used to adjust the intensity of learning new features; is the rank of the residual matrix, used to control the order of magnitude of the parameters in the residual flow. and The first i Input vector of residual vector pairs Input expand global base pool The coordinate projection on the coordinate plane and the corresponding output vector Expanding the global base pool in the output The coordinate projection on the surface. Through the above steps, the present invention successfully reunifies the dual-stream architecture into a single-stream form. It preserves historical precedents while seamlessly integrating newly acquired skills.

[0032] S3. During application, the task that is semantically closest to the input visual question answering task is indexed from multiple stored visual question answering tasks. Based on the mixing matrix corresponding to the found visual question answering task and the final expanded global base pool, the weight update amount corresponding to the current task is obtained. The weight update amount is added to the corresponding parameters of the multimodal large model to obtain the updated multimodal large model. The input visual question answering task obtains the corresponding answer through the updated multimodal large model.

[0033] Specifically, addressing the technical challenge of not being able to pre-obtain task identifiers in practical application scenarios, this invention designs and implements a task-independent, non-parametric dynamic routing reasoning mechanism. This mechanism utilizes task keys constructed during the training phase as semantic anchors to achieve automatic retrieval and accurate matching of historical skills. Specifically, during training, the system caches the visual feature centroid and instruction feature centroid of each task, forming a unique task key for that task. During the reasoning phase, when faced with unknown input images... and text instructions The model first extracts its modality-specific feature representations through a pre-trained encoder. To accurately measure the fit between the current input and historical visual question-answering tasks, this invention employs a multimodal weighted similarity metric consistent with the training initialization phase. Based on this comprehensive similarity metric, the system performs nearest neighbor matching to identify the optimal task index. The calculation method is as follows: ,in, and These represent the instructions to be processed and the image input during the inference phase, respectively. For the first A set of historical task keys for visual question answering tasks are used for semantic matching with the current input. Once the optimal task index is determined... The system will exclusively activate the blending matrix corresponding to that index. and combined with a shared expanded base pool and The reconstructed weight update amount, adapted to the current input, is obtained using the following formula: Through this mechanism, the model can automatically route to the most matching subspace combination based solely on the semantic features of the input data, even in the complete absence of explicit task identifiers. This not only effectively utilizes targeted domain knowledge for reasoning but also minimizes knowledge interference between different tasks, achieving highly automated adaptive reasoning.

[0034] In the preferred embodiment using LLaVA-1.5-7B as the backbone multimodal large model, the visual encoder employs a pre-trained CLIP-ViT-L / 336px, and all linear layers of the language model are inserted into the LoRA module of this invention. The training dataset sequentially includes eight visual question answering task datasets: ScienceQA, TextVQA, ImageNet, GQA, VizWiz, Grounding, VQAv2, and OCR-VQA. The types of visual question answering tasks covered are significantly diverse, including knowledge reasoning question answering with scientific charts as input, text recognition question answering with text-dense images as input, image classification question answering with natural scene images as input, visual reasoning question answering with composite scene images as input, assistive question answering for visually impaired individuals, visual localization question answering with region descriptions as queries, open-domain question answering with open natural language as the question format, and OCR question answering with printed text images as input. The eight visual question-answering tasks described above exhibit significant cross-domain differences in both image visual distribution and question instruction semantic structure, posing a rigorous test to the anti-forgetting ability and task adaptability of multimodal continuous learning methods in visual question-answering scenarios. The training rounds for each visual question-answering task were set to 1, the warm-up ratio to 0.03, and a cosine decay learning rate scheduling strategy was adopted. The learning rate of the LoRA module was set to... The projector's learning rate is set to The batch size for all visual question answering tasks is uniformly set to 128. The rank of the residual stream is set to r=8. The projection fidelity threshold is set to During task key construction, N=1000 instances are randomly sampled for each visual question-answering task. The visual feature centroids and instruction feature centroids are calculated using both the visual encoder and instruction encoder. The modal weights for task perception initialization are set to... During training, the parameters of the visual encoder, language model backbone, and global basal pooling are all frozen; only the residual flow matrix is ​​adjusted. and the hybrid matrix of current visual question answering tasks Perform gradient updates.

[0035] To further verify the technical effectiveness of this invention in continuous learning scenarios across multiple visual question-answering tasks, a comparative experiment was conducted on the CoIN benchmark using LoRA sequential fine-tuning and MoELoRA as baseline methods. LoRA sequential fine-tuning is based on a low-rank adaptation method published at the International Conference on Learning Representations (ICLR 2022) in 2022, while MoELoRA is based on a hybrid expert low-rank adaptation method for multimodal continuous learning published at the NeurIPS Conference on Neural Information Processing Systems (NeurIPS 2024) in 2024. Both baseline methods use the same backbone model LLaVA-1.5-7B as this invention, and the LoRA module is inserted into all linear layers of the language model. Hyperparameters such as learning rate, batch size, and number of training epochs are consistent with this invention. The rank of LoRA sequential fine-tuning is set to... When a new visual question answering task arrives, sequential gradient updates are performed directly on the same set of LoRA parameters without employing any anti-forgetting mechanism; the rank of MoELoRA is also set to... Each LoRA layer is replaced with a hybrid expert structure consisting of two experts. Expert weights are dynamically assigned to the input through a learnable routing network, and all expert parameters are continuously shared and updated in the visual question answering task sequence.

[0036] The evaluation indicators include three items, set as follows: The average accuracy is the model's performance on the i-th task after training on the j-th visual question answering task, for a total of K tasks. The overall performance of the model after it has been trained to complete all visual question-answering tasks in sequence is measured by the forgetting rate. The quantitative model's performance decline on previously learned tasks after learning a new visual question answering task; accuracy on the new task. This reflects the model's immediate learning plasticity for each new visual question-answering task. The comparison results of the three metrics are shown in Table 1.

[0037] Table 1 Comparison of the effects of different models

[0038] After sequentially learning eight visual question answering tasks, LoRA's average accuracy, forgetting rate, and new task accuracy were 28.74%, 37.29%, and 61.36%, respectively, indicating severe catastrophic forgetting during its continuous adaptation to multiple visual question answering tasks. While MoELoRA improved model capacity to some extent through a hybrid expert mechanism, the problem of cross-task knowledge interference still exists due to the continuous sharing and updating of expert parameters across tasks. The three metrics were 35.86%, 25.71%, and 58.36%, respectively, with a forgetting rate still as high as 25.71%, indicating that a fixed-architecture shared expert strategy cannot fundamentally solve the catastrophic forgetting problem. Under the same visual question-answering task sequence and evaluation conditions, this invention achieves 65.34%, 0.00%, and 65.34% accuracy in three metrics, respectively. The average accuracy is 36.60 percentage points higher than LoRA sequential fine-tuning and 29.48 percentage points higher than MoELoRA. The forgetting rate is reduced from 37.29% and 25.71% to 0.00%, respectively. The accuracy on new tasks is also superior to the techniques of the two existing models. This fully demonstrates that in the process of sequentially adapting to multiple cross-domain visual question-answering tasks, this invention can fundamentally cut off the propagation path of cross-task knowledge interference, effectively suppress catastrophic forgetting, and at the same time maintain sufficient learning ability for new visual question-answering tasks. The overall performance is significantly better than the techniques of existing models.

Claims

1. A multimodal large-scale model continuous learning method for visual question answering tasks, characterized in that, include: Construct a LoRA model that includes a multiplexed stream and a residual stream, wherein the multiplexed stream includes an initial mixing matrix, an input global basis pooling, and an output global basis pooling; The training sample set for each visual question answering task is used to update the gradient of the LoRA model's residual flow and initial mixing matrix through a multimodal large model and a LoRA model. The training samples include images, question instructions and corresponding text answers. The input low-rank matrix and output low-rank matrix of the residual stream after gradient update are decomposed to obtain multiple residual vector pairs constructed from input vectors and corresponding output vectors. The input vectors and corresponding output vectors are projected onto the corresponding input or output global basis pooling, and the input or output vectors with fidelity below a threshold are selected. The selected input or output vectors are transformed and added to the corresponding input or output global basis pooling to obtain the expanded global basis pooling. The mixture matrix of the current visual question answering task is obtained based on the expanded global basis pooling, residual vector pairs and gradient-updated mixture matrix. The final expanded global basis pooling is obtained after training on multiple stored visual question answering tasks. In application, the task that is semantically closest to the input visual question answering task is indexed from multiple stored visual question answering tasks. Based on the mixing matrix corresponding to the found visual question answering task and the final expanded global base pool, the weight update amount corresponding to the current task is obtained. The weight update amount is added to the corresponding parameters of the multimodal large model to obtain the updated multimodal large model. The input visual question answering task obtains the corresponding answer through the updated multimodal large model.

2. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, Methods for obtaining the initial mixing matrix include: Based on the image instance and text instruction of the current task, the visual feature centroid and instruction feature centroid of the current visual question answering task are obtained through the visual encoder and instruction encoder. The mixing coefficients of historical tasks are obtained by normalizing the weighted cosine similarity of the visual feature centroids and instruction feature centroids of the current visual question answering task and the historical visual question answering task. The initial mixture matrix is ​​obtained by aligning the mixture matrices of each historical task to the dimension of the current input global base pool or output global base pool, multiplying them by the corresponding mixture coefficients, and then summing them.

3. The multimodal large model continuous learning method for visual question answering tasks according to claim 2, characterized in that, Based on the image instance and text instruction of the current visual question answering task, the visual feature centroid and instruction feature centroid of the current visual question answering task are obtained through the visual encoder and instruction encoder, including: We obtain multiple image instances and text instructions for the current visual question answering task, and then calculate the average value of each image instance to obtain the visual feature centroid. The centroid of the instruction feature is obtained by averaging the values ​​of each text instruction after encoding it using an instruction encoder.

4. The multimodal large model continuous learning method for visual question answering tasks according to claim 2, characterized in that, The mixture coefficients for historical visual question answering tasks are obtained by normalizing the weighted cosine similarity of the visual feature centroids and instruction feature centroids of the current and historical visual question answering tasks, including: The visual feature centroid similarity value is obtained by comparing the cosine similarity between the visual feature centroids of the current visual question answering task and the visual feature centroids of the historical tasks. The similarity value of the instruction centroids is obtained by comparing the cosine similarity between the centroids of the instruction features in the current visual question answering task and the centroids of the instruction features in the historical tasks. The mixture coefficients of historical visual question answering tasks are obtained by weighted summation of the centroid similarity values ​​of visual features and instruction features, followed by normalization using the Softmax function with a temperature coefficient.

5. The multimodal large model continuous learning method for visual question answering tasks according to claim 2, characterized in that, From a storage of multiple visual question answering tasks, the visual question answering task that is semantically closest to the input task is indexed, including: Construct a unique task key for the current visual question answering task based on the visual feature centroid and instruction feature centroid of each visual question answering task, and store the unique task keys for the multiple visual question answering tasks. The input image instances and text instructions for the visual question answering task are semantically matched with the unique task key of each task in the storage to find the task with the closest semantics.

6. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, Decompose the input low-rank matrix and output low-rank matrix of the updated residual stream, including: The input low-rank matrix is ​​decomposed column by column to obtain multiple input vectors, and the output low-rank matrix is ​​decomposed row by row to obtain multiple output vectors corresponding to the input vectors. Each input vector and its corresponding output vector form a residual vector pair.

7. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, After projecting the input vector and its corresponding output vector onto the corresponding global base pooling, input or output vectors with fidelity below a threshold are selected, including: Project the input vector and the corresponding output vector of each residual vector pair onto the corresponding input or output global basis pooling to obtain the best reconstruction approximation on the input or output global basis pooling. The input vector and its corresponding output vector are compared with the best reconstruction approximation on the input or output global basis pooling to obtain the fidelity of the input vector and its corresponding output vector. If the fidelity is lower than the threshold, the corresponding input or output vector is orthogonalized and normalized and then added to the corresponding input or output global base pool to obtain an expanded global base pool. The expanded global base pool includes an input expanded global base pool or an output expanded global base pool. If the fidelity is greater than or equal to the threshold, the corresponding input or output vector is pruned.

8. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, The mixture matrix for the current task is obtained based on the extended global basis pooling, residual vector pairs, and the mixture matrix updated by gradients. k Mixing matrix for: , , ,in, To update the blending matrix after applying the gradients of the current task The dimensions are unified to either the input global base pool or the output global base pool. This is the scaling factor for the residual flow, used to adjust the intensity of learning new features; is the rank of the residual matrix, used to control the order of magnitude of the parameters in the residual flow. and The first i Input vector of residual vector pairs Input expand global base pool The coordinate projection on the coordinate plane and the corresponding output vector Expanding the global base pool in the output Coordinate projection on.

9. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, After training on multiple visual question answering tasks based on storage, the final expanded global base pool is obtained, including: The expanded global basis pooling includes an input expanded global basis pooling or an output expanded global basis pooling. The input expanded global basis pooling or the output expanded global basis pooling corresponding to the previous task is used as the input global basis pooling and the output global basis pooling of the LoRA model for the current task. The LoRA model is trained on multiple visual question answering tasks, and the corresponding input vectors and output vectors are added to the input global basis pooling and the output global basis pooling to obtain the final expanded global basis pooling.

10. The multimodal large model continuous learning method for visual question answering tasks according to claim 1, characterized in that, Gradient updates are performed on the residual stream and initial mixing matrix based on the training sample set for each visual question answering task, including: A training sample set corresponding to each visual question answering is obtained, and a LoRA model is trained using the training sample set corresponding to each task. The LoRA model also includes pre-trained weights. During the training process, the input global base pooling, output global base pooling, and pre-trained weights are frozen, and the gradient of the residual flow and the initial mixing matrix is ​​updated using the cross loss function constructed from the predicted values ​​and the ground truth values.

Images