Visual task processing method and device based on multi-modal large model of low-rank adaptive enhancement

Abstract

The invention provides a visual task processing method and device based on a multi-modal large model of low-rank adaptive enhancement, and relates to the technical field of data processing. The method comprises the following steps: receiving a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences through an orchestrator, and determining a reasoning mode of a low-rank adapter used by each batch of visual input data; and inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic fragmentation calculation on the visual input data and a decomposition matrix of the corresponding low-rank adapter through an adaptive fragmentation matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request. Through the method provided by the invention, diversified visual task requirements are met.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly to a method and device for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement. Background Art

[0002] With the rapid development of artificial intelligence technology, multi-modal large models (LMMs) are increasingly widely used in visual tasks. By combining visual and language information, these models can handle complex multi-modal tasks, such as visual question answering, object detection, and video understanding. However, existing LMMs often show insufficient performance when dealing with specific-domain visual tasks, especially in tasks that require domain-specific knowledge.

[0003] How to meet the diverse visual task requirements is a technical problem that needs to be solved currently. Summary of the Invention

[0004] The present invention provides a method and device for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement to solve the defects existing in the prior art.

[0005] The present invention provides a method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement, including the following steps: Receiving a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; Dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences by an orchestrator, and determining the inference mode of a low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; Inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method.

[0006] According to the method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention, each visual task request further includes a corresponding task type identifier; Inputting the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the visual input data and the decomposed matrix of the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request, including: Inputting the visual input data of each batch, the corresponding task type identifier, and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the visual input data and the decomposed matrix of the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the multi-modal intermediate features corresponding to each visual task request; According to the multi-modal intermediate features corresponding to each visual task request, calling the corresponding visual task head through the task type identifier, and outputting the response result corresponding to each visual task request; wherein, the visual task head is: a trainable linear layer integrated in the low-rank adapter for mapping the multi-modal intermediate features to the discrete output space of the visual task request.

[0007] According to a method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention, the generation process of the low-rank adapter includes: Obtaining the target external knowledge of the target task; wherein, the target external knowledge includes: a specific domain data set and the model parameters of a specific domain small model; Based on a pre-designed accuracy-aware knowledge fusion algorithm, fusing the target external knowledge of the target task into the minimum number of initial low-rank adapters; Integrating the corresponding visual task head into the initial low-rank adapter to obtain the low-rank adapter; wherein, the visual task head is used to replace the language modeling head of the basic multi-modal large model and output the response result corresponding to the visual task request.

[0008] According to a method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention, the specific domain data set includes: a plurality of training subsets and corresponding plurality of validation subsets; The step of fusing the target external knowledge of the target task into the minimum number of initial low-rank adapters based on the pre-designed accuracy-aware knowledge fusion algorithm includes: Determining an initial training subset from the plurality of training subsets, and inputting the initial training subset into the current initial low-rank adapter for training; After each training iteration, evaluating the prediction accuracy of the current initial low-rank adapter for the target task through the corresponding validation subset; When the prediction accuracy is lower than the preset threshold, roll back the parameters of the current initial low-rank adapter to the state where the prediction accuracy was greater than or equal to the preset threshold for the last time; Repeat the above steps until all the multiple training subsets are assigned to the corresponding initial low-rank adapters, and the prediction accuracy of each initial low-rank adapter is greater than or equal to the preset threshold.

[0009] According to a method for processing visual tasks based on a multi-modal large model with low-rank adaptation enhancement provided by the present invention, the method for dynamically scheduling the multiple visual task requests into multiple batches of processing sequences by an orchestrator and determining the inference mode of the low-rank adapter used for the visual input data of each batch includes: Dynamically schedule the multiple visual task requests into multiple batches of processing sequences by an orchestrator, and dynamically determine the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch; wherein, the inference mode includes: a merging mode, a bypass mode, and a hybrid mode; The method of inputting the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request includes: Input the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM based on the determined inference mode to obtain the response result corresponding to each visual task request.

[0010] According to a method for processing visual tasks based on a multi-modal large model with low-rank adaptation enhancement provided by the present invention, the dynamic switching of the inference mode is implemented by the orchestrator; The method of dynamically determining the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch includes: When the load of the visual task request meets the preset merging mode condition, determine the inference mode as the merging mode; wherein, the preset merging mode condition is: the call frequency of the low-rank adapter with the highest frequency request exceeds the preset skew threshold; When the number of unresponded requests is greater than the preset tolerance threshold, switch the inference mode to the hybrid mode through the orchestrator; wherein, the number of unresponded requests is: the number of requests in the current batch that have not been processed; When the number of unresponsive requests is greater than half of the maximum batch capacity, the scheduler switches the inference mode to the bypass mode.

[0011] A method for processing visual tasks based on a multimodal large model with low-rank adaptive enhancement according to the present invention. Through the adaptive sharding matrix multiplication operator ATMM, dynamic sharding calculations are performed on the visual input data and the decomposition matrices of the corresponding low-rank adapters based on the determined inference mode, including: When the determined inference mode is the merging mode, the low-rank adapter with the highest frequency request is superimposed on the weight matrix of the base multimodal large model, and inference is performed through a single matrix calculation; When the determined inference mode is the bypass mode, multiple heterogeneous low-rank adapters are calculated in parallel through the adaptive sharding matrix multiplication operator ATMM to support high-concurrency requests; When the determined inference mode is the hybrid mode, both the merging mode and the bypass mode are executed simultaneously.

[0012] The present invention also provides a device for processing visual tasks based on a multimodal large model with low-rank adaptive enhancement, including the following modules: A receiving module for receiving multiple visual task requests sent by a visual application and the visual input data corresponding to each visual task request in the multiple visual task requests; A determining module for dynamically scheduling the multiple visual task requests into multiple batches of processing sequences through a scheduler and determining the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; A calculation module for inputting the visual input data of each batch and the corresponding low-rank adapter into a target multimodal large model, and performing dynamic sharding calculations on the visual input data and the decomposition matrices of the corresponding low-rank adapters through the adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request; wherein, the target multimodal large model is obtained by fine-tuning the pre-trained multimodal large model through a low-rank adaptive method for domain knowledge fusion.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the method for processing visual tasks based on a multimodal large model with low-rank adaptive enhancement as described in any one of the above.

[0014] The present invention also provides a non - transitory computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for processing visual tasks based on the multi - modal large model with low - rank adaptive enhancement as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for processing visual tasks based on the multi - modal large model with low - rank adaptive enhancement as described in any one of the above.

[0016] A method and apparatus for processing visual tasks based on a multi - modal large model with low - rank adaptive enhancement provided by the present invention, by receiving a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences by an orchestrator, and determining an inference mode of a low - rank adapter used for the visual input data of each batch; wherein, the low - rank adapter used for the visual input data of each batch is pre - determined by the visual application; inputting the visual input data of each batch and the corresponding low - rank adapter into a target multi - modal large model, and performing dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low - rank adapter through an adaptive sharding matrix multiplication operator (ATMM) to obtain a response result corresponding to each visual task request; wherein, the target multi - modal large model is obtained by performing domain - knowledge fusion fine - tuning on a pre - trained multi - modal large model through a low - rank adaptive method. It can be seen that the present invention obtains a target multi - modal large model by performing domain - knowledge fusion fine - tuning on a pre - trained basic multi - modal large model through a low - rank adaptive method. Based on the target multi - modal large model, batch requests from a visual application are received. By introducing the adaptive sharding matrix multiplication operator (ATMM), the parallel computing efficiency of heterogeneous low - rank adapters is significantly improved. According to the performance requirements of the visual application, application requests and low - rank adapters are dynamically managed, and the optimal inference mode is selected, so as to meet the diverse visual task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following - described drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a flowchart of the method for processing visual tasks based on the multi - modal large model with low - rank adaptive enhancement provided by the present invention.

[0019] Figure 2It is a schematic diagram of the data processing flow of the method for processing visual tasks using a multi-modal large model based on low-rank adaptive enhancement provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the accuracy-aware knowledge fusion algorithm provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the process of replacing the language modeling head of the base multi-modal large model with a visual task head provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the hybrid mode provided by the present invention.

[0023] Figure 6 It is one of the schematic diagrams of the algorithm provided by the present invention.

[0024] Figure 7 It is the second schematic diagram of the algorithm provided by the present invention.

[0025] Figure 8 It is a schematic diagram of the structure of the device for processing visual tasks using a multi-modal large model based on low-rank adaptive enhancement provided by the present invention.

[0026] Figure 9 It is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0028] The following will be combined with Figures 1-9 Describe a method and device for processing visual tasks using a multi-modal large model based on low-rank adaptive enhancement of the present invention.

[0029] It should be noted that with the rapid development of artificial intelligence technology, multi-modal large models (LMMs) are increasingly widely used in visual tasks. By combining visual and language information, LMMs can handle complex multi-modal tasks such as visual question answering and image captioning. However, existing LMMs often perform poorly when dealing with visual tasks in specific domains, especially when domain-specific knowledge is required. The Low-Rank Adaptation (LoRA) technology provides an effective method. By introducing low-rank adapters into pre-trained multi-modal large models, it can improve the performance of the model on specific tasks without significantly increasing the computational overhead. Although the LoRA technology performs well in improving model accuracy, it still faces many challenges in practical applications. First, the existing LoRA model service systems have low computational efficiency when dealing with high-concurrency requests, resulting in high latency. Second, different visual tasks have different performance requirements. How to flexibly schedule and manage low-rank adapters to meet the needs of different tasks is still an urgent problem to be solved.

[0030] Based on this, the present invention provides a method for processing visual tasks based on a multi-modal large model enhanced by low-rank adaptation to solve at least one of the above problems.

[0031] Figure 1 It is a schematic flowchart of the method for processing visual tasks based on a multi-modal large model enhanced by low-rank adaptation provided by the present invention, as Figure 1 shown, the method includes the following: Step 100: Receive multiple visual task requests sent by a visual application and the visual input data corresponding to each visual task request in the multiple visual task requests.

[0032] It should be noted that each visual task request includes a task type identifier and visual input data. The task type identifier is used to indicate the type of the target task (such as "detect vehicles in an image" or "answer questions about an image"). The visual input data can be a single image, a sequence of video frames, or other visual modality data. Before inputting into the model, the visual input data needs to be preprocessed (such as normalization, scaling, etc.) to adapt to the input requirements of the model.

[0033] Step 200: Dynamically schedule the multiple visual task requests into multiple batches of processing sequences through an orchestrator, and determine the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application.

[0034] It should be noted that the orchestrator provided in this embodiment will schedule the inputs of the vision application to be executed in batches one after another, and orchestrate the low-rank adapters used by these inputs to execute in what mode (such as merge mode, bypass mode, or hybrid mode).

[0035] Step 300: Input each batch of visual input data and the corresponding low-rank adapter into the target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on the pre-trained multi-modal large model through the low-rank adaptation method.

[0036] It should be noted that after obtaining the visual input data, it is input into the target multi-modal large model. The target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on the pre-trained basic multi-modal large model through the low-rank adaptation (LoRA) method. The LoRA method can improve the performance of the model on specific tasks without significantly increasing the computational overhead by introducing low-rank adapters in the basic model. This process is efficiently completed through the adaptive sharding matrix multiplication operator ATMM for the calculation of the vision application request and LoRA (the vision application request and LoRA each provide a matrix, and ATMM calculates these two matrices).

[0037] Step 300 specifically includes: Step 310: Input each batch of visual input data, the corresponding task type identifier, and the corresponding low-rank adapter into the target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the multi-modal intermediate features corresponding to each visual task request.

[0038] Step 320: According to the multi-modal intermediate features corresponding to each visual task request, call the corresponding visual task head through the task type identifier to output the response result corresponding to each visual task request; wherein, the visual task head is a trainable linear layer integrated in the low-rank adapter and is used to map the multi-modal intermediate features to the discrete output space of the visual task request.

[0039] The above is a step description of the method for processing visual tasks using the multi-modal large model based on low-rank adaptive enhancement provided by the present invention. From the description of the above steps, it can be seen that according to the method for processing visual tasks using the multi-modal large model based on low-rank adaptive enhancement provided by the present invention, multiple visual task requests sent by a visual application and the visual input data corresponding to each visual task request in the multiple visual task requests are received; the multiple visual task requests are dynamically scheduled by an orchestrator into multiple batches of processing sequences, and the inference mode of the low-rank adapter used for the visual input data of each batch is determined; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; the visual input data of each batch and the corresponding low-rank adapter are input into a target multi-modal large model, and dynamic sharding calculation is performed on the decomposed matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method. It can be seen that the present invention obtains a target multi-modal large model by performing domain knowledge fusion fine-tuning on a pre-trained basic multi-modal large model through a low-rank adaptive method, receives batch requests from a visual application based on the target multi-modal large model, significantly improves the parallel computing efficiency of heterogeneous low-rank adapters by introducing an adaptive sharding matrix multiplication operator ATMM, dynamically manages application requests and low-rank adapters according to the performance requirements of the visual application, and selects the optimal inference mode, so as to meet diverse visual task requirements.

[0040] Based on the above embodiments, in this embodiment, the generation process of the low-rank adapter includes: Step 410, obtaining target external knowledge of a target task; wherein, the target external knowledge includes: a specific domain dataset and model parameters of a specific domain small model.

[0041] It should be noted that the specific domain dataset includes: multiple training subsets and corresponding multiple validation subsets.

[0042] Step 420, based on a pre-designed accuracy-aware knowledge fusion algorithm, fusing the target external knowledge into the minimum number of initial low-rank adapters.

[0043] Step 420 specifically includes: Step 421, determining an initial training subset in the multiple training subsets, and inputting the initial training subset into the current initial low-rank adapter for training.

[0044] Step 422, after each training iteration, evaluating the prediction accuracy of the current initial low-rank adapter for the target task through the corresponding validation subset.

[0045] Step 423: When the prediction accuracy is lower than the preset threshold, roll back the parameters of the current initial low-rank adapter to the state where the prediction accuracy was greater than or equal to the preset threshold for the last time.

[0046] Step 424: Repeat the above steps until all the multiple training subsets are assigned to the corresponding initial low-rank adapters and the prediction accuracy of each initial low-rank adapter is greater than or equal to the preset threshold.

[0047] Step 430: Integrate the corresponding vision task head into the initial low-rank adapter to obtain the low-rank adapter; wherein, the vision task head is used to replace the language modeling head of the base multi-modal large model and output the response result corresponding to the vision task request.

[0048] Figure 2 It is a schematic diagram of the data processing flow of the method for processing vision tasks based on a multi-modal large model enhanced by low-rank adaptation provided by the present invention. The following combines Figure 2 , and details the method for processing vision tasks based on a multi-modal large model enhanced by low-rank adaptation provided by the present invention.

[0049] In one embodiment, referring to Figure 2 , in the offline stage, the accuracy-aware LoRA adapter generation method receives external knowledge (from existing domain-specific small models or datasets) and the accuracy requirements specified by the vision application (as shown by the dashed arrow in Figure 2 ) and generates the minimum number of LoRA adapters. The generated LoRA adapters are rich in domain-specific knowledge and can output accurate responses for tasks in the vision application. In the online stage, the flexible LoRA adapter orchestration mechanism receives requests from the vision application (as shown by the solid arrow in Figure 2 ), organizes them into batches, selects the inference mode, and orchestrates the corresponding LoRA adapters to minimize the average response latency while ensuring the latency constraints for each vision application. Each request batch is passed to the corresponding adapter and the base multi-modal large model and is inferred in the selected inference mode. The batch processing and inference mode switcher of the LoRA adapter is implemented through the adaptive tiled matrix multiplication operator (ATMM) to achieve high efficiency.

[0050] Specifically, the accuracy-aware LoRA adapter generation method includes an accuracy-aware knowledge fusion algorithm and a vision task head.

[0051] 1. Precision-Aware Knowledge Fusion Algorithm: For easy management at runtime, the goal is to fuse external knowledge into the minimum number of LoRA adapters without violating the precision requirements of any visual task. To this end, the training method must consider the limited capacity of LoRA adapters and the complex precision changes brought about by knowledge fusion.

[0052] Suppose there is a prophet who can predict in advance the precision of LoRA adapters that fuse any combination of knowledge. Then, this problem can be formulated as a constrained bin-packing problem, with the goal of packing knowledge into the minimum number of LoRA adapter "bins" to ensure that the precision of each adapter meets the requirements for each visual task. However, such a prophet does not exist.

[0053] To solve the original problem, this embodiment proposes a simple and easy-to-implement heuristic algorithm, namely the precision-aware knowledge fusion algorithm, to determine which knowledge can be fused into a LoRA adapter. The algorithm first collects the dataset, Figure 3 is a schematic diagram of the precision-aware knowledge fusion algorithm provided by the present invention, as Figure 3 shown, by executing representative data on the existing domain-specific small model; if the application provides a dataset, use them directly. The training process is a standard supervised learning process, calculating the cross-entropy loss to update the parameters, as can be seen from the following formula.

[0054] Knowledge fusion adopts a greedy and precision-aware heuristic method. It starts from a random dataset and trains LoRA adapters sequentially using each dataset until its precision on a specific task is lower than the required threshold (i.e., the above preset threshold). If this happens, the weights of the adapter will be rolled back, and a new adapter will be initialized to learn from the most recent dataset. The worst-case scenario of this method may generate a LoRA adapter for each dataset, but in actual experiments, each LoRA adapter fuses the knowledge (datasets) of 4 domains on average.

[0055] 2. Visual Task Head: To reduce the inference latency, this embodiment designs a visual task head. It is designed as a trainable linear layer and is part of the LoRA adapter, Figure 4 is a schematic diagram of the process of replacing the language modeling head of the base multimodal large model with the visual task head provided by the present invention, as Figure 4 shown, for predicting task-specific results based on the output features of the LMM. The visual task head can be flexibly customized for various visual tasks during the training of the LoRA adapter, provided that the fused knowledge comes from the same task type. Figure 4The effects of using the original language modeling head (LM head) and visual task heads for action recognition were compared. By replacing the LM head with a visual task head, LMM saved 4 rounds of inference and approximately 180 ms of time cost. The reason for this is that the outputs of most visual tasks are a finite discrete candidate set, such as the number of vehicle counts, the categories of action recognition, and binary queries for specific targets on images or videos.

[0056] For visual applications that require a natural language interface, the LM head is retained. For example, when a video query application asks "a boy wearing a red sweater is lost at the corner" and specifies person detection, VaLoRA will call the corresponding LoRA adapter containing the detection head for an efficient response. By using the LoRA adapter batch processing operator ATMM, these different visual task heads can be batch executed with the LM head.

[0057] The method for processing visual tasks by the multi-modal large model based on low-rank adaptation enhancement provided in this embodiment combines external knowledge with the basic multi-modal large model to generate LoRA adapters containing domain-specific knowledge to meet the specific accuracy requirements of visual applications.

[0058] Based on the above embodiment, in this embodiment, step 200 dynamically schedules the multiple visual task requests into multiple batches of processing sequences through an orchestrator, and determines the inference mode of the low-rank adapter used for the visual input data of each batch, including: Step 210, dynamically schedules the multiple visual task requests into multiple batches of processing sequences through an orchestrator, and dynamically determines the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch; wherein, the inference mode includes: merge mode, bypass mode, and hybrid mode.

[0059] Step 210 specifically includes: Step 211, when the request load of the visual task request meets the preset merge mode condition, determines the inference mode as the merge mode; wherein, the preset merge mode condition is: the call frequency of the low-rank adapter with the highest frequency request exceeds the preset skew threshold; Step 212, in the case where the number of unresponded requests is greater than the preset tolerance threshold, switches the inference mode to the hybrid mode through the orchestrator; wherein, the number of unresponded requests is: the number of requests in the current batch that have not been processed; Step 213, in the case where the number of unresponded requests is greater than half of the maximum batch capacity, switches the inference mode to the bypass mode through the orchestrator.

[0060] Step 220: Input the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and perform dynamic tiling calculations on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive tiling matrix multiplication operator (ATMM) based on the determined inference mode, to obtain the response results corresponding to each visual task request.

[0061] Step 220 specifically includes: Step 221: When the determined inference mode is the merging mode, stack the low-rank adapter with the highest frequency request on the weight matrix of the basic multi-modal large model, and perform inference through a single matrix calculation.

[0062] Step 222: When the determined inference mode is the bypass mode, parallelly calculate multiple heterogeneous low-rank adapters through the adaptive tiling matrix multiplication operator (ATMM) to support high-concurrency requests.

[0063] Step 223: When the determined inference mode is the hybrid mode, execute both the merging mode and the bypass mode simultaneously.

[0064] It should be noted that the LMM serving visual applications is very likely to call multiple heterogeneous LoRA adapters simultaneously. To efficiently calculate these adapters, this embodiment proposes an adaptive tiling matrix multiplication operator (ATMM), which supports efficient bypass mode inference, merging mode inference, and hybrid mode inference. For details, see Figure 2 。

[0065] 1. ATMM: Adaptive-tiling matrix multiplication operator (ATMM): It is feasible to directly batch-process the calculations of heterogeneous adapters on the standard kernel (such as batch GEMM in dLoRA), but it will result in excessive latency and insufficient hardware utilization. This is due to the significant padding caused by the heterogeneity of the application request length and the LoRA adapter rank. Therefore, a customized kernel is required.

[0066] This embodiment first analyzes two existing custom kernels of S-LoRA and Punica. The kernel of S-LoRA uses the sharding technique to avoid significant padding. Different from batch GEMM that pads heterogeneous input matrices into a unified shape, it divides them into fine-grained blocks and computes the output of each block in parallel on CUDA cores. The kernel of Punica also adopts the sharding technique and further improves the efficiency by leveraging the mature CUTLASS library and high-performance Tensor cores. However, neither of these two kernels has achieved satisfactory efficiency. The fundamental reason is that their static sharding configurations are insufficient to handle diverse input shapes, resulting in insufficient utilization of computing resources.

[0067] The key observation of this embodiment is that the computational efficiency varies significantly with different sharding configurations. Based on this, ATMM, an adaptive sharding matrix multiplication CUDA kernel, is proposed, which can fully utilize computing resources to achieve efficient heterogeneous LoRA adapter batching. It has two key design choices: (1) Adaptive sharding according to the input shape. Given two input matrices, ATMM retrieves the optimal sharding configuration from the hash table, divides the matrices into thread block shards according to this configuration, and then further divides them into warp shards. (2) Pipeline data loading and computing. After sharding, ATMM transfers the data of each shard to the corresponding memory and performs computations on CUDA / Tensor cores using the corresponding executable kernels. To hide the data loading latency, ATMM allocates double space for each shard in the shared memory and register file: one for the computation of the current shard and the other for prefetching the data of the next shard. Note that this double buffering refers to using double buffers in the shared memory, which is an on-chip caching strategy independent of GPU memory. Essentially, it is a caching usage strategy that does not cause global memory overhead. This double buffering is feasible because the shard shape at each level is unified, enabling efficient positioning of the next shard.

[0068] 2. Profile-based optimal tiling search: To maintain the advantages of adaptive sharding, the goal of this embodiment is to search for the optimal sharding configuration for each different input and switch between them at runtime. This can be done in the offline phase because it is only affected by the input shape, not their values. Although some previous work has studied the optimal tiling of matrix multiplication, it is still challenging due to the complex GPU thread parallel mechanism and memory access patterns, as well as the large search space of input shapes.

[0069] To address this challenge, this embodiment treats the search as a black-box problem and proposes a search algorithm based on performance analysis. With the help of the CUTLASS Profiler, it performs performance analysis on all possible ATMM input matrix shapes, records the optimal sharding configurations into a hash table, and compiles the corresponding executable kernels. This embodiment uses the following knowledge to reduce the search space: (1) From a hardware perspective, architectural features limit the feasible input shapes (e.g., the limited global memory of the hardware architecture) and sharding configurations (e.g., the dimension of each shard is at least 16 and must be a power of 2). (2) From the perspective of input data, the model dimensions of LMM limit the step size of the input matrix shape changes (e.g., 4096 in Qwen-VL). By doing so, the search space can be reduced by up to 20 times. For example, the total search space for Qwen-VL on A100 is 50,000 configurations, which is reduced to 3,000 according to the CUTLASS documentation and the maximum context length of 2048 of the Qwen-VL model. Appendix A shows the detailed algorithm. With the algorithm of this embodiment, it takes less than 30 minutes to search for all optimal configurations for Qwen-VL's vision tasks on an A100 GPU.

[0070] Furthermore, to meet the diverse performance requirements of vision applications, this embodiment proposes an orchestrator for scheduling requests, managing LoRA adapters, and switching inference modes to achieve efficient and flexible LoRA LLM inference operation. This embodiment first implements two tools using ATMM, namely a fast mode switcher and a hybrid inference mode, to support the orchestrator.

[0071] 1. Swift inference mode switch: Existing systems (such as dLoRA) introduce excessive additional latency during mode switching. To reduce these overheads, a common approach is to pre-compute the LoRA matrices of the entire base model (i.e., matrix Δ W = B × A)( ) and store it in the host memory, and then swap it to the GPU when needed. However, such a large amount of data (e.g., about 3GB for each LoRA adapter) will cause very high latency (about 1 second each time) during swapping. Instead, the fast mode switcher of this embodiment calculates the LoRA matrix during operation and only stores the adapters (i.e., A and B, each 43MB) in the GPU. It has two core designs: (1) Eliminate unnecessary memory copying through continuous memory allocation. The switcher of this embodiment pre-allocates continuous memory for the weight matrix, avoiding memory copying in tensor reshaping and achieving efficient in-place LoRA matrix merging / splitting. (2) Calculate the LoRA matrices of all layers at once and merge / split them. Through ATMM, the switcher can efficiently calculate the LoRA matrices of the entire model and add or subtract them all from the base model weights in one operation. By doing so, the mode switch of this embodiment takes less than 10ms, which is more than 5 times faster than dLoRA.

[0072] 2. Mixture inference mode: Compared with the bypass mode, the merged mode inference only supports processing one LoRA adapter at a time, which will cause requests for other vision tasks to starve. To alleviate the starvation, this embodiment proposes a new inference mode, namely deLoRA, to support simultaneous execution of merged and bypass inferences. Figure 5 is the schematic diagram of the hybrid mode provided by the present invention, as Figure 5 shown, LoRA1 processes requests in the merged mode, while other LoRA adapters (LoRA X ) process their requests in the bypass mode. To maintain the consistent results of LoRA X requests, this embodiment introduces a deLoRA branch to prevent contamination of LoRA1. The weights of deLoRA are the same as those of the merged LoRA. Based on the distributive property of matrix multiplication, its correctness can be verified as follows: where W represents the weights of the base model, deLoRA, and LoRA adapters; input x and output x are the input and output of the LoRA X request. The overhead of calculating W deLoRA1 is exactly the same as LoRA 1, and the overhead introduced by subtracting it is very small.

[0073] The hybrid inference mode has two advantages: (1) It does not incur the cost of switching from the merged mode to the bypass mode. (2) When there are more requests for merged LoRA adapters than other requests, its additional computational overhead is less than that of the bypass inference mode.

[0074] 3. Scheduling policy: To minimize the average response latency and meet the latency constraints of each request, the orchestrator in this embodiment must carefully orchestrate requests, adapters, and inference modes. The policy in this embodiment follows a greedy heuristic and includes two principles: (1) Execute in the merge mode as much as possible because it produces the fastest response and has no additional overhead. (2) When starvation occurs, switch to the hybrid mode first and then to the bypass mode in the order of switching cost and additional computation. Figure 6 is one of the algorithm schematic diagrams provided by the present invention, as Figure 6 shown, Algorithm 1 (scheduling algorithm) shows the pseudocode. In Algorithm 1: Input: Request set R, LoRA adapter set L, inference mode M.

[0075] Output: The next request set B to be batched.

[0076] Function body: 1. Starvation state judgment: Identify and determine the subset of requests currently in a starvation state based on a preset threshold or rule .

[0077] 2. Adapter selection: From the LoRA adapter set L, comprehensively evaluate based on dimensions such as adapter compatibility, load status, and request adaptation requirements, and select the adapter l_optimal that can theoretically merge and process the largest number of requests.

[0078] 3. Condition detection: Calculate the proportion of the number of hungry requests to the total number of requests, and the proportion of the number of mergable requests to the total number of requests, and judge whether the number of R_hungry is less than 50% and the number of mergable requests is greater than 50%.

[0079] 4. Mode switching and request selection (when the conditions in step 3 are met): If R_hungry is empty, switch to the merge execution mode. In this mode, according to the First Come First Serve (FCFS) principle, select requests within the upper limit of the number of requests that the adapter l_optimal can merge and process from the remaining requests in the order of request arrival to form the batch request set B.

[0080] If \(R_{hungry}\) is not empty, switch to the hybrid execution mode. On the one hand, include all the hungry requests \(R_{hungry}\) in the batch request set \(B\); on the other hand, select an appropriate number of requests from the remaining requests that can be merged with \(R_{hungry}\) for processing according to the first-come-first-served principle and that the adapter \(l_{optimal}\) can support, and add them to the batch \(B\) together.

[0081] 5. Mode Switching and Request Selection (when the condition in Step 3 is not met): Switch to the bypass execution mode. Incorporate the hungry requests \(R_{hungry}\) and the remaining requests into the batch request set \(B\) strictly according to the first-come-first-served principle, in the original order of request arrival, to ensure that the requests are processed in order.

[0082] 6. Output Batch: Output the constructed batch request set \(B\) for subsequent inference or processing flow calls.

[0083] To alleviate starvation, it assigns a token to each request, representing its waiting time plus the execution time in the current mode and the mode switching latency (line 2), and sets a tolerance threshold \(\theta\) as the condition for determining whether the current request is starving, and uses the number of starving requests as the condition for switching to the hybrid mode. When the request load meets the condition for switching to the merging mode, the algorithm switches the mode to the merging mode (see Figure 6 lines 5 - 8). When starving requests occur, the algorithm immediately processes them using the hybrid mode (see Figure 6 lines 9 - 12). When it further exceeds half of the maximum batch capacity, it switches to the bypass mode (see Figure 6 lines 13 - 15).

[0084] The prior art may think that the static inference mode determined by a one-time search can work well without switching. However, in practice, this is almost impossible. Most applications will experience dynamic workload changes, making it difficult to define an optimal execution mode or order. For example, video analysis applications serve multiple users. When new registration tasks (specified streams, tasks / adapters, precisions) arrive, the workload of the video analysis application changes, and the workload of the LoRA adapter changes with multiple rounds of VQA in visual retrieval applications.

[0085] Figure 7 is the second schematic diagram of the algorithm provided by the present invention, as Figure 7 shown, Algorithm 2 (tiling search algorithm): Input: Model, hardware environment.

[0086] Output: The optimal configuration for each input shape.

[0087] Function Body: 1. Determine the input size range: Based on the memory limitations of the model and the hardware, obtain the minimum and maximum values of the input size to limit the input size range and avoid problems caused by insufficient memory.

[0088] 2. Obtain the configuration list: According to the hardware characteristics, obtain the possible configuration list, which contains configuration parameters at different levels, such as the tile size and the warp size of the threads.

[0089] 3. Traverse the input shape, the rank of the adapter, the tile, and the warp configuration, and call the measurement program to perform performance analysis on the current input combination; according to the analysis results, update the optimal configuration for the corresponding input.

[0090] 4. Output the optimal configuration: After completing all traversals, output the optimal configuration for each input shape, which contains the optimal configuration for each input shape. These configurations can provide the best performance under the given model and hardware environment and can be directly used for the deployment and inference process of the model to improve the computing efficiency and resource utilization rate.

[0091] The method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided in this embodiment realizes efficient heterogeneous LoRA adapter batch processing and inference mode switching through the adaptive tiled matrix multiplication operator (ATMM) and a flexible LoRA adapter orchestration mechanism. These designs enable the system to efficiently process concurrent requests for multiple visual tasks while ensuring accuracy, significantly improving the throughput and response speed of the system.

[0092] The method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention combines external knowledge (such as domain-specific small models or datasets) with the multi-modal large model (LMM) to generate LoRA adapters containing domain-specific knowledge to meet the specific accuracy requirements of visual applications; through the adaptive tiled matrix multiplication operator (ATMM), it efficiently calculates concurrent heterogeneous LoRA adapters to reduce the computational latency; according to the performance requirements of visual applications, it dynamically manages application requests and LoRA adapters and selects the optimal inference mode (merge mode, bypass mode, or hybrid mode) to minimize the average response latency.

[0093] The device for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention will be described below. The device for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement described below can be mutually referred to the method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement described above.

[0094] Figure 8It is a schematic structural diagram of an apparatus for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention. As Figure 8 shown, the apparatus for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention includes: A receiving module 801, configured to receive a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; A determining module 802, configured to dynamically schedule the plurality of visual task requests into a plurality of batches of processing sequences through an orchestrator, and determine an inference mode of a low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; A computing module 803, configured to input the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and perform dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method.

[0095] The apparatus for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement provided by the present invention receives a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; dynamically schedules the plurality of visual task requests into a plurality of batches of processing sequences through an orchestrator, and determines an inference mode of a low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; inputs the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performs dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method. It can be seen from this that the present invention obtains a target multi-modal large model by performing domain knowledge fusion fine-tuning on a pre-trained basic multi-modal large model through a low-rank adaptive method, receives batch requests from a visual application based on the target multi-modal large model, significantly improves the parallel computing efficiency of heterogeneous low-rank adapters by introducing an adaptive sharding matrix multiplication operator ATMM, dynamically manages application requests and low-rank adapters according to the performance requirements of the visual application, and selects an optimal inference mode, so as to meet diverse visual task requirements.

[0096] Based on the above embodiments, in this embodiment, each visual task request further includes a corresponding task type identifier; The computing module 803 is specifically configured to: Input each batch of visual input data, the corresponding task type identifier, and the corresponding low-rank adapter into the target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the multi-modal intermediate features corresponding to each visual task request; According to the multi-modal intermediate features corresponding to each visual task request, call the corresponding visual task head through the task type identifier, and output the response result corresponding to each visual task request; wherein, the visual task head is: a trainable linear layer integrated in the low-rank adapter for mapping the multi-modal intermediate features to the discrete output space of the visual task request.

[0097] Based on the above embodiments, in this embodiment, the device further includes a generation module, specifically configured to: Obtain the target external knowledge of the target task; wherein, the target external knowledge includes: a specific domain dataset and the model parameters of a specific domain small model; Based on a pre-designed accuracy-aware knowledge fusion algorithm, fuse the target external knowledge of the target task into the minimum number of initial low-rank adapters; Integrate the corresponding visual task head into the initial low-rank adapter to obtain the low-rank adapter; wherein, the visual task head is used to replace the language modeling head of the basic multi-modal large model and output the response result corresponding to the visual task request.

[0098] Based on the above embodiments, in this embodiment, the specific domain dataset includes: a plurality of training subsets and corresponding plurality of validation subsets; The device further includes a fusion module, specifically configured to: Determine an initial training subset from the plurality of training subsets, and input the initial training subset into the current initial low-rank adapter for training; After each training iteration, evaluate the prediction accuracy of the current initial low-rank adapter for the target task through the corresponding validation subset; In the case where the prediction accuracy is lower than a preset threshold, roll back the parameters of the current initial low-rank adapter to the state where the last prediction accuracy is greater than or equal to the preset threshold; Repeat the above steps until all the plurality of training subsets are assigned to the corresponding initial low-rank adapters, and the prediction accuracy of each initial low-rank adapter is greater than or equal to the preset threshold.

[0099] Based on the above embodiments, in this embodiment, the determining module 802 is specifically configured to: Dynamically schedule the multiple visual task requests into multiple batches of processing sequences through an orchestrator, and dynamically determine the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch; wherein, the inference modes include: merge mode, bypass mode, and hybrid mode; The computing module 803 is specifically configured to: Input the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharded matrix multiplication operator (ATMM) based on the determined inference mode to obtain the response result corresponding to each visual task request.

[0100] Based on the above embodiments, in this embodiment, the dynamic switching of the inference mode is implemented through the orchestrator; The determining module 802 is specifically configured to: When the load of the visual task request meets the preset merge mode condition, determine the inference mode as the merge mode; wherein, the preset merge mode condition is: the call frequency of the low-rank adapter with the highest frequency request exceeds a preset skew threshold; In the case where the number of unresponded requests is greater than a preset tolerance threshold, switch the inference mode to the hybrid mode through the orchestrator; wherein, the number of unresponded requests is: the number of requests in the current batch that have not been processed; In the case where the number of unresponded requests is greater than half of the maximum batch capacity, switch the inference mode to the bypass mode through the orchestrator.

[0101] Based on the above embodiments, in this embodiment, the computing module 803 is specifically configured to: In the case where the determined inference mode is the merge mode, stack the low-rank adapter with the highest frequency request and the weight matrix of the basic multi-modal large model, and perform inference through a single matrix calculation; In the case where the determined inference mode is the bypass mode, parallelly calculate multiple heterogeneous low-rank adapters through the adaptive sharded matrix multiplication operator (ATMM) to support high-concurrency requests; In the case where the determined inference mode is the hybrid mode, simultaneously execute the merge mode and the bypass mode.

[0102] Figure 9 Illustrates a schematic diagram of the physical structure of an electronic device, as Figure 9As shown, the electronic device can be a robot or other electronic device, and the electronic device can include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communications interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 can call the logical instructions in the memory 930 to execute a method for processing visual tasks based on a multi-modal large model with low-rank adaptation enhancement, including: Receiving a plurality of visual task requests sent by a visual application and visual input data corresponding to each visual task request in the plurality of visual task requests; Dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences through an orchestrator, and determining the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; Inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptation method.

[0103] In addition, when the logical instructions in the above-mentioned memory 930 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for visual task processing based on the low-rank adaptive enhanced multi-modal large model provided by the above-mentioned various methods, including: Receiving a plurality of visual task requests sent by a visual application and the visual input data corresponding to each visual task request in the plurality of visual task requests; Dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences by an orchestrator, and determining the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; Inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method.

[0105] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for visual task processing based on the low-rank adaptive enhanced multi-modal large model provided by the above-mentioned various methods, including: Receiving a plurality of visual task requests sent by a visual application and the visual input data corresponding to each visual task request in the plurality of visual task requests; Dynamically scheduling the plurality of visual task requests into a plurality of batches of processing sequences by an orchestrator, and determining the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; Inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposition matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptive method.

[0106] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0107] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing visual tasks using a multimodal large model based on low-rank adaptive enhancement, characterized in that Including: Receiving multiple visual task requests sent by a visual application and visual input data corresponding to each visual task request in the multiple visual task requests; Dynamically scheduling the multiple visual task requests into multiple batches of processing sequences by an orchestrator, and determining an inference mode of a low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; Inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by performing domain knowledge fusion fine-tuning on a pre-trained multi-modal large model through a low-rank adaptation method.

2. The method for visual task processing based on a multi-modal large model with low-rank adaptive enhancement according to claim 1, wherein, Each visual task request further includes a corresponding task type identifier; The step of inputting the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through an adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request, includes: Inputting the visual input data of each batch, the corresponding task type identifier, and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain multi-modal intermediate features corresponding to each visual task request; According to the multi-modal intermediate features corresponding to each visual task request, calling a corresponding visual task head through the task type identifier to output a response result corresponding to each visual task request; wherein, the visual task head is a trainable linear layer integrated in the low-rank adapter and is used to map the multi-modal intermediate features to the discrete output space of the visual task request.

3. The method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement according to claim 2, wherein The generation process of the low-rank adapter includes: Obtaining target external knowledge of a target task; wherein, the target external knowledge includes: a specific domain dataset and model parameters of a specific domain small model; Based on a pre-designed precision-aware knowledge fusion algorithm, fusing the target external knowledge of the target task into a minimum number of initial low-rank adapters; Integrating a corresponding visual task head into the initial low-rank adapter to obtain the low-rank adapter; wherein, the visual task head is used to replace the language modeling head of the base multi-modal large model and output a response result corresponding to the visual task request.

4. The method for processing visual tasks based on the multi-modal large model with low-rank adaptive enhancement according to claim 3, wherein, The specific domain dataset includes: multiple training subsets and corresponding multiple validation subsets; The step of fusing the target external knowledge of the target task into a minimum number of initial low-rank adapters based on a pre-designed precision-aware knowledge fusion algorithm includes: Determine an initial training subset from the multiple training subsets and input the initial training subset into the current initial low-rank adapter for training; After each training iteration, evaluate the prediction accuracy of the current initial low-rank adapter for the target task through the corresponding validation subset; If the prediction accuracy is lower than the preset threshold, roll back the parameters of the current initial low-rank adapter to the state where the prediction accuracy was greater than or equal to the preset threshold for the last time; Repeat the above steps until all the multiple training subsets are assigned to the corresponding initial low-rank adapters and the prediction accuracy of each initial low-rank adapter is greater than or equal to the preset threshold.

5. The method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement according to claim 2, wherein The dynamic scheduling of the multiple visual task requests into multiple batches of processing sequences by the orchestrator and determining the inference mode of the low-rank adapter used for the visual input data of each batch includes: The dynamic scheduling of the multiple visual task requests into multiple batches of processing sequences by the orchestrator and dynamically determining the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch; wherein, the inference modes include: merge mode, bypass mode, and hybrid mode; Inputting the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain the response result corresponding to each visual task request, includes: Inputting the visual input data of each batch and the corresponding low-rank adapter into the target multi-modal large model, and performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM based on the determined inference mode to obtain the response result corresponding to each visual task request.

6. The method for processing visual tasks based on a multi-modal large model with low-rank adaptive enhancement according to claim 5, wherein, The dynamic switching of the inference mode is achieved by the orchestrator; The dynamically determining the inference mode of the low-rank adapter used for the visual input data of each batch based on the load characteristics of the visual task requests of each batch includes: When the load of the visual task request meets the preset merge mode condition, determine the inference mode as the merge mode; wherein, the preset merge mode condition is: the call frequency of the low-rank adapter with the highest frequency request exceeds the preset skew threshold; When the number of unresponded requests is greater than the preset tolerance threshold, switch the inference mode to the hybrid mode through the orchestrator; wherein, the number of unresponded requests is: the number of requests in the current batch that have not been processed; When the number of unresponded requests is greater than half of the maximum batch capacity, switch the inference mode to the bypass mode through the orchestrator.

7. The method for processing visual tasks based on a multimodal large model with low-rank adaptive enhancement according to claim 6, wherein The performing dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM based on the determined inference mode includes: When the determined inference mode is the merge mode, superimpose the low-rank adapter with the highest frequency request on the weight matrix of the basic multi-modal large model and perform inference through a single matrix calculation; When the determined inference mode is the bypass mode, multiple heterogeneous low-rank adapters are calculated in parallel through the adaptive sharding matrix multiplication operator ATMM to support high-concurrency requests; When the determined inference mode is the hybrid mode, the merge mode and the bypass mode are executed simultaneously.

8. An apparatus for processing visual tasks using a multi-modal large model based on low-rank adaptive enhancement, characterized in that, It includes: A receiving module, configured to receive multiple visual task requests sent by a visual application and visual input data corresponding to each visual task request in the multiple visual task requests; A determining module, configured to dynamically schedule the multiple visual task requests into multiple batches of processing sequences through an orchestrator, and determine the inference mode of the low-rank adapter used for the visual input data of each batch; wherein, the low-rank adapter used for the visual input data of each batch is pre-determined by the visual application; A calculation module, configured to input the visual input data of each batch and the corresponding low-rank adapter into a target multi-modal large model, and perform dynamic sharding calculation on the decomposed matrices of the visual input data and the corresponding low-rank adapter through the adaptive sharding matrix multiplication operator ATMM to obtain a response result corresponding to each visual task request; wherein, the target multi-modal large model is obtained by fine-tuning the pre-trained multi-modal large model through a low-rank adaptation method for domain knowledge fusion.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for processing visual tasks based on the multi-modal large model enhanced by low-rank adaptation according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for processing visual tasks based on the multi-modal large model enhanced by low-rank adaptation according to any one of claims 1 to 7.

Images