Task reasoning method and device and electronic equipment
By receiving task requests and replacing the linear layer with a customized segmented aggregation matrix multiplication module, a multitasking inference model is formed, which solves the problem of high equipment configuration requirements in the prior art, and realizes efficient utilization of equipment resources and multitasking processing capabilities.
Patent Information
- Application Number
- CN202510317647.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2025-08-08
AI Technical Summary
In the prior art, when multiple large models are deployed on local devices to complete multiple tasks, the device configuration requirements are high, resulting in waste of resources and inefficiency.
By receiving task requests, the fine-tuning weight matrix of the task target is determined, and the preset basic inference model and linear layer are used, combined with the customized segmented aggregation matrix multiplication module, the linear layer of the basic model is replaced, and a multi-task inference model is formed, which reduces the equipment configuration requirements.
While completing multi-task inference, the configuration requirements for local devices are reduced and the processing capabilities and efficiency of devices are improved.
Smart Images
Figure CN120450028A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a task reasoning method, device, electronic device, and readable storage medium. Background Art
[0002] With the rapid development of science and technology, various large models are constantly emerging, providing greater convenience for people. In existing technologies, deploying a large model on a local device can complete a task. To complete another task, another large model must be deployed. In other words, multiple large models are required to complete multiple tasks. However, the deployment and operation of multiple large models places high demands on the local device configuration. Summary of the Invention
[0003] In view of this, embodiments of the present application provide a task reasoning method, apparatus, electronic device, and readable storage medium, which can reduce the requirements for local device configuration while completing multi-task reasoning.
[0004] In a first aspect, an embodiment of the present application provides a task reasoning method, comprising: receiving a task request; wherein the task request includes a task objective and a task content corresponding to each task objective; determining the fine-tuning weight matrix of each task objective based on each task objective and the correspondence between the task objective and the fine-tuning weight matrix; determining the reasoning result corresponding to the task request based on each task content, the fine-tuning weight matrix of each task objective and a preset basic reasoning model; wherein the preset basic reasoning model includes at least one linear layer, and the target linear layer in the at least one linear layer is used to obtain an output feature matrix based on the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic reasoning model that is adjacent to the target linear layer and is located before the target linear layer.
[0005] According to a specific implementation method of an embodiment of the present application, the preset basic reasoning model is determined according to the following steps: determining a new linear layer based on the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model; using the new linear layer to replace the target linear layer in the initial basic model to obtain the final basic reasoning model; wherein, the new linear layer is the target linear layer in the final basic reasoning model.
[0006] According to a specific implementation method of an embodiment of the present application, the preset segmented aggregation matrix multiplication module is a customized segmented aggregation matrix multiplication module; wherein, determining a new linear layer based on the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the basic model includes: registering the customized segmented aggregation matrix multiplication module to the model inference framework; the model inference framework determines a new linear layer based on the customized segmented aggregation matrix multiplication module and the linear layer corresponding to the basic model.
[0007] According to a specific implementation of an embodiment of the present application, the target linear layer in the preset basic inference model includes a first target linear layer; wherein, determining the inference result corresponding to the task request based on the fine-tuning weight matrix of each task content and each task target and the preset basic inference model includes: the first target linear layer determines a first output feature matrix of the first target linear layer based on the fine-tuning weight matrix of each task target and the output feature matrix of the previous network layer of the first target linear layer; the first target linear layer determines a second output feature matrix of the first target linear layer based on the linear layer corresponding to the basic model and the output feature matrix of the previous network layer of the first target line; and the first output feature matrix and the second output feature matrix are added to obtain the output feature matrix of the first target linear layer.
[0008] According to a specific implementation of an embodiment of the present application, determining the first output feature matrix of the first target linear layer based on the fine-tuning weight matrix of each task objective and the output feature matrix of the previous network layer of the first target linear layer includes: segmenting the output feature matrix of the previous network layer of the first target linear layer according to the task objective to obtain multiple feature matrix segments; determining the task objective index of each feature matrix segment according to the correspondence between each feature matrix segment and a preset feature matrix segment and the task objective index; determining the fine-tuning weight matrix corresponding to each feature matrix segment in the fine-tuning weight matrix of each task objective according to the task objective index of each feature matrix segment; and determining the first output feature matrix of the first target linear layer according to each feature matrix segment and the fine-tuning weight matrix corresponding to each feature matrix segment.
[0009] According to a specific implementation manner of an embodiment of the present application, after determining the fine-tuning weight matrix of each task target based on each task target and the correspondence between the task target and the fine-tuning weight matrix, and before determining the inference result corresponding to the task request, the method further includes: obtaining a list corresponding to the target linear layer; the list includes the fine-tuning weight matrix of each task target; distributing the fine-tuning weight matrix of each task target in the list to the target linear layer in the preset basic inference model through the forward propagation interface; the target linear layer in the preset basic inference model receives the fine-tuning weight matrix distributed by the forward propagation interface.
[0010] According to a specific implementation method of an embodiment of the present application, after receiving the task request, the method further includes: splicing the task contents in the task request; wherein, the task request includes a first task request and a second task request; the splicing of the task contents in the task request includes: splicing the task content in the first task request and the task content in the second task request; wherein, the splicing of the task content in the first task request and the task content in the second task request includes: when the task target of the first task request and the task target of the second task request are the same, splicing the task content in the first task request and the task content in the second task request into the same splicing segment; when the task target of the first task request and the task target of the second task request are different, splicing the task content in the first task request and the task content in the second task request into different splicing segments.
[0011] According to a specific implementation method of an embodiment of the present application, the use of a new linear layer to replace the target linear layer in the initial basic model to obtain a final basic reasoning model includes: using a new linear layer to replace the target linear layer in the initial basic model to obtain an intermediate basic model; quantizing the intermediate basic model; and compiling the quantized intermediate basic model to obtain a final basic reasoning model.
[0012] In a second aspect, an embodiment of the present application provides a task reasoning device, comprising: a first receiving module for receiving a task request; wherein the task request includes a task objective and a task content corresponding to each task objective; a first determination module for determining the fine-tuning weight matrix of each task objective based on each task objective and the correspondence between the task objective and the fine-tuning weight matrix; a second determination module for determining the reasoning result corresponding to the task request based on each task content, the fine-tuning weight matrix of each task objective and a preset basic reasoning model; wherein the preset basic reasoning model includes at least one linear layer, and the target linear layer in the at least one linear layer is used to obtain an output feature matrix based on the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic reasoning model that is adjacent to the target linear layer and is located before the target linear layer.
[0013] According to a specific implementation method of an embodiment of the present application, the device also includes: a third determination module, used to determine a new linear layer based on a preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model; a replacement module, used to replace the target linear layer in the initial basic model with the new linear layer to obtain a final basic reasoning model; wherein, the new linear layer is the target linear layer in the final basic reasoning model.
[0014] According to a specific implementation method of an embodiment of the present application, the preset segmented aggregation matrix multiplication module is a customized segmented aggregation matrix multiplication module; wherein the third determination module is specifically used to: register the customized segmented aggregation matrix multiplication module into the model inference framework; the model inference framework determines a new linear layer based on the customized segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model.
[0015] According to a specific implementation of an embodiment of the present application, the target linear layer in the preset basic inference model includes a first target linear layer; wherein the second determination module includes: a first determination submodule, configured to determine, through the first target linear layer, a first output feature matrix of the first target linear layer based on a fine-tuning weight matrix of each task target and an output feature matrix of a previous network layer of the first target linear layer; a second determination submodule, configured to determine, through the first target linear layer, a second output feature matrix of the first target linear layer based on an output feature matrix of a linear layer corresponding to the basic model and a previous network layer of the first target line; and an addition submodule, configured to add the first output feature matrix and the second output feature matrix to obtain an output feature matrix of the first target linear layer.
[0016] According to a specific implementation of an embodiment of the present application, the first determination submodule is specifically configured to: segment the output feature matrix of the previous network layer of the first target linear layer according to the task objective to obtain multiple feature matrix segments; determine the task objective index of each feature matrix segment according to the correspondence between each feature matrix segment and a preset feature matrix segment and the task objective index; determine, according to the task objective index of each feature matrix segment, a fine-tuning weight matrix corresponding to each feature matrix segment in the fine-tuning weight matrix of each task objective; and determine the first output feature matrix of the first target linear layer according to each feature matrix segment and the fine-tuning weight matrix corresponding to each feature matrix segment.
[0017] According to a specific implementation of an embodiment of the present application, the device further includes: an acquisition module, configured to acquire a list corresponding to the target linear layer; the list includes a fine-tuning weight matrix for each task target; a distribution module, configured to distribute the fine-tuning weight matrix for each task target in the list to the target linear layer in the preset basic inference model through the forward propagation interface; and a second receiving module, configured to receive the fine-tuning weight matrix distributed by the forward propagation interface to the target linear layer in the preset basic inference model.
[0018] According to a specific implementation method of an embodiment of the present application, the device also includes: a splicing module, which is used to splice the task contents in the task request after the first receiving module receives the task request; wherein, the task request includes a first task request and a second task request; the splicing module includes: a splicing sub-module, which is used to splice the task content in the first task request and the task content in the second task request; the splicing sub-module is specifically used to, when the task target of the first task request and the task target of the second task request are the same, splice the task content in the first task request and the task content in the second task request into the same splicing segment; when the task target of the first task request and the task target of the second task request are different, splice the task content in the first task request and the task content in the second task request into different splicing segments.
[0019] According to a specific implementation method of an embodiment of the present application, the replacement module is specifically used to: replace the target linear layer in the initial basic model with a new linear layer to obtain an intermediate basic model; quantize the intermediate basic model; and compile the quantized intermediate basic model to obtain a final basic inference model.
[0020] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the task reasoning method described in any of the aforementioned implementation methods.
[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the task reasoning method described in any of the aforementioned implementation methods.
[0022] The task reasoning method, device, electronic device, and readable storage medium of this embodiment combine the fine-tuning weight matrices of multiple task targets with the multi-task reasoning model in the case of multiple task requests, and obtain an output feature matrix in the target linear layer of the multi-task reasoning model. This output feature matrix can be used as the input of the next network layer in the multi-task model to further obtain the reasoning results corresponding to the task request. In this way, while completing multi-task reasoning, the requirements for local device configuration can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0024] Figure 1 A flowchart of a task reasoning method based on a task reasoning model provided in one embodiment of the present application; Figure 2 A calculation diagram of the SGMV operator is provided in a specific embodiment of the present application; Figure 3 This is a schematic diagram of the use of a multi-task reasoning model provided in a specific embodiment of the present application; Figure 4 A schematic diagram of the structure of a task reasoning device provided in one embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0025] The embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described are only a portion of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.
[0026] In order to enable those skilled in the art to better understand the technical concepts, implementation plans and beneficial effects of the embodiments of the present application, specific examples are described in detail below.
[0027] Figure 1 A flowchart of a task reasoning method provided in one embodiment of the present application is shown in FIG. Figure 1 As shown, the task reasoning method of this embodiment includes: S101: Receive a task request.
[0028] The task request includes at least two task objectives and task content corresponding to each task objective.
[0029] Task objectives can also be called task functions, such as solving math problems, emotion recognition, writing Python programs, etc.
[0030] The task content corresponding to solving math problems in the task objective is such as what does 9 plus 3 equal?
[0031] The received task request may be a voice message, text message, etc.
[0032] S102: Determine a fine-tuning weight matrix for each task objective according to each task objective and the corresponding relationship between the task objective and the fine-tuning weight matrix.
[0033] Each task objective has a corresponding fine-tuning weight matrix, which consists of two low-order matrices.
[0034] S103: Determine the reasoning result corresponding to the task request based on the fine-tuning weight matrix of each task content and each task goal and the preset basic reasoning model.
[0035] In this embodiment, the preset basic reasoning model includes at least one linear layer, and the target linear layer in the at least one linear layer is used to obtain an output feature matrix based on the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic reasoning model that is adjacent to the target linear layer and is before the target linear layer.
[0036] The input to the target linear layer of the preset basic inference model includes two input quantities: one is the fine-tuning weight matrix for each task objective, and the other is the output of the previous network layer. In a specific example, the preset basic inference model includes network layer A and network layer B connected to network layer A, where network layer A is located before network layer B, network layer B is a linear layer, network layer A is the previous network layer relative to network layer B, the output of network layer A is a, and the fine-tuning weight matrix of at least two task objectives is b. Then a and b are the inputs of network layer B, and network layer B processes a and b to obtain the corresponding output feature matrix.
[0037] In this embodiment, by splicing multiple fine-tuning weight matrices to the target linear layer of the basic inference model at one time, the model has multi-task processing capabilities.
[0038] In this embodiment, in the case of multiple task requests, the fine-tuning weight matrices of multiple task targets are combined with the multi-task reasoning model, and an output feature matrix is obtained in the target linear layer of the multi-task reasoning model. The output feature matrix can be used as the input of the next network layer in the multi-task model to further obtain the reasoning results corresponding to the task requests. In this way, while completing multi-task reasoning, the requirements for local device configuration can be reduced.
[0039] It is understandable that the output feature matrix obtained in the target linear layer of the multi-task reasoning model can also be directly output as the final result of the multi-task model.
[0040] In some examples, the method of this embodiment is applied to a local device.
[0041] In some examples, a multi-task reasoning model is determined according to the following steps: A1. Determine a new linear layer based on the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model.
[0042] The initial basic model is a trained model with specific functions, such as a large language model.
[0043] The linear layer in this embodiment can be a universal linear model that can be used by all major models (base models) and can be stored in a common tool such as a model inference framework (also known as a tensor virtual machine (TVM)). In this case, the linear layer in the tensor virtual machine can be rewritten according to the preset piecewise aggregation matrix multiplication module to obtain a new linear layer.
[0044] The functionality of the new linear layer can superimpose the functionality of the piecewise aggregation matrix multiplication module on the functionality of the corresponding linear layer of the base model.
[0045] To facilitate deployment of models with multi-task reasoning on any device, in some examples, the preset segmented aggregate matrix multiplication module is a customized segmented aggregate matrix multiplication module.
[0046] The customized segmented aggregate matrix multiplication module is a functional module customized for a specific device. In some examples, the customized segmented aggregate matrix multiplication module is a module compiled for a central processing unit (CPU), and the module can run on the central processing unit.
[0047] In other examples, a corresponding customized segmented aggregate matrix multiplication module capable of running on the device is compiled based on the configuration of the local device.
[0048] Through the customized segmented aggregation matrix multiplication module, it is convenient to deploy and run the multi-task reasoning model in this embodiment on a local device with lower configuration.
[0049] In this embodiment, determining a new linear layer in step A1 according to the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model includes: A11. Register the customized segmented aggregation matrix multiplication module to the model inference framework.
[0050] The TVM framework includes linear layers for use in the model.
[0051] A12. The model inference framework determines a new linear layer based on the customized segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model.
[0052] In a specific example, the customized segmented aggregation matrix multiplication module is called using the call_dsp_packed method of TVM, and the customized segmented aggregation matrix multiplication module is combined with the linear layer corresponding to the initial basic model to obtain a new linear layer.
[0053] A2. Use the new linear layer to replace the target linear layer in the base model to obtain the final base inference model.
[0054] The new linear layer determined in step A1 is used to replace the target linear layer in the base model to obtain a reasoning model with multi-task processing capabilities. The final base reasoning model in this embodiment is the preset base reasoning model in the above embodiment.
[0055] In some examples, new linear layers are used to replace some linear layers in the base model. In this case, these linear layers are the target linear layers. In other examples, new linear layers are used to replace all linear layers in the base model. In this case, all linear layers are the target linear layers.
[0056] In this embodiment, the new linear layer is the target linear layer in the multi-task reasoning model.
[0057] In some examples, the target linear layer in the preset basic inference model includes a first target linear layer.
[0058] The first target linear layer is a target linear layer in the multi-task reasoning model.
[0059] In step S103, the inference result corresponding to the task request is determined based on the fine-tuned weight matrix of each task content and each task goal and the preset basic inference model, including: S103a, the first target linear layer determines a first output feature matrix of the first target linear layer according to the fine-tuning weight matrix of each task target and the output feature matrix of the previous network layer of the first target linear layer.
[0060] For example, in a multi-task reasoning model, the output feature matrix of the previous network layer of the first target linear layer is X, which is also an input of the first target linear layer. The fine-tuning weight matrices A and B of each task target serve as another input of the first target linear layer. In the first target linear layer, the piecewise aggregation matrix multiplication module can be called to multiply the two together to obtain the first output feature matrix, namely X × A × B.
[0061] S103b. The first target linear layer determines a second output feature matrix of the first target linear layer according to the output feature matrix of the linear layer corresponding to the basic model and the previous network layer of the first target line.
[0062] In this embodiment, the first target linear layer is the linear layer obtained after replacement, and the linear layer before replacement is the linear layer corresponding to the base model. The first target linear layer has the function of replacing the linear layer before replacement. Therefore, the output feature matrix X of the network layer before the first target linear layer is processed by the linear layer before replacement by the first target linear layer to obtain the first output feature matrix X×K, where K is the weight of the linear layer before replacement.
[0063] S103c: Add the first output feature matrix and the second output feature matrix to obtain the output feature matrix of the first target linear layer.
[0064] The output feature matrix of the first target linear layer is X×K+X×A×B.
[0065] To improve the efficiency of multi-task reasoning, in some examples, determining the first output feature matrix of the first target linear layer according to the fine-tuned weight matrix of each task target and the output feature matrix of the previous network layer of the first target linear layer in step S103a includes: B1. According to the task objectives, the output feature matrix of the previous network layer of the first target linear layer is segmented to obtain multiple feature matrix segments.
[0066] If there are two task objectives, after receiving the output feature matrix of the network layer before the first target linear layer, the output feature matrix is divided into two parts according to the different task objectives, that is, two feature matrix segments, each feature matrix segment corresponds to one task objective.
[0067] B2. Determine the task target index of each feature matrix segment according to the correspondence between each feature matrix segment and the preset feature matrix segment and task target index.
[0068] B3. According to the task target index of each feature matrix segment, determine the fine-tuning weight matrix corresponding to each feature matrix segment in the fine-tuning weight matrix of each task target.
[0069] B4. Determine the first output feature matrix of the first target linear layer based on each feature matrix segment and the fine-tuning weight matrix corresponding to each feature matrix segment.
[0070] Multiply each feature matrix segment with its corresponding fine-tuning weight matrix to obtain the result corresponding to each feature matrix segment, and concatenate the results corresponding to each feature matrix segment to obtain the first output feature matrix of the first target linear layer.
[0071] In order to enable the multi-task reasoning model to have multi-task reasoning capabilities while retaining the functions of the basic model, in some examples, after determining the fine-tuning weight matrix for each task objective based on the task objectives and the correspondence between the task objectives and the fine-tuning weight matrix in step S102, and before determining the inference result corresponding to the task request in step S103, the method of this embodiment further includes: S104. Obtain the list corresponding to the target linear layer.
[0072] The list includes the fine-tuning weight matrix of each task objective.
[0073] The fine-tuning weight matrix of each task objective in this embodiment is organized in the form of a list, and all major models can use the fine-tuning weight matrix organized in the form of a list.
[0074] The parameters of the list can be standardized through the configuration file.
[0075] S106. Distribute the fine-tuning weight matrix of each task target in the list to the target linear layer in the preset basic inference model through the forward propagation interface.
[0076] During use, the fine-tuning weight matrix can be distributed through the forward propagation interface.
[0077] In this embodiment, by distributing the fine-tuning weight matrix, the model is given multi-tasking processing capabilities, avoiding the problem of modifying the basic model in the morning due to the need to merge the basic model and the fine-tuning weight matrix (Lora weight).
[0078] S107. The target linear layer in the preset basic inference model receives the fine-tuning weight matrix distributed by the forward propagation interface.
[0079] In this embodiment, the fine-tuning weight matrix of each task target is distributed to the target linear layer in the preset basic reasoning model through the forward propagation interface. After receiving the fine-tuning weight matrix distributed by the weight matrix distribution interface, the target linear layer in the basic reasoning model processes the fine-tuning weight matrix of each task target.
[0080] In order to increase throughput and reduce the number of traps in each matrix multiplication kernel, in some examples, after receiving the task request, the method of this embodiment further includes: S108: Splice the task contents in the task request.
[0081] The amount of concatenating the task contents in the task request to form an input model.
[0082] The task request includes a first task request and a second task request; The task contents in the splicing task request in step S108 include: S108a: Concatenate the task content in the first task request and the task content in the second task request.
[0083] Specifically, the step S108a of combining the task content in the first task request and the task content in the second task request includes: C1. When the task target of the first task request and the task target of the second task request are the same, the task content in the first task request and the task content in the second task request are spliced into the same splicing segment.
[0084] Placing tasks with the same objectives in adjacent positions so that they are in the same splicing segment can improve the task throughput of the multi-task reasoning model and facilitate improving reasoning efficiency.
[0085] C2. When the task target of the first task request and the task target of the second task request are different, the task content in the first task request and the task content in the second task request are spliced into different splicing segments.
[0086] For task contents in task requests with different task objectives, the task contents corresponding to different task objectives are spliced into different splicing segments.
[0087] To improve the inference efficiency of the multi-task inference model, in some examples, the target linear layer in the initial base model is replaced with a new linear layer in step A2 to obtain a final base inference model, including: A21. Use a new linear layer to replace the target linear layer in the initial base model to obtain an intermediate base model.
[0088] A22. Quantify the intermediate basic model.
[0089] Quantifying the intermediate basic inference model can reduce the number of parameters of the intermediate basic model, making the model smaller in size and the computing resources it relies on smaller.
[0090] In some examples, the quantization tool is TVM.
[0091] A23. Compile the quantized intermediate basic model to obtain the final basic inference model.
[0092] In some examples, the compilation tool is TVM.
[0093] After compilation, the final basic inference model is obtained, which is deployed on the local device in the form of a link library.
[0094] After the quantization and compilation process, the initial task reasoning model is optimized, and the final basic reasoning model obtained has high reasoning efficiency.
[0095] The solution of this application is described in detail below with a specific embodiment.
[0096] The task reasoning method of this embodiment may include: 1. Write the SGMV operator (customized segmented aggregate matrix multiplication module) on general devices.
[0097] The general purpose device may be a CPU.
[0098] The SGMV operator (Segmented Gather Matrix-Vector Multiplication) is an operator that batch-calculates the multiplication of multiple fine-tuned (Lora) weight matrices. It can effectively improve the throughput of computing devices and enhance the processing efficiency of multiple tasks.
[0099] Input of the SGMV operator: feature matrix X, segment vector S, Lora index vector idx and Lora weight W.
[0100] Among them, the feature matrix X: the input of the target linear layer.
[0101] Segment vector S: Each element is a critical point of a segment in X, representing the segment information in X. Each segment requests the same Lora weight matrix.
[0102] Lora index vector idx: The length is the same as S, indicating the Lora index of the corresponding segment request.
[0103] Lora weight W: A multidimensional tensor consisting of multiple lora weights of the same rank.
[0104] The algorithm of the SGMV operator is: X[S[i], S[i+1]]+= X[S[i], S[i+1]]@ W[idx[i]]; where i represents the index of S and @ represents matrix multiplication.
[0105] Specifically, the input X is sliced (segmented) using the elements in the segment vector S. The matrix corresponding to each segment uses the same lora weights for matrix multiplication. For the i-th segment, the slice of X is S[i] to S[i+1]. The index idx[i] of the corresponding lora is obtained from the index vector, and then the weight matrix is obtained from W using the index for multiplication.
[0106] SGMV adopts the strategy of in-place output, and the output is also in X. This can avoid the problem of resource waste caused by opening up new storage space for output.
[0107] See also Figure 2 ,For each segment after segmentation, since there is no dependency between each segment, each segment can be calculated in parallel.
[0108] 2. Register the SGMV operator to TVM.
[0109] 3. For the linear layer of TVM, use the SGMV operator to rewrite the linear layer to obtain a new linear layer.
[0110] The function of the new linear layer is: Linear(X, W1, A, B, S, idx, W): X1 = X @ W1 SGMV(X, S, idx, lora_A) SGMV(X, S, idx, lora_B) RETURN X+X1 Where W1 is the weight matrix of the original linear layer in TVM; A and B are two low-order matrices of the fine-tuning weight matrix (lora weight matrix); X, S, idx, and W have the same meanings as the parameters in step 1.
[0111] The Lora calculation of a target linear layer is decomposed into using two SGMV operators.
[0112] If quantitative reasoning is required, SGMV needs to be integrated into the linear layer of the quantized version. 4. Replace all linear layers of the base model with the new linear layers to obtain a model with multi-task reasoning capabilities.
[0113] 5. Define the weight matrix distribution interface for the multi-task reasoning model.
[0114] This embodiment designs an abstract mechanism that leverages TVM's parameter passing design to unify the LoRa weight matrices of various models into a single list. All model LoRa parameters appear as a list to upper layers, and the caller only needs to implement a single interface to use them. While the list parameters vary by model, they can be standardized through configuration files.
[0115] Define the weight matrix distribution interface as follows: FORWARD(X: Tensor, kv_cache:PagedKVCache, loras:list[Tensor]) Where X is the tensor corresponding to the received task content; kv_cache is the key-value cache; and loras is a list including the fine-tuning weight matrix.
[0116] Inside the FORWARD function, the lora weight matrix can be distributed for the task inference model.
[0117] By defining Loras parameters, the differences between various models after Lora fine-tuning are shielded. Different numbers of parameters can be passed in according to different models, thereby unifying the calling interfaces of multiple models.
[0118] 6. Quantize and compile models with multi-task reasoning capabilities.
[0119] After defining the model, use TVM to compile and quantize the model with multi-task reasoning capabilities, generate the model link library, and then use the FORWARD method to use the model.
[0120] When using the model, in order to increase throughput and reduce the number of times the matrix multiplication kernel is trapped, requests for the same Lora are usually merged to adjacent positions before input to form the same segment.
[0121] Users can use one task target (Lora) to process tasks in each request, or use multiple Loras to process multiple tasks in one request.
[0122] See also Figure 3 In a specific example, a user can specify one or more LoRas in a request to perform specific tasks. In the figure, there are three LoRas (three tasks). Assume that their functions are as follows:
[0123] If a user needs to determine the sentiment of a sentence, they can specify Lora 3 in the request, and the model will generate the sentence's emotional tendency. In another scenario, if a user needs to write Python code to solve a math problem, they can specify both Lora 1 and Lora 2 in the request with different weight coefficients (for example, 0.5 and 0.5). They will work at a certain level of power (the two power levels are the same) based on the user-specified weight ratio, and use the Python code to solve the problem.
[0124] The request uses different weight coefficients to multiply the weight coefficients with the corresponding fine-tuning weight matrix to obtain a new fine-tuning weight matrix. The target linear layer in the multi-task reasoning model processes the new fine-tuning weight matrix.
[0125] Figure 4 This is a structural diagram of a task reasoning device provided in one embodiment of the present application, as shown in FIG. Figure 4 As shown, the multi-task reasoning device based on the task reasoning model of this embodiment includes: a first receiving module 11, used to receive a task request; wherein the task request includes at least two task objectives and task content corresponding to each task objective; a first determining module 12, used to determine the fine-tuning weight matrix of each task objective according to each task objective and the correspondence between the task objective and the fine-tuning weight matrix; a second determining module 13, used to determine the reasoning result corresponding to the task request according to each task content, the fine-tuning weight matrix of each task objective and a preset basic reasoning model; wherein the preset basic reasoning model includes at least one linear layer, and the target linear layer in the at least one linear layer is used to obtain an output feature matrix according to the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic reasoning model that is adjacent to the target linear layer and is located before the target linear layer.
[0126] The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0127] The device of this embodiment, in the case of multiple task requests, combines the fine-tuning weight matrices of multiple task targets with the multi-task reasoning model, obtains an output feature matrix in the target linear layer of the multi-task reasoning model, and the output feature matrix can be used as the input of the next network layer in the multi-task model to further obtain the reasoning results corresponding to the task requests. In this way, while completing multi-task reasoning, the requirements for local device configuration can be reduced.
[0128] As an optional embodiment, the device also includes: a third determination module, used to determine a new linear layer based on a preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model; a replacement module, used to replace the target linear layer in the initial basic model with the new linear layer to obtain a final basic reasoning model; wherein the new linear layer is the target linear layer in the final basic reasoning model.
[0129] As an optional implementation, the preset segmented aggregation matrix multiplication module is a customized segmented aggregation matrix multiplication module; wherein the third determination module is specifically used to: register the customized segmented aggregation matrix multiplication module into the model inference framework; the model inference framework determines a new linear layer based on the customized segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model.
[0130] As an optional implementation, the target linear layer in the preset basic inference model includes a first target linear layer; wherein the second determination module includes: a first determination submodule, configured to determine, through the first target linear layer, a first output feature matrix of the first target linear layer based on a fine-tuning weight matrix of each task target and an output feature matrix of a previous network layer of the first target linear layer; a second determination submodule, configured to determine, through the first target linear layer, a second output feature matrix of the first target linear layer based on a linear layer corresponding to the basic model and an output feature matrix of a previous network layer of the first target line; and an addition submodule, configured to add the first output feature matrix and the second output feature matrix to obtain an output feature matrix of the first target linear layer.
[0131] As an optional implementation, the first determination submodule is specifically configured to: segment the output feature matrix of the previous network layer of the first target linear layer according to the task objective to obtain a plurality of feature matrix segments; determine the task objective index of each feature matrix segment according to the correspondence between each feature matrix segment and a preset feature matrix segment and the task objective index; determine, according to the task objective index of each feature matrix segment, a fine-tuning weight matrix corresponding to each feature matrix segment in the fine-tuning weight matrix of each task objective; and determine the first output feature matrix of the first target linear layer according to each feature matrix segment and the fine-tuning weight matrix corresponding to each feature matrix segment.
[0132] As an optional embodiment, the device further includes: an acquisition module for acquiring a list corresponding to the target linear layer; the list includes a fine-tuning weight matrix for each task target; a distribution module for distributing the fine-tuning weight matrix for each task target in the list to the target linear layer in the preset basic inference model through the forward propagation interface; and a second receiving module for the target linear layer in the preset basic inference model to receive the fine-tuning weight matrix distributed by the forward propagation interface.
[0133] As an optional embodiment, the device also includes: a splicing module, which is used to splice the task contents in the task request after the first receiving module receives the task request; wherein, the task request includes a first task request and a second task request; the splicing module includes: a splicing sub-module, which is used to splice the task content in the first task request and the task content in the second task request; the splicing sub-module is specifically used to splice the task content in the first task request and the task content in the second task request into the same splicing segment when the task target of the first task request and the task target of the second task request are the same; and splice the task content in the first task request and the task content in the second task request into different splicing segments when the task target of the first task request and the task target of the second task request are different.
[0134] As an optional implementation, the replacement module is specifically used to: replace the target linear layer in the initial basic model with a new linear layer to obtain an intermediate basic model; quantize the intermediate basic model; and compile the quantized intermediate basic model to obtain a final basic inference model.
[0135] The device of the above embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.
[0136] Figure 5 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application is shown in FIG. Figure 5As shown, it may include: a shell 61, a processor 62, a memory 63, a circuit board 64 and a power supply circuit 65, wherein the circuit board 64 is placed inside the space enclosed by the shell 61, and the processor 62 and the memory 63 are arranged on the circuit board 64; the power supply circuit 65 is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory 63 is used to store executable program code; the processor 62 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 63, and is used to execute any one of the task reasoning methods provided in the aforementioned embodiments, thereby also achieving corresponding beneficial technical effects, which have been described in detail above and will not be repeated here.
[0137] The aforementioned electronic devices exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and are primarily designed to provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.
[0138] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.
[0139] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.
[0140] (4) Other electronic devices with data interaction functions.
[0141] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement any task reasoning method provided in the aforementioned embodiments, thereby also achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.
[0142] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.
[0143] Each embodiment in this specification is described in a related manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.
[0144] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0145] For the convenience of description, the above device is described as being divided into various units / modules based on their functions. Of course, when implementing this application, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.
[0146] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0147] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A task reasoning method, characterized in that: include: Receive a task request; wherein the task request includes at least two task objectives and task content corresponding to each task objective; Determine the fine-tuning weight matrix for each task objective based on the corresponding relationship between the task objective and the fine-tuning weight matrix; Determine an inference result corresponding to the task request based on the fine-tuning weight matrix of each task content and each task objective and a preset basic inference model; wherein the preset basic inference model includes at least one linear layer, and a target linear layer in the at least one linear layer is used to obtain an output feature matrix based on the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic inference model that is adjacent to the target linear layer and is located before the target linear layer.
2. The method according to claim 1, characterized in that The preset basic reasoning model is determined according to the following steps: Determine a new linear layer based on the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model; The target linear layer in the initial basic model is replaced by a new linear layer to obtain a final basic reasoning model; wherein the new linear layer is the target linear layer in the final basic reasoning model.
3. The method according to claim 2, characterized in that The preset segmented aggregation matrix multiplication module is a customized segmented aggregation matrix multiplication module; The step of determining a new linear layer based on the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the basic model includes: Registering the customized segmented aggregate matrix multiplication module into the model inference framework; The model inference framework determines a new linear layer based on the customized segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model.
4. The method according to claim 2, characterized in that The target linear layer in the preset basic inference model includes a first target linear layer; The step of determining the reasoning result corresponding to the task request based on the fine-tuned weight matrix of each task content and each task objective and a preset basic reasoning model includes: The first target linear layer determines a first output feature matrix of the first target linear layer according to the fine-tuning weight matrix of each task target and the output feature matrix of the previous network layer of the first target linear layer; The first target linear layer determines a second output feature matrix of the first target linear layer according to an output feature matrix of a linear layer corresponding to the base model and a previous network layer of the first target line; The first output feature matrix and the second output feature matrix are added to obtain an output feature matrix of the first target linear layer.
5. The method according to claim 4, characterized in that The step of determining a first output feature matrix of the first target linear layer according to the fine-tuning weight matrix of each task objective and the output feature matrix of a previous network layer of the first target linear layer includes: According to the task objective, segmenting the output feature matrix of the network layer preceding the first target linear layer to obtain a plurality of feature matrix segments; Determine the task target index of each feature matrix segment according to the correspondence between each feature matrix segment and the preset feature matrix segment and the task target index; According to the task target index of each feature matrix segment, in the fine-tuning weight matrix of each task target, the fine-tuning weight matrix corresponding to each feature matrix segment is determined; Determine a first output feature matrix of the first target linear layer according to each feature matrix segment and a fine-tuning weight matrix corresponding to each feature matrix segment.
6. The method according to claim 1, characterized in that After determining the fine-tuning weight matrix for each task objective based on the corresponding relationship between the task objectives and the fine-tuning weight matrix, and before determining the inference result corresponding to the task request, the method further includes: Obtain a list corresponding to the target linear layer; the list includes a fine-tuning weight matrix for each task target; Distribute the fine-tuning weight matrix of each task target in the list to the target linear layer in the preset basic inference model through the forward propagation interface; The target linear layer in the preset basic inference model receives the fine-tuning weight matrix distributed by the forward propagation interface.
7. The method according to claim 1, characterized in that After receiving the task request, the method further includes: Splicing the task contents in the task request; Wherein, the task request includes a first task request and a second task request; The step of splicing the task contents in the task request includes: concatenating the task content in the first task request and the task content in the second task request; The step of combining the task content in the first task request and the task content in the second task request includes: If the task objective of the first task request and the task objective of the second task request are the same, splicing the task content in the first task request and the task content in the second task request into the same splicing segment; In a case where the task target of the first task request and the task target of the second task request are different, the task content in the first task request and the task content in the second task request are spliced into different splicing segments.
8. The method according to claim 2, characterized in that The method of replacing the target linear layer in the initial base model with the new linear layer to obtain the final base inference model includes: Replace the target linear layer in the initial base model with the new linear layer to obtain an intermediate base model; quantifying the intermediate basic model; The quantized intermediate basic model is compiled to obtain a final basic inference model.
9. A task reasoning device, characterized in that: include: A first receiving module is configured to receive a task request, wherein the task request includes a task objective and task content corresponding to each task objective; A first determination module is used to determine the fine-tuning weight matrix of each task objective according to each task objective and the corresponding relationship between the task objective and the fine-tuning weight matrix; The second determination module is used to determine the inference result corresponding to the task request based on the fine-tuning weight matrix of each task content and each task objective and a preset basic inference model; wherein the preset basic inference model includes at least one linear layer, and the target linear layer in the at least one linear layer is used to obtain an output feature matrix based on the fine-tuning weight matrix and the output of the previous network layer; the previous network layer is a network layer in the preset basic inference model that is adjacent to the target linear layer and is located before the target linear layer.
10. The device according to claim 9, characterized in that The device further comprises: A third determination module is used to determine a new linear layer according to the preset segmented aggregation matrix multiplication module and the linear layer corresponding to the initial basic model; A replacement module is used to replace the target linear layer in the initial basic model with a new linear layer to obtain a final basic reasoning model; wherein the new linear layer is the target linear layer in the final basic reasoning model.