Task processing method, device and apparatus, and computer storage medium

Abstract

Embodiments of the present application provide a task processing method, device and equipment and computer storage medium; the method comprises: obtaining a word set to be processed, the word set to be processed comprising a plurality of words from a plurality of tasks to be processed; based on the plurality of tasks to be processed corresponding to the word set to be processed, determining the weight information corresponding to the linear layer in the plurality of expert network models, the weight information comprising a plurality of weight fusion parameters of a plurality of linear layers belonging to the same hierarchical structure in the plurality of expert network models, the weight fusion parameters being determined by fusion calculation on the basis weight parameters of the expert basis network and the low rank adaptation matrix corresponding to the task to be processed; dividing the weight information and the word set to be processed into a plurality of data blocks, the data block comprising part of the weight fusion parameters and part of the words satisfying the calculation relationship; parallel processing the plurality of data blocks to determine the process inference result of the linear layer, the process inference result being used for input to the next linear layer for inference operation.

Description

Technical Field

[0001] This application relates to the field of network model technology, and in particular to a task processing method, apparatus, device, and computer storage medium. Background Technology

[0002] Mixture of Experts (MoE) is a neural network architecture that dynamically selects a combination of multiple expert subnetworks to process input through a gating network. With the development of MoE models, traditional solutions can deploy independent model instances (including a base network and a low-rank adaptation network) for each downstream task and request different model services based on the task type.

[0003] However, when multiple tasks need to be processed simultaneously, since different tasks correspond to different model instances, multiple model instances need to be used for task processing operations. This not only causes the memory usage to increase linearly, but also easily reduces the efficiency of task processing operations. Summary of the Invention

[0004] This application provides a task processing method, apparatus, device, and computer storage medium that can achieve parallel processing of multiple tasks through a hybrid expert model, which not only reduces the amount of video memory required but also improves the efficiency of task processing.

[0005] In a first aspect, embodiments of the present invention provide a task processing method, including: Obtain a set of lexical units to be processed, wherein the set of lexical units includes multiple lexical units from multiple tasks to be processed; Based on the multiple tasks to be processed corresponding to the lexical set, the weight information corresponding to the linear layers in multiple expert network models is determined. The multiple expert network models are used to process the multiple tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the task to be processed. The weight information and the word set are divided into multiple data blocks, and each data block includes some weight fusion parameters and some word elements that satisfy the calculation relationship. The multiple data blocks are processed in parallel to determine the process inference result corresponding to the linear layer. The process inference result is used to input the inference operation to the next linear layer to determine the multiple task inference results corresponding to multiple tasks to be processed.

[0006] Secondly, embodiments of the present invention provide a task processing apparatus, comprising: The first acquisition module is used to acquire a set of lexical units to be processed, wherein the set of lexical units includes multiple lexical units from multiple tasks to be processed; The first grouping module is used to group multiple words in the word set and determine the word stack corresponding to multiple tasks to be processed; The first determining module is used to determine the weight information corresponding to the linear layers in multiple expert network models based on the multiple tasks to be processed corresponding to the word set. The multiple expert network models are used to process the multiple tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the task to be processed. The first grouping module is further configured to divide the weight information and the word set into multiple data blocks, wherein the data blocks include some weight fusion parameters and some words that satisfy the calculation relationship; The first processing module is used to process the multiple data blocks in parallel, determine the process inference result corresponding to the linear layer, and the process inference result is used to input to the next linear layer for inference operation, so as to determine the multiple task inference results corresponding to multiple tasks to be processed.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method in the first aspect described above.

[0008] Fourthly, embodiments of the present invention provide a computer storage medium for storing a computer program, which, when executed by a computer, implements the method described in the first aspect above.

[0009] Fifthly, embodiments of the present invention provide a computer program product, comprising: a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps of the method described in the first aspect.

[0010] The task processing method, apparatus, device, and computer storage medium provided in this embodiment obtain a set of terms to be processed, and then determine the weight information corresponding to the linear layers in multiple expert network models based on the multiple tasks to be processed corresponding to the set of terms. The weight information and the set of terms are divided into multiple data blocks, and the multiple data blocks are processed in parallel. This allows for accurate determination of the process inference result corresponding to the linear layer. By enabling a single instance model of the hybrid expert model, multiple different tasks to be processed can be processed. This effectively overcomes the defect in related technologies that require the deployment of different model instances for different tasks, which increases the memory usage. Furthermore, it enables parallel processing of multiple tasks to be processed, thereby effectively improving the efficiency of task processing and further ensuring the practicality of the method. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic diagram of a task processing method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating a task processing method provided for an exemplary embodiment of this application; Figure 3 A schematic diagram of multiple expert network models provided for an exemplary embodiment of this application; Figure 4 This is a flowchart illustrating a specific embodiment of the present application, which describes how to determine the weight information of linear layers in multiple expert network models based on multiple tasks to be processed corresponding to the lexical set. Figure 5 A flowchart illustrating the process of dividing the weight information and the word set into multiple data blocks, provided for an exemplary embodiment of this application; Figure 6 A flowchart illustrating the process of performing parallel computation on multiple data blocks to determine the inference result corresponding to the linear layer, provided as an exemplary embodiment of this application; Figure 7 A schematic diagram of the principle of a hybrid reasoning method for multiple tasks provided in an exemplary application embodiment of this application. Figure 1 ; Figure 8 A schematic diagram of the principle of a hybrid reasoning method for multiple tasks provided in an exemplary application embodiment of this application. Figure 2 ; Figure 9A schematic diagram of the structure of a task processing apparatus provided for an exemplary embodiment of this application; Figure 10 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0014] The various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards. Furthermore, the technical solutions provided in the embodiments of this application can employ deep learning models with relatively large parameter scales. The large model is merely an example, and the embodiments of this application do not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in the embodiments of this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0015] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.

[0016] Terminology definition: Mixture of Experts (MOE) is a neural network architecture that uses a gating network to dynamically select a combination of multiple expert subnetworks to process the input.

[0017] Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that achieves model adaptation through low-rank matrix decomposition (ΔW=B@A), where A can be called the first matrix and B can be called the second matrix.

[0018] To facilitate understanding of the task processing method, apparatus, device, and computer storage medium provided in the embodiments of this application, the relevant technologies are briefly described below: In e-commerce platforms, with the emergence of new scenarios, new data, and new demands, algorithmic models often need to have rapid iteration and responsiveness. The traditional model of "retraining a complete model for each new demand" is no longer able to meet the agile development requirements of applications due to its high time, computing power, and human resource costs. At the same time, the limited online graphics processing unit (GPU) resources also limit the feasibility of deploying dedicated models independently for different application scenarios.

[0019] As can be seen from the above, enterprises often face the challenge that the same expert network needs to serve multiple downstream tasks, such as Optical Character Recognition (OCR) tasks, image description tasks, object detection tasks, and so on.

[0020] Currently, with the widespread adoption of Mixture of Experts (MoE) architectures, MoE architectures achieve decoupling of parameters and computation through sparse activation, improving model capacity and performance without significantly increasing computational costs. Specifically, related technology 1 provides a multi-model instance deployment scheme. This scheme can deploy independent model instances for each downstream task. Independent model instances can include an expert base network (Base) and a LoRA low-rank adaptation matrix. Then, different models can be requested for task processing operations based on the task type of the downstream task.

[0021] However, the above solution has the following drawbacks: 1) The memory usage grows linearly, and N tasks require N times the memory; 2) The request traffic of different tasks is uneven, and some instances are idle for a long time, thus reducing resource utilization; 3) In a real production environment, the same batch may contain requests from different tasks, and different task requests in the same batch cannot be processed in parallel; 4) When adding new tasks, new instances need to be redeployed, which makes the scalability of the solution poor and increases the operation and maintenance costs.

[0022] Related technology 2 provides a scheme for serially switching the LoRA low-rank adaptation matrix. Specifically, after obtaining multiple tasks in a single mixed task batch, since the deployed model is a single model instance, the parameters of the LoRA low-rank adaptation matrix can be dynamically loaded and unloaded according to the task, and samples of the same task in the same batch can be merged.

[0023] However, the above solution has the following drawbacks: 1) High model loading latency: Each switch of the LoRA adapter takes 100-200ms, resulting in significant cumulative latency; 2) Low GPU utilization: Serial processing can easily lead to increased idle time of the GPU, resulting in low utilization; 3) Decreased throughput: The size of mixed task batches is forcibly split, making it impossible to take advantage of the parallel processing of large batches.

[0024] To address the aforementioned technical problems, embodiments of this application provide a task processing method, apparatus, device, and computer storage medium, as detailed in the appendix. Figure 1 As shown, the execution entity of this task processing method can be a task processing device 200, which can be implemented as a local server, a cloud server, or an edge server. When the task processing device 200 is implemented as a cloud server, the task processing method can be executed in the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, each with computing, storage, and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide this service by providing a service interface, which users can call to use the corresponding service. Service interfaces include Software Development Kits (SDKs) and Application Programming Interfaces (APIs).

[0025] The task processing device 200 is communicatively connected to a client set 100, which may include multiple clients, such as client 1, client 2, and client 3, etc. These clients are used by users to trigger task processing operations. The clients can be any computing device with a certain information interaction capability; specifically, clients can be mobile phones, personal computers (PCs), tablets, application programs, etc. Furthermore, the basic structure of the client may include at least one processor. The number of processors depends on the client's configuration and type. The client may also include memory, which can be volatile, such as random access memory (RAM), or non-volatile, such as read-only memory (ROM), flash memory, etc., or both types. The memory typically stores the operating system (OS), one or more applications, and may also store program data. In addition to the processing unit and memory, the client also includes some basic configurations, such as a network interface card (NIC) chip, I / O bus, display components, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, mouse, stylus, printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.

[0026] Task processing device 200 refers to a device capable of performing task processing operations in a network virtual environment, typically a device that utilizes a network for information planning and task processing. Task processing device 200 can be a task processing model used to implement task processing operations. Physically, task processing device 200 can be any device capable of providing computing services and performing corresponding task processing operations, such as a processor, server, etc. The main components of task processing device 200 include a processor, hard disk, memory, system bus, etc., and its architecture is similar to that of a general-purpose computer.

[0027] In this embodiment described above, the task processing device 200 establishes a network connection with any one of the clients in the client set 100. This network connection can be a wireless or wired network connection. If the task processing device 200 and the client have a communication connection, the mobile network standard can be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), Global Microwave Access Interoperability (WiMax), 5G, 6G, etc.

[0028] In this embodiment, any one client in the client set 100 is used by a user to generate a set of lexical units to be processed to trigger task processing operations. The lexical set includes multiple lexical units from multiple different tasks to be processed. To enable task processing operations, the lexical set can be sent to the task processing device 200, allowing the task processing device 200 to perform corresponding task processing operations based on the set of lexical units to be processed.

[0029] The task processing device 200 is used to receive the set of words to be processed sent by the client set 100. Since the multiple words in the word set come from multiple different tasks to be processed, in order to ensure the quality and efficiency of the task processing operation, the weight information corresponding to the linear layers in the multiple expert network models in the hybrid expert model can be determined based on the multiple tasks to be processed corresponding to the word set. The multiple expert network models are used to process different tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters corresponding to the expert base network and the low-rank adaptation matrix corresponding to the task to be processed.

[0030] For example, given multiple tasks to be processed based on the word set, including Task 1, Task 2, and Task 3, we can then determine the LoRA low-rank adaptation matrices 1, 2, and 3 corresponding to Task 1, Task 2, and Task 3, respectively. When the expert foundation network includes Expert Foundation Network 1, Expert Foundation Network 2, and Expert Foundation Network 3, each of these expert foundation networks includes two linear layers, namely Linear Layer 1 and Linear Layer 2. Then, based on the LoRA low-rank adaptation matrix 1 corresponding to Task 1 and the basic weights of Expert Foundation Network 1... The parameters are fused to obtain weight information 11 corresponding to linear layer 1 in expert network model 1 for processing task 1. Similarly, the LoRA low-rank adaptation matrix 2 corresponding to task 2 is fused with the basic weight parameters of expert network 1 to obtain weight information 12 corresponding to linear layer 1 in expert network model 1 for processing task 2. The LoRA low-rank adaptation matrix 3 corresponding to task 3 is fused with the basic weight parameters of expert network 1 to obtain weight information 13 corresponding to linear layer 1 in expert network model 1 for processing task 3. Following the above method, weight information 21, weight information 22, and weight information 23 corresponding to linear layer 1 in expert network model 2, and weight information 31, weight information 32, and weight information 33 corresponding to linear layer 1 in expert network model 3 can be obtained respectively.

[0031] In this system, the weight information corresponding to the linear layers in multiple expert network models reflects the reasoning behavior of these models. These multiple expert network models are used to process multiple tasks. The resulting weight information of the linear layers in these models is then used to analyze and process the word set. To further improve the quality and efficiency of task processing, the weight information and word set can be divided into multiple data blocks. Specifically, these blocks can be divided according to the correspondence during computation. Each data block includes some weight fusion parameters and some words that satisfy the computational relationship. These multiple data blocks can then be processed in parallel to determine the process reasoning result corresponding to the linear layer. This process reasoning result is then input into the next linear layer for reasoning operations to determine the task reasoning result corresponding to each task.

[0032] In this embodiment, when analyzing and processing multiple tasks, by enabling a single-instance model of the hybrid expert model, processing operations can be performed on multiple different tasks. This effectively overcomes the defect in related technologies that require the deployment of different model instances for different tasks, which increases the memory usage. Furthermore, it can also perform parallel processing operations on multiple tasks, thereby effectively improving the efficiency of task processing operations and further ensuring the practicality of the method.

[0033] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0034] Figure 2 A flowchart illustrating a task processing method provided for an exemplary embodiment of this application; see attached diagram. Figure 2 As shown, this embodiment provides a task processing method. The execution subject of this method is a task processing device, which can be implemented as software or a combination of software and hardware. When the task processing device is implemented as hardware, it can specifically be various electronic devices capable of performing task processing operations. In some instances, the task processing device can be implemented as an application client, server, cloud server, etc. When the task processing device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the task processing method provided in this embodiment may include: Step S201: Obtain the set of lexical units to be processed, which includes multiple lexical units from multiple tasks to be processed.

[0035] Step S202: Based on the multiple tasks to be processed corresponding to the word set, determine the weight information corresponding to the linear layers in multiple expert network models. Multiple expert network models are used to process different tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters corresponding to the expert base network and the low-rank adaptation matrix corresponding to the task to be processed.

[0036] Step S203: Divide the weight information and word set into multiple data blocks. Each data block includes some weight fusion parameters and some words that satisfy the calculation relationship.

[0037] Step S204: Perform parallel processing on multiple data blocks to determine the process inference result corresponding to the linear layer. The process inference result is used as input to the next linear layer for inference operation to determine the multiple task inference results corresponding to multiple tasks to be processed.

[0038] The specific implementation methods and principles of each of the above steps are explained in detail below: Step S201: Obtain the set of lexical units to be processed, which includes multiple lexical units from multiple tasks to be processed.

[0039] When there are multiple different task processing requirements, the task processing device can obtain a set of words to be processed. This set of words includes multiple words from multiple different tasks to be processed. The same task to be processed can include multiple different words. That is, one word can correspond to a task identifier of one task to be processed, and different words can correspond to the same task identifier.

[0040] In some instances, the lexical set can be obtained by analyzing and processing task description information or requests from multiple different tasks to be processed. In this case, obtaining the lexical set to be processed may include: identifying multiple clients communicating with the task processing device; obtaining multiple different tasks to be processed through these clients, wherein these multiple tasks to be processed can be determined by task processing requests obtained through human-computer interaction operations at the same time or within the same time period; and then performing word segmentation on these multiple different tasks to obtain a lexical set corresponding to each task. This lexical set may include multiple lexical units. Furthermore, for the lexical set, the multiple lexical units can be stored in random order within the lexical set, or the multiple lexical units can be stored sequentially based on multiple different tasks to be processed. This effectively ensures the accuracy and reliability of obtaining the lexical set.

[0041] Step S202: Based on the multiple tasks to be processed corresponding to the word set, determine the weight information corresponding to the linear layers in multiple expert network models. Multiple expert network models are used to process different tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters corresponding to the expert base network and the low-rank adaptation matrix corresponding to the task to be processed.

[0042] After obtaining the set of terms to be processed, in order to efficiently and in parallel process multiple different tasks, the weight information of the linear layers in multiple expert network models can be determined based on the multiple tasks to be processed corresponding to the set of terms. Among them, multiple different tasks to be processed often correspond to different LoRA low-rank adaptation matrices, and different LoRA low-rank adaptation matrices can be connected to multiple expert base networks.

[0043] For example, such as Figure 3 As shown, the LoRA low-rank adaptation matrices determined by multiple different task types (task 1, task 2, and task 3) include: LoRA low-rank adaptation matrices 11 and 21 corresponding to task 1, LoRA low-rank adaptation matrices 12 and 22 corresponding to task 2, and LoRA low-rank adaptation matrices 13 and 23 corresponding to task 3. When the expert foundation network includes expert foundation network 1 and expert foundation network 2, the above-mentioned expert foundation... Network 1 is communicatively connected to LoRA low-rank adaptation matrices 11, 12, and 13. When expert network 1 is connected to LoRA low-rank adaptation matrix 11 for computation, an expert network model for task 1 can be obtained; when connected to LoRA low-rank adaptation matrix 12, an expert network model for task 2 can be obtained; and when connected to LoRA low-rank adaptation matrix 13, an expert network model for task 3 can be obtained. Similarly, when expert network 2 is connected to LoRA low-rank adaptation matrix 21 for computation, an expert network model for task 1 can be obtained; when connected to LoRA low-rank adaptation matrix 22, an expert network model for task 2 can be obtained; and when connected to LoRA low-rank adaptation matrix 23, an expert network model for task 3 can be obtained. Therefore, the same task can be assigned to different expert network models for analysis and processing simultaneously.

[0044] Furthermore, for expert network models, multiple expert network models can correspond to the same model architecture, and any expert network model includes multiple linear layers. In order to achieve stable processing operations for multiple different tasks, the weight information corresponding to the linear layers in multiple expert network models used for task inference operations can be determined based on the identifiers of multiple tasks corresponding to the word set. This weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in multiple expert network models. The weight fusion parameters can be determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the task.

[0045] For example, when multiple expert network models include expert base network 1, expert base network 2, and expert base network 3, and each expert base network includes two linear layers, in order to stably process multiple different tasks, multiple low-rank adaptation matrices can be determined based on the multiple tasks corresponding to the word set. Then, by fusing the basic weight parameters of each expert base network with the multiple LoRA low-rank adaptation matrices, weight fusion parameters corresponding to each task can be obtained. Then, the weight information corresponding to the linear layers in the multiple expert network models used to process multiple tasks can be obtained through the weight fusion parameters corresponding to each task. For example, the weight information includes: weight information 1 corresponding to the linear layer of the first layer in multiple expert network models and weight information 2 corresponding to the linear layer of the second layer in multiple expert network models. The aforementioned weight information 1 includes: a1, a set of weight fusion parameters (including weight fusion parameters used to process different tasks) obtained by fusing the layer weight parameters corresponding to the linear layer of the first layer in expert basic network 1 with multiple different low-rank adaptation matrices; b1, a set of weight fusion parameters obtained by fusing the layer weight parameters corresponding to the linear layer of the first layer in expert basic network 2 with multiple different low-rank adaptation matrices; and c1, a set of weight fusion parameters obtained by fusing the layer weight parameters corresponding to the linear layer of the first layer in expert basic network 3 with multiple different low-rank adaptation matrices. The weight information 2 includes: a2, a set of weight fusion parameters obtained by fusing the layer weight parameters corresponding to the linear layer of the second layer in expert foundation network 1 with multiple different low-rank adaptation matrices; b2, a set of weight fusion parameters obtained by fusing the layer weight parameters corresponding to the linear layer of the second layer in expert foundation network 2 with multiple different low-rank adaptation matrices; and c3, a set of weight fusion parameters obtained by fusing the layer weight parameters corresponding to the linear layer of the second layer in expert foundation network 3 with multiple different low-rank adaptation matrices.

[0046] Step S203: Divide the weight information and word set into multiple data blocks. Each data block includes some weight fusion parameters and some words that satisfy the calculation relationship.

[0047] Since the weight information corresponding to the linear layers in multiple expert network models can identify the data reasoning behavior of the expert network models to a certain extent, and the word set can reflect the task content of multiple different tasks to be processed, in order to improve the quality and efficiency of task processing operations to a certain extent, after obtaining the weight information and word set corresponding to the linear layers in multiple expert network models, the weight information and word set can be divided into multiple data blocks. Each data block can include some weight fusion parameters and some words that satisfy the calculation relationship. Specifically, the weight fusion parameters and some words included in the same data block can refer to the corresponding parts that need to be reasoned and calculated when performing task reasoning operations.

[0048] In some instances, multiple data blocks can be obtained by manually dividing the weight information and word set. In this case, dividing the weight information and word set into multiple data blocks may include: displaying a partitioning configuration page for the weight information and word set; obtaining the partitioning configuration operation entered by the user on the partitioning configuration page; and dividing the weight information and word set into multiple data blocks based on the partitioning configuration operation. This effectively ensures the accuracy and reliability of determining multiple data blocks.

[0049] Step S204: Perform parallel processing on multiple data blocks to determine the process inference result corresponding to the linear layer. The process inference result is used as input to the next linear layer for inference operation to determine the multiple task inference results corresponding to multiple tasks to be processed.

[0050] Since multiple data blocks not only include information related to multiple tasks to be processed, but also the weight parameter information corresponding to the expert network model used to process multiple tasks to be processed, and the multiple data blocks are independent of each other, in order to improve the efficiency of processing multiple tasks to be processed to a certain extent, after obtaining multiple data blocks, multiple data blocks can be processed in parallel, thereby determining the process inference result corresponding to the linear layer.

[0051] In some instances, parallel processing operations can be implemented using a Graphics Processing Unit (GPU) that communicates with a task processing device. In this case, parallel processing of multiple data blocks to determine the process inference result corresponding to the linear layer can include: sending multiple data blocks to the GPU for parallel processing. Since the multiple data blocks come from multiple different tasks to be processed, parallel computation operations can be performed on different tasks to be processed, and the process inference result corresponding to the linear layer can be obtained. The process inference result can be obtained by fusing the parallel processing results of the multiple data blocks. The process inference result corresponding to the linear layer is then returned to the task processing device, so that the task processing device can reliably obtain the process inference result corresponding to the linear layer.

[0052] Furthermore, the obtained process inference results are used as input to subsequent network layers for inference operations, enabling the stable determination of multiple task inference results corresponding to multiple pending tasks, where one pending task corresponds to one task inference result. For example, in any expert network model that includes a first linear layer and a second linear layer following the first linear layer, multiple data blocks corresponding to the first linear layer can be determined based on the weight information and word set corresponding to the first linear layer. Then, multiple data blocks can be processed in parallel to obtain the process inference result 1 corresponding to the first linear layer. This process inference result 1 is then used as the input word set for the second linear layer. Based on the weight information and input word set of the second linear layer, multiple data blocks corresponding to the second linear layer are determined. These data blocks are then processed in parallel again to obtain the process inference result 2 corresponding to the second linear layer. This process inference result 2 is then input into subsequent network layers for inference operations, ensuring a stable determination of the task inference result corresponding to each task to be processed. Different tasks can have different task inference results. For example, when multiple tasks include task 1, task 2, and task 3, after multiple expert network models perform inference operations on the word sets corresponding to the multiple tasks, task inference result 1 corresponding to task 1, task inference result 2 corresponding to task 2, and task inference result 3 corresponding to task 3 can be obtained. This effectively improves the quality and efficiency of task processing.

[0053] The task processing method provided in this embodiment obtains a set of terms to be processed, and then determines the weight information corresponding to the linear layers in multiple expert network models based on the multiple tasks to be processed corresponding to the set of terms. The weight information and the set of terms are divided into multiple data blocks, and the multiple data blocks are processed in parallel. This allows for accurate determination of the process inference results corresponding to the linear layers. It also enables the processing of multiple different tasks by enabling a single instance model of the hybrid expert model. This effectively overcomes the defect in related technologies that require the deployment of different model instances for different tasks, which increases the consumption of video memory. Furthermore, it enables parallel processing of multiple tasks, thereby effectively improving the efficiency of task processing and further ensuring the practicality of the method.

[0054] Figure 4 This application provides an exemplary embodiment of a flowchart illustrating the process of determining the weight information of linear layers in multiple expert network models based on multiple tasks to be processed corresponding to a set of lexical units. Based on the above embodiment, refer to the appendix... Figure 4 As shown, based on the multiple tasks to be processed corresponding to the word set, determining the weight information corresponding to the linear layers in multiple expert network models can include: Step S401: Determine the task identifiers of the multiple tasks to be processed corresponding to the lexical set. Different tasks to be processed correspond to different task identifiers.

[0055] Since the lexical set includes multiple lexicals from multiple tasks to be processed, each lexical can correspond to a task identifier for identifying the task to be processed. Different tasks to be processed correspond to different task identifiers. In order to perform efficient parallel processing operations on the tasks to be processed, the task identifiers of multiple tasks to be processed corresponding to the lexical set can be determined. The task identifier can be determined by the task to be processed corresponding to each lexical.

[0056] Step S402: Based on multiple task identifiers, determine the model weight parameters corresponding to the reasoning operations of multiple expert network models for different tasks to be processed.

[0057] Different task identifiers can identify the relevant information of different tasks to be processed. Different tasks to be processed often require expert network models with different low-rank adaptation matrices for analysis and processing. Each expert network has a one-to-one correspondence with a task to be processed. Different task identifiers can correspond to different low-rank adaptation matrices, and different low-rank adaptation matrices can correspond to different matrix parameters. For example, low-rank adaptation matrix 1 includes matrix A1 and matrix B1, where matrix A1 and matrix B1 include matrix parameters used for inference operations on task 1; low-rank adaptation matrix 2 can include matrix A2 and matrix B2, where matrix A2 and matrix B2 include matrix parameters used for inference operations on task 2. In addition, an expert base network can be connected to multiple low-rank adaptation matrices, meaning that an expert base network can call any one of the multiple low-rank adaptation matrices to perform task processing operations. Therefore, in order to accurately process multiple different tasks corresponding to a word set, the model weight parameters of expert network models with different low-rank adaptation matrices can be determined based on multiple task identifiers. These model weight parameters can be obtained by fusing the basic weight parameters of the expert network model and the corresponding low-rank adaptation matrix.

[0058] In some instances, the model weight parameters corresponding to each of the expert network models after training can be stored in a preset cache. In this case, multiple expert network models corresponding to multiple tasks to be processed can be determined from all expert network models. These multiple expert network models can be at least a part of all expert network models. Then, by accessing the preset cache, the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed can be obtained. This effectively ensures the stability and reliability of the obtained model weight parameters.

[0059] In other instances, the model weight parameters corresponding to each of the multiple expert network models can be determined based on the model types to which the multiple expert base networks belong. In this case, determining the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks based on multiple task identifiers may include: determining the model type to which the expert base networks belong, where the model type includes at least one of the following: a first-type expert network and a second-type expert network, with the activation frequency of the first-type expert network being higher than that of the second-type expert network; and determining the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks based on the model types to which the multiple expert base networks belong and the multiple task identifiers.

[0060] Since expert network models of different types can have different model weight parameters, in order to flexibly determine the model weight parameters corresponding to each of the multiple expert network models, we can first determine the model type to which each expert base network belongs. The model type to which the expert base network belongs can be either a first-class expert network or a second-class expert network. The first-class expert network and the second-class expert network mentioned above can have different activation frequency characteristics. In some instances, the activation frequency of the first-class expert network is higher than that of the second-class expert network. In this case, the first-class expert network can be called a high-frequency expert model, and the second-class expert network can be called a low-frequency expert model.

[0061] In some instances, the model type to which each expert base network belongs can be determined based on the activation frequencies of each of the multiple expert base networks. In this case, obtaining the model type to which an expert base network belongs may include: obtaining the activation frequencies of multiple expert base networks; sorting the multiple expert base networks based on the magnitude of the activation frequencies to obtain model sorting information; and determining the model type to which the expert base network belongs based on the model sorting information.

[0062] Specifically, since different model types can correspond to different activation frequency characteristics, in order to accurately determine the model type to which multiple expert network models belong, the activation frequencies corresponding to each of the multiple expert base networks can be obtained. In some instances, the activation frequencies corresponding to each of the multiple expert base networks can be determined by the activation frequency distribution obtained from offline experiments on multiple expert network models.

[0063] Since activation frequency can identify the proportion or number of times a corresponding expert network model is selected by the routing mechanism on a given dataset, the model type of each expert basic network can be determined not only by its activation frequency but also by its selection proportion or number of selections. The specific determination principles are similar. The following explanation uses the determination of model type based on the activation frequency of each expert network model as an example: After obtaining the activation frequencies of multiple expert foundation networks, the model type of each expert foundation network can be determined based on these activation frequencies. In some instances, determining the model type of each expert foundation network based on its activation frequencies may include: determining a frequency threshold for analyzing the activation frequencies; comparing the activation frequencies of each expert foundation network with the frequency threshold; if the activation frequency is greater than or equal to the frequency threshold, classifying the expert network model corresponding to that activation frequency as a first-class expert network; if the activation frequency is less than the frequency threshold, classifying the expert network model corresponding to that activation frequency as a second-class expert network. This effectively achieves a stable determination of the model category of each expert network model.

[0064] In other instances, after obtaining the activation frequencies of multiple expert base networks, the multiple expert base networks can be sorted based on the magnitude of the activation frequencies to obtain model ranking information. In the model ranking information, the activation frequencies of expert network models located at the beginning of the ranking are greater than those of expert network models located at the end of the ranking. Then, based on the model ranking information, the model type to which each expert base network belongs can be determined.

[0065] The model type to which each expert basic network belongs can be determined based on the quantity information used to determine the first type of expert network. In this case, determining the model type to which multiple expert basic networks belong based on model ranking information may include: obtaining the quantity information used to determine the first type of expert network; determining the expert network models located at the position of the quantity information and those located before the quantity information as the first type of expert network based on model ranking information; and determining the expert basic network located after the quantity information as the second type of expert network based on model ranking information.

[0066] Specifically, to accurately determine the model type of multiple expert foundation networks, we can first obtain the quantity information used to determine the first type of expert networks. This quantity information is positively correlated with the computing resources used to perform data processing operations. Therefore, the quantity information used to define the first type of expert networks can be automatically obtained based on computing resources. Alternatively, the quantity information of the first type of expert networks can be determined based on user-input human-computer interaction. After obtaining the quantity information for determining the first type of expert networks, we can determine the expert network models located at or before the quantity information as first-type expert networks, i.e., expert network models with high activation frequencies, based on model ranking information. The expert foundation networks located after the quantity information are determined as second-type expert networks, i.e., expert network models with low activation frequencies. This effectively achieves flexible and reliable determination of the model type of each expert foundation network.

[0067] Furthermore, the model type to which each expert foundation network belongs can be determined not only based on the number of expert networks used to determine the first type of expert network, but also based on the median value in the model ranking information. In this case, determining the model type to which multiple expert foundation networks belong based on the model ranking information can include: determining the median value in the ranking information based on the model ranking information; identifying expert network models located at the median value and those located before the median value as first type expert networks; and identifying expert network models located after the median value as second type expert networks. This also ensures the flexibility and reliability of determining the model type to which each expert foundation network belongs.

[0068] After obtaining the model types to which multiple expert base networks belong, the model weight parameters corresponding to each expert network model can be determined based on these model types. In some instances, the model weight parameters corresponding to each expert network model can be determined based on a pre-defined mapping table, which includes the mapping relationship between expert network models, the model types to which the expert base networks belong, and the model weight parameters.

[0069] In other instances, since the model weight parameters of expert network models of different model types can correspond to different caching strategies, in some scenarios, different caching strategies may include a first caching strategy and a second caching strategy. The first caching strategy is used to indicate that the model weight parameters of the expert network model are pre-calculated and stored in a preset memory. The second caching strategy is used to indicate that the model weight parameters of the expert network model are not pre-calculated and need to be calculated and obtained in real time.

[0070] To alleviate the storage pressure on GPU memory, different caching strategies can be used to obtain the model weight parameters corresponding to expert network models of different model types through data reading or data retrieval operations. In this case, based on the model types of multiple expert base networks and multiple task identifiers, determining the model weight parameters corresponding to multiple expert network models when performing inference operations on different tasks can include: when the expert base network is a first type of expert network, determining the linear layer identifier corresponding to the linear layer in the expert base network; based on the task identifier and the linear layer identifier, using a first caching strategy to obtain the model weight parameters corresponding to the expert network models when performing inference operations on different tasks. This first caching strategy is used to identify that the model weight parameters of the expert network models are pre-calculated and stored in GPU memory.

[0071] After obtaining the model types corresponding to multiple expert network models, the model weight parameters of each expert network model can be cached based on their respective model types. Specifically, when the model type corresponding to the expert network model is a first-type expert network, the model weight parameters corresponding to the first-type expert network can be stored in the GPU memory using a first caching strategy. Specifically, the task identifier "task_id", the linear layer identifier "Layer_id", and the first caching strategy identifier "L1" corresponding to the first-type expert network are used as the key "Key", where "Key" is "L1 + task_id + Layer_id". Simultaneously, the model weight parameters corresponding to the first-type expert network when inferring different tasks are stored in the GPU memory as the value "Value" using the first caching strategy.

[0072] For example, when the model type of the expert base network 1 is a first-class expert network, and the low-rank adaptation matrix determined based on multiple tasks includes low-rank adaptation matrix 1 and low-rank adaptation matrix 2, model weight parameter 1 can be obtained by fusing the basic weight parameters of expert base network 1 and low-rank adaptation matrix 1; model weight parameter 2 can be obtained by fusing the basic weight parameters of expert base network 1 and low-rank adaptation matrix 2. Then, the calculated model weight parameters 1 and 2 are stored in the video memory using the first caching strategy, so that the model weight parameters of the expert network model composed of expert base network 1 and low-rank adaptation matrix can be quickly called to perform inference operations on the corresponding tasks.

[0073] When data inference operations are required using the expert network model of the first type of expert network, the linear layer identifier corresponding to the linear layer in the expert network model can be determined first. Then, based on the task identifier and the linear layer identifier, the model weight parameters corresponding to the expert network model can be obtained using the first caching strategy. Specifically, the task identifier "task_id", the linear layer identifier "Layer_id", and the first caching strategy identifier "L1" corresponding to the first type of expert network are used as the key "Key". The first caching strategy is used to perform data reading operations in the video memory, thereby reliably obtaining the model weight parameters corresponding to the expert network model.

[0074] In some other instances, since the first caching strategy and the second caching strategy have different policy characteristics, for the model weight parameters of the expert network model, when the expert base network in the expert network model is a second type of expert network, the model weight parameters corresponding to the expert network model when performing inference operations on different tasks can also be obtained through the second caching strategy. In this case, based on the model types to which multiple expert base networks belong and multiple task identifiers, determining the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks can include: when the expert network model is a second type of expert network, determining the linear layer identifier corresponding to the linear layer in the expert base network; based on the task identifier and the linear layer identifier, using the second caching strategy to obtain the low-rank adaptation matrix corresponding to the expert base network, which includes a first matrix and a second matrix, the above-mentioned second caching strategy is used to identify the temporary calculation of the model weight parameters of the expert network model; based on the basic weight parameters of the expert base network, the first matrix and the second matrix, determining the model weight parameters corresponding to the expert network model when performing inference operations on different tasks.

[0075] In the case of a second-type expert network as the expert foundation network, to alleviate the pressure on the GPU memory, a second caching strategy can be used to store the low-rank adaptation matrix corresponding to the second-type expert network in the GPU memory. The low-rank adaptation matrix can include a first matrix A and a second matrix B, which are used to determine the low-rank adaptation increment parameters corresponding to the second-type expert network. In some instances, Specifically, the task identifier "task_id" corresponding to the second type of expert network, the linear layer identifier "Layer-id" corresponding to the linear layer in the second type of expert network, and the second caching strategy identifier "L2" corresponding to the second type of expert network are used as the key "Key". The key is then "L2 + task_id + layer_id". Simultaneously, the first matrix corresponding to the second type of expert network's inference for different tasks is used in conjunction with the second caching strategy. Second matrix It is stored in video memory as the value "Value".

[0076] When data inference operations are required using the expert foundation network of a second-type expert network, the task identifier of the task to be processed and the linear layer identifier corresponding to the linear layer in the second-type expert network can be determined first. Then, based on the task identifier and the linear layer identifier, the first matrix and the second matrix corresponding to the expert network model can be obtained using the second caching strategy. Specifically, the task identifier "task_id", the linear layer identifier "Layer-id", and the second caching strategy identifier "L2" corresponding to the second-type expert network are used as keys for data reading operations in the video memory. This allows for the stable acquisition of the first matrix required by the expert network model for inference operations on the aforementioned task. Second matrix .

[0077] The first matrix in the LoRA low-rank adaptation matrix is ​​obtained. Second matrix Then, the basic weight parameters and the first matrix of the expert basic network can be analyzed. Second matrix The analysis and processing are performed to determine the model weight parameters corresponding to the expert network model. The basic weight parameters of the expert base network can be pre-stored in the video memory using a first caching strategy. Therefore, before determining the model weight parameters corresponding to the expert network model based on the basic weight parameters of the expert base network, the first matrix, and the second matrix, the method in this embodiment may further include: determining the basic weight parameters corresponding to the expert base network based on the task identifier using the first caching strategy.

[0078] In the case of multiple expert network models, the expert base network in the multiple expert network models is the component that needs to be passed through when all words are inferred. Therefore, the expert base network corresponding to multiple expert network models can also be regarded as the high-frequency activated model part. Therefore, in order to improve the efficiency of data inference operations to a certain extent, not only can the model weight parameters corresponding to the first type of expert network be stored in the first caching strategy, but the base weight parameters corresponding to the expert base network can also be stored in the first caching strategy.

[0079] When it is necessary to determine the model weight parameters corresponding to the expert network model of the second type of expert network, since the model weight parameters are determined by real-time fusion calculation based on the basic weight parameters corresponding to the expert base network and the low-rank adaptation matrix, in order to accurately determine the model weight parameters, the basic weight parameters corresponding to the expert base network can be determined based on the task identifier and linear layer identifier corresponding to the second type of expert network using the first caching strategy. Then, the first matrix, the second matrix, and the basic weight parameters can be analyzed and processed to stably determine the model weight parameters corresponding to the expert network model.

[0080] In some instances, the model weight parameters corresponding to the expert network model can be determined by analyzing and processing the first and second matrices using a weight parameter calculation model. In this case, determining the model weight parameters corresponding to the expert network model based on the first and second matrices can include: determining a pre-trained weight parameter calculation model; inputting the basic weight parameters of the expert network, the first and second matrices into the weight parameter calculation model for inference operations, and obtaining the model weight parameters corresponding to the expert network model output by the weight parameter calculation model. This effectively ensures the accuracy and reliability of determining the model weight parameters corresponding to each expert network model.

[0081] Alternatively, the model weight parameters corresponding to the expert network model can also be determined by calculating the first matrix and the second matrix. For example, in the first matrix... The second matrix is Basic weight parameters At that time, we can first base it on the first matrix. Second matrix Determine the low-rank adaptation incremental parameters , Then, incremental parameters can be adapted based on low-rank parameters. and basic weight parameters To determine the model weight parameters corresponding to the expert network model. The model's weight parameters It can be determined using the following formula: ;or, ,in, These are preset weighting coefficients.

[0082] Step S403: Determine the weight information based on the model weight parameters corresponding to the inference operations of multiple expert network models for different tasks to be processed.

[0083] After obtaining the model weight parameters corresponding to each of the multiple expert network models, the model weight parameters corresponding to each of the multiple expert network models can be analyzed and processed to determine the weight information corresponding to the linear layers in the multiple expert network models.

[0084] In some instances, weight information can be determined by fusing the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in multiple expert network models. In this case, determining the weight information based on the model weight parameters corresponding to the inference operations of multiple expert network models on different tasks can include: determining the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in multiple expert network models based on the model weight parameters corresponding to the inference operations of multiple expert network models on different tasks; and then concatenating the weight fusion parameters of multiple linear layers with the same hierarchical structure to obtain the weight information corresponding to the linear layers in multiple expert network models. This effectively ensures the accuracy and reliability of determining the weight information corresponding to the linear layers in multiple expert network models.

[0085] For example, multiple expert network models include expert network model 1, expert network model 2, and expert network model 3. When each expert network model includes two linear layers, the weight fusion parameters belonging to the first linear layer and the weight fusion parameters belonging to the second linear layer can be obtained separately. For example, the weight fusion parameters belonging to the first linear layer include: weight fusion parameter W_1_1 corresponding to the first linear layer in expert network model 1, weight fusion parameter W_2_1 corresponding to the first linear layer in expert network model 2, and weight fusion parameter W_3_1 corresponding to the first linear layer in expert network model 3; the weight fusion parameters belonging to the second linear layer include: weight fusion parameter W_1_2 corresponding to the second linear layer in expert network model 1, weight fusion parameter W_2_2 corresponding to the second linear layer in expert network model 2, and weight fusion parameter W_3_2 corresponding to the second linear layer in expert network model 3.

[0086] For the first linear layer, the weight fusion parameters W_1_1, W_2_1, and W_3_1 mentioned above can be concatenated to obtain the weight parameters W_1 = W_1_1||W_2_1||W_3_1 corresponding to the first linear layer in the multiple expert network model. The "||" symbol is the concatenation operator. Similarly, for the second linear layer, the weight fusion parameters W_1_2, W_2_2, and W_3_2 mentioned above can be concatenated to obtain the weight parameters W_2 = W_1_2||W_2_2||W_3_2 corresponding to the second linear layer in the multiple expert network model. The "||" symbol is the concatenation operator.

[0087] In other instances, the weight information corresponding to the linear layers in multiple expert network models can be implemented as a stacked matrix. In this case, determining the weight information based on the model weight parameters of each of the multiple expert network models can include: determining the weight fusion parameters of each linear layer in the multiple expert network models based on the model weight parameters of each of the multiple expert network models; determining the weight fusion parameters of each linear layer in the multiple expert network models based on the model weight parameters of each of the multiple expert network models; and grouping and aggregating the weight fusion parameters of the linear layers with the same hierarchical structure based on the multiple tasks to be processed corresponding to the word set to obtain the stacked weight information.

[0088] After obtaining the model weight parameters corresponding to each of the multiple expert network models, these parameters can be analyzed and processed to determine the weight fusion parameters for each linear layer in the multiple expert network models. For example, the weight fusion parameters corresponding to the first linear layer in the multiple expert network models include: W_base_L_1_1+dW_L_1_1_task1, W_base_L_1_1+dW_L_1_1_taskN, W_L_2_1_task1, W_base_L_N_1+dW_L_N_1_task1, W_L_2_1_taskN, and W_base_L_N_1+dW_L_N_1_taskN, where dW is used to identify the incremental model weights. It is determined by calculation using the low-rank adaptation matrix corresponding to the task to be processed.

[0089] Since the aforementioned multiple weight fusion parameters are used for inference operations on different tasks, meaning different tasks can correspond to different weight parameters of the expert network model, to ensure processing operations on multiple different tasks, after obtaining the multiple weight fusion parameters corresponding to the first linear layer in the aforementioned multiple expert network models, the weight fusion parameters corresponding to the linear layers with the same hierarchical structure can be grouped based on the multiple tasks corresponding to the word set. For example, the weight fusion parameters used for processing the same task can be grouped as one group. The parameter group refers to the parameter group corresponding to different tasks to be processed. Parameter group 1 may include: (W_base_L_1_1+dW_L_1_1_task1)||(W_L_2_1_task1)||(W_base_L_N_1+dW_L_N_1_task1); parameter group 2 may include: (W_L_2_1_taskN)||(W_base_L_N_1+dW_L_N_1_taskN)||(W_base_L_1_1+dW_L_1_1_taskN), and so on.

[0090] After obtaining the parameter sets corresponding to different tasks to be processed, multiple parameter sets can be aggregated to obtain stacked weight information. This weight information can be the weight matrix obtained by stacking multiple parameter sets, thus ensuring the accuracy and reliability of determining the weight information.

[0091] In this embodiment, by determining the task identifiers of multiple tasks to be processed corresponding to the word set, multiple expert network models are determined based on the multiple task identifiers, and then the model weight parameters corresponding to each of the multiple expert network models are determined. Based on the model weight parameters corresponding to each of the multiple expert network models, the weight information is determined. This effectively ensures the accuracy and reliability of the weight information determination, and then facilitates the processing operation of the tasks to be processed based on the determined weight information, thereby ensuring the quality and efficiency of the task processing operation.

[0092] Figure 5 A flowchart illustrating the division of weight information and lexical sets into multiple data blocks is provided as an exemplary embodiment of this application; based on any of the above embodiments, refer to the appendix. Figure 5 As shown, for multiple data blocks, they can be obtained not only by dividing the weight information and word set based on a pre-trained data partitioning module, but also by grouping multiple words in the word set. In this case, dividing the weight information and word set into multiple data blocks can include: Step S501: Group the multiple lexical units in the lexical set to obtain the lexical group corresponding to the task to be processed.

[0093] Since the lexical set includes multiple lexical units from multiple different tasks to be processed, in order to achieve efficient and parallel processing of multiple different tasks to be processed, after obtaining the lexical set, the multiple lexical units in the lexical set can be grouped to obtain lexical groups corresponding to each task to be processed.

[0094] In some instances, the word groups corresponding to each task to be processed can be obtained by grouping the word set using a pre-trained grouping model. In this case, grouping multiple words in the word set to obtain word groups corresponding to the tasks to be processed may include: determining the pre-trained grouping model; inputting the word set into the grouping model for grouping operations to obtain word groups corresponding to each task to be processed. Each word group includes task description words corresponding to each task to be processed, which effectively ensures the accuracy and reliability of obtaining word groups corresponding to each task to be processed.

[0095] Step S502: Aggregate the word groups corresponding to multiple tasks to be processed to determine word stacking.

[0096] Since different tasks to be processed can correspond to different word groups, in order to perform efficient parallel processing operations on multiple different tasks to be processed at the same time, we can first obtain the word groups corresponding to the tasks to be processed through the routing operation of multiple expert network models belonging to the same hybrid expert layer (MOE) in the hybrid expert network model. Then, we can perform aggregation operations on the word groups corresponding to multiple tasks to be processed. Specifically, we can perform aggregation and concatenation operations on multiple word groups according to the receiving order of multiple tasks to be processed, so as to obtain word stacks.

[0097] Step S503: Divide the weight information and word stacking into multiple data blocks.

[0098] Since the lexical stack can represent multiple lexical elements that represent the task content of multiple tasks to be processed, and the weight information includes multiple layer weight information corresponding to the linear layers in multiple expert network models, in order to use multiple expert network models to perform efficient and parallel processing operations on multiple tasks to be processed, the weight information and lexical stack can be divided into multiple data blocks. There is a corresponding calculation relationship between some weight parameters and some lexical elements included in the divided data blocks.

[0099] In some instances, the partitioning operation can be implemented using partitioning granularity parameters. In this case, dividing the weight information and word stacks into multiple data blocks can include: determining the partitioning granularity parameters of the data blocks, wherein the partitioning granularity parameters can include at least one of the following: the size of the data block, the computational parallelism of the data block, and the aforementioned data block size and computational parallelism are positively correlated with the utilization rate of computing resources; then, the word stacks and weight information can be divided into multiple data blocks based on the partitioning granularity parameters, thus effectively ensuring the accuracy and reliability of determining multiple data blocks.

[0100] In other instances, the partitioning operation can be achieved not only through partitioning granularity parameters, but also through a pre-trained partitioning network model. In this case, partitioning the weight information and word stacks into multiple data blocks can include: determining the pre-trained partitioning network model; inputting the weight information and word stacks into the partitioning network model for partitioning, thereby obtaining multiple data blocks output by the partitioning network model. This also ensures the flexibility and reliability of determining multiple data blocks.

[0101] In this embodiment, multiple lexical units in the lexical set are grouped to obtain lexical groups corresponding to the tasks to be processed. Then, the lexical groups corresponding to the multiple tasks to be processed are aggregated to determine the lexical stack. This divides the weight information and lexical stack into multiple data blocks, which effectively ensures the stability and reliability of determining multiple data blocks. This facilitates parallel processing of multiple data blocks, thereby realizing efficient and parallel task processing of multiple tasks to be processed, and further improving the quality and efficiency of task processing.

[0102] Figure 6 A flowchart illustrating the parallel computation of multiple data blocks to determine the inference result corresponding to the linear layer, provided as an exemplary embodiment of this application; based on any of the above embodiments, refer to the appendix. Figure 6 As shown, parallel processing of multiple data blocks can be achieved not only through a graphics processor communicating with the task processing device, but also through multiple computing units. In this case, parallel computation of multiple data blocks to determine the process inference result corresponding to the linear layer can include: Step S601: Use multiple computing units to perform parallel computing on multiple data blocks to obtain computing results corresponding to the multiple data blocks.

[0103] After acquiring multiple data blocks, multiple computing units can be used to perform parallel computing operations on the multiple data blocks, thereby obtaining the computing results corresponding to the multiple data blocks. The multiple computing units can be deployed in the task processing device, or the multiple computing units can communicate with the task processing device.

[0104] For example, when the task processing device is implemented as a GPU, the GPU can deploy 20 computing units. In this case, multiple data blocks can be allocated to the 20 computing units for parallel computing operations on the data blocks, thereby obtaining the computing results corresponding to each data block. When the number of multiple data blocks is greater than the number of computing units, one computing unit can process one or more data blocks, and the obtained computing results correspond to each data block. That is, when performing inference computing on one data block, one computing result can be obtained.

[0105] Step S602: Based on the calculation results corresponding to multiple data blocks, determine the process inference result corresponding to the linear layer.

[0106] Since any one of the multiple data blocks can often represent part of the information of the task to be processed, the calculation result corresponding to any one data block can only represent part of the result of the reasoning operation on the task to be processed. Therefore, after obtaining the calculation results corresponding to multiple data blocks, the calculation results corresponding to multiple data blocks can be analyzed and processed to stably determine the process reasoning result corresponding to the linear layer.

[0107] In some instances, the procedural inference result corresponding to the linear layer can be obtained by fusing the calculation results corresponding to multiple data blocks. In this case, determining the procedural inference result corresponding to the linear layer based on the calculation results corresponding to multiple data blocks may include: determining the original positions corresponding to multiple words in the word set; and summarizing the calculation results corresponding to each word in the data block based on the original positions corresponding to multiple words to obtain the procedural inference result corresponding to the linear layer.

[0108] After obtaining the lexical set, the original position corresponding to each lexical in the lexical set can be recorded and obtained. Since the calculation result corresponding to the data block includes the lexical inference result corresponding to each lexical in the data block, in order to accurately determine the process inference result corresponding to the linear layer, after obtaining the original positions corresponding to multiple lexicals, the calculation result corresponding to each lexical in the data block can be summarized based on the original positions corresponding to multiple lexicals. Specifically, the calculation result corresponding to each lexical in the data block can be reordered and summarized according to the original positions corresponding to multiple lexicals, so as to stably obtain the process inference result corresponding to the linear layer. This effectively ensures the accuracy and reliability of determining the process inference result corresponding to each linear layer.

[0109] In other instances, the process inference result corresponding to the linear layer can be obtained not only by fusing the calculation results corresponding to multiple data blocks, but also based on the word set and a pre-trained aggregation model. In this case, determining the process inference result corresponding to the linear layer based on the calculation results corresponding to multiple data blocks can include: determining the pre-trained aggregation model; inputting the word set and the calculation results corresponding to multiple data blocks into the aggregation model for aggregation operation, outputting and determining the process inference result corresponding to the linear layer. This also ensures the flexibility and reliability of determining the process inference result corresponding to the linear layer.

[0110] In this embodiment, multiple computing units are used to perform parallel computing on multiple data blocks to obtain computing results corresponding to the multiple data blocks. Then, based on the computing results corresponding to the multiple data blocks, the process inference result corresponding to the linear layer is determined. This effectively realizes the efficient parallel processing operation of multiple tasks to be processed by parallel computing operations based on multiple data blocks, and further improves the quality and efficiency of task processing operations.

[0111] In practical applications, this embodiment provides a hybrid inference method for multiple tasks based on the MoE architecture [a unified high-performance inference architecture with single instance, multiple tasks, and multiple LoRA low-rank adaptation matrices]. This method balances memory usage and computational resource overhead by using a multi-level weighted caching strategy that is aware of expert activation frequency and computational resource costs. In addition, through a weight management mechanism that combines static caching and dynamic caching, efficient parallel inference of multiple tasks within a single batch is achieved, thereby improving computational efficiency. Furthermore, this scheme realizes a multi-task extended inference architecture that decouples the training phase from the inference phase, thereby enabling the addition of inference capabilities for new tasks at a low cost and providing extremely low marginal costs for subsequent task expansion. This solves the problem in multi-task hybrid inference scenarios where multiple tasks are included in a single batch for inference, which not only increases inference costs but also addresses the problems of computational fragmentation, low GPU utilization, and slow inference speed caused by the need to load multiple LoRA low-rank adaptation matrices (or low-rank adapters) simultaneously.

[0112] Specifically, this method may include a model parameter caching phase and a model application phase, wherein, see Appendix Figure 7 As shown, the model parameter caching stage may include the following steps: Step 11: Determine the model type of each trained expert network model.

[0113] The model weight parameters for any Expert network model are determined by fusing the base weight parameters of the base expert network and the LoRA low-rank adaptation matrix for the corresponding task. Different tasks correspond to different LoRA low-rank adaptation matrices. To accurately determine the model type of the trained Expert network models, the activation frequency distribution of each Expert network model can be determined first. This activation frequency distribution can be determined by conducting offline experiments on a given dataset. This distribution indicates the proportion or number of times each Expert network model is selected by the routing mechanism on the given dataset.

[0114] After determining the activation frequency distributions of multiple expert network models (Experts), the model type of each expert network model is then determined based on the activation frequency distributions of each of the multiple expert base networks. The model type can include high-frequency expert network models ("Hot Experts") and low-frequency expert network models. Specifically, the multiple expert base networks can be sorted from high to low based on their activation frequency distributions to obtain network model ranking information. Then, the expert network models in the top-N of the ranking information are identified as high-frequency expert network models, and those below the top-N are identified as low-frequency expert network models. This effectively achieves stable and reliable determination of the model type of multiple trained expert network models.

[0115] Furthermore, the specific value corresponding to Top-N can be determined based on actual computing resources. When computing resources are sufficient, Top-N can be set to a higher value; when computing resources are insufficient, Top-N can be set to a lower value. In some scenarios, Top-N can be defined as the top 10% of the network model ranking information. In this case, the expert network models in the top 10% of the network model ranking information can be identified as high-frequency expert network models. These high-frequency expert network models typically handle 20%-30% of the word load in the entire word set.

[0116] Step 12: Based on the model type of each expert network model, perform differentiated caching operations on the model weight parameters of multiple expert network models.

[0117] When multiple task inference services are triggered or started, the full weight parameters of the expert base networks in multiple expert network models, as well as the incremental weight parameters corresponding to multiple LoRA low-rank adaptation matrices, can be loaded first. These full weight parameters and incremental weight parameters can be determined through model training. Then, the model weight parameters of multiple expert network models can be stored differentially based on their model types. In some instances, the full model parameters for each task corresponding to frequently activated, high-frequency expert network models can be pre-calculated and cached in GPU memory, while only the incremental weight parameters corresponding to the corresponding LoRA low-rank adaptation matrices are cached for other expert network models. Specifically, the differential caching operations include: For scenarios involving multiple tasks and multiple low-rank adaptation matrices, to balance memory usage and computational resource overhead, multi-level caching operations can be performed on model weight parameters based on the model type and computational resource cost corresponding to each expert network model. Specifically, a first-level caching strategy (corresponding to the first caching strategy in the above embodiment) and a second-level caching strategy (corresponding to the second caching strategy in the above embodiment) are pre-set. The first-level caching strategy is used to identify and pre-calculate and store the full model weight parameters for each task corresponding to the shared model components and the high-frequency expert network model. The shared model components can refer to the shared components that all tokens must pass through when using the expert network model for task processing operations, which may include attention layers, shared expert networks, etc. Specifically, when storing the full model weights in GPU memory using the first-level caching strategy, the task identifier "task_id", the linear layer identifier "Layer-id", and the first-level caching strategy identifier "L1" corresponding to the high-frequency expert network model are used as keys, and the tensor of the full model weights is used as the value to reside in GPU memory.

[0118] The aforementioned second-level caching strategy is used to identify and store partial weight parameters corresponding to low-frequency expert network models (which can be other expert network models besides those covered in the first-level cache). These partial weight parameters can be LoRA low-rank adaptation matrices corresponding to the low-frequency expert network models when processing different tasks. These LoRA low-rank adaptation matrices can include low-rank matrices. and low-rank matrix When the aforementioned low-frequency expert network model performs inference operations on multiple tasks, the total data volume of all LoRA incremental weight parameters is typically less than 1% of the total weights in the complete model. Specifically, when storing the incremental weight parameters of the LoRA low-rank adaptation matrix in GPU memory using a two-level caching strategy, the task identifier "task_id" corresponding to the low-frequency expert network model, the linear layer identifier "Layer-id" corresponding to the linear layer in the low-frequency expert network model, and the two-level caching strategy identifier "L2" corresponding to the low-frequency expert network are used as keys, and the parameter tensor of the LoRA low-rank adaptation matrix is ​​used as the value for storage.

[0119] When a low-frequency expert network model is needed for task processing, the low-rank matrix can be read in real time based on the task identifier "task_id", the linear layer identifier "Layer-id", and the second-level cache strategy identifier "L2" corresponding to the low-frequency expert network model. and low-rank matrix Then it can be dynamically calculated. This is then superimposed on the base weight parameters corresponding to the expert base network stored in video memory using a level-one caching strategy. This allows us to obtain the full model weight parameters corresponding to the low-frequency expert network model when performing task inference operations.

[0120] By performing differentiated caching operations on the model weight parameters of multiple expert network models, the following effects can be achieved: 1) The first-level caching strategy eliminates the LoRA fusion computation overhead on high-frequency paths, achieving "zero runtime latency"; the second-level caching strategy ensures the controllability of GPU memory usage; 2) Safe and controllable: When high-frequency expert network models perform task inference operations, the proportion of the full model weight parameters is usually low (<3%). Combined with configurable Top-N thresholds (such as Top5, Top10, or Top15%), even in multiple concurrent task scenarios, the total cache increment can be controlled within 15% of the GPU memory occupied by the complete model.

[0121] In addition, see attached document Figures 7-8 As shown, the application phase of the model may include the following steps: Step 21: Obtain a mixed batch of task requests consisting of multiple different tasks to be processed.

[0122] The mixed task request batch can include multiple different request prompts corresponding to pending tasks. For example, when the batch size corresponding to the batch is 5, it means that the batch can include 5 request prompts. The 5 request prompts can correspond to 5 different pending tasks, or the 5 request prompts can correspond to 3 different pending tasks.

[0123] Step 22: Perform task parsing on the mixed task request batch to obtain a token set, which may include multiple tokens from multiple different tasks to be processed.

[0124] After obtaining the batch of mixed task requests, word parsing operations can be performed on all requests in the batch to obtain a lexicon set corresponding to the batch of mixed task requests. This lexicon set can include multiple lexicon tokens from multiple different tasks to be processed.

[0125] Step 23: Determine the task identifier (task_id) corresponding to each word in the word set.

[0126] The tokens in the lexical set can come from requests corresponding to different pending tasks. Since different pending tasks often require expert network models with different LoRA low-rank adaptation matrices for processing, in order to accurately determine the inference operation for multiple different pending tasks, after obtaining the lexical set, we can first determine the task identifier corresponding to each lexical in the lexical set. The task identifier corresponding to each lexical can be determined based on the task category to which the request prompt belongs.

[0127] For example, when the batch size of the mixed task request batch is 5, three tasks to be processed are passed in. These three tasks need to be processed using the following three network models: "base model", "LoRA low-rank adaptation matrix 1", and "LoRA low-rank adaptation matrix 2". The task identifiers (task_ids) corresponding to the mixed task request matching can be [0,1,1,2,2]. That is, the task identifier (task_id) of the first request prompt is "0", the task identifiers (task_id) of the second and third request prompts are "1", and the task identifiers (task_id) of the fourth and fifth request prompts are "2". Therefore, it can be seen that the task identifiers (task_id) corresponding to all words in the first request prompt are "0", the task identifiers (task_id) corresponding to multiple words in the second and third request prompts are "1", and the task identifiers (task_id) corresponding to multiple words in the fourth and fifth request prompts are "2".

[0128] Step 24: Based on the task identifier (task_id) corresponding to each word, perform rearrangement and aggregation operations on multiple words in the word set to obtain a word stack, and store the original order index map of each word in the word set.

[0129] Multiple expert network models belonging to the same hybrid expert layer (MOE) in the hybrid expert network model are routed through the routing module to obtain multiple tokens in the token set. Then, all tokens obtained from the hybrid task request batch are grouped and aggregated according to their task identifier (task_id). Specifically, multiple tokens belonging to the same task identifier are first divided into a token block, that is, multiple tokens of the same task are arranged continuously in the memory to form a compact calculation block. The multiple token blocks corresponding to multiple tasks to be processed are aggregated to obtain the token stack (Token_stack). At the same time, an original order index (Original_Index_Map) is maintained to record the position of each token in the original batch for subsequent restoration operations.

[0130] Step 25: Determine the full model weight parameters corresponding to multiple expert network models based on the model type corresponding to each expert network.

[0131] To accurately and flexibly process multiple different tasks, multiple LoRA low-rank adaptation matrices can be determined based on the task identifier (task_id) corresponding to each lexical unit, used to process the multiple tasks separately. Since the full model weight parameters of the expert network model are determined by fusing the LoRA low-rank adaptation matrices and the basic weight parameters of the expert base network, different expert base networks can correspond to different model types, and different model types of expert base networks can have different methods for determining the full model weight parameters of the expert network model. Therefore, in order to quickly determine the full model weight parameters of the expert network model, we can first determine the model type of the expert base network. When the model type of the expert base network is a high-frequency expert network, since the full model weight parameters corresponding to the high-frequency expert network model are pre-cached in the video memory using a first-level caching strategy, the full model weight parameters corresponding to different tasks to be processed for the high-frequency expert network model can be directly read from the video memory. Specifically, the full model weight parameters can be read from the video memory based on the task identifier "task_id" of the task to be processed and the linear layer identifier "Layer-id" corresponding to the linear layer in the expert base network.

[0132] When the expert network model is a low-frequency expert network, the full model weight parameters for multiple different tasks to be processed corresponding to the low-frequency expert network model are not pre-calculated and stored in the GPU memory. Therefore, it is necessary to temporarily or in real-time calculate the full model weight parameters of the current expert network model. Thus, multiple low-rank adaptation matrices can be determined based on the task identifier (task_id) corresponding to each word, and then fused based on these low-rank adaptation matrices and the basic weight parameters of the low-frequency expert network to obtain the full model weight parameters of the current expert network model. For example, when the expert network model is a low-frequency expert network model, the low-rank matrix corresponding to the low-frequency expert network model can be obtained by reading it from the video memory using the task identifier "task_id" of the task to be processed, the linear layer identifier "Layer-id" corresponding to the linear layer in the low-frequency expert network model, and the L2 cache strategy identifier "L2" corresponding to the low-frequency expert network model. and low-rank matrix Then, using the task identifier "task_id" of the task to be processed and the linear layer identifier "Layer-id" corresponding to the linear layer in the expert foundation network, the basic weight parameters corresponding to the expert foundation network are read from the video memory. It can start the computing unit to perform calculations based on the low-rank matrix. low-rank matrix and basic weight parameters Calculate the temporary fusion weights, specifically, the temporary fusion weights. It can be determined using the following formula: ,in, This temporary fusion weight is based on preset weight parameters. This refers to the full model weight parameters corresponding to the low-frequency expert network model.

[0133] Step 26: Based on the full model weight parameters and word stacking corresponding to multiple expert network models, use multiple expert network models to perform inference operations on mixed task request batches corresponding to multiple different tasks to be processed, so as to determine the task inference results corresponding to each task to be processed.

[0134] When using multiple expert network models to perform inference operations on mixed task request batches corresponding to multiple different tasks to be processed, in order to achieve parallel processing operations on multiple different tasks to be processed, the full model weight parameters and word stacks can be divided to obtain multiple data blocks. Any data block can include a portion of the full model weight parameters and a portion of the word stacks corresponding to the calculation operations required.

[0135] Specifically, in order to achieve parallel accelerated computation for multiple tasks, the linear weight fusion parameters (which are part of the total model weight parameters) corresponding to each linear layer in multiple expert network models can be flattened according to the hierarchical structure of the linear layers. That is, the linear weight fusion parameters corresponding to all first linear layers in all expert network models can be flattened, and the linear weight fusion parameters corresponding to all second linear layers can be flattened, so as to obtain the weight information corresponding to each linear layer after flattening. Next, the weight information corresponding to each linear layer is grouped according to the task identifier (task_ids) corresponding to the weight fusion parameter and then aggregated. Different tasks to be processed can correspond to different grouping results. Then, multiple grouping results are stacked and aggregated according to multiple different tasks to be processed, so as to obtain the weight parameter stack W_stack corresponding to each linear layer. The weight parameter stack W_stack can be a matrix structure that conforms to [Num_Tasks,Input_Dim,Output_Dim], where Num_Tasks is the total number of tasks to be processed, Input_Dim is the input tensor dimension of the expert network model, and Output_Dim is the output tensor dimension of the expert network model.

[0136] After obtaining the weight parameter stack W_stack and word stack Token_stack corresponding to each linear layer in multiple expert network models, the weight parameter stack W_stack and word stack Token_stack can be divided into multiple data blocks. Specifically, the weight parameter stack W_stack and word stack Token_stack can be divided into multiple small data blocks according to the corresponding parts during computation. Then, multiple blocks are computed in parallel on multiple computing units (StreamingMultiprocessor, or GPUSM for short). For example, when multiple blocks include Task1Block1, Task1Block2, ..., TaskNBlockM, Task1Block1 can be assigned to one GPUSM computing unit for processing, Task1Block2 can be assigned to another GPUSM computing unit for processing, and so on, with TaskNBlockM assigned to yet another GPUSM computing unit for processing, thereby obtaining the block computation results corresponding to each of the multiple data blocks. The size of the data block and the computational parallelism of the data block can be configured according to the actual computing power of the computing unit. Specifically, the utilization rate of the current GPU SM can be maximized, and all computing units can continuously perform computational operations to maximize the GPU throughput and improve inference speed.

[0137] For the data block, it includes a partial weight parameter stack W_stack and a token stack Token_stack. The weight parameter stack W_stack is obtained by grouping and aggregating the linear weight fusion parameters based on the task identifiers corresponding to multiple tasks to be processed and the linear layers (Linear) in multiple expert network models. The token stack Token_stack is obtained by grouping and aggregating according to task type. This achieves grouping and aggregating tokens and Linear layer parameters by task type, thus directly realizing parallel computation on the "Linear layer × task" dimension. Specifically, since an expert network model "Expert" actually corresponds to a feedforward neural network (FFN) structure, it is essentially composed of several Linear layers serially (e.g., up-proj / down-proj, etc.). LoRA training only applies to the parameters of the Linear layer, i.e., it performs low-rank incremental modeling on W. Therefore, in the inference implementation, the abstract dimension of "Expert" can be removed, and parallel computation can be organized directly on the "Linear layer × task" dimension. This effectively overcomes the problem of "not only easily causing severe divergence of computational kernel branches, but also easily causing low GPU utilization" that exists in related technologies through the two-level routing method ("Token" -> expert network model "Expert" and expert network model "Expert" -> task low-rank adaptation matrix "TaskLoRA"). By using the idea of ​​"dimensionality reduction", the two-dimensional routing is transformed into a one-dimensional task-level parallel computation operation, thus making full use of the GPU's computing resources.

[0138] In addition, when using multiple computing units to perform parallel computing operations on multiple data blocks, since a separate computing unit is not started for each task to be processed, but the parallel computing operations of multiple tasks are implemented in a single computing kernel, the quality and efficiency of parallel processing of multiple tasks to be processed are effectively improved. After obtaining the block computation results of multiple data blocks, the block computation results of multiple data blocks can be reordered according to the original index_Index_Map corresponding to each word. That is, according to the Original_Index_Map, the token results in the block computation results of each data block are written back to the corresponding positions of the original batch, ensuring that the output order is consistent with the input order. This allows us to obtain the process inference results corresponding to the current linear layer in multiple expert network models. Then, the process inference results can be used as new word stacks and input into the next linear layer for inference operations. This process is repeated to determine the task inference results corresponding to each task to be processed. For example, we can obtain task inference result 1 output by expert network model Expert_1, task inference result 2 output by expert network model Expert_2, ..., task inference result N output by expert network model Expert_N.

[0139] The technical solution provided in this application embodiment effectively achieves a balance between resource efficiency, inference speed, and scalability through architectural innovation. Specifically: 1) In scenarios where inference operations are performed on multiple tasks within a single task batch, a multi-level caching strategy based on expert activation frequency awareness and computational resource cost is used to cache the expert network model and the model weight parameters (accounting for >90%) corresponding to the expert base network. A very small number of LoRA parameters and high-frequency fusion weights are stored in multiple copies. Specifically, taking advantage of the long-tail characteristics of the MoE activation distribution, a multi-level caching strategy of "common module, high-frequency full fusion + low-frequency dynamic computation" is proposed. The number of model weight parameters corresponding to the high-frequency expert network model can be dynamically adjusted according to the actual computational resources, thereby achieving high-efficiency computational performance under a memory-safe budget and completely eliminating runtime weight I / O and repeated loading latency.

[0140] 2) Multi-level routing dimensionality reduction and dynamic weighted task-level micro-batch parallel processing: By removing the abstract dimension of "expert network" from the "2D routing to 1D parallel" scheduling algorithm, parallel computing is organized directly on the "Linear layer × task" dimension. Then, by recombining and weighting tokens according to tasks at runtime, multiple data blocks are obtained. This integrates the originally fragmented multi-task inference computing into a single kernel-level efficient parallel computing of multiple data blocks. That is, different tasks can share a set of GPU computing units, maximizing the utilization and throughput of computing units. For a task batch with multiple different tasks, the throughput can be increased by at least 2 times. The more task categories in the batch, the greater the throughput increase. At the same time, it also solves the GPU computing fragmentation problem caused by heterogeneous requests from multiple tasks, thereby significantly improving device utilization.

[0141] 3) By keeping the expert base network fixed, each task only performs incremental learning through the LoRA low-rank adaptation matrix. New tasks only need to provide the trained LoRA parameters and update the configuration, without retraining the expert base network or routing protocol. In addition, the LoRA parameters corresponding to each task are physically isolated, ensuring 100% reproduction of the single-task fine-tuning effect, no accuracy loss, and no interference between tasks. This effectively realizes a unified inference architecture with single instance, multiple tasks, and multiple LoRAs, providing extremely low marginal cost for subsequent task expansion and further improving the practicality of the method.

[0142] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0143] Figure 9 A schematic diagram of a task processing apparatus provided for an exemplary embodiment of this application; see attached diagram. Figure 9 As shown, this embodiment provides a task processing device for performing the above-described tasks. Figure 2 The task processing method shown herein, specifically, the task processing device may include: The first acquisition module 11 is used to acquire a set of lexical units to be processed, which includes multiple lexical units from multiple tasks to be processed. The first grouping module 12 is used to group multiple words in the word set and determine the word stack corresponding to multiple tasks to be processed; The first determining module 13 is used to determine the weight information corresponding to the linear layers in multiple expert network models based on the multiple tasks to be processed corresponding to the word set. The multiple expert network models are used to process the multiple tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters of the expert basic network and the low-rank adaptation matrix corresponding to the task to be processed. The first grouping module 12 is also used to divide the weight information and word set into multiple data blocks, and the data blocks include some weight fusion parameters that satisfy the calculation relationship and some word elements. The first processing module 14 is used to process multiple data blocks in parallel, determine the process inference result corresponding to the linear layer, and input the process inference result to the next linear layer for inference operation, so as to determine the multiple task inference results corresponding to multiple tasks to be processed.

[0144] In some instances, before determining the model weight parameters corresponding to the expert network model based on the basic weight parameters of the expert base network, the first matrix, and the second matrix, the first determining module 13 is further used to: determine the basic weight parameters corresponding to the expert base network based on the task identifier and the linear layer identifier using a first caching strategy.

[0145] The task processing device in this embodiment can also perform the above-described... Figures 1-8 The description of the embodiments shown is for reference only, and will not be elaborated upon here.

[0146] like Figure 10 As shown, this embodiment provides an electronic device for performing the above-described... Figure 2 The task processing method shown may include a memory 24 and a processor 25.

[0147] Memory 24 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0148] Processor 25, coupled to memory 24, executes a computer program in memory 24 to: acquire a set of terms to be processed, the set including multiple terms from multiple tasks to be processed; based on the multiple tasks to be processed corresponding to the set of terms, determine the weight information corresponding to the linear layers in multiple expert network models, the multiple expert network models being used to process the multiple tasks to be processed, the weight information including weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models, the weight fusion parameters being determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the tasks to be processed; divide the weight information and the set of terms into multiple data blocks, the data blocks including partial weight fusion parameters and partial terms satisfying the calculation relationship; process the multiple data blocks in parallel to determine the process inference result corresponding to the linear layer, the process inference result being used as input to the next linear layer for inference operation to determine the multiple task inference results corresponding to the multiple tasks to be processed.

[0149] Furthermore, such as Figure 10 As shown, the electronic device also includes other components such as a communication component 26, a display 27, a power supply component 28, and an audio component 29. Figure 10 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 10 The components shown. Additionally... Figure 10 The components within the center frame are optional, not mandatory, and their specific requirements depend on the product form of the work node. In this embodiment, the work node can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the work node in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 10 The components within the center frame; if the working node in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may not include... Figure 10 The component within the center frame.

[0150] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0151] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0152] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0153] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0154] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0155] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device can be implemented as a means to implement the corresponding functions in the above method embodiments.

[0156] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A task processing method, characterized in that, include: Obtain a set of lexical units to be processed, wherein the set of lexical units includes multiple lexical units from multiple tasks to be processed; Based on the multiple tasks to be processed corresponding to the lexical set, the weight information corresponding to the linear layers in multiple expert network models is determined. The multiple expert network models are used to process the multiple tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the task to be processed. The weight information and the word set are divided into multiple data blocks, and each data block includes some weight fusion parameters and some word elements that satisfy the calculation relationship. The multiple data blocks are processed in parallel to determine the process inference result corresponding to the linear layer. The process inference result is used to input the inference operation to the next linear layer to determine the multiple task inference results corresponding to multiple tasks to be processed.

2. The method according to claim 1, characterized in that, Based on the multiple tasks to be processed corresponding to the aforementioned lexical set, the weight information corresponding to the linear layers in multiple expert network models is determined, including: Determine the task identifiers of multiple tasks to be processed corresponding to the word set, with different tasks corresponding to different task identifiers; Based on the multiple task identifiers, determine the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed; The weight information is determined based on the model weight parameters corresponding to the inference operations performed by the multiple expert network models on different tasks to be processed.

3. The method according to claim 2, characterized in that, The weight information is determined based on the model weight parameters corresponding to the inference operations performed by the multiple expert network models on different tasks, including: Based on the model weight parameters corresponding to the reasoning operations of the multiple expert network models for different tasks to be processed, the weight fusion parameters of the linear layers in the multiple expert network models are determined. Based on the multiple tasks to be processed corresponding to the lexical set, the weight fusion parameters corresponding to the linear layers with the same hierarchical structure are grouped and aggregated to obtain weight information in a stacked form.

4. The method according to claim 2, characterized in that, Based on the multiple task identifiers, the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed are determined, including: Determine the model type to which the expert basic network belongs, wherein the model type includes at least one of the following: a first type of expert network and a second type of expert network, wherein the activation frequency of the first type of expert network is higher than the activation frequency of the second type of expert network; Based on the model type to which the multiple expert network base networks belong and the multiple task identifiers, the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed are determined.

5. The method according to claim 4, characterized in that, Determining the model type to which the expert foundation network belongs includes: Obtain the activation frequencies corresponding to the multiple expert basic networks; The multiple expert base networks are ranked based on the activation frequency to obtain model ranking information; Based on the model ranking information, the model type to which the expert basic network belongs is determined.

6. The method according to claim 5, characterized in that, Based on the model ranking information, the model type to which the expert base network belongs is determined, including: Obtain information used to determine the number of the first type of expert networks; Based on the model ranking information, the expert network models located at the position of the quantity information and those located before the quantity information are identified as the first type of expert network. Based on the model ranking information, the expert basic network located after the quantity information is identified as the second type of expert network.

7. The method according to claim 4, characterized in that, Based on the model types to which the multiple expert network base networks belong and the multiple task identifiers, the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed are determined, including: When the expert foundation network is a first type of expert network, the linear layer identifier corresponding to the linear layer in the expert foundation network is determined; Based on the task identifier and the linear layer identifier, the model weight parameters corresponding to the inference operation of the expert network model for different tasks to be processed are obtained using the first caching strategy. The first caching strategy is used to identify that the model weight parameters of the expert network model are pre-calculated and stored in the video memory.

8. The method according to claim 4, characterized in that, Based on the model types to which the multiple expert network base networks belong and the multiple task identifiers, the model weight parameters corresponding to the multiple expert network models when performing inference operations on different tasks to be processed are determined, including: When the expert foundation network is a second type of expert network, determine the linear layer identifier corresponding to the linear layer in the expert foundation network; Based on the task identifier and the linear layer identifier, the low-rank adaptation matrix corresponding to the expert base network is obtained using the second caching strategy. The low-rank adaptation matrix includes a first matrix and a second matrix. The second caching strategy is used to identify the temporary calculation of the model weight parameters of the expert network model. Based on the basic weight parameters of the expert network, the first matrix, and the second matrix, the model weight parameters corresponding to the expert network model when performing inference operations on different tasks are determined.

9. The method according to claim 8, characterized in that, Before determining the model weight parameters corresponding to the inference operations of the expert network model for different tasks based on the basic weight parameters of the expert network, the first matrix, and the second matrix, the method further includes: Based on the task identifier and the linear layer identifier, the basic weight parameters corresponding to the expert basic network are determined using the first caching strategy.

10. The method according to any one of claims 1-9, characterized in that, The weight information and the lexical set are divided into multiple data blocks, including: Grouping multiple lexical units in the lexical set to obtain lexical groups corresponding to the task to be processed; Aggregate the word groups corresponding to the task to be processed to determine the word stack; The weight information and the word stack are divided into multiple data blocks.

11. The method according to claim 10, characterized in that, The weight information and the lexical stack are divided into multiple data blocks, including: Determine the granularity parameters for data block partitioning; The word stack and weight information are divided into multiple data blocks based on the partitioning granularity parameter.

12. The method according to any one of claims 1-9, characterized in that, Parallel computation is performed on multiple data blocks to determine the process inference result corresponding to the linear layer, including: Multiple computing units are used to perform parallel computations on the multiple data blocks to obtain computation results corresponding to the multiple data blocks; Based on the calculation results corresponding to the multiple data blocks, the process inference result corresponding to the linear layer is determined.

13. The method according to claim 12, characterized in that, Based on the calculation results corresponding to the multiple data blocks, the process inference result corresponding to the linear layer is determined, including: Determine the original positions corresponding to multiple lexical units in the lexical set; Based on the original positions corresponding to the multiple lexical units, the calculation results corresponding to the lexical units in the data block are summarized to obtain the process inference results corresponding to the linear layer.

14. A task processing device, characterized in that, include: The first acquisition module is used to acquire a set of lexical units to be processed, wherein the set of lexical units includes multiple lexical units from multiple tasks to be processed; The first grouping module is used to group multiple words in the word set and determine the word stack corresponding to multiple tasks to be processed; The first determining module is used to determine the weight information corresponding to the linear layers in multiple expert network models based on the multiple tasks to be processed corresponding to the word set. The multiple expert network models are used to process the multiple tasks to be processed. The weight information includes the weight fusion parameters of multiple linear layers belonging to the same hierarchical structure in the multiple expert network models. The weight fusion parameters are determined by fusing the basic weight parameters of the expert base network and the low-rank adaptation matrix corresponding to the task to be processed. The first grouping module is further configured to divide the word stack and the weight stack into multiple data blocks, wherein the data blocks include some weight fusion parameters and some words that satisfy the calculation relationship; The first processing module is used to process the multiple data blocks in parallel, determine the process inference result corresponding to the linear layer, and the process inference result is used to input to the next linear layer for inference operation, so as to determine the multiple task inference results corresponding to multiple tasks to be processed.

15. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method of any one of claims 1-13.

16. A computer storage medium, characterized in that, Used to store a computer program that, when executed by a computer, implements the method of any one of claims 1-13.

17. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the steps of the method of any one of claims 1-13.

Images