Hybrid expert model deployment method and device, equipment, medium and product

Abstract

The invention discloses a deployment method and device of a hybrid expert model, equipment, a medium and a product, and relates to the technical field of artificial intelligence. The hybrid expert model is pre-deployed to monitor activation state parameters of an expert network in the hybrid expert model when a reasoning calculation task is executed based on the hybrid expert model; therefore, the activation demand of the expert network during actual execution of the reasoning calculation task is determined; according to the activation state parameter of the expert network and the resource state information of the computing node, computing a computing cost parameter when the computing node executes a reasoning computing task after the expert network is deployed to the computing node, thereby solving a deployment position of the expert network with the minimum computing cost; the hybrid expert model deployment scheme which meets the actual reasoning calculation requirement and is low in cost is obtained, the problem that the cost is high when a hybrid expert model is deployed in a heterogeneous calculation system in the related technology is solved, and the resource utilization rate of the heterogeneous calculation system is increased.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deployment method, device, equipment, medium and product of a hybrid expert model. Background Art

[0002] With the development of artificial intelligence (AI) technology, the scale of neural network models used continues to increase, leading to an increasing demand for computing power. A mixture of experts (MoE) is a machine learning method that divides an AI model into separate subnetworks (also called "expert networks"), each of which specializes in a subset of the input data to jointly perform a task. Due to the large scale of a mixture of expert models, a single computing device often cannot meet the requirements. Different expert networks of a mixture of expert models need to be deployed on multiple computing devices, with each computing device responsible for running part of the expert model. This leads to high equipment costs and low resource utilization. Summary of the Invention

[0003] The present invention provides a deployment method, apparatus, device, medium and product of a hybrid expert model to at least solve the problems of high equipment operating cost and low resource utilization in deploying hybrid expert models in heterogeneous computing systems in related technologies.

[0004] The present invention provides a deployment method of a hybrid expert model, comprising: Pre-deploy the hybrid expert model and monitor activation state parameters of the expert network in the hybrid expert model when performing reasoning computing tasks based on the hybrid expert model; Obtain resource status information of computing nodes of the heterogeneous computing system to be deployed; Calculating, based on the resource status information of the computing node and the activation status parameter of the expert network, a computing cost parameter when the computing node executes the inference computing task after the expert network is deployed to the computing node; With the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and the optimization goal of minimizing the computing cost parameter, optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

[0005] The present invention also provides a deployment device for a hybrid expert model, comprising: A task information collection module is used to pre-deploy the hybrid expert model and monitor the activation state parameters of the expert network in the hybrid expert model when performing reasoning and computing tasks based on the hybrid expert model; A system information collection module is used to obtain resource status information of computing nodes of the heterogeneous computing system to be deployed; A computing module is used to calculate the computing cost parameters when the computing node executes the inference computing task after the expert network is deployed to the computing node based on the resource status information of the computing node and the activation status parameters of the expert network; with the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and the minimization of the computing cost parameters as the optimization goal, to perform optimization calculations to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned methods for deploying a hybrid expert model when executing the computer program.

[0007] The present invention also provides a non-volatile storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned deployment methods of the hybrid expert model are implemented.

[0008] The present invention also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned deployment methods of the hybrid expert model when executed by a processor.

[0009] Through the present invention, the hybrid expert model is pre-deployed to monitor the activation state parameters of the expert network in the hybrid expert model when executing reasoning computing tasks based on the hybrid expert model, thereby determining the activation requirements of the expert network when actually executing reasoning computing tasks; according to the activation state parameters of the expert network and the resource status information of the computing nodes of the heterogeneous computing system to be deployed, the computing cost parameters when the computing nodes execute reasoning computing tasks after the expert network is deployed to the computing nodes are calculated, and the memory capacity of the computing nodes is constrained and the optimization calculation is performed with the goal of minimizing the computing cost, so as to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system, thereby obtaining a hybrid expert model deployment solution that meets the actual reasoning computing needs and has low cost, solves the problem of high cost when deploying hybrid expert models in heterogeneous computing systems in related technologies, and improves the resource utilization of heterogeneous computing systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A flowchart of a first method for deploying a hybrid expert model provided by an embodiment of the present invention; Figure 2 A schematic diagram of the deployment architecture of the first hybrid expert model provided by an embodiment of the present invention; Figure 3 A schematic diagram of the deployment architecture of the second hybrid expert model provided by an embodiment of the present invention; Figure 4 This is a flowchart of a second method for deploying a hybrid expert model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0014] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0015] Here, some key terms used in the embodiments of the present invention are explained.

[0016] Mixture of experts (MoE) is a machine learning approach that partitions an artificial intelligence (AI) model into separate subnetworks (or "experts"), each specializing in a subset of the input data to collectively perform a task. MoE enables large-scale models, even those containing billions of parameters, to significantly reduce computational costs during pre-training and achieve faster performance during inference time. Generally speaking, it achieves this efficiency by selectively activating the specific experts needed for a particular task, rather than activating the entire neural network for each task.

[0017] The hybrid expert model mainly consists of two parts: the gating network and the expert network.

[0018] Each expert network is an independent neural network that has been trained to excel at specific reasoning and computational tasks. The expert network is typically a neural network whose algorithmic structure can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, etc. It can also be a model built on an attention network, such as a transformer model, a bidirectional encoder representation from transformers (BERT) model, or a contrastive language-image pre-training (CLIP) model, etc., but the present invention is not limited to this.

[0019] The gating network is a key component of the hybrid expert model. Its role is to select the appropriate expert network for each input and assign weights to the output of each expert network, ensuring that only the expert model most suitable for the current task participates in the final prediction.

[0020] When using a hybrid expert model to perform inference computing tasks, multiple expert networks share a large model space. The input data for each inference computing task only activates a subset of the expert networks for computation. The gating network processes the input feature vector, calculating the weight of each expert network through linear transformation and the softmax function. The top k expert networks are then selected based on the weights. These selected expert networks process the assigned input data, and the outputs of all activated expert networks are weighted and summed according to the weights assigned by the gating network to produce the final output.

[0021] Expert parallel computing is a type of model parallel computing. By deploying different expert networks of a hybrid expert model on different computing nodes, parallel computing between expert networks can be achieved. This parallelization can significantly improve the inference speed and scalability of the model.

[0022] Hybrid expert models can handle distributed data with more complex distributions. However, the parallel deployment of experts within the hybrid expert model's expert network, tailored to the needs of varying inference computing tasks, remains a challenge. This is particularly true in recent years, with the increasing adoption of multi-heterogeneous computing systems. In these systems, heterogeneous computing power with varying computational performance (such as different types of computing chips or cards) is integrated into a single distributed computing environment to collaboratively complete the distributed inference tasks of the hybrid expert model. Because the computing performance and operating costs of each computing device in a multi-heterogeneous computing system vary, the actual inference computing tasks they target may also differ. In this complex environment, controlling the inference computing cost of the deployed hybrid expert model, thereby improving computing device resource utilization and reducing energy consumption, has become a key issue facing related technologies.

[0023] To solve the problem of cost-based deployment of hybrid expert models in heterogeneous computing systems, an embodiment of the present invention provides a deployment solution for a hybrid expert model. By pre-deploying the hybrid expert model, the activation state parameters of the expert network in the hybrid expert model when executing inference computing tasks based on the hybrid expert model are monitored, thereby determining the activation requirements of the expert network when actually executing inference computing tasks. Based on the activation state parameters of the expert network and the resource status information of the computing nodes of the heterogeneous computing system to be deployed, the computational cost parameters of the computing nodes when executing inference computing tasks after the expert network is deployed to the computing nodes are calculated. With the memory capacity of the computing nodes as a constraint and the goal of minimizing the computational cost, an optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system, thereby optimizing the deployment of the hybrid expert model in the heterogeneous computing system. The expert network can be reasonably deployed in the heterogeneous computing system or even in a multi-heterogeneous computing system to achieve reasonable deployment of inference computing tasks, thereby reducing the hardware cost of hybrid expert model computing, improving the resource utilization of computing equipment, and saving energy and reducing emissions.

[0024] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the deployment method of the hybrid expert model depends, the specific application environment architecture or specific hardware architecture is described herein.

[0025] The hybrid expert model deployment solution provided by this invention can be deployed on heterogeneous computing systems. Heterogeneous computing systems include multiple computing devices (computing nodes), and different computing nodes may have different parameters such as computing core type, computing power, memory size, and communication bandwidth.

[0026] In the heterogeneous computing system targeted by the present invention, the computing core type of the computing node may include, but is not limited to, one or more of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a neural network processor (NPU), a microcontroller unit (MCU), and an application-specific integrated circuit (ASIC).

[0027] In the heterogeneous computing system targeted by the present invention, different computing nodes can be interconnected through a network, such as Ethernet; or they can be interconnected through a bus, such as a high-speed serial computer expansion bus (Peripheral Component Interconnect Express, PCIe) or an NVIDIA high-speed interconnect bus.

[0028] The deployment method of the hybrid expert model provided by the embodiment of the present invention can be applied to one or more computing nodes in a heterogeneous computing system, and can also be applied to a control node outside the heterogeneous computing system.

[0029] An embodiment of the present invention provides a method for deploying a hybrid expert model. The method is described in detail below in conjunction with the execution flow of the method for deploying the hybrid expert model.

[0030] Figure 1 A flowchart of a first method for deploying a hybrid expert model provided by an embodiment of the present invention; Figure 2 A schematic diagram of the deployment architecture of the first hybrid expert model provided by an embodiment of the present invention; Figure 3 A schematic diagram of the deployment architecture of the second hybrid expert model provided in an embodiment of the present invention.

[0031] like Figure 1 As shown, the deployment method of the hybrid expert model provided by the embodiment of the present invention may include: S101: pre-deploying the hybrid expert model and monitoring the activation state parameters of the expert network in the hybrid expert model when performing reasoning computing tasks based on the hybrid expert model.

[0032] S102: Obtain resource status information of computing nodes of the heterogeneous computing system to be deployed.

[0033] S103: Calculating, based on the resource status information of the computing node and the activation status parameters of the expert network, a computing cost parameter when the computing node performs the inference computing task after the expert network is deployed to the computing node.

[0034] S104: With the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and with minimization of the computing cost parameter as the optimization goal, an optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

[0035] In terms of the deployment of the gated network, the embodiments of the present invention can be oriented towards two deployment modes of the gated network.

[0036] In some optional implementations of the present invention, a gated network can be deployed on each computing node. The gated network on each computing node will dynamically decide which input data to assign to the local expert network based on the local input data, or forward the input data to the expert network on other computing nodes. Figure 2 As shown, expert network 1, expert network 2, and gating network are deployed on computing nodes 1 and 2. After input data 1 enters computing node 1, the gating network decides to activate the expert network 1 of the hybrid expert model, and then distributes input data 1 to the local expert model 1 and the expert model 1 of computing node 2. The outputs of the two expert networks 1 are then aggregated to obtain output data 1. After input data 2 enters computing node 2, the gating network decides to activate the expert network 2 of the hybrid expert model, and then distributes input data 2 to the local expert model 2 and the expert network 2 of computing node 1. The outputs of the two expert networks 2 are then aggregated to obtain output data 2.

[0037] In some other optional implementations of the present invention, the gated network can also be deployed on a centralized node. All input data of the hybrid expert model are first sent to the centralized node, and the gated network makes a unified routing decision, and then distributes the input data to the computing nodes where each expert network is located. Figure 3 As shown, expert network 1 and expert network 2 are deployed on computing nodes 1 and 2. After the centralized node makes a decision on the input data 3, it determines to activate expert network 1. Then, the input data 3 is sent to computing node 1 and computing node 2 to activate the two expert networks 1, and then the outputs of the two expert networks 1 are aggregated to obtain output data 3.

[0038] It should be noted that the above Figure 2 、 Figure 3 The deployment architecture shown is only used to illustrate the deployment method of the gated network. In the embodiment of the present invention, one computing node can deploy one or more expert networks, and the same expert network can be deployed on different computing nodes.

[0039] This embodiment of the present invention aims to determine the activation requirements of each expert network when the hybrid expert model actually executes the inference computing tasks it is required to perform after deployment by monitoring the activation status parameters of each expert network. Combined with the resource status information on the computing nodes, the deployment goal is to minimize the cost while satisfying the activation requirements of the expert networks.

[0040] That is to say, the distribution of inference computing tasks faced by the hybrid expert model after deployment is regular, and this regularity affects the activation requirements of each expert network. Therefore, the hybrid expert model can be pre-deployed and the activation status parameters of each expert network can be monitored.

[0041] The embodiment of the present invention can be applied before deploying the hybrid expert model in a heterogeneous computing system.

[0042] For S101, the hybrid expert model is pre-deployed, and the input data of the actual inference computing task to be processed is input into the hybrid expert model to monitor the activation state parameters of each expert network.

[0043] Since the deployment of a hybrid expert network requires relatively high hardware costs, in an embodiment of the present invention, S101 pre-deploys the hybrid expert model and monitors the activation state parameters of the expert network in the hybrid expert model when performing inference computing tasks based on the hybrid expert model. This may include: pre-deploying the gating network of the hybrid expert model; inputting the input data of the inference computing task into the gating network, and monitoring the weight of the expert network output by the gating network; and determining the activation state parameters of the expert network based on the weight of the expert network.

[0044] Monitoring the activation state parameters of the expert networks in the hybrid expert model when executing an inference computing task based on the hybrid expert model in S101 may include monitoring the number of expert network activations per unit time as the expert network activation state parameter. By monitoring the number of expert network activations per unit time, it is determined how many times each expert network needs to be activated per unit time when the hybrid expert model executes the actual inference computing task.

[0045] For S102, the resource status information of the computing node can be obtained by accessing the cluster management node of the heterogeneous computing system or accessing each computing node separately. The resource status information of the computing node can be the initial resource status information of the computing node. If the heterogeneous computing system is also used to deploy other tasks, the resource status information of the computing node is the current remaining resource status information of the computing node. The type of resource status information of the computing node may include one or more of computing power resource parameters, memory resource parameters, and network resource parameters. Among them, the computing power resource parameter can be represented by the number of floating point operations per second (FLOPS) of the computing node. The network resource parameter can be represented by the communication bandwidth of the external link of the computing node. The memory resource parameter can be represented by the available memory capacity of the computing node.

[0046] For S103, based on the resource status information of the computing node and the activation status parameters of the expert network, the computing cost parameters of the computing node when performing the actual reasoning computing task after the expert network is deployed on the computing node can be calculated.

[0047] In an embodiment of the present invention, S103 calculates the computing cost parameters when the computing node performs inference computing tasks after the expert network is deployed to the computing node based on the resource status information of the computing node and the activation status parameters of the expert network, which may include: calculating the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node based on the activation status parameters of the expert network and the hardware resource parameters of the computing node; calculating the computing cost parameters when the computing node performs inference computing tasks after the expert network is deployed to the computing node based on the actual number of activations of the expert network and the resource status information of the computing node.

[0048] The computing cost parameter can be determined based on the unit operating cost of the computing node. Determining the unit operating cost can include: monitoring the computing node's power consumption parameters and the node's operation and maintenance costs during operation; and calculating the node's unit operating cost based on the power consumption parameters, the operation and maintenance costs, and the node's operating time. In other words, after monitoring and calculating the node's unit operating cost, the computing cost parameter can be calculated by predicting the time it takes the node to perform inference computing tasks after deploying the expert network.

[0049] In S104, embodiments of the present invention aim to obtain a solution for deploying a hybrid expert model in a heterogeneous computing system. Given the availability of hardware resources in a heterogeneous computing system, there are numerous possible deployment options for the hybrid expert model. Embodiments of the present invention aim to obtain a deployment solution that minimizes computational cost. This deployment solution is represented by the deployment locations of each expert network in the hybrid expert model in the heterogeneous computing system, specifically, which expert network is deployed on which computing node.

[0050] Since the parameters of other networks in the hybrid expert model are relatively small except for the expert network, the parameters of other networks can be flexibly deployed on each computing node or on a centralized node.

[0051] The deployment method of the hybrid expert model provided by the embodiment of the present invention pre-deploys the hybrid expert model to monitor the activation state parameters of the expert network in the hybrid expert model when performing reasoning computing tasks based on the hybrid expert model, thereby determining the activation requirements of the expert network when actually performing reasoning computing tasks; based on the activation state parameters of the expert network and the resource status information of the computing nodes of the heterogeneous computing system to be deployed, the computing cost parameters when the computing nodes perform reasoning computing tasks after the expert network is deployed to the computing nodes are calculated, and the memory capacity of the computing nodes is used as a constraint and the optimization calculation is performed with the goal of minimizing the computing cost to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system, thereby obtaining a hybrid expert model deployment solution that meets actual reasoning computing requirements and is low in cost, solves the problem of high cost when deploying hybrid expert models in heterogeneous computing systems in related technologies, and improves the resource utilization of heterogeneous computing systems.

[0052] Based on the above embodiment, the embodiment of the present invention continues to introduce the steps of calculating the computation cost parameters when the computing node performs an inference computing task after the expert network is deployed to the computing node.

[0053] In the above embodiment, after the unit operating cost of the computing node is obtained by monitoring and calculating, the computing cost parameter of the computing node can be calculated by predicting the time it takes for the computing node to perform the inference computing task after the expert network is deployed.

[0054] Based on this, in an embodiment of the present invention, the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node is calculated based on the activation status parameters of the expert network and the hardware resource parameters of the computing node, which may include: using the computing power resource parameters of the computing node as a constraint, calculating the actual number of activations of the expert network after the expert network is deployed on the computing node.

[0055] Specifically, taking the computing power resource parameters of the computing node as a constraint, calculating the actual number of activations of the expert network after the expert network is deployed on the computing node can include: calculating the allowed number of activations of the expert network on the computing node based on the remaining computing power resource parameters of the computing node and the computational complexity parameters of the expert network; obtaining the smaller value between the activation requirement parameter of the expert network and the allowed number of activations of the expert network on the computing node as the actual number of activations of the expert network.

[0056] Among them, the computational complexity parameter of the expert network can be the number of floating-point operations that need to be performed per second after the expert network is activated once; the computing power resource parameter of the computing node can be the number of floating-point operations per second of the computing node.

[0057] That is, using the computing power resource parameters of the computing node, specifically the remaining computing power resource parameters of the computing node, as the constraint condition, if the remaining computing power of the computing node can meet the activation requirements of the expert network, the number of expert network activations per unit time is used as the actual number of expert network activations on the computing node. If the remaining computing power of the computing node cannot meet the activation requirements of the expert network, the remaining computing power of the computing node is used as the limit to determine the allowed number of expert network activations.

[0058] When determining the number of times the expert network is allowed to be activated on a computing node, the computational complexity parameter of the expert network can be obtained. For example, the number of floating-point operations required for each forward propagation calculation performed by the expert network is the computational complexity parameter. The computational complexity parameter of the expert network is divided by the number of floating-point operations per second of the computing node to obtain the number of times the expert network is allowed to be activated on the computing node.

[0059] After predicting the actual number of times the expert network is activated per unit time when the computing node performs inference computing tasks after deploying the expert network on the computing node, the computational time required for these computations can be calculated based on this, and the computational cost parameters can be calculated in combination with the unit operating cost of the computing node. The computational cost parameters for the computing node performing inference computing tasks after the expert network is deployed on the computing node are calculated based on the actual number of expert network activations and the resource status information of the computing node. This may include: calculating the computational amount of the computing node performing inference computing tasks based on the expert network based on the actual number of expert network activations and the computational complexity parameters of the expert network; calculating the time consumption prediction result for the computing node performing inference computing tasks based on the expert network based on the computational amount and the computing power resource parameters of the computing node; and calculating the computational cost parameter based on the time consumption prediction result and the unit operating cost of the computing node.

[0060] Among them, the steps for determining the unit operating cost of the computing node may include: monitoring the power consumption parameters of the computing node during operation and the operation and maintenance costs of the computing node; calculating the unit operating cost of the computing node based on the power consumption parameters, operation and maintenance costs and the running time of the computing node.

[0061] In addition to the calculation steps of the computing cost parameters when the computing node performs the inference computing task after the expert network is deployed to the computing node as introduced in the above embodiment, in other optional implementations of the embodiments of the present invention, the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node is calculated based on the activation state parameters of the expert network and the hardware resource parameters of the computing node. It may also include: calculating the actual number of activations of the expert network after the expert network is deployed on the computing node based on the computing power resource parameters and the network resource parameters of the computing node as constraints.

[0062] Specifically, with the computing power resource parameters and network resource parameters of the computing node as constraints, calculating the actual number of activations of the expert network after the expert network is deployed on the computing node can include: calculating the first allowed number of activations of the expert network on the computing node based on the remaining computing power resource parameters of the computing node and the computational complexity parameters of the expert network; calculating the second allowed number of activations of the expert network on the computing node based on the remaining network resource parameters of the computing node and the amount of input and output data of the expert network in an inference computing task; obtaining the minimum value of the activation requirement parameter of the expert network, the first allowed number of activations, and the second allowed number of activations as the actual number of activations of the expert network.

[0063] Among them, the computational complexity parameter of the expert network can be the number of floating-point operations that need to be performed per second after the expert network is activated once; the computing power resource parameter of the computing node can be the number of floating-point operations per second of the computing node.

[0064] That is to say, the computing power resource parameters and network resource parameters of the computing node, specifically the remaining computing power resource parameters and the remaining network resource parameters of the computing node are used as constraints. If the remaining computing power and remaining network resources of the computing node can meet the activation requirements of the expert network, the number of activations of the expert network per unit time is used as the actual number of activations of the expert network on the computing node. Otherwise, the number of allowed activations of the expert network is determined based on the remaining computing power and remaining network resources of the computing node.

[0065] When determining the allowed number of activations of the expert network based on the remaining computing power, the computational complexity parameter of the expert network can be obtained. For example, the number of floating-point operations required for each forward propagation calculation performed by the expert network is the computational complexity parameter. By dividing the computational complexity parameter of the expert network by the number of floating-point operations per second of the computing node, the allowed number of activations of the expert network under the computing power constraint of the computing node can be obtained.

[0066] When determining the allowed number of activations of the expert network based on the remaining network resources, the amount of input data of the expert network in executing an inference computing task can be obtained, and the allowed number of activations of the expert network under the network resource constraints of the computing node can be determined based on the amount of input data and the remaining network resource parameters of the computing node.

[0067] Then, the smaller value of the two allowed activation times is taken as the actual allowed activation times.

[0068] After predicting the actual number of times the expert network is activated per unit time when the computing node performs an inference computing task after deploying the expert network on the computing node, the computational time required for these computations can be calculated based on this, and the computational cost parameter can be calculated in combination with the unit operating cost of the computing node. The computational cost parameter for the computing node performing an inference computing task after the expert network is deployed on the computing node is calculated based on the actual number of expert network activations and the resource status information of the computing node. This may include: calculating the computational amount of the computing node performing the inference computing task based on the expert network based on the actual number of expert network activations and the computational complexity parameter of the expert network; calculating the computational time prediction result of the computing node performing the inference computing task based on the expert network based on the computational amount and the computing power resource parameter of the computing node; calculating the communication time prediction result of the computing node performing the inference computing task based on the expert network based on the actual number of expert network activations, the amount of input and output data when the expert network is activated once, and the remaining network resource parameter of the computing node; and calculating the computational cost parameter based on the computational time prediction result, the communication time prediction result, and the unit operating cost of the computing node.

[0069] Among them, the steps for determining the unit operating cost of the computing node may include: monitoring the power consumption parameters of the computing node during operation and the operation and maintenance costs of the computing node; calculating the unit operating cost of the computing node based on the power consumption parameters, operation and maintenance costs and the running time of the computing node.

[0070] Based on the above embodiment, the embodiment of the present invention further introduces the overall process of the deployment method of the hybrid expert model.

[0071] Figure 4 This is a flowchart of a second method for deploying a hybrid expert model provided by an embodiment of the present invention.

[0072] like Figure 4 As shown, the deployment method of the hybrid expert model provided by the embodiment of the present invention can be implemented based on four modules, namely: a task information collection module, a system information collection module, a calculation module and a task delivery module.

[0073] The task information collection module is used to parse or calculate the necessary input information based on the user-entered hybrid expert distributed inference task information. This information may include the forward propagation computational complexity of the expert network in the hybrid expert model, expressed in floating-point operations (FLOPs). The computational complexity of the forward propagation can be estimated based on the neural layer composition of the neural network, where the computational complexity is equal to the sum of the computational complexity of all neural layers. The computational complexity of a neural layer can be estimated using mathematical methods. For example, for the forward computation of a fully connected layer, assuming the input data dimensions are (N, D), the hidden layer weight dimensions are (D, out), and the output is (N, out), the computational complexity is FLOPs = N*(2×D-1)*out. Different neural layers have different computational methods, which are not listed here. Generally, the network shape of the expert networks is the same, so the computational complexity of only one expert network needs to be calculated.

[0074] The total number of expert networks in the mixture of experts model. This information can be directly obtained from the model structure.

[0075] The amount of video memory used by a single expert network during inference. This information can be directly calculated based on the expert network's structure or compiled using existing open-source tools such as torchstat. This information is the same for all expert networks.

[0076] The number of times each expert network is activated per unit time can be expressed as the average activation frequency per second. By pre-deploying the hybrid expert model and setting a time threshold (for example, 30 minutes), the number of times each expert network is activated per unit time can be calculated.

[0077] The system information collection module is used to collect relevant information of the heterogeneous computing system, which may include: the number of available computing nodes in the heterogeneous computing system.

[0078] The resource status information of each computing node may include one or more of the computing power resource parameters, memory resource parameters, and network resource parameters of each computing node. This resource status information can be obtained from the computing node product manual or actual testing.

[0079] The unit operating cost of each computing node can be calculated by monitoring the power consumption parameters of the computing node during operation and the operation and maintenance costs of the computing node, and according to the power consumption parameters, operation and maintenance costs and the running time of the computing node.

[0080] Thus, the input data of the calculation module may include variables of the expert network and variables of the calculation nodes.

[0081] Among them, the expert network ( =1,2,…, ) of the variable ( is the total number of expert networks) can include: expert networks Activation state parameters (unit: times / second), expert network The computational complexity parameter (Unit: FLOPs / time), Expert Network The number of parameters (or the memory capacity occupied, in MB).

[0082] Compute nodes ( =1,2,…, ) of the variable ( is the total number of computing nodes) can include: computing nodes Floating point operations per second (FLOPS), computing nodes Remaining memory capacity (MB), computing nodes Unit operating cost (The operating cost per second can be quantified by electricity and operation and maintenance costs).

[0083] The calculation module is used to calculate the deployment plan that minimizes the computational cost of deploying the hybrid expert model in the heterogeneous computing system based on the information collected by the task information collection module and the system information collection module. After the calculation is completed, the calculation module sends the calculation results to the task delivery module. The output of the calculation module can be used as the allocation matrix Indicates that the elements Representing expert networks On the compute node The number of activations per unit time (times / second).

[0084] The task dispatch module is based on the allocation matrix , deploy the corresponding expert network on the compute nodes of the heterogeneous computing system and send the expert network's activation count per unit time on the compute nodes to the heterogeneous computing system's load balancer. This allows the load balancer to distribute computational requests in the inference computing task according to the specified activation count per unit time. For example, if expert network a is deployed on both compute nodes 1 and 2, and the computational modules output activation counts of 3 and 7, respectively, the load balancer will distribute computational requests to activate expert network a to compute nodes 1 and 2 in a ratio of 3:7.

[0085] The following is a detailed introduction to the steps of performing calculations according to input information by a calculation module provided by an embodiment of the present invention.

[0086] First, initialize according to the input information, including: initializing the remaining computing power of all computing nodes: Initialize the remaining memory of all compute nodes: Initialize the activation requirements of the expert network: . n*k distribution matrix , initialize all elements to 0.

[0087] In some optional implementations of the present invention, global optimization calculations can be performed on all expert networks and all computing nodes based on the above input information. However, due to the large amount of input information, the solution complexity is relatively high.

[0088] In order to improve the solution efficiency, in some other optional implementations of the embodiments of the present invention, S104 is based on the constraint condition that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node, and the optimization goal is to minimize the computing cost parameters, and perform optimization calculations to obtain the deployment location of the expert network of the hybrid expert model in the heterogeneous computing system. It can include: performing optimization calculations on each expert network separately to obtain the target computing node where the expert network is deployed.

[0089] Specifically, performing optimization calculations on each expert network to obtain the target computing node deployed by the expert network may include: determining whether there is an undeployed expert network; if so, obtaining an undeployed expert network to perform optimization calculations to obtain the target computing node deployed by the expert network.

[0090] Among them, obtaining an undeployed expert network to perform optimization calculations to obtain the target computing node deployed by the expert network can include: initializing the activation state parameters of the expert network to the activation requirement parameters of the expert network; calculating the computational cost parameters of deploying the expert network on the computing nodes from the computing nodes whose remaining memory capacity is greater than the parameter amount of the expert network, and selecting the computing node with the smallest computational cost parameter as the target computing node of the expert network; judging whether the target computing node meets the activation requirement parameters of the expert network; if so, determining that the optimization calculation of the expert network is completed; if not, updating the activation requirement parameters of the expert network according to the determined target computing node, and returning to the step of determining the target computing node.

[0091] The deployment method of the hybrid expert model provided by an embodiment of the present invention may also include: if the determined target computing node does not meet the activation requirement parameters of the expert network, and there is no computing node among the other computing nodes whose remaining memory capacity is greater than the parameter amount of the expert network, then outputting information that the deployment of the hybrid expert model has failed.

[0092] In actual calculation, for each pair Expert network and each compute node , if the computing node Remaining memory capacity (i.e., parameters sufficient to deploy the expert network), then the expert network will be calculated Assign to compute nodes The calculation cost parameters after for: ; in, Represents a compute node Assign expert network After that, the expert network The maximum number of times per second that can be activated can be recorded as , limited by the expert network The remaining activation requirements and compute nodes The remaining computing power per second. Representative expert network After each activation, the compute node The running time on , so the cost parameter is calculated Can be calculated as: computing nodes The unit operating cost multiplied by the operating time of each activation Multiply by the actual number of activations .

[0093] if , then the expert network Unable to compute node On deployment, at this time .

[0094] A greedy allocation algorithm can be used to assign All calculated , choose an allocation with the lowest computational cost: ; in, are the expert network, computing node, and the number of activations per unit time of the expert network on the computing node when the computational cost parameter is minimized, respectively. argmin represents the minimum value calculation.

[0095] Updating compute nodes Remaining memory capacity: .

[0096] Updating compute nodes The remaining computing power: .

[0097] Update expert network Remaining activation requirements: .

[0098] Update the allocation matrix :Will Join the allocation matrix In, that is .

[0099] At this time, if all expert networks Remaining activation demand , it is considered that the allocation has been completed, and the output allocation matrix .

[0100] If there is an expert network of Is not equal to 0, then make the following judgment: If , or there is no computing node that meets the remaining computing power to accommodate the remaining activation requirements of an expert network, the result returned is that the hybrid expert model deployment fails, otherwise, return to the step of determining the computing nodes assigned to the expert network.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0102] An embodiment of the present invention also provides a deployment device for a hybrid expert model, comprising: a task information collection module for pre-deploying the hybrid expert model and monitoring the activation state parameters of the expert network in the hybrid expert model when executing reasoning computing tasks based on the hybrid expert model; a system information collection module for obtaining resource status information of computing nodes of a heterogeneous computing system to be deployed; a computing module for calculating, based on the resource status information of the computing nodes and the activation state parameters of the expert network, the computing cost parameters when the computing nodes execute reasoning computing tasks after the expert network is deployed to the computing nodes; with the constraint that the number of parameters of the expert network deployed on the computing nodes does not exceed the memory capacity of the computing nodes and the optimization goal of minimizing the computing cost parameters, an optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

[0103] For the description of the features in the embodiment corresponding to the deployment device of the hybrid expert model, please refer to the relevant description of the embodiment corresponding to the deployment method of the hybrid expert model, which will not be repeated here.

[0104] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned embodiments of the deployment method of the hybrid expert model.

[0105] An embodiment of the present invention further provides a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned hybrid expert model deployment method embodiments when running.

[0106] In an exemplary embodiment, the non-volatile storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0107] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned embodiments of the method for deploying a hybrid expert model are implemented.

[0108] An embodiment of the present invention further provides another computer program product, including a non-volatile storage medium, wherein the non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned hybrid expert model deployment method embodiments are implemented.

[0109] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0110] The above is a detailed introduction to the deployment method, device, equipment, medium and product of a hybrid expert model provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A deployment method of a hybrid expert model, characterized in that: include: Pre-deploy the hybrid expert model and monitor activation state parameters of the expert network in the hybrid expert model when performing reasoning computing tasks based on the hybrid expert model; Obtain resource status information of computing nodes of the heterogeneous computing system to be deployed; Calculating, based on the resource status information of the computing node and the activation status parameter of the expert network, a computing cost parameter when the computing node executes the inference computing task after the expert network is deployed to the computing node; With the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and the optimization goal of minimizing the computing cost parameter, optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

2. The method for deploying a hybrid expert model according to claim 1, wherein: Monitoring activation state parameters of an expert network in the hybrid expert model when performing an inference computing task based on the hybrid expert model includes: The number of activations of the expert network within a unit time is monitored as an activation state parameter of the expert network.

3. The deployment method of the hybrid expert model according to claim 2, characterized in that: Calculating, based on the resource status information of the computing node and the activation status parameter of the expert network, a computing cost parameter when the computing node performs the inference computing task after the expert network is deployed to the computing node, including: Calculating the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node according to the activation state parameters of the expert network and the hardware resource parameters of the computing node; According to the actual activation times of the expert network and the resource status information of the computing node, a computing cost parameter when the computing node performs the inference computing task after the expert network is deployed to the computing node is calculated.

4. The method for deploying a hybrid expert model according to claim 3, wherein: Calculating, based on the activation state parameters of the expert network and the hardware resource parameters of the computing node, the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node, includes: Taking the computing power resource parameters of the computing node as a constraint, the actual number of activations of the expert network after the expert network is deployed on the computing node is calculated.

5. The method for deploying a hybrid expert model according to claim 4, characterized in that: Calculating the actual number of activations of the expert network after the expert network is deployed on the computing node, with the computing power resource parameter of the computing node as a constraint, includes: Calculating the number of times the expert network is allowed to be activated on the computing node according to the remaining computing power resource parameter of the computing node and the computational complexity parameter of the expert network; The smaller value of the activation requirement parameter of the expert network and the allowed activation times of the expert network on the computing node is obtained as the actual activation times of the expert network.

6. The method for deploying a hybrid expert model according to claim 5, characterized in that: Calculating, based on the actual activation times of the expert network and the resource status information of the computing node, a computational cost parameter when the computing node executes the inference computing task after the expert network is deployed on the computing node, including: Calculating the computational effort of the computing node in executing the inference computing task based on the expert network according to the actual activation times of the expert network and the computational complexity parameter of the expert network; Calculating a time-consuming prediction result of the computing node executing the inference computing task based on the expert network according to the computing amount and the computing power resource parameters of the computing node; The computing cost parameter is calculated based on the time consumption prediction result and the unit operating cost of the computing node.

7. The method for deploying a hybrid expert model according to claim 3, wherein: Calculating, based on the activation state parameters of the expert network and the hardware resource parameters of the computing node, the actual number of activations of the expert network per unit time after the expert network is deployed on the computing node, includes: The actual number of activations of the expert network after the expert network is deployed on the computing node is calculated with the computing power resource parameters of the computing node and the network resource parameters of the computing node as constraints.

8. The method for deploying a hybrid expert model according to claim 7, wherein: Calculating the actual number of activations of the expert network after deploying the expert network on the computing node, based on the computing power resource parameters of the computing node and the network resource parameters of the computing node as constraints, includes: Calculating a first allowed number of activations of the expert network on the computing node according to a remaining computing resource parameter of the computing node and a computational complexity parameter of the expert network; Calculating a second allowed activation number of the expert network on the computing node according to the remaining network resource parameter of the computing node and the amount of input and output data of the expert network in one inference computing task; A minimum value among the activation requirement parameter of the expert network, the first allowed activation times, and the second allowed activation times is obtained as the actual activation times of the expert network.

9. The method for deploying a hybrid expert model according to claim 8, wherein: Calculating, based on the actual activation times of the expert network and the resource status information of the computing node, a computational cost parameter when the computing node executes the inference computing task after the expert network is deployed on the computing node, including: Calculating the computational effort of the computing node in executing the inference computing task based on the expert network according to the actual activation times of the expert network and the computational complexity parameter of the expert network; Calculating a prediction result of the computation time required for the computation node to execute the inference computation task based on the expert network according to the computation amount and the computing power resource parameters of the computation node; Calculate, based on the actual number of activations of the expert network, the amount of input and output data when the expert network is activated once, and the remaining network resource parameters of the computing node, a predicted result of the communication time consumption of the computing node performing the inference computing task based on the expert network; The computing cost parameter is obtained by calculation according to the computing time prediction result, the communication time prediction result and the unit operation cost of the computing node.

10. The method for deploying a hybrid expert model according to claim 5 or 8, characterized in that: The computational complexity parameter of the expert network is the number of floating-point operations required to be performed per second after the expert network is activated once; The computing power resource parameter of the computing node is the number of floating-point operations per second of the computing node.

11. The method for deploying a hybrid expert model according to claim 6 or 9, characterized in that: The step of determining the unit operating cost of the computing node includes: Monitoring the power consumption parameters of the computing nodes during operation and the operation and maintenance costs of the computing nodes; The unit operating cost of the computing node is calculated according to the power consumption parameter, the operation and maintenance cost, and the operating time of the computing node.

12. The method for deploying a hybrid expert model according to claim 1, wherein: With the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and with minimization of the computing cost parameter as the optimization goal, an optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system, including: The optimization calculation is performed on each expert network respectively to obtain the target computing node deployed by the expert network.

13. The method for deploying a hybrid expert model according to claim 12, wherein: Executing the optimization calculation on each expert network respectively to obtain the target computing node deployed by the expert network includes: Determining whether there is an undeployed expert network; If yes, an undeployed expert network is acquired to perform the optimization calculation, and a target computing node deployed by the expert network is obtained.

14. The method for deploying a hybrid expert model according to claim 13, wherein: Obtaining an undeployed expert network to perform the optimization calculation to obtain a target computing node deployed by the expert network, including: Initializing the activation state parameters of the expert network to activation requirement parameters of the expert network; calculating, from the computing nodes having a remaining memory capacity greater than the number of parameters of the expert network, the computing cost parameter for deploying the expert network on the computing nodes, and selecting the computing node with the smallest computing cost parameter as the target computing node for the expert network; Determining whether the target computing node meets activation requirement parameters of the expert network; If yes, determining that the optimization calculation of the expert network is finished; If not, after updating the activation requirement parameters of the expert network according to the determined target computing node, the process returns to the step of determining the target computing node.

15. The method for deploying a hybrid expert model according to claim 14, wherein: Also includes: If the determined target computing node does not meet the activation requirement parameters of the expert network, and there is no computing node among the other computing nodes whose remaining memory capacity is greater than the parameter amount of the expert network, information indicating that the hybrid expert model deployment has failed is output.

16. The method for deploying a hybrid expert model according to claim 1, wherein: Pre-deploying the hybrid expert model and monitoring the activation state parameters of the expert network in the hybrid expert model when performing an inference computing task based on the hybrid expert model include: Pre-deploying a gating network of the hybrid expert model; Inputting the input data of the inference computing task into the gating network, and monitoring the weight of the expert network output by the gating network; An activation state parameter of the expert network is determined according to the weight of the expert network.

17. A deployment device for a hybrid expert model, characterized in that: include: A task information collection module is used to pre-deploy the hybrid expert model and monitor the activation state parameters of the expert network in the hybrid expert model when performing reasoning and computing tasks based on the hybrid expert model; A system information collection module is used to obtain resource status information of computing nodes of the heterogeneous computing system to be deployed; a calculation module, configured to calculate, based on the resource status information of the computing node and the activation status parameter of the expert network, a calculation cost parameter when the computing node executes the inference computing task after the expert network is deployed to the computing node; With the constraint that the number of parameters of the expert network deployed on the computing node does not exceed the memory capacity of the computing node and the optimization goal of minimizing the computing cost parameter, optimization calculation is performed to obtain the deployment position of the expert network of the hybrid expert model in the heterogeneous computing system.

18. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the method for deploying a hybrid expert model according to any one of claims 1 to 16 when executing the computer program.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the method for deploying the hybrid expert model according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for deploying the hybrid expert model according to any one of claims 1 to 16 are implemented.

Images