Distributed deployment method and device of model, chip, storage medium and program product
By segmenting the network structure and tensors of the model to be deployed and deployed to multiple nodes to be deployed, the problem of low hardware utilization in the existing technology is solved, and efficient inference speed and video memory management are achieved.
Patent Information
- Application Number
- CN202411335892.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art has low hardware utilization when performing distributed deployment of models, resulting in slowing inference speed and increasing memory demand.
Through the model deployment node, the network structures of the model to be deployed are divided into p modules to be deployed, and the tensors are divided into t segmented tensors. Based on these segmented modules and tensors, the model is deployed to N waiting for Deploy node, N=p*t.
Pipeline parallelism between layers and tensor parallelism within layers is realized, which improves hardware utilization, reduces video memory requirements and speeds up inference.
Smart Images
Figure CN119443172A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of distributed deployment of neural network models used in power systems, and in particular, relates to a distributed deployment method, distributed deployment device, chip, computer-readable storage medium and computer program product of the model. Background Art
[0002] With the popularity of deep learning models and their implementation in practical applications, we are facing a surge in video memory and a slowdown in inference speed. We increasingly need to do some optimization work in the inference deployment phase to reduce video memory and speed up inference. When the model is too large, we may encounter the problem of being unable to deploy it on a single machine.
[0003] At this time, the model needs to be deployed in a distributed manner and deployed to different devices for running. However, the hardware utilization rate during distributed deployment is generally low. Summary of the invention
[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application proposes a distributed deployment method, a distributed deployment device, a chip, a computer-readable storage medium and a computer program product of a model, which are conducive to improving the hardware utilization rate of the distributed deployment.
[0005] In a first aspect, the present application provides a distributed deployment method of a model, which is applied to a computer system, wherein the computer system includes a model deployment node and nodes to be deployed, and the model includes a multi-layer network structure. The method includes dividing each layer of the network structure into p modules to be deployed through the model deployment node, and the modules to be deployed include one or more layers of the network structure; dividing the tensor of the model to be deployed into t divided tensors through the model deployment node; based on the p modules to be deployed and the t divided tensors, the model deployment node deploys the model to be deployed to N nodes to be deployed, N=p*t, and the nodes to be deployed include devices with data processing capabilities.
[0006] In some embodiments, the method also includes obtaining a preset deployment configuration model, wherein the deployment configuration model determines p, t and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time.
[0007] In some embodiments, obtaining a preset deployment configuration model includes:
[0008] Obtain a training set, wherein the training set includes p, t, and batch numbers corresponding to models of different complexity under different deployment configurations, wherein the deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed, and the proportion of cavitation time;
[0009] Model training is performed based on the training set to obtain the model to be deployed.
[0010] In some embodiments, the void time ratio is determined based on the forward execution time of the model to be deployed for each batch, the number of batches, and p, and the void time ratio is less than a preset threshold.
[0011] In some embodiments, the cavitation time ratio is determined based on the cavitation time and the predicted execution time, the cavitation time is determined based on the forward execution time of each batch and the number of modules to be deployed in the waiting state, and the predicted execution time is determined based on the forward execution time of each batch and p.
[0012] In some embodiments, the cavitation time ratio is determined based on the following formula:
[0013]
[0014] t bubble =(p-1)×t f ;
[0015] t ideal =m×t f ;
[0016] Where, bubbleration is the proportion of cavitation time, t bubble is the cavitation time, t ideal is the predicted execution time, t f is the forward execution time for each batch, and m is the number of batches.
[0017] In some embodiments, the complexity of the model to be deployed is determined based on at least one of the number of parameters and the amount of calculation of the model to be deployed, and the hardware performance of the node to be deployed is determined based on the data processing performance of the node to be deployed.
[0018] In some embodiments, the number of batches of the model to be deployed is greater than or equal to 4p.
[0019] In some embodiments, 4≤t≤8.
[0020] In some embodiments, the value of t is proportional to the communication occupancy time of the model to be deployed, and the communication occupancy time is less than a predetermined threshold.
[0021] In some embodiments, before the to-be-deployed model is deployed to N to-be-deployed nodes, the method further includes:
[0022] Performing operator fusion on the operators of the module to be deployed, so as to merge multiple consecutive operations or operators into a single operator;
[0023] The deploying the model to be deployed to N nodes to be deployed based on the p modules to be deployed and the t segmentation tensors includes:
[0024] Based on the p modules to be deployed and the t segmented tensors after operator fusion, the model to be deployed is deployed to N nodes to be deployed.
[0025] In some embodiments, before the to-be-deployed model is deployed to N to-be-deployed nodes, the method further includes:
[0026] quantizing the model to be deployed to obtain p quantized models to be deployed;
[0027] The deploying the model to be deployed to N nodes to be deployed based on the p modules to be deployed and the t segmentation tensors includes:
[0028] Based on the quantized p modules to be deployed and the t segmentation tensors, the model to be deployed is deployed to N nodes to be deployed.
[0029] In a second aspect, the present application provides a distributed deployment device, which is applied to a computer system, wherein the computer system includes a model deployment node and a node to be deployed, and the distributed deployment device includes:
[0030] A model segmentation module, used to segment each layer of the network structure of the model into p modules to be deployed through the model deployment node, wherein the modules to be deployed include one or more layers of the network structure;
[0031] A tensor segmentation module, used to segment the tensor of the model to be deployed into t segmented tensors through a model deployment node;
[0032] A deployment module is used for, based on the p modules to be deployed and the t segmentation tensors, the model deployment node deploys the model to be deployed to N nodes to be deployed, N=p*t, and the nodes to be deployed include devices with data processing capabilities.
[0033] In some embodiments, the model segmentation module is specifically used to segment each layer of the network structure into p modules to be deployed, and the modules to be deployed include one or more layers of the network structure.
[0034] In some embodiments, the device also includes an acquisition module, which is used to obtain a preset deployment configuration model, and the deployment configuration model determines p, t and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time.
[0035] In some embodiments, the acquisition module is specifically used to acquire a training set, wherein the training set includes p, t and batch number corresponding to models of different complexity under different deployment configurations, and the deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time; model training is performed based on the training set to obtain the model to be deployed.
[0036] In some embodiments, the device also includes an operator fusion module; the operator fusion module is used to perform operator fusion on the operators of the module to be deployed before the model to be deployed is deployed to N nodes to be deployed, so as to merge multiple consecutive operations or operators into a single operator; the deployment module is specifically used to deploy the model to be deployed to N nodes to be deployed based on the p modules to be deployed and the t segmentation tensors after operator fusion.
[0037] In some embodiments, the device also includes a quantization module, which is used to quantize the model to be deployed before the model to be deployed to N nodes to be deployed, so as to obtain p quantized models to be deployed; the deployment module is specifically used to deploy the model to be deployed to N nodes to be deployed based on the quantized p modules to be deployed and the t segmentation tensors.
[0038] In a third aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed deployment method in any of the above-mentioned embodiments.
[0039] In a fourth aspect, the present application provides a chip, which is used to execute the distributed deployment method in any of the above embodiments.
[0040] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the distributed deployment method in any of the above embodiments.
[0041] The distributed deployment method, distributed deployment device, chip, computer-readable storage medium and computer program product provided in the embodiments of the present application realize pipeline parallelism between layers by dividing the model to be deployed into p modules to be deployed, and realize tensor parallelism within the layer by dividing the tensor of the model to be deployed into t divided tensors. It can be understood that the larger the number of tensor parallelism, the longer the communication occupancy time, and the larger the number of pipeline parallelism, the longer the idle time of the node to be deployed is in waiting. Therefore, while ensuring that all nodes to be deployed are utilized, the communication time and idle time can be balanced by adjusting the ratio of pipeline parallelism and tensor parallelism, thereby improving the utilization rate of hardware.
[0042] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0044] Figure 1 It is a schematic diagram of an application scenario of the distributed deployment method provided in an embodiment of the present application;
[0045] Figure 2 It is a flow chart of the distributed deployment method provided in the embodiment of the present application;
[0046] Figure 3 It is a flow chart of the distributed deployment method provided in the embodiment of the present application;
[0047] Figure 4 It is a flow chart of the distributed deployment method provided in the embodiment of the present application;
[0048] Figure 5 It is a flow chart of the distributed deployment method provided in the embodiment of the present application;
[0049] Figure 6 is a flow chart of a distributed deployment method provided in an embodiment of the present application; and
[0050] Figure 7 It is a module schematic diagram of the distributed deployment device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.
[0052] For ease of understanding, the application scenarios of this application are first introduced below:
[0053] See also Figure 1 , Figure 1 1 is an application scenario diagram of a distributed deployment method provided in an embodiment of the present application. The application scenario provided in the present application includes a terminal device 101 and a server 102, and the distributed deployment method provided in the present application can be executed by at least one of the terminal device 101 and the server 102.
[0054] The terminal equipment may include but is not limited to:
[0055] (1) Internet of Things devices: smart meters, robots, sensor devices (such as temperature sensors, humidity sensors, light sensors, etc.), actuator devices (smart light bulbs, smart sockets, electronic locks, etc.), embedded devices (smart TVs, smart speakers, smart refrigerators, etc.), vehicle-mounted devices (smart vehicles, vehicle-mounted navigation systems, etc.), etc., which are not limited to the embodiments of the present application.
[0056] (2) Other devices: smart phones (such as Android phones, IOS phones, etc.), tablet computers, laptops, desktop computers, smart speakers, smart watches, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., which are not limited to the embodiments of the present application.
[0057] The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but the embodiments of the present application do not limit this.
[0058] In the power system scenario, the server is used to obtain power data collected by various terminal devices, and process the power data through a preset neural network model used to implement various power system functions, thereby realizing corresponding power system functions, such as electricity price forecasting, power consumption anomaly detection, etc.
[0059] It should be noted that Figure 1 The number of terminal devices and servers is only used as an example, and the number of terminal devices and servers can be more or less, which is not limited here. The terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.
[0060] The distributed deployment method involved in this application can be implemented based on cloud technology.
[0061] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data calculation, storage, processing and sharing.
[0062] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.
[0063] The distributed deployment method of the present application can be implemented based on cloud computing. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.
[0064] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.
[0065] The distributed deployment method in the embodiment of the present application can be executed by a chip, and the chip can be installed in at least one of the server and the terminal device, that is, the method can be executed by the server or the terminal device alone, or by the server and the terminal device together. Therefore, the execution subject of each step will not be repeatedly described below.
[0066] Based on the introduction of the above application scenarios, the embodiment of the present application provides a distributed deployment method, which is described in detail below:
[0067] See also Figure 2 A distributed deployment method provided in an embodiment of the present application is implemented by steps 011 to 013, which are described in detail below.
[0068] Step 011: Divide each layer of the network structure into p modules to be deployed through the model deployment node, and the modules to be deployed include one or more layers of the network structure.
[0069] The model deployment node may be one or more servers in the above application scenarios.
[0070] The model to be deployed may be a neural network model. The module to be deployed is a part of the model to be deployed, and different modules to be deployed are used to implement different data processing flows of the model to be deployed.
[0071] Among them, the neural network model is a computational model that imitates the structure and function of the biological nervous system and is used to recognize patterns, make decisions, or solve complex input-output mapping problems. It consists of a large number of artificial neurons (nodes to be deployed), which interact with each other through weighted connections to form a multi-layer network structure. The core components of the neural network model include the input layer, hidden layer, and output layer, as well as the connection weights and activation functions between them.
[0072] Optionally, based on the network structure of the model to be deployed, the model to be deployed may be divided into p modules to be deployed, where p is a positive integer.
[0073] For example, each layer of the network structure can be divided into p modules to be deployed according to the number of network layers included in the model to be deployed, and the modules to be deployed include one or more layers of the network structure.
[0074] It can be understood that the model to be deployed processes the input data in turn through each layer of the network until the final processing result is output. Therefore, by segmenting the network layer of the model to be deployed, the operation of the model can be maintained without affecting the model while achieving model segmentation.
[0075] After the model to be deployed is divided into p modules to be deployed, the p modules to be deployed can be deployed to different nodes to be deployed respectively, so as to achieve pipeline parallelism between layers. After a batch of data is input into the model, different modules to be deployed in the model process the data they need to process respectively, so as to achieve pipeline parallelism.
[0076] For the same input data, p modules to be deployed need to process it in sequence, that is, the next module to be deployed needs to wait for the output of the previous module to be deployed to output the output result of the input data. For different input data, they do not interfere with each other. For example, after the first module to be deployed processes one input data, it can immediately process the next input data. When the second module to be deployed processes the data output by the first module to be deployed, the first module to be deployed can process other input data in parallel. This improves the parallelism of each node to be deployed and improves the utilization rate of each node to be deployed.
[0077] Step 012: Split the tensor of the model to be deployed into t split tensors through the model deployment node, where t is a positive integer.
[0078] Among them, a tensor is a multidimensional array, which is a core concept in modern mathematics, physics, and computer science, especially in the fields of machine learning and deep learning. A tensor can be regarded as a generalized concept of scalars, vectors, and matrices, which allows data organization and operation in higher dimensions.
[0079] Specifically, tensor parallelism can be achieved by splitting the large tensor in the model (usually the tensor of weights or activation functions), and then having multiple nodes to be deployed process the split tensors in parallel.
[0080] The key idea of tensor parallelism is to split the model parameters or intermediate activation tensors along a certain dimension (for example, along the rows or columns of the weight matrix), and then calculate them in parallel on multiple nodes to be deployed. The advantage of doing so is that it can reduce the memory requirements on each device and make full use of the performance of each node to be deployed, thereby improving the utilization rate of each node to be deployed.
[0081] Step 013: Based on p modules to be deployed and t segmentation tensors, the model deployment node deploys the model to be deployed to N nodes to be deployed, where N=p*t.
[0082] The nodes to be deployed include devices with data processing capabilities. The nodes to be deployed and the model deployment nodes together constitute a computer system.
[0083] For example, a computer system includes multiple servers in an application scenario, some of which are used as model deployment nodes, and the other servers are used as nodes to be deployed. Alternatively, the servers used as model deployment nodes can also be used as nodes to be deployed.
[0084] Optionally, the nodes to be deployed include a graphics processing unit (GPU), a neural network processor unit (NPU), a tensor processing unit (TPU), a central processing unit (CPU), etc.
[0085] Optionally, the number of nodes to be deployed can be determined based on the number of available devices, specifically based on the number of devices and the number of processing units (such as GPU, NPU, TPU, CPU, etc.) in each device that can independently perform data processing, that is, the number of nodes to be deployed is the number of processing units that can independently perform data processing.
[0086] Specifically, after performing model segmentation between layers and tensor segmentation within a layer, the total number of segmented parts, i.e., p*t, can be obtained. Each segmented part can be deployed to a node to be deployed, i.e., the required number of nodes to be deployed N=p*t.
[0087] In this way, through inter-layer segmentation and intra-layer segmentation, each segmented part that can be fully deployed to the available N nodes to be deployed is obtained, so that each segmented part is deployed to the N nodes to be deployed one by one, so that each node to be deployed runs one of the segmented parts, realizing distributed deployment.
[0088] Optionally, the value of t is proportional to the communication occupancy time of the model to be deployed, and the communication occupancy time is less than a predetermined threshold.
[0089] For the models to be deployed, due to different real-time requirements, the communication occupancy time when the model is allowed to process data is also different. By setting the communication occupancy time of the model to be deployed, the parameters of the tensor splitting of the model to be deployed (that is, the number of split tensors of the tensor splitting) can be configured.
[0090] It can be understood that the more the number t of tensor splits, the more GPUs need to communicate to synchronize data at the end of each round of synchronization, and the more time is spent on communication. Therefore, by setting the communication time of the model to be deployed to be less than the predetermined threshold, the number t of split tensors can be limited, thereby ensuring that the real-time performance of the model after deployment will not be affected by the long communication time.
[0091] Optionally, 4≤t≤8. In this way, by setting the number t of split tensors within a reasonable range, it can be ensured that the communication time of the model will not be too long.
[0092] The distributed deployment method provided in the embodiment of the present application realizes pipeline parallelism between layers by dividing the model to be deployed into p modules to be deployed, and realizes tensor parallelism within the layer by dividing the tensor of the model to be deployed into t divided tensors. It can be understood that the larger the number of tensor parallelism, the longer the communication occupancy time, and the larger the number of pipeline parallelism, the longer the idle time of the node to be deployed is in waiting. Therefore, while ensuring that all nodes to be deployed are utilized, the communication time and idle time can be balanced by adjusting the ratio of pipeline parallelism and tensor parallelism, thereby improving the utilization rate of hardware.
[0093] See also Figure 3 In some embodiments, the distributed deployment method further includes:
[0094] Step 014: Obtain a preset deployment configuration model, where the deployment configuration model determines p, t, and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed, and the proportion of cavitation time.
[0095] The deployment configuration model may be a neural network model. The deployment configuration model is a pre-trained model used to automatically set the configuration of distributed deployment (ie, p, t, and batch number).
[0096] In the context of machine learning and deep learning, batch size refers to the number of data samples fed into the model for training in one iteration. Batch size is an important hyperparameter that affects the training speed, memory usage, and convergence process of the model.
[0097] Specifically, the deployment configuration model can comprehensively determine p, t and the number of batches by inputting at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed and the number of nodes to be deployed, and by calculating the proportion of cavitation time.
[0098] For example, the deployment configuration model can first determine multiple groups of combinations of p, t and batch numbers by inputting at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, and the number of nodes to be deployed, and then calculate the cavitation time proportion corresponding to each combination, so as to determine the combination with the smallest cavitation time proportion or less than a preset threshold (such as 5%, 10%, etc.) as the final output.
[0099] Optionally, p, t and the number of batches form a set of configuration parameters, the configuration parameters include multiple groups, and the deployment configuration model is also used to determine the calculation time and communication time of each module to be deployed corresponding to each group of p, t and batch number, and to determine the target configuration parameters that minimize the sum of the calculation time and the communication time among the multiple groups of configuration parameters.
[0100] Specifically, there may be multiple groups of configuration parameter combinations whose cavitation time ratio output by the deployment configuration model is the smallest or less than a preset threshold (such as 5%, 10%, etc.). At this time, in order to further improve the performance of distributed deployment, the calculation time and communication time of each module to be deployed corresponding to each group of p, t and batch number can be further determined, so as to determine the target configuration parameters that minimize the sum of the calculation time and the communication time. In this way, the final target configuration parameters can not only reduce the cavitation time, but also minimize the total time of the calculation time and the communication time.
[0101] The specific calculation process of the calculation time of each module to be deployed corresponding to each group of p, t and batch number is as follows:
[0102] After determining p, the division of pipeline stages is completed, and the operators contained in each pipeline stage are also determined, so that the number of linear operation units and attention operation units can be determined. Based on p and t, the operation time of the linear operation unit can be estimated, which includes multiplication time, data transmission time, memory data access time, etc.; based on p and t, the operation time of the attention operation unit can also be estimated, which includes the operation time of the attention operation after using p and t to split the dimensions related to the input matrix of the attention operation. In this way, based on the operation time of all linear operation units and attention operation units of each module to be deployed, the calculation time of each module to be deployed can be determined respectively, thereby determining the total calculation time of each module to be deployed.
[0103] The specific calculation process of the communication duration of each module to be deployed corresponding to each group of p, t and batch number is as follows:
[0104] The communication time between adjacent pipeline stages can be estimated by using prior knowledge (for example, estimating the communication time based on the size of the matrix known in advance) or by using a timer to measure the simulated communication time (for example, simulating the data transmission process between pipeline stages according to the data scale and measuring the communication time). For example, based on the parallelization scheme and the dimension of the result matrix of the pipeline stage, the dimension of this result matrix can indicate the output data volume of the pipeline stage, and multiple communication durations can be determined by measurement or estimation, and the communication duration represents the communication time between adjacent pipeline stages; the total communication duration between pipeline stages can be determined based on the multiple communication durations.
[0105] In this way, by measuring or estimating the communication time between adjacent pipeline stages, we can more accurately understand the communication time of the entire processing process when the model performs inference tasks (that is, the total communication time of each module to be deployed).
[0106] See also Figure 4 Optionally, step 014 includes:
[0107] Step 0141: Obtain a training set, where the training set includes p, t, and batch numbers corresponding to models of different complexity under different deployment configurations. The deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed, and the proportion of cavitation time.
[0108] Step 0142: Perform model training based on the training set to obtain a model to be deployed.
[0109] The complexity of the model to be deployed may be determined based on at least one of the number of parameters and the amount of computation of the model to be deployed, and the hardware performance of the node to be deployed may be determined based on the data processing performance of the node to be deployed.
[0110] It can be understood that the larger the number of parameters of the model to be deployed, the greater the complexity of the model to be deployed. The greater the amount of computation required for the model to be deployed, the greater the complexity of the model to be deployed. The stronger the data processing performance of the node to be deployed, the stronger the hardware performance of the node to be deployed. For example, the stronger the GPU data processing performance, the stronger the hardware performance of the node to be deployed.
[0111] Specifically, in order to obtain a deployment configuration model, model training is required in advance. During training, a training set that meets the requirements is first obtained. The training set includes p, t, and batch numbers corresponding to models of different complexity under different deployment configurations, and the models deployed in the training set can achieve results that meet user requirements after deployment, such as balancing communication time and idle time. In this way, by training the model with the training set, a deployment configuration model that can determine p, t, and batch numbers based on at least one of the complexity of the model to be deployed, the hardware performance of the node to be deployed, the number of nodes to be deployed, and the proportion of cavitation time can be obtained, thereby realizing automated distributed deployment configuration.
[0112] Optionally, the void time ratio is determined based on the forward execution time of the model to be deployed for each batch, the number of batches, and p, and the void time ratio is less than a preset threshold.
[0113] Optionally, the preset threshold is an empirical value, such as a preset threshold of 5%, 10%, 15%, etc.
[0114] For example, the void time ratio is determined based on the void time and the predicted execution time. The void time is determined based on the forward execution time of each batch and the number of modules to be deployed in the waiting state. The predicted execution time is determined based on the forward execution time of each batch and p.
[0115] Specifically, the cavitation time ratio can be determined based on the following formula:
[0116]
[0117] t bubble =(p-1)×t f ;
[0118] t ideal =m×t f ;
[0119] Where, bubbleration is the proportion of cavitation time, t bubble is the cavitation time, t ideal is the predicted execution time, t f is the forward execution time for each batch, and m is the number of batches.
[0120] It can be understood that when the batch number m is larger, the proportion of vacancy time will be smaller, so that the idle time of the nodes to be deployed is less, which can improve the hardware utilization of the nodes to be deployed.
[0121] Optionally, the number of batches of the model to be deployed is greater than or equal to 4p. That is, m is greater than or equal to 4p.
[0122] In this way, by reasonably setting the batch number m so that the batch number is more than 4 times the number of modules to be deployed, it is helpful to reduce idle time.
[0123] It is understandable that during the training process of the model to be deployed, the model output can be restricted to accelerate the convergence speed of the model. For example, limit m to be greater than or equal to 4p, 4≤t≤8.
[0124] See also Figure 5 In some embodiments, before the model to be deployed is deployed to the N nodes to be deployed, the method further includes:
[0125] Step 015: Perform operator fusion on the operators of the module to be deployed, so as to merge multiple consecutive operations or operators into a single operator;
[0126] Step 013: Based on the p modules to be deployed and the t segmentation tensors, the model to be deployed is deployed to the N nodes to be deployed, including:
[0127] Step 0131: Based on the p modules to be deployed and t segmented tensors after operator fusion, the model to be deployed is deployed to N nodes to be deployed.
[0128] Specifically, in order to improve model performance, operators of the modules to be deployed can be fused to merge multiple consecutive operations or operators into a single operator.
[0129] Among them, operator fusion is a common optimization technique, which is often used in deep learning and other computationally intensive tasks. The basic operation is to merge multiple consecutive operations or operators into a single operator to reduce computational and memory overhead. Using operator fusion is very helpful to improve execution efficiency, reduce the need for data transmission and temporary storage, and improve cache utilization. In fact, it is a direct optimization of "less data movement and more computation", which achieves more utilization of computing resources in this way.
[0130] After the operator fusion is performed, the distributed deployment of the model to be deployed can be realized based on the p modules to be deployed and t segmentation tensors after the operator fusion, and the model to be deployed can be deployed to N nodes to be deployed.
[0131] See also Figure 6In some embodiments, before the model to be deployed is deployed to the N nodes to be deployed, the method further includes:
[0132] Step 016: quantize the models to be deployed to obtain p quantized models to be deployed;
[0133] Step 013: Based on the p modules to be deployed and the t segmentation tensors, the model to be deployed is deployed to the N nodes to be deployed, including:
[0134] Step 0132: Based on the quantized p modules to be deployed and t segmentation tensors, the model to be deployed is deployed to N nodes to be deployed.
[0135] Specifically, it can be understood that in order to improve the performance of the model to be deployed, before deployment, in addition to performing operator fusion on the module to be deployed, the model to be deployed can also be quantized.
[0136] Among them, quantization refers to the process of approximating the continuous value of a signal to a finite number of discrete values. It can be understood as a method of information compression. When considering this concept in a computer system, it is generally represented by "low bits". Some people also call quantization "fixed-point", but strictly speaking, the range of representation is reduced. Fixed-point refers specifically to linear quantization with a scale of 2, which is a more practical quantization method.
[0137] In this way, by quantizing the model, model quantization can alleviate problems such as large number of parameters, large amount of calculation, and high memory usage of the neural network model.
[0138] After the model is quantized, distributed deployment can be achieved based on the quantized p modules to be deployed and t partitioned tensors, and the model to be deployed can be deployed to N nodes to be deployed.
[0139] According to the method described in the above embodiment, the present application embodiment also provides a distributed deployment device 300, which is used to execute the steps in the above distributed deployment method. Figure 7 , Figure 7 : is a schematic diagram of the structure of a distributed deployment device 300 provided in an embodiment of the present application. The distributed deployment device 300 includes:
[0140] The model segmentation module 301 is used to segment each layer of the network structure of the model into p modules to be deployed through the model deployment node, and the modules to be deployed include one or more layers of network structure;
[0141] A tensor splitting module 302 is used to split the tensor of the model to be deployed into t split tensors through the model deployment node;
[0142] The deployment module 303 is used for the model deployment node to deploy the model to be deployed to N nodes to be deployed based on p modules to be deployed and t segmentation tensors, where N=p*t, and the nodes to be deployed include devices with data processing capabilities.
[0143] In some embodiments, the model segmentation module 301 is specifically used to segment each layer of the network structure into p modules to be deployed, and the modules to be deployed include one or more layers of the network structure.
[0144] In some embodiments, the device 300 also includes an acquisition module 304, which is used to obtain a preset deployment configuration model, and the deployment configuration model determines p, t and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time.
[0145] In some embodiments, the acquisition module 304 is specifically used to obtain a training set, which includes p, t and batch numbers corresponding to models of different complexities under different deployment configurations, and the deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time; model training is performed based on the training set to obtain the model to be deployed.
[0146] In some embodiments, the device also includes an operator fusion module 305; the operator fusion module 305 is used to perform operator fusion on the operators of the modules to be deployed before the models to be deployed are deployed to N nodes to be deployed, so as to merge multiple consecutive operations or operators into a single operator; the deployment module 303 is specifically used to deploy the model to be deployed to N nodes to be deployed based on the p modules to be deployed and t segmentation tensors after operator fusion.
[0147] In some embodiments, the device also includes a quantization module 306, which is used to quantize the model to be deployed before the model to be deployed to N nodes to be deployed, so as to obtain p quantized models to be deployed; the deployment module 303 is specifically used to deploy the model to be deployed to N nodes to be deployed based on the quantized p modules to be deployed and t segmentation tensors.
[0148] It should be noted that the specific details of each module unit in the above-mentioned distributed deployment device 300 have been described in detail in the embodiment of the above-mentioned distributed deployment method, and will not be repeated here.
[0149] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0150] In some embodiments, the distributed deployment device in the embodiments of the present application can be implemented in hardware, such as a chip, or a component in a chip, such as an integrated circuit or a chip; the distributed deployment device can also be implemented in software, such as as an application installed in a chip.
[0151] In some embodiments, the chip of the embodiment of the present application can execute each process of the embodiment of the above-mentioned distributed deployment method and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0152] Among them, chips, also known as integrated circuits (ICs), microcircuits or microchips, are the core components of modern electronic devices. They are made of semiconductor materials (usually silicon) and integrate millions or even billions of microelectronic components (such as transistors, resistors, capacitors, etc.) through a series of complex manufacturing processes to form an electronic circuit with specific functions. These components and their interconnections are manufactured on the same small silicon wafer. They are small in size, light in weight, but powerful in function. They are the basis of various electronic products such as computers, smart phones, home appliances, automobiles, medical equipment, etc.
[0153] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the embodiment of the above-mentioned distributed deployment method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0154] The processor may be the chip in the above embodiment. The computer readable storage medium may be a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0155] Computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 504 and mass storage device 507 can be collectively referred to as memory.
[0156] The embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned distributed deployment method when executed by a processor. The processor may be a chip in the above-mentioned embodiment. When the computer program is executed by the processor, each process of the embodiment of the above-mentioned distributed deployment method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0157] It is understandable that in the specific implementation of this application, data related to user identity or characteristics is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0158] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.
Claims
1. A distributed deployment method of a model, characterized in that: Applied to a computer system, the computer system includes a model deployment node and a node to be deployed, the model includes a multi-layer network structure, and the method includes: Divide each layer of the network structure into p modules to be deployed through a model deployment node, wherein the modules to be deployed include one or more layers of the network structure; Splitting the tensor of the model to be deployed into t split tensors through the model deployment node; Based on the p modules to be deployed and the t segmented tensors, the model deployment node deploys each of the models to be deployed to N nodes to be deployed, N=p*t, and the nodes to be deployed include devices with data processing capabilities.
2. The distributed deployment method according to claim 1, characterized in that: Also includes: A preset deployment configuration model is obtained, wherein the deployment configuration model determines p, t, and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed, and the proportion of cavitation time.
3. The distributed deployment method according to claim 2, characterized in that: The p, t and batch number form a set of configuration parameters, and the configuration parameters include multiple groups. The deployment configuration model is also used to determine the calculation time and communication time of each module to be deployed corresponding to each group of the p, t and batch number, and to determine the target configuration parameters that minimize the sum of the calculation time and the communication time.
4. The distributed deployment method according to claim 3, characterized in that: The obtaining of the preset deployment configuration model includes: Obtain a training set, wherein the training set includes p, t, and batch numbers corresponding to models of different complexity under different deployment configurations, wherein the deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed, and the proportion of cavitation time; Model training is performed based on the training set to obtain the model to be deployed.
5. The distributed deployment method according to any one of claims 2 to 4, characterized in that: The void time ratio is determined based on the forward execution time of the model to be deployed for each batch, the number of batches, and p, and the void time ratio is less than a preset threshold.
6. The distributed deployment method according to claim 5, characterized in that: The cavitation time proportion is determined based on the cavitation time and the predicted execution time. The cavitation time is determined based on the forward execution time of each batch and the number of the modules to be deployed in the waiting state. The predicted execution time is determined based on the forward execution time of each batch and p.
7. The distributed deployment method according to claim 6, characterized in that: The cavitation time ratio is determined based on the following formula: t bubble =(p-1)×t f ; t ideal =m×t f ; Where, bubbleration is the proportion of cavitation time, t bubble is the cavitation time, t ideal is the predicted execution time, t f is the forward execution time for each batch, and m is the number of batches.
8. The distributed deployment method according to claim 3 or 4, characterized in that: The complexity of the model to be deployed is determined based on at least one of the parameter quantity and the computational complexity of the model to be deployed, and the hardware performance of the node to be deployed is determined based on the data processing performance of the node to be deployed.
9. The distributed deployment method according to any one of claims 1 to 8, characterized in that: The batch number of the model to be deployed is greater than or equal to 4p.
10. The distributed deployment method according to any one of claims 1 to 8, characterized in that: 4≤t≤8。 11. The distributed deployment method according to claim 1 or 10, characterized in that: The value of t is proportional to the communication occupancy time of the model to be deployed, and the communication occupancy time is less than a predetermined threshold.
12. The distributed deployment method according to claim 1, characterized in that: Before the to-be-deployed model is deployed to the N to-be-deployed nodes, the method further includes: Performing operator fusion on the operators of the module to be deployed, so as to merge multiple consecutive operations or operators into a single operator; The deploying the model to be deployed to N nodes to be deployed based on the p modules to be deployed and the t segmentation tensors includes: Based on the p modules to be deployed and the t segmented tensors after operator fusion, the model to be deployed is deployed to N nodes to be deployed.
13. The distributed deployment method according to claim 1 or 12, characterized in that: Before the to-be-deployed model is deployed to the N to-be-deployed nodes, the method further includes: quantizing the model to be deployed to obtain p quantized models to be deployed; The deploying the model to be deployed to N nodes to be deployed based on the p modules to be deployed and the t segmentation tensors includes: Based on the quantized p modules to be deployed and the t segmentation tensors, the model to be deployed is deployed to N nodes to be deployed.
14. A distributed deployment device, characterized in that: Applied to a computer system, the computer system includes a model deployment node and a node to be deployed, and the device includes: A model segmentation module, used to segment each layer of the network structure of the model into p modules to be deployed through the model deployment node, wherein the modules to be deployed include one or more layers of the network structure; A tensor segmentation module, used to segment the tensor of the model to be deployed into t segmented tensors through a model deployment node; A deployment module is used for, based on the p modules to be deployed and the t segmentation tensors, the model deployment node deploys the model to be deployed to N nodes to be deployed, N=p*t, and the nodes to be deployed include devices with data processing capabilities.
15. The distributed deployment device according to claim 14, characterized in that: The device also includes an acquisition module, which is used to obtain a preset deployment configuration model, and the deployment configuration model determines p, t and the number of batches based on at least one of the complexity of the model to be deployed, the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time.
16. The distributed deployment device according to claim 15, characterized in that: The acquisition module is specifically used to acquire a training set, wherein the training set includes p, t and batch numbers corresponding to models of different complexity under different deployment configurations, wherein the deployment configuration includes at least one of the hardware performance of the nodes to be deployed, the number of nodes to be deployed and the proportion of cavitation time; Model training is performed based on the training set to obtain the model to be deployed.
17. The distributed deployment device according to claim 14, characterized in that: The device further includes an operator fusion module; the operator fusion module is used to perform operator fusion on the operators of the module to be deployed before the model to be deployed is deployed to the N nodes to be deployed, so as to merge multiple consecutive operations or operators into a single operator; The deployment module is specifically used to deploy the model to be deployed to N nodes to be deployed based on the p modules to be deployed after operator fusion and the t segmentation tensors.
18. The distributed deployment device according to claim 14 or 17, characterized in that: The device also includes a quantization module, which is used to quantize the model to be deployed before the model to be deployed to N nodes to be deployed, so as to obtain p quantized models to be deployed; the deployment module is specifically used to deploy the model to be deployed to N nodes to be deployed based on the quantized p modules to be deployed and the t segmentation tensors.
19. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed deployment method as described in any one of claims 1 to 13 is implemented.
20. A chip, characterized in that: The chip is used to implement the distributed deployment method as described in any one of claims 1-13.
21. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the distributed deployment method as described in any one of claims 1 to 13.