A model acceleration method, apparatus and related device

Abstract

The present application provides a model acceleration method, device and related equipment, wherein the model acceleration method comprises: in the deployment strategy corresponding to the model, deploying the forward calculation of a certain computing unit in the second iteration round to the bubble time of the computing unit in the first iteration round, so that the computing unit completes the forward calculation of the second iteration round in the bubble time of the first iteration round, wherein the second iteration round is an iteration round after the first iteration round, and the bubble time of the first iteration round is the time between the forward calculation and the backward calculation in the first iteration round. In the above method, for each computing unit participating in model training, by advancing the forward calculation in the subsequent iteration round to the bubble time of the current iteration round for processing, the idle time of the computing unit is utilized to process future tasks, thereby reducing the idle time of the computing resource, and improving the utilization rate of the computing resource.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model acceleration method, apparatus and related equipment. Background Technology

[0002] With the continuous and in-depth development of artificial intelligence technology, the scale and complexity of model training are increasing daily, and the computing and memory resources required for model training are also growing exponentially. Multimodal models, as a highly challenging type of model in the field of artificial intelligence, are used to process data from different modalities (such as text, images, speech, and video). For example, in industrial manufacturing scenarios, multimodal models can identify incorrectly sized parts from multiple images corresponding to an assembly line to ensure the quality of the final product.

[0003] Currently, multimodal model training employs a distributed training approach, where multiple computing units (multiple processors or nodes in an inference cluster) corresponding to the multimodal model process multiple datasets in parallel. Because these multimodal models process data with varying sizes and resolutions (e.g., different images require different memory sizes and resolutions), the computation time required by each computing unit in parallel processing is not entirely the same. Furthermore, due to the requirement for gradient data synchronization during multimodal model training, it is highly likely that in a given iteration, some computing units will finish processing their assigned data and then become idle while waiting for other computing units to process it. This situation leads to idle and wasted computing resources. Summary of the Invention

[0004] This application provides a model acceleration method, apparatus, and related equipment. For each computing unit participating in model training, by advancing the forward computation in subsequent iterations to the idle time of the current iteration, the current idle time of the computing unit is used to process future tasks, thereby reducing the idle time of computing resources and improving the utilization rate of computing resources.

[0005] In a first aspect, this application provides a model acceleration method, the method comprising: obtaining an initial deployment strategy for the model, the initial deployment strategy instructing a first computing unit to perform computation using the model in a first iteration round and a second iteration round; in the initial deployment strategy, deploying the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round, so that the first computing unit completes the forward computation of the second iteration round during the vacuolation time of the first iteration round, wherein the second iteration round is after the first iteration round, and the vacuolation time of the first iteration round is the time between the forward computation and the backward computation in the first iteration round.

[0006] In the above scheme, both the first and second iterations are iterations for model training. An iteration is the number of times data is completely transmitted within the model and the model parameters are updated. After each iteration, the system updates the model parameters based on the backpropagation results corresponding to multiple data points in that iteration. As the number of iterations increases, the model parameters gradually adapt to the characteristics and patterns of the input data, improving the model's accuracy and generalization ability. When multiple computational units run the model iteratively, due to the requirement of gradient data synchronization, the model parameters need to be updated based on the backpropagation results of each computational unit in that iteration, thereby improving the correctness of the updated model parameters.

[0007] In the above scheme, forward computation refers to the process of calculating the input data according to the operational rules defined by the model. Corresponding to forward computation is backward computation, which refers to the process of calculating the gradient of the model parameters based on the error between the calculation result obtained from forward computation and the actual calculation result. The system can adjust the model parameters according to the gradient of the model parameters, thereby reducing the error between the model's forward computation result and the actual calculation result. In one iteration, the computation unit first uses the model to perform forward computation on the data, and then performs backward computation.

[0008] Gradient data synchronization refers to the synchronized integration of the back-calculation results from multiple computational units within a single iteration of a model. In other words, gradient data synchronization synchronizes and integrates the gradients corresponding to multiple model parameters calculated by multiple computational units in a single iteration, thereby uniformly adjusting the model parameters and ensuring consistency across the multiple computational units when running the model. Using only the back-calculation results from a subset of computational units for synchronization and parameter adjustment will cause the updated model parameters to deviate from the true values, thus affecting the model's analytical and predictive capabilities. As described above, gradient data synchronization occurs after the back-calculation of multiple computational units in each iteration.

[0009] Currently, when multiple computing units are running a model for iteration, due to the requirement of gradient data synchronization, in one iteration, the computing units that have finished processing first need to wait for the unprocessed units. This means the previously processed units enter an idle state, generating idling time and reducing the utilization of computing resources. Implementing the scheme described in the first aspect involves processing each computing unit participating in model training during the idling time of the current iteration, thus utilizing the unit's current idle time to handle future tasks. This reduces idle computing resources and improves their utilization.

[0010] This explains that since gradient data synchronization integrates the results of backward computation by multiple computation units in one iteration, the gradient data synchronization occurs after the backward computation of multiple computation units in each iteration. Therefore, each computation unit can process the forward computation in subsequent iterations in advance, leaving the backward computation in subsequent iterations.

[0011] In one possible implementation of the first aspect, the initial deployment strategy further instructs the second computing unit to perform calculations using the model in the first and second iteration rounds. The method further includes: allocating input data for the calculations of the first and second computing units in multiple iteration rounds, wherein in the same iteration round, the calculation time corresponding to the first input data allocated to the first computing unit is less than the calculation time corresponding to the second input data allocated to the second computing unit, wherein the calculation time includes forward calculation time and backward calculation time, and the multiple iteration rounds include the first iteration round and the second iteration round.

[0012] In the current scheme, the computation time corresponding to the data processed by multiple computing units in different iteration rounds varies, resulting in uncontrollable cavitation time for each computing unit in multiple iteration rounds, making it difficult to utilize cavitation time. For example, in the first iteration round, the first computing unit has cavitation time of 2 seconds, while the second computing unit has no cavitation time; in the second iteration round, the first computing unit has no cavitation time, while the second computing unit has cavitation time of 3 seconds.

[0013] By implementing the above scheme, in the same iteration round, data with longer computation time is deliberately allocated to the second computing unit, and data with shorter computation time is also allocated to the second computing unit. This causes the first computing unit to experience idle time while waiting for the second computing unit in multiple iteration rounds. By deploying the forward computation of the first computing unit in subsequent iteration rounds to these idle times, the idle computing resources are reduced and the utilization rate of computing resources is improved.

[0014] In one possible implementation of the first aspect, allocating input data for the computation of the first computing unit and the second computing unit in multiple iterations includes: obtaining the load characteristics of each data in multiple data, wherein the multiple data are data that the model needs to process, and the load characteristics include forward computation time and backward computation time; dividing the multiple data into multiple clusters based on the load characteristics of the multiple data, the multiple clusters including a first cluster and a second cluster; selecting a portion of data from the first cluster as first input data, and selecting a portion of data from the second cluster as second input data.

[0015] By implementing the above scheme, the load characteristics (time required for forward computation and time required for backward computation) of the data participating in model training are obtained. Then, the data is divided into multiple clusters according to the load characteristics. The load characteristics of the data in each cluster are similar. Finally, the data in different clusters are assigned to different computing units, thereby ensuring that the computation time corresponding to the data processed by each computing unit in different iteration rounds remains relatively constant, thus ensuring the generation of cavitation time.

[0016] For example, after dividing multiple data into two clusters (a first cluster and a second cluster), where the computation time corresponding to the data in the first cluster is smaller and the computation time corresponding to the data in the second cluster is larger, by allocating the data in the first cluster to a first computing unit and the data in the second cluster to a second computing unit, the first computing unit will experience cavitation time in each iteration due to waiting for the second computing unit.

[0017] This means that when allocating data from the first cluster to the first computing unit and data from the second cluster to the second computing unit, the amount of data allocated to the first computing unit is equal to the amount of data allocated to the second computing unit each time.

[0018] In one possible implementation of the first aspect, allocating input data for the computation of the first computing unit and the second computing unit in multiple iterations includes: obtaining the load characteristics of each data in the multiple data, wherein the multiple data are the data that the model needs to process, and the load characteristics include forward computation time and backward computation time; dividing the multiple data into multiple clusters based on the load characteristics of the multiple data, the multiple clusters including a first cluster and a second cluster; dividing the data in the first cluster into multiple micro-batches, and dividing the data in the second cluster into multiple micro-batches; using the multiple micro-batches corresponding to the first cluster as first input data, and using the multiple micro-batches corresponding to the second cluster as second input data, wherein the first computing unit and the second computing unit process one micro-batch respectively in one iteration.

[0019] In one possible implementation of the first aspect, after dividing the multiple data into multiple clusters based on the load characteristics of multiple data, the method further includes: calculating the distance vector of each cluster in the multiple clusters, where the distance vector is the vector from the center point of the cluster to the farthest point or from the farthest point to the center point; if the magnitude of the distance vector of the third cluster is greater than a first threshold, establishing a fourth cluster with the farthest point as the initial center point, and the multiple clusters include the third cluster; if the magnitude of the distance vector of the third cluster is less than the first threshold, updating the center point of the third cluster until the magnitude of the distance vector of the third cluster is greater than or equal to the first threshold.

[0020] By implementing the above scheme, after dividing multiple data into multiple clusters based on the load conditions of multiple data, the appropriateness of the multiple clusters is determined by judging the magnitude of the vector from the center point to the farthest point or from the farthest point to the center point in each cluster. If it is determined that the multiple clusters are inappropriate, the division method is adjusted in time to improve the accuracy of clustering.

[0021] In one possible implementation of the first aspect, the load characteristics also include one or more of memory loading time, memory unloading time, and dynamic memory usage, wherein memory loading time is the time required to load the computation results of the forward computation; memory unloading time is the time required to unload the computation results of the forward computation; and dynamic memory usage is the memory size occupied by the computation results of the forward computation.

[0022] Implementing the above scheme involves unloading and loading computation results during the computational unit's data processing. For example, when the computational unit's cache is limited, it can unload the results of the forward computation to other memory locations after completing the forward computation of a certain data point, and then load the results back from those memory locations before performing the reverse computation of that data. This approach also improves clustering accuracy by using more diverse load features to cluster multiple datasets.

[0023] In one possible implementation of the first aspect, deploying the forward computation of the first computing unit in the second iteration round to the bubbling time of the first computing unit in the first iteration round includes: obtaining a first time, a second time, and a third time, wherein the first time is the time required for the first computing unit to perform forward and backward computation on the input data using the model in the first iteration round; the second time is the time required for the second computing unit to perform forward and backward computation on the input data using the model in the first iteration round; the third time is the time required for the first computing unit to perform forward computation on the input data using the model in the second iteration round; based on the third time being less than or equal to the difference between the second time and the first time, deploying the forward computation of the first computing unit in the second iteration round to the bubbling time of the first computing unit in the first iteration round.

[0024] Implementing the above scheme, before deploying the forward computation of the first computing unit in the second iteration into the vacuolation time of the first computing unit in the first iteration, it is determined whether the vacuolation time of the first computing unit in the first iteration is greater than or equal to the forward computation time of the second iteration. If it is determined that the vacuolation time of the first computing unit in the first iteration is greater than or equal to the forward computation time of the second iteration, then the forward computation of the first computing unit in the second iteration is deployed into the vacuolation time of the first computing unit in the first iteration. This avoids the situation where, after deploying the forward computation of the first computing unit in the second iteration into the vacuolation time of the first computing unit in the first iteration, the total time required by the first computing unit in the first iteration exceeds the time required by the second computing unit in the first iteration, thus preventing the second computing unit from experiencing vacuolation time in the first iteration.

[0025] In one possible implementation of the first aspect, deploying the forward computation of the first computing unit in the second iteration round to the bubbling time of the first computing unit in the first iteration round includes: acquiring a first time, a second time, a third time, and a fourth time, wherein the first time is the time required for the first computing unit to perform forward and backward computation on the input data using the model in the first iteration round; the second time is the time required for the second computing unit to perform forward and backward computation on the input data using the model in the first iteration round; the third time is the time required for the first computing unit to perform forward computation on the input data using the model in the second iteration round; and the fourth time is the time required for the first computing unit to unload the forward computation result of the input data in the second iteration round; based on the fact that the sum of the third time and the fourth time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration round is deployed to the bubbling time of the first computing unit in the first iteration round.

[0026] In the above scheme, if the cache of the first computing unit is small, forcibly deploying the forward computation of the first computing unit in the second iteration round during the idling time of the first computing unit in the first iteration round may cause the first computing unit to experience a cache overflow problem. Therefore, after the first computing unit completes the forward computation result in the second iteration round in the first iteration round, it can unload the forward computation result in the second iteration round from the cache to other memory. When the reverse computation in the second iteration round is calculated later, the forward computation result in the second iteration round can be loaded from other memory.

[0027] Implementing the above scheme, before deploying the forward computation of the first computing unit in the second iteration into the vacuolation time of the first computing unit in the first iteration, it is determined whether the vacuolation time of the first computing unit in the first iteration is greater than or equal to the sum of the forward computation time of the second iteration and the time required to unload the forward computation result of the second iteration. If it is determined that the vacuolation time of the first computing unit in the first iteration is greater than or equal to the sum of the forward computation time of the second iteration and the time required to unload the forward computation result of the second iteration, then the forward computation of the first computing unit in the second iteration is deployed into the vacuolation time of the first computing unit in the first iteration. This avoids the situation where, after deploying the forward computation of the first computing unit in the second iteration into the vacuolation time of the first computing unit in the first iteration, the total time required by the first computing unit in the first iteration exceeds the time required by the second computing unit in the first iteration, thus preventing the second computing unit from experiencing vacuolation time in the first iteration.

[0028] In one possible implementation of the first aspect, after deploying the forward computation of the first computing unit in the second iteration round to the idling time of the first computing unit in the first iteration round, the method further includes: instructing the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round to the first memory, wherein both the first computing unit and the second computing unit can access the first memory; before the first computing unit performs the reverse computation of the second iteration round, instructing the first computing unit to load the computation result of the forward computation of the first computing unit in the second iteration round from the first memory.

[0029] In one possible implementation of the first aspect, instructing the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round into the first memory includes: based on the remaining space of the first memory being greater than or equal to a first threshold, instructing the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round into the first memory.

[0030] By implementing the above scheme, the calculation results of the first computing unit in the second iteration are unloaded into the first memory only when there is sufficient remaining space in the first memory, thereby reducing the risk of data loss or corruption due to insufficient memory space.

[0031] Secondly, this application provides a model acceleration device, comprising: an acquisition module for acquiring an initial deployment strategy of the model, the initial deployment strategy instructing a first computing unit to perform calculations using the model in a first iteration round and a second iteration round; and an adjustment module for, in the initial deployment strategy, deploying the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round, so that the first computing unit completes the forward computation of the second iteration round in the vacuolation time of the first iteration round, wherein the second iteration round is after the first iteration round, and the vacuolation time of the first iteration round is the time between the forward computation and the backward computation in the first iteration round.

[0032] In one possible implementation of the second aspect, the initial deployment strategy further instructs the second computing unit to perform calculations using the model in the first and second iteration rounds. The acquisition module is further configured to: allocate input data for the calculations of the first and second computing units in multiple iteration rounds, wherein in the same iteration round, the calculation time corresponding to the first input data allocated to the first computing unit is less than the calculation time corresponding to the second input data allocated to the second computing unit, wherein the calculation time includes forward calculation time and backward calculation time, and the multiple iteration rounds include the first iteration round and the second iteration round.

[0033] In one possible implementation of the second aspect, the acquisition module is specifically used to: acquire the load characteristics of each data in a plurality of data, wherein the plurality of data are data that the model needs to process, and the load characteristics include forward computation time and backward computation time; divide the plurality of data into a plurality of clusters based on the load characteristics of the plurality of data, the plurality of clusters including a first cluster and a second cluster; select a portion of data from the first cluster as first input data, and select a portion of data from the second cluster as second input data.

[0034] In one possible implementation of the second aspect, the load characteristics also include one or more of memory loading time, memory unloading time, and dynamic memory usage, wherein memory loading time is the time required to load the computation results of the forward computation; memory unloading time is the time required to unload the computation results of the forward computation; and dynamic memory usage is the memory size occupied by the computation results of the forward computation.

[0035] In one possible implementation of the second aspect, the adjustment module is specifically used to: acquire a first time, a second time, and a third time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; based on the third time being less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration is deployed to the bubbling time of the first computing unit in the first iteration.

[0036] In one possible implementation of the second aspect, the adjustment module is specifically used to: acquire a first time, a second time, a third time, and a fourth time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; and the fourth time is the time required for the first computing unit to unload the forward computation results of the input data in the second iteration. Based on the fact that the sum of the third time and the fourth time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration is deployed to the bubbling time of the first computing unit in the first iteration.

[0037] In one possible implementation of the second aspect, after deploying the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round, the apparatus further includes an unloading module for: instructing the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round to a first memory, wherein both the first computing unit and the second computing unit can access the first memory; and instructing the first computing unit to load the computation result of the forward computation of the first computing unit in the second iteration round from the first memory before the first computing unit performs the reverse computation of the second iteration round.

[0038] In one possible implementation of the second aspect, the unloading module is specifically used to: based on the remaining space of the first memory being greater than or equal to a first threshold, instruct the first computing unit to unload the calculation result of the forward calculation of the first computing unit in the second iteration round to the first memory.

[0039] Thirdly, this application provides a computing device including a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the model acceleration method as described in the first aspect and any possible implementation thereof.

[0040] Fourthly, this application provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the model acceleration method as described in the first aspect and any possible implementation thereof.

[0041] Fifthly, this application provides a computing device cluster, which includes at least one computing device. Each computing device includes a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to enable the computing device cluster to implement the model acceleration method as described in the first aspect and any possible implementation of the first aspect.

[0042] In a sixth aspect, this application provides a computer-readable storage medium including computer program instructions that, when executed by a computing device, enable the computing device to perform a model acceleration method as described in the first aspect and any possible implementation thereof.

[0043] The second, third, fourth, fifth and sixth aspects mentioned above all have various possible designs similar to the first aspect and any possible implementation of the first aspect, and can produce corresponding technical effects, which will not be elaborated here.

[0044] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the architecture of a model acceleration system provided in this application;

[0046] Figure 2 This is a schematic diagram of a model acceleration method provided in this application;

[0047] Figure 3 This is a schematic diagram of a process provided in this application for allocating multiple micro-batches to multiple iteration rounds corresponding to multiple computing units;

[0048] Figure 4 This application provides a schematic diagram of an iterative cycle;

[0049] Figure 5 This application provides a schematic diagram of another iteration round;

[0050] Figure 6This is a schematic diagram of the architecture of a model acceleration device provided in this application;

[0051] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application;

[0052] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0053] Figure 9 This is a schematic diagram of another structure of a computing device cluster provided in this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0055] In distributed training of multimodal models, the time required for different computational units to complete forward and backward computations varies significantly due to differences in the size and resolution of the input data processed by each unit. Since gradient data synchronization requires waiting for all computational units to complete backward computation, some units may finish early and remain idle in each iteration, resulting in wasted computational resources. Here, an iteration refers to the number of times data is completely transmitted through the model and parameters are updated. Each computational unit in the model performs both forward and backward computations in each iteration. Gradient data synchronization involves synchronizing and integrating the results of multiple backward computations from various computational units within a single iteration to adjust the model's parameters. Without gradient data synchronization, using only the results of backward computations from a subset of units for synchronization and parameter adjustment will cause the updated model parameters to deviate from their true values, thus affecting the model's analytical and predictive capabilities.

[0056] Existing model acceleration methods mainly achieve load balancing when each computing unit processes data, by eliminating differences in features such as size and resolution among multiple input data.

[0057] Eliminating resolution differences between different data refers to preprocessing data of varying resolutions to a uniform resolution. However, this method suffers from several drawbacks. For high-resolution data, preprocessing to a uniform resolution results in the loss of significant detail, leading to inaccurate predictions or analysis results. For example, preprocessing a 720P image to 480P will result in the loss of detail from the original image. Conversely, preprocessing low-resolution data to high resolution can lead to redundancy in image information, increasing the processing load. Therefore, this approach aims to reduce the processing time differences between individual computational units by increasing the overall computation time.

[0058] Eliminating size differences between data points involves padding data of varying sizes to a uniform size. However, this method introduces unnecessary computation by padding smaller data points with blank spaces. Furthermore, when multiple data points exhibit a long-tailed distribution, most data points require padding with a large number of blank spaces, resulting in significant unnecessary computation.

[0059] In summary, while current model acceleration methods can reduce the time difference in data processing among multiple computing units in a single round of computation, thereby reducing idle computing resources, they also introduce new problems for model training, both in terms of handling data resolution and data size.

[0060] To reduce the situation where some computing units complete their computations prematurely and enter idle waiting periods when multiple computing units are processing multiple data in parallel, resulting in idle and wasted computing resources, this application provides a model acceleration method, apparatus, and related equipment. The model acceleration method includes: in the deployment strategy corresponding to the model, deploying the forward computation of a certain computing unit in the second iteration round to the idle time of that computing unit in the first iteration round, so that the computing unit can complete the forward computation of the second iteration round during the idle time of the first iteration round. The second iteration round is the iteration round following the first iteration round, and the idle time of the first iteration round is the time between the forward and backward computations in the first iteration round. In the above method, for each computing unit participating in model training, by advancing the forward computation of subsequent iteration rounds to the idle time of the current iteration round, the current idle time of the computing unit is used to process future tasks, thereby reducing the idle time of computing resources and improving the utilization rate of computing resources.

[0061] For example, computing unit 1 and computing unit 2 process data 1 and data 2 respectively in iteration round 1, and computing unit 2 processes data 3 in iteration round 2. The time for computing unit 1 to process data 1 is 5 seconds, the time for computing unit 2 to process data 2 is 2 seconds, the forward computation time for computing unit 2 to process data 3 is 2.5 seconds, and the backward computation time for computing unit 2 to process data 3 is 3.5 seconds. Therefore, computing unit 2 will generate a 3-second cavitation time in iteration round 1. By using the model acceleration method provided in this application, the forward computation of data 3 in iteration round 2 is advanced to the cavitation time of computing unit 2 in iteration round 1, reducing the cavitation time of computing unit 2 in iteration round 1 to 3 - 2.5 = 0.5 seconds.

[0062] The aforementioned iteration rounds refer to the number of times data is completely transmitted within the model and the model parameters are updated. After each iteration round, the system updates the model parameters based on the backpropagation results corresponding to multiple data points in that iteration round. As the number of iteration rounds increases, the model parameters gradually adapt to the characteristics and patterns of the input data, thereby improving the model's accuracy and generalization ability. When multiple computational units run the model iteratively, due to the requirement of gradient data synchronization, the model parameters need to be updated based on the backpropagation results of each of the multiple computational units in that iteration round, thereby improving the correctness of the updated model parameters.

[0063] The aforementioned forward computation refers to the process of calculating input data using the operational rules defined by the model. Corresponding to forward computation is backward computation, which refers to the process of calculating the gradient of model parameters based on the error between the forward computation result and the actual computation result, as well as calculating the loss function. The system can adjust the model parameters based on the gradient of the model parameters, thereby reducing the error between the model's forward computation result and the actual computation result. In one iteration, the computation unit first uses the model to perform forward computation on the data, and then performs backward computation.

[0064] The system architecture involved in the model acceleration method provided in this application will be introduced below. (See also...) Figure 1 , Figure 1 This is a schematic diagram of the architecture of a model acceleration system provided in this application, such as... Figure 1 As shown, the architecture includes a model acceleration device 100, a data provider 110, and a model training device 120 (including computing unit 1, computing unit 2, ..., computing unit n), wherein there is a communication connection between the model acceleration device 100 and the model training device 120; there is a communication connection between the data provider 110 and the model training device 120.

[0065] Optional, Figure 1The model acceleration device 100 and the model training device 120 can be combined into one device. For example, the model acceleration device 100 can be a module unit in the model training device 120.

[0066] The communication connections between the model acceleration device 100 and the model training device 120, and between the data provider 110 and the model training device 120, can be wired or wireless. Wired connections include Ethernet, wired connections, cables, etc.; wireless connections include wireless local area networks (Wi-Fi), cellular networks, etc. Furthermore, connections that simultaneously support wired and wireless connections may also include the Internet, local area networks (LANs), etc. Other connection types may also be included, but this application does not specifically limit them.

[0067] The data provider 110 can be deployed on computing devices. These devices can use virtualization technology to create virtual environments such as virtual machines and containers to manage and utilize computing resources. A virtual machine is a virtualization technology implemented at the computer software level. Its working principle is to create an abstraction layer on top of the physical computer hardware, dividing and managing physical hardware resources, including memory and hard disks, so that each virtual machine can independently use its corresponding hardware resources, meaning each virtual machine can independently run its operating system and applications. A container is a lightweight, portable software encapsulation technology. Its working principle is to encapsulate an application and all its dependencies (including databases, frameworks, and files) together to form an independent unit, allowing applications within the container to run in the same way in different environments without being affected by the external environment. Compared to virtual machines, containers do not need to simulate complete hardware and operating systems; instead, they directly utilize the operating system kernel of the device they reside on. Therefore, containers also have advantages over virtual machines, such as faster startup speeds and more flexible deployment.

[0068] The model acceleration device 100 and the model training device 120 can also be deployed on computing devices, including bare metal servers (BMS). BMS refers to general-purpose physical servers, including servers based on Advanced RISC Machines (ARM) architecture and x86 servers, which are used to deliver complete server resources directly to users, allowing users to operate and manage them completely autonomously. Compared with virtual machines, BMS does not need to simulate multiple virtual computing environments through software, so there is no performance loss after virtualization. Compared with containers, BMS does not need to encapsulate and isolate each application, so there are no encapsulation limitations of containers.

[0069] The model acceleration device 100, the data provider 110, and the model training device 120 can also be deployed on terminal devices, including computer terminal devices, mobile terminal devices, network terminal devices, and virtual terminal devices. Among them, computer terminal devices include personal computers, laptops, and tablets; mobile terminal devices include smartphones, smartwatches, and smart bracelets; network terminal devices include routers, switches, and modems; and virtual terminal devices include virtual reality (VR) devices.

[0070] The model acceleration device 100 and the model training device 120 can also be deployed in a computing device cluster, which includes multiple computing devices as described above.

[0071] This document explains that the model acceleration device 100 and the model training device 120 described above can be deployed on the same computing device or terminal device, or they can be deployed on different computing devices or terminal devices. For example, the model acceleration device 100 and the model training device 120 can be deployed on different computing devices in a computing device cluster. Whether the model acceleration device 100 and the model training device 120 are deployed on the same computing device or terminal device depends on the specific application environment, and this application does not make specific limitations here.

[0072] Optionally, the code or source code used to implement the functions of the model acceleration device 100, the data provider 110, and the model training device 120 can be deployed in a storage device or a storage array. The storage device is a hardware device used to store and retrieve digital data, specifically a hard disk drive (HDD), a solid state drive (SSD), a universal serial bus flash drive, or a memory card, etc. The storage array refers to a storage system formed by combining multiple storage devices in a specific way, specifically a redundant array of independent disks (RAID), a storage area network (SAN), or network attached storage (NAS), etc. This application does not specifically limit the specific form of the storage device and storage array, or the device on which the storage device and storage array are located.

[0073] The model acceleration device 100, data provider 110, and model training device 120 each include a chip system composed of one or more chips to achieve the corresponding functions. Specifically, the model acceleration device 100, data provider 110, and model training device 120 can be devices implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), or a generic array logic (GAL). While data processing units (DPUs), neural network processing units (NPUs), graphics processing units (GPUs), systems on chips (SoCs), offload cards, and accelerator cards are not traditionally considered PLDs, they can also be used to construct these devices, and this application does not impose specific limitations on them.

[0074] It should be noted here that the specific architecture of the model acceleration device 100, the data provider 110, and the model training device 120 is only for illustrative purposes. The actual architecture of each device can be flexibly deployed according to actual business needs, and will not be illustrated here.

[0075] The above describes the system architecture that may be involved in this application. The following is a brief explanation of how this system architecture implements the model acceleration method provided in this application.

[0076] In the model acceleration method provided in this application, the data provider 110 sends training data and a model name to the model training device 120, instructing the model training device 120 to load the corresponding model file according to the model name and train the model using the training data; the model training device 120 constructs an initial deployment strategy based on the model name and training data, combined with its internal configuration, and sends the training data and the initial deployment strategy to the model acceleration device 100; the model acceleration device 100 incorporates the training data into the initial deployment strategy and optimizes the initial deployment strategy to obtain a final deployment strategy; the model training device 120 obtains and instructs multiple computing units to run a specified model according to the final deployment strategy, and trains the specified model using the training data.

[0077] The following is based on Figure 1 The system architecture shown, combined with Figure 2 To introduce the model acceleration method provided in this application, see [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of a model acceleration method provided in this application, such as... Figure 2 As shown, the steps of this model acceleration method include:

[0078] S201: The model acceleration device 100 receives the training data sent by the data provider 110.

[0079] The training data mentioned above is multimodal data used for model training, specifically covering image data, text data, audio data, and video data. There are differences between different types of data, and there are also differences between multiple data of the same type. For example, image data generally has a smaller memory footprint than video data, and there are also differences in memory footprint and resolution between different types of video data.

[0080] The aforementioned image data can include natural scene images, medical images, and satellite images. Natural scene images describe natural landscapes and are used to train the model, helping it learn the characteristics of different natural environments. This facilitates the model's subsequent recognition of input natural scene images and its application in creating new natural scene images. Medical images specifically include X-ray films, CT scans, and MRI images. Using medical images to train the model enables it to assist doctors in diagnosing diseases based on the input medical images. Satellite images provide a macroscopic view of terrain or atmospheric flow information. Using satellite images to train the model allows it to predict the weather for specific areas based on the input satellite images.

[0081] The aforementioned text data can include articles, papers, texts, and e-books. Training the model with text data allows it to accumulate more knowledge. Audio data can include music, ambient sounds, and speech. Training the model with audio data allows it to accumulate more speech material. Video data can include movie clips or video clips. Training the model with video data improves its ability to analyze videos.

[0082] This explains that the training data mentioned above can be combined in various ways when participating in model training. It can include a single type of data or multiple types. For example, training data can consist of only multiple medical images, or it can consist of multiple natural scene images, multiple texts, and multiple video clips. Training the model using training data of a single data type allows for deeper learning and optimization in a specific domain. For instance, training the model with a large number of medical images enables it to directly determine whether a patient has a tumor based on their MRI images. Training the model with training data of different data types provides more comprehensive information, thereby improving the model's generalization ability. For example, training the model with a large amount of video and audio about monkeys enables it to quickly determine whether a monkey appears in a video with sound.

[0083] Optionally, the training data can also be multiple sets, each containing multiple data of different types. By integrating data with similar content and then inputting the integrated data into the model for training, the trained model can achieve higher performance and better application results. For example, an image paired with descriptive text can be used, where the image provides intuitive visual information, while the descriptive text describes the content of the image, and the two complement each other.

[0084] The specific way in which the model acceleration device 100 receives the training data sent by the data provider 110 can be as follows: the model provider 110 packages and encapsulates the training data and sends it to the model training device 120; after receiving the training data, the model training device 120 forwards the training data to the model acceleration device 100.

[0085] S202: The model acceleration device 100 receives the initial deployment strategy sent by the model training device 120.

[0086] The aforementioned initial deployment strategy is constructed by the model training device 120 based on its internal configuration after receiving the model name and training data sent by the data provider 110. The initial deployment strategy includes how many iterations the model training device 120 needs to spend training the corresponding model and the amount of data each computing unit needs to process in each iteration. For example, after receiving the name of model A and 1000 training data points, the model training device 120 constructs an initial deployment strategy based on its internal configuration (including the existence of 10 computing units that can participate in model training). This initial deployment strategy includes that it takes 100 iterations to train model A and that each computing unit needs to process 1 piece of data in each iteration.

[0087] It should be noted that the initial deployment strategy constructed by the aforementioned model training device 120 is also related to many factors, including but not limited to the parameters of multiple computing units in the model training device 120, the strategy during model training (distributed parallel strategy, communication-computation masking strategy, recomputation strategy, and offloading strategy, etc.), the characteristics of the training data (text length of text data, size and resolution of image data, etc.), and the maximum amount of data that each computing unit can process in each iteration. This application does not specifically limit how the model training device 120 constructs the initial deployment strategy.

[0088] The aforementioned computing units are units in the model training device 120 used to load model files and perform calculations using the model. Specifically, the computing unit can be the core of a processor or a node in an inference cluster. The core of a processor is the center of processor operation and control, capable of interpreting and executing tasks or instructions, and can specifically be the CPU core. The inference cluster is a computing cluster established to perform inference tasks (predicting or analyzing based on data input into the model). Each node in the inference cluster has certain computing and storage capabilities, and the nodes cooperate with each other to ensure the efficient execution of inference tasks.

[0089] When a computing unit acts as a processor, the parameters of multiple computing units include the number of cores, clock speed, cache size, and memory bandwidth of each processor, among which:

[0090] The number of cores refers to the number of independent computing cores in a processor. The more cores a processor has, the more tasks it can process in parallel at the same time. For example, an 8-core processor can process 8 different tasks at the same time.

[0091] Clock speed refers to the clock frequency of a processor. The higher the clock speed, the more instructions the processor can execute or process per unit of time. For example, a processor with a clock speed of 2GHz can execute or process up to 3.5 billion instructions per second.

[0092] Cache refers to a high-speed storage area inside the processor. The processor can retrieve data from the cache at a faster rate than it can retrieve data from memory. The larger the cache, the higher the processing efficiency of the processor.

[0093] Memory is the area where the processor stores data for a long time. The processor can store data in memory and retrieve data from memory. For processors with large memory bandwidth, the time to store data in memory and load data from memory will be shorter.

[0094] When a computing unit acts as a node in an inference cluster, the parameters for multiple computing units include the number of nodes, the bandwidth between nodes, and the bandwidth for persistent storage, among others:

[0095] The number of nodes refers to the total number of nodes contained in the inference cluster. The larger the number of nodes, the more tasks the inference cluster can process in parallel at the same time, thereby improving the running efficiency of the model. For example, when the inference cluster has 50 nodes, all 50 nodes can load and run the file corresponding to a certain model. Assuming that the model needs to input 1 piece of data each time it runs, then these 50 nodes can process 50 pieces of data at the same time.

[0096] Inter-node bandwidth refers to the rate of network communication or data transmission between multiple nodes. The higher the inter-node bandwidth, the lower the latency of data transmission between multiple nodes. For example, when node 1 loads the file of model 1, node 1 can send the file of model 1 to node 2 through network communication. When the size of the file of model 1 is 50GB and the inter-node bandwidth is 10GB per second, the time required for the file of model 1 to be transmitted from node 1 to node 2 is 50 ÷ 10 = 5 seconds.

[0097] Persistent storage is an area that stores files of multiple models for a long period of time. Nodes in the inference cluster load the required model files from persistent storage to run the corresponding models. The greater the bandwidth of persistent storage, the shorter the time it takes for each node to load the model files from persistent storage. For example, if the file size of model 2 is 100GB and the bandwidth of persistent storage is 50GB per second, and 100 nodes need to run model 2, then the time required for these 100 nodes to load the file of model 2 from persistent storage is 100 × 100 ÷ 50 = 200 seconds.

[0098] Optionally, the aforementioned computing unit can also consist of a multimodal encoder, a backbone base model, a projection module, and connectors. These components cooperate with each other, enabling the computing unit to process and fuse various types of data, wherein:

[0099] A multimodal encoder is used to transform different types of input data (such as image data, text data, audio data, and video data) into features that can be processed. Each type of input data requires a specific encoder for transformation; therefore, a multimodal encoder can be understood as a combination of multiple encoders for different types of data. For example, in image data processing, the encoder for image data can be a Convolutional Neural Network (CNN). CNNs extract local features of image data, such as texture or edges, through convolutional and pooling layers.

[0100] The parameters of a multimodal encoder are related to the type of encoder. Taking an image encoder as an example, the parameters corresponding to a CNN include the kernel size, the number of convolutional layers, and the pooling method.

[0101] The backbone model receives features from the multimodal encoder and fuses, infers, and generates these features. It is typically built on the Transformer architecture. The parameters of the backbone model are the basic parameters of the model to be trained, including the self-attention mechanism, the number of computational layers, the batch size, and the hidden layer dimension.

[0102] The projection module maps the features output by the multimodal encoder to a common feature space, enabling the comparison and fusion of features from different types of data within the same space, thereby improving the efficiency of the backbone model. The parameters of the projection module include the projection matrix and bias vectors.

[0103] Connectors are used to establish relationships between features of different types of data, thereby promoting the fusion of different features and further improving the efficiency of the backbone model. Connector parameters include fusion weights.

[0104] This explains that, Figure 2 There is no strict order between S201 and S202. That is, the model acceleration device 100 can receive the initial deployment strategy first and then receive the training data, or it can receive the initial deployment strategy and the training data at the same time.

[0105] S203: The load characteristics of the model acceleration device 100 for acquiring training data.

[0106] The aforementioned load characteristics include forward computation time, reverse computation time, memory unloading time, memory loading time, and dynamic memory usage. Forward computation time is the time required for the computing unit to complete the forward computation of a certain data; reverse computation time is the time required for the computing unit to complete the reverse computation of a certain data; memory unloading time is the time required for the computing unit to unload the processing result of the forward computation of a certain data; memory loading time is the time required for the computing unit to load the processing result of the forward computation of a certain data; and dynamic memory usage is the memory usage of the processing result of the forward computation of a certain data.

[0107] The model acceleration device 100 calculates the load characteristics of the training data based on the training data and the parameters of multiple computing units. Specifically, the model acceleration device 100 constructs a load characteristic calculation model based on the parameters of multiple computing units, the strategy during model training, and the characteristics of the training data. The model acceleration device 100 then inputs some features of the training data (such as size, resolution, etc.) into the load characteristic calculation model to obtain the load characteristics of the training data.

[0108] The parameters of the aforementioned computing units, the strategies for model training, and the characteristics of the training data can be found in the relevant content in S202 above, and will not be elaborated upon here.

[0109] Existing solutions tokenize the representational features of training data and then calculate the load features of the training data based on the token length. These representational features include data size, number of words, number of images, and resolution, among others. The model acceleration method provided in this application employs load modeling, specifically operator-level fine-grained modeling. By modeling all factors affecting data processing time, the final computation time obtained under a specific hardware configuration is more accurate, thereby improving the method's accuracy.

[0110] S204: The model acceleration device 100 divides the training data into multiple micro-batches based on the load characteristics of the training data.

[0111] The aforementioned micro-batch is the smallest unit that a computational unit needs to process in each iteration of the model. A micro-batch can include one or more training data sets. The training data that a single computational unit needs to process in each iteration is collectively referred to as a micro-batch.

[0112] The aforementioned model acceleration device 100 divides the training data into multiple micro-batches based on the load characteristics of the training data, specifically including: ① The model acceleration device 100 clusters the training data according to the load characteristics of the training data to obtain multiple clusters; ② The model acceleration device 100 divides the data in each cluster into multiple micro-batches, and each micro-batch contains at least one piece of data from its corresponding cluster.

[0113] ① The model acceleration device 100 clusters the training data according to the load characteristics of the training data to obtain multiple clusters.

[0114] As shown in S203 above, the load characteristics of each data point include forward computation time, backward computation time, memory unloading time, memory loading time, and dynamic memory usage. Therefore, these five load characteristics can be used to represent a data point. For example, if the forward computation time, backward computation time, memory unloading time, memory loading time, and dynamic memory usage of data 1 are 3 seconds, 5 seconds, 2 seconds, 2 seconds, and 2GB respectively, then data 1 can be represented as (3, 5, 2, 2, 2). By constructing a 5-dimensional coordinate system and placing the training data into this coordinate system, multiple clusters can be obtained according to the corresponding clustering algorithm.

[0115] The clustering algorithm mentioned above can be the K-Means algorithm, hierarchical clustering algorithm, or DBSCAN algorithm. This application does not limit the specific method of clustering the training data.

[0116] Optionally, the above-mentioned single data is represented by five load features: forward computation time, backward computation time, memory unloading time, memory loading time, and dynamic memory usage. In some possible embodiments, the training data participating in clustering can also be represented by one or more of the following: forward computation time, backward computation time, memory unloading time, memory loading time, and dynamic memory usage. For example, if the forward computation time and backward computation time of data 2 are 2 seconds and 3 seconds respectively, then data 2 can be represented by (2, 3). By constructing a 2D coordinate system and placing the training data into the coordinate system, multiple clusters can be obtained according to the corresponding clustering algorithm.

[0117] This explains that when clustering training data based on its load characteristics, theoretically, the more features the data includes, the better the clustering algorithm will perform. However, as the number of features increases, the computational load of the clustering algorithm will also increase. The appropriate selection of the coordinate system to represent the data participating in clustering should be made in conjunction with the specific application scenario, finding a suitable balance between accuracy and practicality.

[0118] Optionally, after clustering the training data into multiple clusters, the model acceleration device 100 can also optimize each cluster. The specific optimization process includes: calculating the distance vector between the center point and the farthest point of each cluster; if the magnitude of the distance vector is greater than a threshold, adding a new cluster and positioning the farthest point as the initial center point of the new cluster; if the magnitude of the distance vector is less than the threshold, updating the center point according to the points in the cluster until the center point remains unchanged.

[0119] This section explains that after dividing multiple data points into multiple clusters based on the load conditions of multiple data points, the appropriateness of the multiple clusters is determined by judging the magnitude of the vector from the center point to the farthest point or from the farthest point to the center point in each cluster. If the multiple clusters are determined to be inappropriate, the division method is adjusted in a timely manner to improve the accuracy of clustering.

[0120] ② The model acceleration device 100 divides the data in each cluster into multiple micro-batches, each micro-batch containing at least one piece of data from its corresponding cluster.

[0121] After dividing the training data into multiple clusters based on the load characteristics of the training data, each cluster contains training data. The number of training data in each micro-batch is determined according to the training strategy, and the data in each cluster is divided into multiple micro-batches.

[0122] S205: The model acceleration device 100 distributes multiple micro-batches to multiple iteration rounds corresponding to multiple computing units.

[0123] Since the initial deployment strategy has already planned the multiple iterations of model training for multiple computing units and the amount of data that each computing unit needs to process in each iteration, the model acceleration device 100, after acquiring multiple micro-batches in S204, with each micro-batch being the same size as the amount of data that each computing unit needs to process in each iteration, can allocate the multiple micro-batches corresponding to the training data to the multiple iterations corresponding to each of the multiple computing units. This allows the multiple computing units to use the model to compute on the training data in multiple iterations.

[0124] For example, suppose the model acceleration device 100 clusters the training data according to the load characteristics of the training data, and finally obtains 3 clusters, where cluster 1 contains 100 data points, cluster 2 contains 150 data points, and cluster 3 contains 200 data points. There are 3 computing units, and the initial deployment strategy requires each computing unit to process 2 data points in each iteration round. Therefore, the model acceleration device 100 divides the data in cluster 1 into 50 micro-batches, the data in cluster 2 into 75 micro-batches, and the data in cluster 3 into 100 micro-batches. Then, the micro-batches corresponding to cluster 1 are assigned to computing unit 1, the micro-batches corresponding to cluster 2 are assigned to computing unit 2, and the micro-batches corresponding to cluster 3 are assigned to computing unit 3. Based on the allocation method in the example above, the amount of data processed by each computing unit in each iteration can be similar, making the training or computation of the entire model more stable. After computing unit 1 has finished processing the micro-batch corresponding to cluster 1, the unprocessed micro-batch in cluster 3 can be allocated to computing unit 1, thereby ensuring that multiple computing units can participate in data processing in each iteration round.

[0125] It should be noted that the above examples are merely examples used in this application and should not constitute specific limitations. In actual application scenarios, the above computing units and the data of the clusters may not be the same, and the specific allocation method should be appropriately selected based on the actual application scenario.

[0126] When the number of computing units is less than the number of clusters, i.e., in the example above, there may be only two computing units (computing unit 1 and computing unit 2), the model acceleration device 100 can allocate the micro-batch corresponding to cluster 1 to computing unit 1 and the micro-batch corresponding to cluster 2 to computing unit 2; after computing unit 1 has finished processing the micro-batch corresponding to cluster 1, it can allocate the micro-batch corresponding to cluster 3 to computing unit 1; after computing unit 2 has finished processing the micro-batch corresponding to cluster 2, it can allocate the micro-batch corresponding to cluster 3 to both computing unit 1 and computing unit 2.

[0127] When the number of computing units is greater than the number of clusters, i.e., in the example above, computing unit 4 may also exist. In this case, the model acceleration device 100 can allocate the micro-batch corresponding to cluster 1 to computing unit 1, allocate the micro-batch corresponding to cluster 2 to computing unit 2, and allocate the micro-batch corresponding to cluster 3 to both computing units 3 and 4. After computing unit 1 has finished processing the micro-batch corresponding to cluster 1, and computing units 3 and 4 have finished processing the micro-batch corresponding to cluster 3, the micro-batch corresponding to cluster 2 is allocated to computing units 1, 2, 3, and 4.

[0128] Existing solutions primarily group data into similar-sized or complementary-length groups based on data size, resolution, or the length of the tokenized sequence. However, these factors do not simply or linearly reflect the data's characteristics; even groups grouped in this way can still exhibit differences in processing time. The model acceleration method provided in this application divides the data into multiple groups through modeling and clustering, ensuring that the processing time within each group is similar. This facilitates the subsequent deployment of multiple computing units to process the data.

[0129] The following is combined Figure 3 To describe in detail the process by which the model acceleration device 100 allocates multiple micro-batches to multiple iterations corresponding to multiple computing units, see [link to relevant documentation]. Figure 3 , Figure 3 This application provides a schematic diagram of a process for allocating multiple micro-batches to multiple iterations corresponding to multiple computing units, as shown in the diagram. Figure 3 As shown, in Figure 3 The dataset contains a total of 16 training data points, 3 clusters, 8 micro-batches, and multiple computational units (computational unit 1, computational unit 2, and computational unit 3).

[0130] Model acceleration device 100 will Figure 3 The process of dividing the data into multiple micro-batches includes: the model acceleration device 100 first obtains the load characteristics corresponding to each of the 16 data points according to the steps in S203; then, based on the load characteristics corresponding to each of the 16 data points, it clusters the 16 data points to obtain three clusters, where cluster 1 contains 4 data points, cluster 2 contains 2 data points, and cluster 3 contains 10 data points; the model acceleration device 100 then determines, according to the model's training strategy, that each computing unit processes 2 data points in one iteration, dividing the multiple clusters into multiple micro-batches containing two data points, such as... Figure 3 In the process, cluster 1 is divided into micro-batch 1 and micro-batch 2, cluster 2 is directly used as micro-batch 3, and cluster 3 is divided into micro-batch 4, micro-batch 5, micro-batch 6, micro-batch 7 and micro-batch 8.

[0131] After dividing the training data into multiple micro-batches, the model acceleration device 100 distributes these batches of data to multiple computing units, so that multiple computing units can process the data during model training or computation. For example, micro-batch 1 and micro-batch 2 are distributed to computing unit 1, micro-batch 3 is distributed to computing unit 2, and micro-batch 4, micro-batch 5, micro-batch 6, micro-batch 7 and micro-batch 8 are distributed to computing unit 3.

[0132] S206: The model acceleration device 100 determines the cavitation time and forward computation time of each computing unit in each iteration based on the processing time of multiple micro-batches.

[0133] The processing time of the micro-batch mentioned above is the sum of the forward computation time and the backward computation time of all training data contained in the micro-batch.

[0134] The aforementioned cavitation time is the time between when the computing unit completes the forward computation of a microbatch in a certain iteration round and begins the reverse computation of that microbatch.

[0135] The aforementioned forward computation time is the time required for the computation unit to complete the forward computation of a micro-batch in a certain iteration round.

[0136] Since, after S205, the model acceleration device 100 has already allocated multiple micro-batches to multiple iteration rounds corresponding to multiple computing units, the model acceleration device 100 can directly calculate the time spent by each computing unit in that iteration round, the forward computation time, and the backward computation time based on the processing time of the micro-batches in each iteration round. The time spent by each computing unit in an iteration round is determined by the micro-batch with the longest processing time in that iteration round.

[0137] For example, see Figure 4 , Figure 4 This application provides a schematic diagram of an iterative cycle. Figure 4 The system contains computing unit 1, computing unit 2, micro-batches (X1 and X2), and micro-batches (Y1 and Y2). Data X1 and data X2 come from the same cluster, and data Y1 and data Y2 come from the same cluster.

[0138] according to Figure 4It can be seen that the time spent by computing unit 1 and computing unit 2 in this iteration is determined by the micro-batch (X1 and X2). That is, the sum of the forward computation time of data X1 and data X2 plus the sum of the backward computation time of data X1 and data X2 is the time required by all computing units in this iteration. The forward computation time of computing unit 1 is the sum of the forward computation time of data X1 and data X2, and the backward computation time of computing unit 1 is the sum of the backward computation time of data X1 and data X2. The forward computation time of computing unit 2 is the sum of the forward computation time of data Y1 and data Y2, and the backward computation time of computing unit 2 is... The sum of the reverse computation times of data Y1 and data Y2; since the micro-batch processed by computing unit 1 is the micro-batch with the longest processing time in this iteration, computing unit 1 has no vacancy time. The vacancy time of computing unit 2 is the time spent in this iteration minus the sum of the forward computation time and the reverse computation time of computing unit 2 in this iteration. That is, the vacancy time of computing unit 2 is the sum of the forward computation time of data X1 and data X2 plus the sum of the reverse computation time of data X1 and data X2, minus the sum of the forward computation time of data Y1 and data Y2 plus the sum of the reverse computation time of data Y1 and data Y2.

[0139] S207: The model acceleration device 100 determines whether the cavitation time of the first computing unit in the first iteration is greater than or equal to the forward computation time of the first computing unit in the second iteration.

[0140] The first and second iteration rounds mentioned above are any two iteration rounds in the initial deployment strategy, and the second iteration round appears after the first iteration round. For example, if the first iteration round is the 5th iteration round, the second iteration round could be the 6th iteration round, the 7th iteration round, the 8th iteration round, and so on.

[0141] The first computing unit mentioned above is any one of the multiple computing units participating in model training.

[0142] After step S205, the model acceleration device 100 has determined the forward computation time, backward computation time, and cavitation time of multiple iterations and multiple computing units in these iterations. At this point, the model acceleration device 100 analyzes all iterations to determine whether the cavitation time of a certain computing unit in a certain iteration is greater than or equal to the forward computation time of a subsequent iteration of that computing unit. For computing units and iterations where the above situation exists, step S208 is executed; if it is determined that the above situation does not exist in multiple iterations, step S209 is executed.

[0143] For example, the cavitation time of computation unit 1 in iteration 3 is 5 seconds; the forward computation time of computation unit 1 in iteration 1 is 4 seconds; the forward computation time of computation unit 1 in iteration 6 is 3 seconds; and the forward computation time of computation unit 1 in iteration 5 is 6 seconds. Since iteration 1 is before iteration 3, the processing in iteration 1 is not considered; since the forward computation time in iteration 5 is greater than the cavitation time in iteration 3, the processing in iteration 5 is not considered; since iteration 6 is after iteration 3, and the forward computation time of computation unit 1 in iteration 6 is greater than the cavitation time of computation unit 1 in iteration 3, the model acceleration device 100 processes the deployment of computation unit 1 in iteration 6 and iteration 3.

[0144] This section explains that the model acceleration device 100 determines whether the vacuolation time of the current iteration of each computational unit is greater than or equal to the forward computation time of subsequent iterations. It does not compare the vacuolation time and forward computation time between different computational units. Furthermore, when analyzing multiple iterations, the model acceleration device 100 may find that the vacuolation time of the current iteration of a certain computational unit is greater than the forward computation time of multiple subsequent iterations. In this case, the iteration closest to the logical position of the current iteration or the iteration with the shortest forward computation time can be selected.

[0145] S208: In the initial deployment strategy, the model acceleration device 100 deploys the forward computation of the first computing unit in the second iteration round to the cavitation time of the first computing unit in the first iteration round.

[0146] The model acceleration device 100 analyzes multiple iterations. If it is determined that the vacuolation time of a certain computing unit in a certain iteration is greater than the forward computation time of a subsequent iteration, the model acceleration device 100 directly deploys the forward computation of the computing unit in the subsequent iteration into the vacuolation time of the computing unit in the current iteration, so that the computing unit can complete the forward computation of the subsequent iteration in the current iteration during model training.

[0147] It should be understood here that after the model acceleration device 100 performs the above-mentioned S207 process on multiple iterations, the forward computation time of the first computing unit in the second iteration and the cavitation time of the first computing unit in the first iteration will also change. At this time, it is necessary to re-analyze the multiple iterations to determine whether there is a computing unit whose cavitation time in a certain iteration is greater than or equal to the forward computation time of a certain iteration after that computing unit. That is, the model acceleration device 100 re-executes S207.

[0148] For example, see Figure 5, Figure 5 This application provides a schematic diagram of another iteration round. Figure 5 The system comprises computation unit 1, computation unit 2, micro-batches (X1 and X2), micro-batches (Y1 and Y2), and micro-batches (Y3 and Y4). Data X1 and data X2 come from the same cluster, and data Y1, data Y2, data Y3, and data Y4 come from the same cluster.

[0149] according to Figure 5 It is known that the vacuolation time of computing unit 2 in the current iteration round is greater than the forward computation time of computing unit 2 in subsequent iteration rounds. Therefore, the model acceleration device 100 deploys the forward computation of micro-batches (Y3 and Y4) to the vacuolation time of computing unit 2 in the current iteration round, so that computing unit 2 can complete the forward computation of micro-batches (Y3 and Y4) in the current iteration round during model training.

[0150] After the above steps, the model acceleration device 100 advances the forward computation of a certain computing unit in subsequent iterations. The computation results of the forward computation in subsequent iterations are stored in this computing unit until the reverse computation of the subsequent iterations is executed. Therefore, when the cache of this computing unit is small, the model acceleration device 100 can, after executing S207 for multiple iterations, instruct the first computing unit to unload the computation results of the forward computation in the second iteration to a designated memory when the forward computation in the second iteration is completed, and instruct the first computing unit to reload the computation results of the forward computation in the second iteration from the designated memory before processing the reverse computation of the second iteration. The designated memory here can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR) or a DDR memory pool.

[0151] It should be understood here that since the first computing unit has an additional action of unloading the calculation result of the forward calculation in the first iteration, when it is necessary to unload the calculation result of the forward calculation of the first computing unit in the second iteration to the designated memory, in the above S206, the model acceleration device 100 determines whether the vacuolation time of the first computing unit in the first iteration is greater than or equal to the forward calculation time of the first computing unit in the second iteration. This should be updated to the model acceleration device 100 determining whether the vacuolation time of the first computing unit in the first iteration is greater than or equal to the sum of the forward calculation time and the memory unloading time of the first computing unit in the second iteration.

[0152] For example, in Figure 5In the initial deployment strategy, after deploying the forward computation of micro-batches (Y3 and Y4) to the current iteration vacancy time of computing unit 2, the model acceleration device 100 further deploys computing unit 2, instructing computing unit 2 to unload the computation results of the forward computation to DDR memory after processing the forward computation of micro-batches (Y3 and Y4), and to load the computation results of the forward computation of micro-batches (Y3 and Y4) from DDR memory before processing the reverse computation of micro-batches (Y3 and Y4) in the future.

[0153] Optionally, before instructing the first computing unit to unload the computation results of the forward computation in the second iteration round to the designated memory, the model acceleration device 100 will also determine whether the remaining memory of the designated memory is greater than a preset threshold. If the model acceleration device 100 determines that the remaining memory of the designated memory is greater than or equal to the preset threshold, it will then instruct the first computing unit to unload the computation results of the forward computation in the second iteration round to the designated memory. If the model acceleration device 100 determines that the remaining memory of the designated memory is less than the preset threshold, it will not further deploy the computing unit 2 in the initial deployment strategy, and the model acceleration device 100 will execute S209.

[0154] This explains that by detecting the remaining memory space, we can prevent memory overflow from occurring in multiple computing units during model training, thereby ensuring the efficient implementation of model training.

[0155] S209: The model acceleration device 100 stops updating the initial deployment strategy and obtains the final deployment strategy.

[0156] S210: The model acceleration device 100 sends the final deployment strategy to the model training device 120.

[0157] After the above steps, the model acceleration device 100 has updated the initial deployment strategy and obtained the final deployment strategy. The model acceleration device 100 sends the final deployment strategy to the model training device 120, so that multiple computing units in the model training device 120 train the model according to the final deployment strategy.

[0158] In summary, this application provides a model acceleration method, which includes: the model acceleration device 100 deploying the forward computation of a computing unit in a subsequent iteration to the idle time of the computing unit in the current iteration, so that the computing unit completes the forward computation in the subsequent iteration during the idle time of the current iteration. In the above method, for each computing unit participating in model training, the model acceleration device 100 processes the forward computation in the subsequent iteration by deploying it to the idle time of the current iteration, utilizing the current idle time of the computing unit to process future tasks, thereby reducing the idle time of computing resources and improving the utilization rate of computing resources.

[0159] For the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application. Other reasonable combinations of steps that those skilled in the art can conceive of based on the above description also fall within the scope of protection of this application. Again, those skilled in the art should be familiar with the fact that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0160] See Figure 6 , Figure 6 This is a schematic diagram of the architecture of a model acceleration device provided in this application, such as... Figure 6 As shown, the model acceleration device 600 is used to input the initial deployment strategy and multiple data, and output the final deployment strategy and uninstallation command.

[0161] The model acceleration device 600 can be further divided into multiple unit modules, such as Figure 6 As shown, the model acceleration device 600 also includes an acquisition module 601, an adjustment module 602, and an unloading module 603. It should be understood that... Figure 6 The number and names of the unit modules included in the model acceleration device 600 are merely examples provided in this application. The model acceleration device 600 may include more or fewer unit modules, and the names of the unit modules are not limited to [specific examples would be inserted here]. Figure 6 The name of the unit module in the above, for example, the model acceleration device 600 may also include a data processing module, and the unloading module is used to receive multiple data sent by the acquisition module 601, preprocess the multiple data, and then send the processed data to the acquisition module 601. Figure 6 The name of the acquisition module 601 can be changed to that of the receiving module. It should be understood that the above examples are for illustrative purposes only and should not be considered as specific limitations.

[0162] The acquisition module 601, adjustment module 602, and unloading module 603 can all be implemented in software or hardware. For example, the implementation of the acquisition module 601 will be described below. Similarly, the implementation of the adjustment module 602 and the unloading module 603 can refer to the implementation of the acquisition module 601.

[0163] As an example of a software functional unit, module 601 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 601 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region or in different regions. Further, the multiple hosts, virtual machines, or containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Similarly, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a VPC is set up within one region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway within each VPC to achieve interconnection between VPCs.

[0164] As an example of a hardware functional unit, the acquisition module 601 may include at least one computing device, such as a server. Alternatively, the acquisition module 601 may be implemented using a CPU, an ASIC, or a PLD, where the PLD can be a CPLD, FPGA, GAL, DPU, NPU, SoC, offload card, accelerator card, or any combination thereof. It should be noted that the multiple computing devices included in the acquisition module 601 can be distributed in the same Availability Zone (AZ) or different AZs. Similarly, the multiple computing devices included in the acquisition module 601 can be distributed in the same VPC or multiple VPCs, where the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and inference cards.

[0165] The functions of the acquisition module 601, adjustment module 602, and unloading module 603 in the above-mentioned model acceleration device 600 are described below.

[0166] The acquisition module 601 is used to acquire the initial deployment strategy of the model. The initial deployment strategy instructs the first computing unit to use the model for calculation in the first iteration round and the second iteration round.

[0167] The adjustment module 602 is used to deploy the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round in the initial deployment strategy, so that the first computing unit completes the forward computation of the second iteration round in the vacuolation time of the first iteration round, wherein the second iteration round is after the first iteration round, and the vacuolation time of the first iteration round is the time between the forward computation and the backward computation in the first iteration round.

[0168] In one possible implementation, the initial deployment strategy further instructs the second computing unit to perform calculations using the model in the first and second iteration rounds. The acquisition module 601 is further configured to: allocate input data for the calculations of the first and second computing units in multiple iteration rounds, wherein in the same iteration round, the calculation time corresponding to the first input data allocated to the first computing unit is less than the calculation time corresponding to the second input data allocated to the second computing unit, wherein the calculation time includes forward calculation time and backward calculation time, and the multiple iteration rounds include the first iteration round and the second iteration round.

[0169] In one possible implementation, the acquisition module 601 is specifically used to: acquire the load characteristics of each data in multiple data sets, wherein the multiple data sets are the data that the model needs to process, and the load characteristics include forward computation time and backward computation time; divide the multiple data sets into multiple clusters based on the load characteristics of the multiple data sets, the multiple clusters including a first cluster and a second cluster; select a portion of data from the first cluster as first input data, and select a portion of data from the second cluster as second input data.

[0170] In one possible implementation, the load characteristics also include one or more of memory loading time, memory unloading time, and dynamic memory usage, wherein memory loading time is the time required to load the computation results of the forward computation; memory unloading time is the time required to unload the computation results of the forward computation; and dynamic memory usage is the memory size occupied by the computation results of the forward computation.

[0171] In one possible implementation, the adjustment module 602 is specifically used to: obtain a first time, a second time, and a third time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; based on the third time being less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration is deployed to the bubbling time of the first computing unit in the first iteration.

[0172] In one possible implementation, the adjustment module 602 is specifically used to: acquire a first time, a second time, a third time, and a fourth time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; the fourth time is the time required for the first computing unit to unload the forward computation results of the input data in the second iteration; based on the fact that the sum of the third time and the fourth time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration is deployed to the bubbling time of the first computing unit in the first iteration.

[0173] In one possible implementation, after deploying the forward computation of the first computing unit in the second iteration round to the idling time of the first computing unit in the first iteration round, the device further includes an unloading module 603, configured to: instruct the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round to the first memory, wherein both the first computing unit and the second computing unit can access the first memory; and before the first computing unit performs the reverse computation of the second iteration round, instruct the first computing unit to load the computation result of the forward computation of the first computing unit in the second iteration round from the first memory.

[0174] In one possible implementation, the unloading module 603 is specifically used to: based on the remaining space in the first memory being greater than or equal to a first threshold, instruct the first computing unit to unload the calculation result of the forward calculation of the first computing unit in the second iteration round to the first memory.

[0175] In summary, this application provides a model acceleration device 600. This device acquires an initial deployment strategy and multiple data sets via an acquisition module 601, allocates data to computing units via an adjustment module 602, and, in the initial deployment strategy, deploys the forward computation of computing units in subsequent iterations to the idle time of the computing unit in the current iteration, so that the computing unit completes the forward computation in subsequent iterations during the idle time of the current iteration. In the aforementioned model acceleration device 600, for each computing unit participating in model training computation, the device 600 processes the forward computation in subsequent iterations by deploying it to the idle time of the current iteration, utilizing the current idle time of the computing unit to handle future tasks, thereby reducing the idle time of computing resources and improving the utilization rate of computing resources.

[0176] The above text combines Figures 2 to 5 The method for accelerating the model provided in this application is described in detail below. Figures 7 to 9 This application describes the computing device and computing device cluster provided in accordance with this application.

[0177] Figure 7 This is a schematic diagram of the structure of a computing device provided in this application. The computing device 700 may be the model acceleration device 600 mentioned above.

[0178] Furthermore, the computing device 700 includes a bus 701, a processor 702, a memory 703, and a communication interface 704. The processor 702, the memory 703, and the communication interface 704 communicate with each other via the bus 701. The computing device 700 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0179] The 701 bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0180] The processor 702 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, or an accelerator card.

[0181] The memory 703 is used to store program code, and its execution is controlled by the processor 702 to perform the above-mentioned tasks. Figures 2 to 5 The processing steps of the model acceleration method in any embodiment. The program code includes one or more software units, which are... Figure 6 In this embodiment, the acquisition module 601, adjustment module 602, and unloading module 603 are specifically configured to execute... Figure 2 The optional steps in steps S201 and S202 of the embodiment are executed by the adjustment module 602. Figure 2 The optional steps in steps S203 to S209 of the embodiment will not be described again here.

[0182] Memory 703 may include volatile memory, such as random access memory (RAM). Processor 702 may also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD. Furthermore, memory 703 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0183] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 703, or two or more types of storage media can be configured to realize the function of memory 703. This application does not limit this.

[0184] The communication interface 704 is a wired interface (e.g., an Ethernet interface), an internal interface (e.g., a Peripheral Component Interconnect express (PCIe) bus interface), a wired interface (e.g., an Ethernet interface), or a wireless interface (e.g., a cellular network interface or a wireless LAN interface) for communicating with other servers or units.

[0185] As one possible implementation, the computing device 700 may include multiple types of processors 702, meaning the computing device 700 is a heterogeneous device. For example, the computing device 700 may include a CPU and a GPU, and the operation steps corresponding to the model acceleration method can be executed by at least one of the processors 702. For the sake of brevity, further details will not be provided here.

[0186] It needs to be explained that, Figure 7 This is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 700 may include more or fewer components, and this is not a limitation. For content not shown or described in the embodiments of this application, please refer to the foregoing. Figures 2 to 5 The relevant descriptions in the embodiments will not be repeated here.

[0187] Figure 8 This is a schematic diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0188] like Figure 8 As shown, the computing device cluster includes at least one computing device 800. The memory 803 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the model acceleration method.

[0189] In some possible implementations, the memory 803 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the model acceleration method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the model acceleration method.

[0190] It should be noted that the memories 803 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the resource migration system's functions. That is, the instructions stored in the memories 803 of different computing devices 800 can implement the acquisition module 601, the adjustment module 602, and the unloading module 603. Specifically, the acquisition module 601 is used to execute... Figure 2 The optional steps in steps S201 and S202 of the embodiment are executed by the adjustment module 602. Figure 2 The optional steps in steps S203 to S209 of the embodiment will not be described again here.

[0191] The computing device 800 includes a bus 801, a processor 802, a memory 803, and a communication interface 804. Further descriptions of the bus 801, processor 802, memory 803, and communication interface 804 can be found in [reference needed]. Figure 7 The descriptions of bus 701, processor 702, memory 703 and communication interface 704 in the embodiments will not be repeated here.

[0192] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc., as described below. Figure 9 One possible implementation method is shown.

[0193] Figure 9 This is another schematic diagram of a computing device cluster provided in this application, such as... Figure 9 As shown, the two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 803 in computing device 800A stores instructions for implementing the acquisition module 601. Meanwhile, the memory 803 in computing device 800B stores instructions for implementing the adjustment module 602 and the unloading module 603.

[0194] It should be understood that Figure 9 The functions of the computing device 800A shown can also be performed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be performed by multiple computing devices 800.

[0195] It needs to be explained that, Figure 9 The implementation shown may be implemented when the processing power of the computing device 800A is insufficient, or when the storage space of the computing device 800A is insufficient, or in other business scenarios. This application does not make any specific limitations.

[0196] This application also provides another type of computing device cluster. The interconnection relationships between the computing devices in this computing device cluster can be similarly referenced. Figure 8 and Figure 9 The connection method of the computing device cluster. The difference is that the memory 803 of one or more computing devices 800 in the computing device cluster can store the same instructions for executing the model acceleration method.

[0197] In some possible implementations, the memory 803 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the model acceleration method. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the model acceleration method.

[0198] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a model acceleration method.

[0199] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to execute a model acceleration method.

[0200] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model acceleration method, characterized in that, The method includes: Obtain the initial deployment strategy of the model, which instructs the first computing unit to use the model for computation in the first iteration round and the second iteration round; In the initial deployment strategy, the forward computation of the first computing unit in the second iteration round is deployed during the vacuolation time of the first computing unit in the first iteration round, so that the first computing unit completes the forward computation of the second iteration round during the vacuolation time of the first iteration round. The second iteration round is after the first iteration round, and the vacuolation time of the first iteration round is the time between the forward computation and the backward computation in the first iteration round.

2. The method according to claim 1, characterized in that, The initial deployment strategy further instructs the second computing unit to perform calculations using the model in both the first and second iteration rounds, and the method further includes: Input data is allocated to the first computing unit and the second computing unit in multiple iteration rounds. In the same iteration round, the computing time corresponding to the first input data allocated to the first computing unit is less than the computing time corresponding to the second input data allocated to the second computing unit. The computing time includes forward computing time and backward computing time. The multiple iteration rounds include the first iteration round and the second iteration round.

3. The method according to claim 2, characterized in that, The process of allocating input data for the calculations of the first and second computing units in multiple iterations includes: Obtain the load characteristics of each data point from a plurality of data points, wherein the plurality of data points are the data that the model needs to process, and the load characteristics include forward computation time and backward computation time; Based on the load characteristics of the multiple data, the multiple data are divided into multiple clusters, including a first cluster and a second cluster; Select a portion of data from the first cluster as the first input data, and select a portion of data from the second cluster as the second input data.

4. The method according to claim 3, characterized in that, The load characteristics also include one or more of memory loading time, memory unloading time, and dynamic memory usage, wherein memory loading time is the time required to load the computation results of the forward computation; memory unloading time is the time required to unload the computation results of the forward computation; and dynamic memory usage is the memory size occupied by the computation results of the forward computation.

5. The method according to any one of claims 2 to 4, characterized in that, The step of deploying the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round includes: The system acquires a first time, a second time, and a third time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; and the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration. Based on the fact that the third time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration round is deployed to the cavitation time of the first computing unit in the first iteration round.

6. The method according to any one of claims 2 to 4, characterized in that, The step of deploying the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round includes: The system acquires a first time, a second time, a third time, and a fourth time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; and the fourth time is the time required for the first computing unit to unload the forward computation results of the input data in the second iteration. Based on the fact that the sum of the third time and the fourth time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration round is deployed to the cavitation time of the first computing unit in the first iteration round.

7. The method according to any one of claims 2 to 6, characterized in that, After deploying the forward computation of the first computing unit in the second iteration into the cavitation time of the first computing unit in the first iteration, the method further includes: The first computing unit is instructed to unload the calculation result of the forward calculation of the first computing unit in the second iteration into the first memory, and both the first computing unit and the second computing unit can access the first memory; Before the first computing unit performs the reverse calculation of the second iteration, the first computing unit is instructed to load the calculation result of the first computing unit's forward calculation in the second iteration from the first memory.

8. The method according to claim 7, characterized in that, The instruction to the first computing unit to unload the computation result of the forward computation of the first computing unit in the second iteration round into the first memory includes: Based on the fact that the remaining space in the first memory is greater than or equal to the first threshold, the first computing unit is instructed to unload the calculation result of the forward calculation of the first computing unit in the second iteration into the first memory.

9. A model acceleration device, characterized in that, The device includes: An acquisition module is used to acquire the initial deployment strategy of the model, wherein the initial deployment strategy instructs the first computing unit to use the model for computation in the first iteration round and the second iteration round; An adjustment module is configured to, in the initial deployment strategy, deploy the forward computation of the first computing unit in the second iteration round to the vacuolation time of the first computing unit in the first iteration round, so that the first computing unit completes the forward computation of the second iteration round during the vacuolation time of the first iteration round, wherein the second iteration round is after the first iteration round, and the vacuolation time of the first iteration round is the time between the forward computation and the backward computation in the first iteration round.

10. The apparatus according to claim 9, characterized in that, The initial deployment strategy also instructs the second computing unit to perform calculations using the model in both the first and second iteration rounds, and the acquisition module is further configured to: Input data is allocated to the first computing unit and the second computing unit in multiple iteration rounds. In the same iteration round, the computing time corresponding to the first input data allocated to the first computing unit is less than the computing time corresponding to the second input data allocated to the second computing unit. The computing time includes forward computing time and backward computing time. The multiple iteration rounds include the first iteration round and the second iteration round.

11. The apparatus according to claim 10, characterized in that, The acquisition module is specifically used for: Obtain the load characteristics of each data point from a plurality of data points, wherein the plurality of data points are the data that the model needs to process, and the load characteristics include forward computation time and backward computation time; Based on the load characteristics of the multiple data, the multiple data are divided into multiple clusters, including a first cluster and a second cluster; Select a portion of data from the first cluster as the first input data, and select a portion of data from the second cluster as the second input data.

12. The apparatus according to claim 11, characterized in that, The load characteristics also include one or more of memory loading time, memory unloading time, and dynamic memory usage, wherein memory loading time is the time required to load the computation results of the forward computation; memory unloading time is the time required to unload the computation results of the forward computation; and dynamic memory usage is the memory size occupied by the computation results of the forward computation.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The adjustment module is specifically used for: The system acquires a first time, a second time, and a third time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; and the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration. Based on the fact that the third time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration round is deployed to the cavitation time of the first computing unit in the first iteration round.

14. The apparatus according to any one of claims 10 to 12, characterized in that, The adjustment module is specifically used for: The system acquires a first time, a second time, a third time, and a fourth time, wherein the first time is the time required for the first computing unit to perform forward and backward computations on the input data using the model in the first iteration; the second time is the time required for the second computing unit to perform forward and backward computations on the input data using the model in the first iteration; the third time is the time required for the first computing unit to perform forward computations on the input data using the model in the second iteration; and the fourth time is the time required for the first computing unit to unload the forward computation results of the input data in the second iteration. Based on the fact that the sum of the third time and the fourth time is less than or equal to the difference between the second time and the first time, the forward computation of the first computing unit in the second iteration round is deployed to the cavitation time of the first computing unit in the first iteration round.

15. The apparatus according to any one of claims 10 to 14, characterized in that, After deploying the forward computation of the first computing unit in the second iteration into the cavitation time of the first computing unit in the first iteration, the apparatus further includes an unloading module for: The first computing unit is instructed to unload the calculation result of the forward calculation of the first computing unit in the second iteration into the first memory, and both the first computing unit and the second computing unit can access the first memory; Before the first computing unit performs the reverse calculation of the second iteration, the first computing unit is instructed to load the calculation result of the first computing unit's forward calculation in the second iteration from the first memory.

16. The apparatus according to claim 15, characterized in that, The unloading module is specifically used for: Based on the fact that the remaining space in the first memory is greater than or equal to the first threshold, the first computing unit is instructed to unload the calculation result of the forward calculation of the first computing unit in the second iteration into the first memory.

17. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the operational steps of the method as described in any one of claims 1 to 8.

18. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each of the at least one computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the operational steps of the method as described in any one of claims 1 to 8.

Images