Model calculation system, method and device and related equipment

Abstract

The invention provides a model calculation system, method and device and related equipment, and the method comprises the steps: segmenting a first model according to the number of xPU acceleration cards, and generating a plurality of second models; segmenting the CPU according to the number of the xPU acceleration cards to generate a plurality of CPU resource groups; each xPU accelerator card is bound with the corresponding CPU resource group and the second model, and a plurality of calculation groups are generated; a plurality of second models are calculated through the plurality of calculation groups. According to the method, the model and the CPU are reasonably segmented according to the number of the xPU acceleration cards, and then the segmented model, CPU resources and the xPU acceleration cards are bound, so that the respective computing advantages of the plurality of xPU acceleration cards can be played, the time delay caused by data transmission among different memory areas can be reduced, and the data transmission efficiency is improved. And the calculation efficiency of the whole model is improved.

Description

Technical Field

[0001] This application relates to the field of heterogeneous acceleration, and in particular to a model computing system, method, apparatus and related equipment. Background Technology

[0002] With the continuous development of technology, the demand for model computing capabilities in cutting-edge fields such as artificial intelligence and big data analysis is increasing day by day, and the requirements are becoming more and more stringent.

[0003] Traditional model computation architectures are increasingly showing significant performance bottlenecks when dealing with increasingly complex model computation tasks. Taking the use of a Central Processing Unit (CPU) for model computation as an example, the CPU's computing power becomes relatively limited when faced with large-scale model computation tasks, especially massively parallel computing tasks. For instance, during the training of deep learning models, the massive amount of data and complex computational requirements make it difficult for the CPU to complete the computation tasks within the allotted time, resulting in low overall model computational efficiency. Summary of the Invention

[0004] This application provides a model computing system, method, apparatus, and related equipment, which can not only enable independent processing of the segmented model and leverage the computing advantages of multiple xPU accelerator cards, but also reduce the latency caused by data transfer between different memory regions by binding CPU resources to xPU cards, thereby improving the overall computing efficiency of the model.

[0005] In a first aspect, this application provides a model computing system for computing a first model, which is divided into multiple second models. The system includes multiple computing groups, each of which includes a central processing unit (CPU) and a heterogeneous processor unit (xPU) accelerator card. The number of second models is equal to the number of computing groups. Each computing group is used to compute one of the multiple second models. The number of CPU resources in each computing group is proportional to the computing power of the xPU accelerator card in that computing group, and the amount of computation of the second model allocated to each computing group is proportional to the computing power of the xPU accelerator card in that computing group.

[0006] In the above approach, xPU encompasses processors other than CPUs used to accelerate specific types of computation. These processors include, but are not limited to, Graphics Processing Units (GPUs), Neural Processing Units (NPUs), and Tensor Processing Units (TPUs). Different processors can provide more efficient computing power than CPUs in specific computing scenarios. Taking GPUs as an example, GPUs have a large number of cores that can process multiple tasks in parallel. Therefore, when the computational model has high requirements for data parallelism, using GPUs can significantly accelerate the model's computation speed. For instance, deep learning models often require a large number of matrix multiplication and addition operations, which demand high data parallelism; GPUs can handle these operations simultaneously.

[0007] In the above scheme, the xPU accelerator card is a hardware computing device obtained by integrating the xPU and other auxiliary components. These other auxiliary components include high-speed memory for data storage and interface circuitry for connection with the host system. It should be understood that since the xPU accelerator card is already a complete computing device, during model computation, only the model and its corresponding operators need to be set on the xPU accelerator card, and the xPU accelerator card can complete the model computation.

[0008] In the above system, when computation is required on the model, the parent model (first model) is divided into multiple sub-models (second models), and these sub-models are processed in parallel through multiple computation groups, thereby improving the computational efficiency of the model. Each computation group consists of CPU resources and xPU accelerator cards, which can leverage the computational advantages of different xPU accelerator cards according to the model structure. For example, a deep learning model includes multiple layers, and different layers may correspond to different computational features. The convolutional layers in the deep learning model involve a large number of matrix multiplication and matrix addition operations. By dividing the different layers in the deep learning model, the model corresponding to the convolutional layer is separated from the deep learning model, and the computation group containing the GPU accelerator card is bound to the model corresponding to the convolutional layer.

[0009] In one possible implementation of the first aspect, the number of xPU accelerator cards in each computing group is 1, and the CPU resources in each computing group include one or more CPU cores; wherein, when the CPU resources in a computing group include multiple CPU cores, the multiple CPU cores are located on one or more CPUs.

[0010] In one possible implementation of the first aspect, the system further includes a partitioning processor for partitioning the first model according to the number of xPU accelerator cards to generate multiple second models, wherein the computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model; the partitioning processor is also used to partition the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group.

[0011] In the aforementioned system, binding the xPU accelerator card to a CPU resource group creates a computing group. This allows for faster data transfer between the xPU and its corresponding CPU. For example, in deep learning models, model parameters and data often need to be frequently transferred between the CPU and xPU, which can easily cause data congestion and latency. By binding the xPU accelerator card to the CPU resource group, a high-speed transmission channel can be established between the xPU and CPU, such as a high-speed Peripheral Component Interconnect Express (PCIe) bus, thereby reducing latency caused by transmission. Furthermore, by binding the xPU accelerator card and CPU resources, data storage and processing are more likely to occur in the local memory areas associated with the xPU accelerator card and CPU resources, thus reducing latency caused by data transfer between different storage areas and further improving the computational efficiency of the model.

[0012] In one possible implementation of the first aspect, the partitioning processor is further configured to partition the first operator to generate multiple second operators. The first operator is a computational operator of the first model, and one of the multiple second operators is a computational operator of one of the multiple second models. Each of the multiple second operators includes a forward computational operator, a backward computational operator, an optimizer operator, and an accompanying operator. The optimizer operator is used to update the model parameters based on the gradient information obtained from the backward computational operator. The accompanying operator is used to assist model computation, including one or more functions such as overflow detection, gradient scaling, and precision transformation. The partitioning processor is further configured to deploy each second operator to a computation group.

[0013] In one possible implementation of the first aspect, the processor splitting is specifically used for: deploying the forward computation operator and the backward computation operator of the second operator corresponding to the first computation group on the xPU accelerator card in the first computation group; wherein, the first computation group is one of multiple computation groups, and the second operator corresponding to the first computation group is the computation operator of the second model corresponding to the first computation group; splitting the optimizer operator in the second operator corresponding to the first computation group into a first optimizer operator and a second optimizer operator according to the function of the optimizer operator; splitting the along-path operator in the second operator corresponding to the first computation group into a first along-path operator and a second along-path operator according to the function of the along-path operator; deploying the first optimizer operator and the first along-path operator on the CPU resources in the first computation group, and deploying the second optimizer operator and the second along-path operator on the xPU accelerator card in the first computation group.

[0014] In the above scheme, the training operators are divided into forward computation operators, backward computation operators, optimizer operators, and follow-up operators. The forward computation operators and backward computation operators are used to calculate the output results and the error between the calculated output results and the expected results based on the given input and model parameters. The optimizer operator is used to adjust the model parameters based on the error between the output results and the expected results. The follow-up operators are used to perform auxiliary operations during the model calculation process.

[0015] This explains that due to the differences in structure and function between CPU and xPU, the effects of deploying operators on CPU and xPU are also different.

[0016] In the aforementioned system, the segmentation processor, based on the structural characteristics of the CPU and xPU, segments the training operators and rationally deploys the segmented operators to further improve the computational efficiency of the entire model. For example, for computationally intensive operators, since the xPU typically has a large number of computational units that can execute the same instructions simultaneously, deploying computationally intensive operators on the xPU can improve their computational efficiency. For logic processing operators, since the CPU has a diverse instruction set capable of executing various logical instructions, including judgments and comparisons, deploying logic processing operators on the xPU can improve their computational efficiency.

[0017] In one possible implementation of the first aspect, the system further includes an operator fusion unit for fusing operators on the xPU accelerator cards to be deployed to the first computing group before the split processor deploys each second operator to a computing group.

[0018] In one possible implementation of the first aspect, the operator fusion unit is also used to fuse the operators to be deployed on the CPU before the split processor deploys each second operator to a computing group.

[0019] The aforementioned operator fusion refers to merging multiple consecutive computational operators with data dependencies into a single operator. For example, in deep learning models, operators used for convolution and normalization can be combined into a composite operator that can be used for both convolution and normalization.

[0020] This explains that most operators involve data reading and writing during execution, which requires access to Dynamic Random Access Memory (DRAM). Excessive DRAM accesses can lead to significant latency, impacting the overall computational efficiency of the model. Due to differences in computing architecture, instruction sets, data storage, and data processing between xPUs and CPUs, operators deployed on xPUs are not suitable for merging with those deployed on CPUs.

[0021] In the above system, by adding an operator fusion unit to the model computation system, multiple operators deployed on the xPU and CPU are merged into a large operator, reducing the number of accesses to dynamic random access memory and thus further improving the computational efficiency of the model.

[0022] In one possible implementation of the first aspect, the system further includes a CPU utilization calculator, which is used to calculate the utilization of a first CPU resource and a second CPU resource, wherein the first CPU resource is the CPU resource used in the system for calculation of multiple second models, and the second CPU resource is the CPU resource in the system other than the first CPU resource; the CPU utilization calculator is also used to allocate a portion of the CPU resources in the second CPU resource to the first CPU resource when the utilization of the second CPU resource is less than or equal to the utilization of the first CPU resource.

[0023] In the above system, by adding a CPU utilization calculator to the model calculation system, the system can dynamically adjust CPU resources based on the relationship between the CPU utilization used for model calculation and the CPU utilization not used for model calculation. This enables the rational allocation of CPU resources. For example, when the CPU utilization not used for model calculation is low, more CPU resources can be allocated to model calculation, and the model calculation system can then distribute the CPU resources to multiple calculation groups, thereby improving the model's calculation efficiency.

[0024] Secondly, this application provides a model calculation method, which includes: dividing a first model according to the number of xPU accelerator cards to generate multiple second models, wherein the computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model; dividing the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group; binding each xPU accelerator card to its corresponding CPU resource group to generate multiple computing groups; and calculating multiple second models through multiple computing groups.

[0025] By implementing the above scheme, the model and CPU are reasonably divided according to the number of xPU accelerator cards, and then the divided model, CPU resources and xPU accelerator cards are bound together. This not only enables parallel processing of multiple divided models and leverages the computing advantages of each xPU accelerator card, but also reduces the latency caused by data transfer between different memory regions by binding CPU resources with xPU accelerator cards, thereby improving the overall computational efficiency of the model.

[0026] Thirdly, this application provides a model computing device, comprising: a model segmentation unit for segmenting a first model according to the number of xPU accelerator cards to generate multiple second models, wherein the computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model; a resource segmentation unit for segmenting CPUs according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group; a processing unit for binding each xPU accelerator card to its corresponding CPU resource group to generate multiple computing groups; and a processing unit for computing multiple second models through the multiple computing groups.

[0027] Fourthly, this application provides a computing device including a processor and a memory, the memory for storing instructions and the processor for executing instructions, so that the computing device implements the method as described in the second aspect.

[0028] Fifthly, this application provides a computing device cluster, which includes at least one computing device. Each computing device includes a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to enable the computing device cluster to implement the method as described in the second aspect.

[0029] In a sixth aspect, this application provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored in any available medium, such that when the computer program product is run on at least one computing device, the at least one computing device performs the method as described in the second aspect.

[0030] In a seventh aspect, this application provides a computer-readable storage medium storing instructions that are executed by a computing device or a cluster of computing devices to implement the method of the second aspect.

[0031] The third, fourth, fifth, sixth and seventh aspects mentioned above all have various possible designs similar to the first aspect and any possible implementation of the first aspect, and can produce corresponding technical effects, which will not be elaborated here. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of a model calculation system provided in this application;

[0033] Figure 2 This is a schematic diagram illustrating the steps of a model calculation method provided in this application;

[0034] Figure 3 This is a schematic diagram of a model segmentation process provided in this application;

[0035] Figure 4 This is a schematic diagram of a CPU partitioning process provided in this application;

[0036] Figure 5 This is a schematic diagram of the structure of a computing group provided in this application;

[0037] Figure 6 This is a schematic diagram of another computing group provided in this application;

[0038] Figure 7 This is an architectural diagram of a model computing device provided in this application;

[0039] Figure 8 This is a schematic diagram of the structure of a computing device provided in this application;

[0040] Figure 9 This is a schematic diagram of the structure of a computing device cluster provided in this application;

[0041] Figure 10 This is a schematic diagram of another structure of a computing device cluster provided in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0043] To improve the computational efficiency of the model, this application provides a model computation system, method, apparatus, and related equipment. The model computation method includes: dividing a first model according to the number of xPU accelerator cards to generate multiple second models; dividing the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups; binding each xPU accelerator card to its corresponding CPU resource group and the second model to generate multiple computation groups; and computing multiple second models through the multiple computation groups. In the above model calculation method, by reasonably dividing the model and computing resources, and binding the divided model, CPU resources, and xPU accelerator cards, compared with using the CPU alone to calculate the model, firstly, the multiple xPU accelerator cards used in this application can meet multiple computing needs. By dividing the model into multiple sub-models, a suitable xPU accelerator card can be selected for calculation according to the corresponding computing type of each sub-model, thereby improving the overall computing efficiency of the model; secondly, by binding the CPU resources and xPU accelerator cards, the data required by the CPU resources and xPU accelerator cards when calculating the sub-models can come from local or nearby memory areas, thereby reducing the latency caused by the transfer of computing data between different memory areas, and further improving the overall computing efficiency of the model.

[0044] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the model calculation system provided in this application, such as... Figure 1 As shown, the model computing system 100 includes a segmentation processor 101, an operator fusion unit 102, a CPU utilization calculator 103, and computing resources.

[0045] The segmentation processor 101, operator fusion unit 102, and CPU utilization calculator 103 can be any of the following: computing devices, computing device clusters, or terminal devices. Computing devices include bare metal servers (BMS), virtual machines, containers, or edge computing devices. A BMS refers to a general-purpose physical server, such as an ARM server or an x86 server. A virtual machine refers to a complete computer system simulated by software, possessing full hardware system functionality and running in a completely isolated environment. Any task that can be performed on a physical computer can be performed on a virtual machine. When creating a virtual machine on a computing device, a portion of the physical machine's hard drive and memory capacity is used as the virtual machine's hard drive and memory capacity. Each virtual machine has an independent basic input / output system (BIOS), hard drive, and operating system, and can be operated like a physical machine. A container is a type of virtualization software that can merge an application and all its dependencies into a single software package. This package is not limited by the underlying host operating system, thus eliminating the need to build complex environments and simplifying the application development and deployment process. Edge computing devices refer to devices that are closer to the data source and end users, featuring low latency and high bandwidth, such as intelligent routers and edge servers. The computing device cluster may include multiple of the aforementioned computing devices, such as a data center, which is not specifically limited in this application; the description of the terminal devices can be referred to the foregoing content, and will not be repeated here. It can be understood that the above example is based on the example that the partition processor 101, the operator fusion unit 102, and the CPU utilization calculator 103 are three independent processors. In practical applications, the partition processor 101, the operator fusion unit 102, and the CPU utilization calculator 103 can be implemented by the same processor in a time-sharing manner, or the partition processor 101 and the operator fusion unit 102 can be implemented by the same processor in a time-sharing manner, the operator fusion unit 102 and the CPU utilization calculator 103 can be implemented by the same processor in a time-sharing manner, and the partition processor 101 and the CPU utilization calculator 103 can be implemented by the same processor in a time-sharing manner.

[0046] Computing resources include xPU accelerator cards and CPUs deployed on servers or computing devices. An xPU accelerator card is a hardware computing device that integrates an xPU and other auxiliary components. An xPU encompasses processors other than a CPU used to accelerate specific types of computations. These processors include, but are not limited to, GPUs, TPUs, and NPUs. Different processors can provide more efficient computing power than CPUs in specific computing scenarios. Taking GPUs as an example, GPUs have a large number of cores that can process multiple tasks in parallel. Therefore, when the computational model has high requirements for data parallelism, using a GPU can significantly accelerate the model's computation speed. For example, deep learning models often require a large number of matrix multiplication and addition operations, which have high requirements for data parallelism. Using a GPU can handle these operations simultaneously. Other auxiliary components include high-speed memory for storing data and interface circuits for continuous connection with the host system. It should be understood that since the xPU accelerator card is already a complete computing device, during model computation, only the model and its corresponding operators need to be set on the xPU accelerator card, and the xPU accelerator card can complete the model computation. This indicates that in... Figure 1 The computing resources shown also include other computing resources besides the CPU and xPU accelerator card. For example, the computing resources also include random-access memory (RAM), which is used to store the programs and data being executed by the CPU.

[0047] The segmentation processor 101, operator fusion unit 102, CPU utilization calculator 103, and computing resources can be distributed on the same device or on different devices. When the segmentation processor 101, operator fusion unit 102, CPU utilization calculator 103, and computing resources are distributed on the same device, they can all be located on a server. When the segmentation processor 101, operator fusion unit 102, and CPU utilization calculator 103 are distributed on different devices, the CPU utilization calculator 103 can be located on a terminal device, such as a mobile phone, tablet, laptop, or desktop computer, while the segmentation processor 101, operator fusion unit 102, and computing resources can be located on a server.

[0048] The following section uses the model calculation system 100 as the execution entity and combines it with... Figures 2 to 6 This application will introduce the model calculation method provided in this application.

[0049] See Figure 2 , Figure 2 This is a schematic diagram illustrating the steps of a model calculation method provided in this application, as shown below. Figure 2 As shown, the steps of this model calculation method include:

[0050] S201: The segmentation processor 101 segments the first model according to the number of xPU accelerator cards to generate multiple second models.

[0051] Both the first and second models are models from multiple academic fields, including mathematics and statistics, computer science, engineering and physics. For example, a model in the field of data and statistics could be a time series model for predicting price trends; a model in the field of computer science could be a convolutional neural network model for image recognition; and a model in the field of engineering and physics could be a car crash model for simulating car collisions.

[0052] Since different models have different model structures, a large model can be divided into multiple smaller models based on its structure. For example, taking a deep learning model as an example, we can illustrate how to divide a model into multiple sub-models based on its structure. The architecture of a deep learning model includes input layers, convolutional layers, pooling layers, and fully connected layers. Based on the architecture of the deep learning model, the deep learning model (i.e., the first model) can be divided into models corresponding to the input layer (i.e., the second model), models corresponding to the convolutional layers (i.e., the second model), models corresponding to the pooling layers (i.e., the second model), and models corresponding to the fully connected layers (i.e., the second model). It should be noted that due to the structure of the model, the ratio of computational cost among the multiple sub-models obtained after dividing the parent model is difficult to control. Therefore, the ratio of computational cost among the multiple sub-models obtained after dividing the parent model may not be the same as the ratio of computing power of multiple xPU accelerator cards. In this case, the segmentation processor 101 only needs to adopt a segmentation method that is closest to the ratio of computing power of multiple xPU accelerator cards.

[0053] The partitioning processor 101 partitions the first model based on the number of xPU accelerator cards, generating multiple second models. Specifically, the partitioning processor 101 acquires the computing power of the multiple xPU accelerator cards; the partitioning processor 101 partitions the first model according to the ratio between the computing power of the multiple xPU accelerator cards, so that each xPU accelerator card corresponds to one second model, and the computational load of the second model corresponding to each xPU accelerator card is proportional to the computing power of that accelerator card. Here, different xPU accelerator cards can have different computing powers or the same computing power. When different xPU accelerator cards have different computing powers, partitioning the parent model according to the ratio between the computing powers of the xPU accelerator cards and assigning a sub-model with a corresponding computational load according to the computing power of each xPU accelerator card can fully utilize the performance of each xPU accelerator card, thereby improving the computational efficiency of the model.

[0054] The computing power of multiple xPU accelerator cards obtained by the aforementioned segmentation processor 101 can be manually input or calculated by the segmentation processor 101 according to the corresponding algorithm program. This application does not limit the method by which the segmentation processor 101 obtains the computing power of the xPU accelerator cards.

[0055] The following is combined Figure 3 The following describes the specific process by which the aforementioned partitioning processor 101 partitions the first model according to the ratio between the computing power of multiple xPU accelerator cards, so that each xPU accelerator card corresponds to a second model, and the computational amount of the second model corresponding to each xPU accelerator card is proportional to the computing power of that accelerator card.

[0056] See Figure 3 , Figure 3 This is a schematic diagram of a model segmentation process provided in this application. Figure 3 It includes three xPU accelerator cards (xPU accelerator card A, xPU accelerator card B, and xPU accelerator card C), with the computing power ratio between xPU accelerator card A, xPU accelerator card B, and xPU accelerator card C being 1:2:3.

[0057] The partitioning processor 101 partitions the parent model into three sub-models—sub-model A, sub-model B, and sub-model C—based on the ratio of their computing power to that of xPU accelerator cards A, B, and C. The computational load ratio among sub-models A, B, and C is 1:2:3. After partitioning the parent model and obtaining sub-models A, B, and C, the partitioning processor 101 allocates sub-model A to xPU accelerator card A, sub-model B to xPU accelerator card B, and sub-model C to xPU accelerator card C, thus achieving a reasonable partitioning and allocation of the model.

[0058] It should be understood here that... Figure 3 This is merely an example provided in this application. In real-world applications, due to the difficulty in controlling the model segmentation results, the segmentation processor 101 can adopt a segmentation method that most closely approximates the ratio of computing power of multiple xPU accelerator cards. For example, in... Figure 3 If the computational ratio between sub-model A, sub-model B, and sub-model C can be 5:3:7 or 4:3:8, the partitioning processor 101 will adopt a partitioning method with a computational ratio of 5:3:7 between sub-model A, sub-model B, and sub-model C, and allocate the partitioned sub-model A to xPU accelerator card B, sub-model B to xPU accelerator card A, and sub-model C to xPU accelerator card C.

[0059] S202: The partitioning processor 101 partitions the CPU according to the number of xPU accelerator cards, generating multiple CPU resource groups.

[0060] The aforementioned CPU resource group includes some or all of the CPU resources and memory resources for storing or providing data to the CPU resources. CPU resources include CPU cores, which are the number of physical cores in a CPU. Each core is an independent processing unit capable of executing commands independently. Each CPU contains one or more CPU cores; for example, a quad-core CPU contains four CPU cores. In a multitasking environment, these four CPU cores process multiple tasks in parallel, thereby improving the overall system response speed. It should be understood that since a CPU contains a certain number of CPU resources, these resources can be divided into different groups. For example, if a CPU has 16 cores and needs to process four tasks, it can be divided into four CPU resource groups with a CPU resource ratio of 8:3:3:2.

[0061] Optionally, CPU resources can also exist as NUMA nodes. A NUMA node is a node within a NUMA framework, where the computer's physical memory and CPU resources are divided into multiple groups (the aforementioned NUMA nodes). Each NUMA node contains a certain amount of CPU resources and local memory. Within the same NUMA node, CPU resources can quickly access their corresponding local memory. It should be understood that the number of NUMA nodes corresponding to each xPU accelerator card is determined based on the ratio of computing power among multiple xPU accelerator cards. During subsequent calculations of the model, the advantage of CPU resources in NUMA nodes being able to quickly access their corresponding local memory is utilized to reduce the performance overhead caused by non-local memory access, thereby improving the overall computational efficiency of the model.

[0062] The partitioning processor 101 partitions the CPUs according to the number of xPU accelerator cards, generating multiple CPU resource groups. Specifically, the partitioning processor 101 acquires the computing power of multiple xPU accelerator cards; the partitioning processor 101 partitions the CPUs available for model computation according to the ratio between the computing power of the multiple xPU accelerator cards, so that each xPU accelerator card corresponds to a CPU resource group, and the number of CPU resources corresponding to each xPU accelerator card is proportional to the computing power of that accelerator card. This illustrates that different xPU accelerator cards have different computing power; the higher the computing power of an xPU accelerator card, the greater the amount of model computation it can handle. Therefore, by partitioning the CPUs according to the ratio between the computing power of the xPU accelerator cards and allocating a corresponding number of CPU resources according to the computing power of each xPU accelerator card, a reasonable allocation of CPU resources is achieved, thereby improving the computational efficiency of the model.

[0063] The following is combined Figure 4This section describes the process by which the partitioning processor 101 partitions the CPU according to the ratio between the computing power of multiple xPU accelerator cards, generating multiple CPU resource groups.

[0064] See Figure 4 , Figure 4 This is a schematic diagram of a CPU partitioning process provided in this application. Figure 4 The system includes three xPU accelerator cards (xPU accelerator card A, xPU accelerator card B, and xPU accelerator card C), with the computing power ratio between xPU accelerator card A, xPU accelerator card B, and xPU accelerator card C being 1:2:3.

[0065] As described above, the partitioning processor 101 partitions the CPUs available for model computation according to the ratio of computing power among xPU accelerator cards A, B, and C, dividing them into three CPU resource groups: CPU resource group 1, CPU resource group 2, and CPU resource group 3. The ratio of the number of CPU resources contained in CPU resource group 1, CPU resource group 2, and CPU resource group 3 is 1:2:3. After partitioning the CPUs available for model computation into CPU resource groups 1, 2, and 3, the partitioning processor 101 allocates CPU resource group 1 to xPU accelerator card A, CPU resource group 2 to xPU accelerator card B, and CPU resource group 3 to xPU accelerator card C, thereby achieving reasonable partitioning and allocation of the CPUs.

[0066] When the CPU resources are NUMA nodes, the above-mentioned partitioning processor 101 partitions the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups. Alternatively, the partitioning processor 101 obtains the number of NUMA nodes; the partitioning processor 101 determines the number of NUMA nodes corresponding to each xPU accelerator card according to the ratio between the computing power of multiple xPU accelerator cards; the number of NUMA nodes corresponding to each xPU accelerator card is proportional to the computing power corresponding to that accelerator card.

[0067] After allocating NUMA nodes to each xPU accelerator card, the aforementioned partitioning processor 101 can further fine-tune the number of CPU resources contained in different NUMA nodes according to the actual application scenario, so that the number of CPU resources obtained by each xPU accelerator card is equivalent to the computing power corresponding to that xPU accelerator card, thereby improving the overall computing efficiency of the model. For example, consider existing NUMA node 1, NUMA node 2, xPU accelerator card A, and xPU accelerator card B, where NUMA node 1 contains 4 CPU resources, NUMA node 2 contains 4 CPU resources, and the computing power ratio between xPU accelerator card A and xPU accelerator card B is 3:5. After allocating NUMA node 1 to xPU accelerator card A and NUMA node 2 to xPU accelerator card B, the partitioning processor 101 moves one CPU resource from NUMA node 1 to NUMA node 2, so that the number of CPU resources corresponding to xPU accelerator card A is 3, and the number of CPU resources corresponding to xPU accelerator card B is 5.

[0068] S203: The segmentation processor 101 segments the first operator corresponding to the first model, generates multiple second operators, and deploys the second operators to the CPU and multiple xPU accelerator cards.

[0069] The first and second operators mentioned above are both operators. An operator is an operational unit corresponding to a model, used to transform, process, or calculate data. For example, in a neural network model, the addition operator is used to add the outputs of each neuron, and the multiplication operator is used to calculate the product of the input of each neuron and the weight corresponding to that neuron. Operators can be used for simple geometric operations or for calculating complex mathematical functions. For example, operators can be simple geometric operations such as addition, subtraction, multiplication, and division; operators can be complex mathematical function calculations such as trigonometric functions, exponential functions, logarithmic functions, and integral functions.

[0070] Different operators have different structures. Similar to models, operators can combine multiple smaller operators into a larger one, or a large operator can be divided into multiple smaller operators. For example, in a deep learning model, the operator corresponding to the forward propagation of a fully connected layer can be decomposed into multiple smaller operators. The forward propagation operator can be represented as output = input × weight + biases, where output is the output; input is the input matrix; weight is the weight matrix; and biases are the bias vectors. This operator includes matrix multiplication and addition, so the forward propagation operator can be decomposed into a matrix multiplication operator and an addition operator. The matrix multiplication operator can be represented as mid_output = input × weight, where mid_output represents the result of multiplying the input matrix and the weight matrix. The meanings of input and weight are the same as in the forward propagation operator. The addition operator can be represented as output = mid_output + biases, where output and biases have the same meanings as in the forward propagation operator, and mid_output has the same meaning as mid_output in the matrix multiplication operator. The method of combining multiple small operators into a large operator can be reasonably derived based on the above examples, and will not be elaborated further in this application.

[0071] In this application, the first operator is the computation operator of the first model, and one of the multiple second operators is the computation operator of one of the multiple second models. The second operators include a forward computation operator, a backward computation operator, an optimizer operator, and an accompanying operator. The forward computation operator and the backward computation operator are used to calculate the output result and the error between the calculated output result and the expected result based on the given input and model parameters. The optimizer operator is used to adjust the model parameters based on the error between the output result and the expected result. The accompanying operator is used to perform auxiliary operations during the model computation process.

[0072] The partitioning processor 101 deploys the second operators onto the CPU and multiple xPU accelerator cards. Specifically, the partitioning processor 101 partitions the first operator to obtain multiple second operators, including forward computation operators, backward computation operators, optimizer operators, and in-path operators. The partitioning processor 101 deploys the forward computation operators and backward computation operators in each second operator onto their corresponding xPU accelerator cards. The partitioning processor 101 further partitions the in-path operators and optimizer operators in each second operator, and deploys the partitioned in-path operators onto their corresponding CPUs and xPU accelerator cards, and deploys the partitioned optimizer operators onto their corresponding CPUs and xPU accelerator cards. The partitioning processor 101 adjusts the partitioning strategy of the optimizer operators and in-path operators using linear programming.

[0073] This section explains that due to the structural and functional differences between CPUs and xPUs, deploying operators on the CPU and xPU will have different effects. By segmenting operators based on their structural characteristics and rationally deploying these smaller operators, the overall computational efficiency of the model can be improved. For example, for computationally intensive operators, since xPUs typically have a large number of computational units that can execute the same instructions simultaneously, deploying computationally intensive operators on the xPU can improve computational efficiency. Similarly, for logic processing operators, since CPUs have diverse instruction sets capable of executing various logical instructions, including judgments and comparisons, deploying logic processing operators on the xPU can improve their computational efficiency.

[0074] Optionally, before deploying the second operator to the CPU and multiple xPU accelerator cards, the split processor 101 will also use the operator fusion processor 102 to first fuse the operators corresponding to the multiple xPU accelerator cards and the operators corresponding to the CPU. This is because most operators involve data reading and writing during execution, which involves accessing Dynamic Random Access Memory (DRAM). Excessive DRAM accesses can lead to long access latency, thus affecting the overall computational efficiency of the model. Due to differences in computing architecture, instruction sets, data storage, and data processing between xPUs and CPUs, operators deployed on xPUs are not suitable for merging with operators deployed on CPUs.

[0075] Optionally, the partitioning processor 101 may first partition the CPU resources according to the number of xPU accelerator cards, then partition the first model according to the number of xPU accelerator cards, and then partition the first operator corresponding to the first model. That is, the partitioning processor 101 may execute S202 first, then S201, and finally S203. This application does not limit the execution order of S202, S201 and S203 by the partitioning processor 101.

[0076] S204: The partitioning processor 101 binds each xPU accelerator card to its corresponding CPU resource group, generating multiple computing groups.

[0077] For a description of the above calculation group, please refer to [link / reference]. Figure 2 The details regarding the segmentation processor 101 will not be elaborated upon here.

[0078] The aforementioned binding of the xPU accelerator card to a CPU resource group creates a computing group. This allows for faster data transfer between the xPU and its corresponding CPU. For example, in deep learning models, model parameters and data often need to be frequently transferred between the CPU and xPU, which can easily cause data congestion and latency. By binding the xPU accelerator card to the CPU resource group, a high-speed transmission channel can be established between the xPU and CPU, such as a PCIe bus or a Peripheral Component Interconnect express switch (PCIe switch), thereby reducing latency caused by transmission. Furthermore, by binding the xPU accelerator card to CPU resources, data storage and processing are more likely to occur in the local memory area associated with the xPU accelerator card and CPU resources, thus reducing latency caused by data transfer between different storage areas and further improving the computational efficiency of the model.

[0079] The following section describes the structure of the partition processor 101 after binding each CPU resource among multiple CPU resources to its corresponding computing resource, using CPU resource groups as NUMA nodes and CPU resources as CPU kernels.

[0080] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a computing group provided in this application. Figure 5 It includes two NUMA nodes (NUMA1 and NUMA2), two xPU accelerator cards (xPU accelerator card 1 and xPU accelerator card 2), and two dynamic random access memories (DRAM). The DRAM is connected to the CPU resources, xPU accelerator card 1 is connected to the CPU resources in NUMA1, and xPU accelerator card 2 is connected to the CPU resources in NUMA2. In each NUMA node, the connection channel between DRAM and CPU is a DRAM channel (not shown in the figure), and the connection channel between CPU resources and xPU accelerator cards is a PCIe switch.

[0081] The DRAM mentioned above in the NUMA node serves to store data processed by the CPU resources. The CPU resources, together with the DRAM, are used to perform calculations on the model. It should be understood that since the xPU accelerator card itself is a complete computing unit, it does not need to be connected to additional DRAM.

[0082] This explains that, Figure 5The structure of the computing group generated after binding the CPU resources and xPU accelerator cards shown is merely an example provided in this application and should not be considered a specific limitation. Since each xPU accelerator card can also correspond to multiple NUMA nodes, the number of CPUs available for model computation can also be multiple. Therefore, according to... Figure 5 The structural diagrams of other computational groups can be easily derived. For example, Figure 6 This is a schematic diagram of another computing group structure provided in this application. Figure 6 It includes two NUMA nodes (NUMA3 and NUMA4), one xPU accelerator card (xPU accelerator card 3), and two dynamic random access memories (DRAM). The DRAM is connected to the CPU resources, and the xPU accelerator card 3 is connected to the CPU resources in NUMA3 and NUMA4 through a PCIe switch.

[0083] S205: The model calculation system 100 calculates multiple second models through multiple calculation groups.

[0084] The model computing system 100 calculates multiple second models through multiple computing groups. Specifically, the model computing system 100 starts the CPU and each xPU accelerator card to perform parallel computing on multiple second models. After all the second models have been calculated, the model computing system 100 merges the calculation results of the multiple second models to obtain the calculation result of the first model.

[0085] During the process of starting the CPU and multiple xPU accelerator cards to compute multiple second models, the model computing system 100 also uses a CPU utilization calculator 103 to calculate the utilization rates of the first CPU and the second CPU. The first CPU is the CPU resource used for model computing, and the second CPU is the CPU resource not used for the first model computing. When the utilization rate of the second CPU is less than or equal to the utilization rate of the first CPU, the CPU utilization calculator 103 allocates some CPU resources from the second CPU to the first CPU. When the utilization rate of the second CPU is greater than the utilization rate of the first CPU, the CPU utilization calculator 103 allocates some CPU resources from the first CPU to the second CPU. It should be understood that during model computing, the CPU resources are dynamically adjusted based on the relationship between the CPU utilization rates used for model computing, the CPU utilization rates not used for model computing, and the total CPU utilization rate. This improves CPU utilization efficiency. Furthermore, when the CPU utilization rate not used for model computing is low, allocating more CPU resources to model computing can reduce the overall computing time and improve the computational efficiency of the model.

[0086] In summary, this application provides a model computation method. This method involves partitioning a first model using a processor 101 based on the number of xPU accelerator cards to generate multiple second models; partitioning the CPU based on the number of xPU accelerator cards to generate multiple CPU resource groups; binding each xPU accelerator card to its corresponding CPU resource group and the second model to generate multiple computation groups; and then the model computation system 100 starts multiple computation groups to compute multiple second models. In this method, by rationally partitioning the model and CPU based on the number of xPU accelerator cards, and then binding the partitioned models, CPU resources, and xPU accelerator cards, parallel processing of the partitioned models can be achieved, leveraging the computational advantages of each xPU accelerator card. Furthermore, by binding CPU resources to xPU accelerator cards, latency caused by data transfer between different memory regions can be reduced, thereby improving the overall computational efficiency of the model.

[0087] See Figure 7 , Figure 7 This is an architectural diagram of a model computing device provided in this application, such as... Figure 7 As shown, the architecture includes a model computing device 700, which is used to perform calculations on the first model.

[0088] The model computing device 700 can be deployed on computing devices, including virtual machines, containers, or servers. A virtual machine is a virtualization technology implemented at the computer software level, enabling a single physical computer to create multiple virtual operating systems and application environments, each running its own operating system and applications independently. A container is a lightweight software packaging method used to package applications and their application environments, allowing applications to run in the same way across different environments. Unlike virtual machines, containers do not contain a complete operating system but share the operating system of the physical computer they reside on, making them more lightweight. A server refers to a general-purpose physical server, including ARM servers or x86 servers.

[0089] The model computing device 700 can also be deployed in a computing device cluster, which includes multiple of the aforementioned computing devices.

[0090] This document explains that the aforementioned model computing device 700 can be deployed on the same computing device or mobile terminal, or on different computing devices or mobile terminals. For example, the aforementioned model computing device 700 can be deployed on different computing devices in a computing device cluster. Whether the model computing device 700 is deployed on the same computing device depends on the specific application environment, and this application does not make specific limitations here.

[0091] The model computing device 700 can be further divided into multiple units, such as Figure 7 As shown, the model computing device 700 also includes a model segmentation unit 701, a resource segmentation unit 702, and a processing unit 703. It should be understood here that... Figure 7 The number and names of the unit modules included in the model computing device 700 are merely examples provided in this application. The model computing device 700 may include more or fewer unit modules, and the names of the unit modules are not limited to [specific examples]. Figure 7 The names of the unit modules in the above-mentioned model computing device 700 can be varied. For example, the model computing device 700 may also include an operator segmentation unit and an operator fusion unit, which are used to segment the training operators corresponding to the first model and deploy the segmented operators to the corresponding CPU or xPU. The name of the processing unit 703 in the above-mentioned model computing device 700 may be changed to a computing unit. The above examples are for illustration only and should not be regarded as specific limitations.

[0092] The aforementioned model segmentation unit 701, resource segmentation unit 702, and processing unit 703 can be implemented in software or hardware. The software and hardware implementation methods of model segmentation unit 701 are described below. The software and hardware implementation methods of resource segmentation unit 702 and processing unit 703 can refer to the software and hardware implementation methods of model segmentation unit 701.

[0093] When the model segmentation unit 701 is implemented by software, the model segmentation unit 701 can be code running on the aforementioned computing device or terminal device. That is, the model segmentation unit 701 can be code running on a personal computer, smartphone, or server, and the number of computing devices or terminal devices can be one or more. That is, the model segmentation unit 701 can also be code running on a cluster of computing devices.

[0094] When the model segmentation unit 701 is implemented in hardware, it can be implemented by at least one computing device or terminal device. Alternatively, the model segmentation unit 701 can also be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), wherein the PLD includes one or more of complex programmable logic devices (CPLD), field-programmable gate arrays (FPGA), and generic array logic (GAL).

[0095] The functions of the model segmentation unit 701, resource segmentation unit 702, and processing unit 703 in the above-mentioned model computing device 700 will be described below.

[0096] The model segmentation unit 701 is used to segment the first model according to the number of xPU accelerator cards to generate multiple second models.

[0097] The aforementioned model segmentation unit 701 is used to implement the above. Figure 2 S201, the model calculation method in the text.

[0098] For a description of the aforementioned xPU accelerator card and the first model, please refer to the above. Figure 2 The relevant description at S201 will not be repeated here.

[0099] The model segmentation unit 701 segments the first model based on the number of xPU accelerator cards, generating multiple second models. Specifically, the model segmentation unit 701 acquires the computing power of the multiple xPU accelerator cards; the model segmentation unit 701 segments the first model according to the ratio between the computing power of the multiple xPU accelerator cards, so that each xPU accelerator card corresponds to one second model, and the computational load of the second model corresponding to each xPU accelerator card is proportional to the computing power of that accelerator card; and the multiple second models are sent to the processing unit 703. This illustrates that different xPU accelerator cards have different computing powers. Segmenting the parent model according to the ratio between the computing powers of the xPU accelerator cards and assigning sub-models with corresponding computational loads based on the computing power of each xPU accelerator card can fully utilize the performance of each xPU accelerator card, thereby improving the computational efficiency of the model.

[0100] The resource partitioning unit 702 is used to partition the CPU according to the number of xPU accelerator cards and generate multiple CPU resource groups.

[0101] The resource partitioning unit 702 described above is used to implement the above. Figure 2 S202, the model calculation method in the text.

[0102] For a description of the CPU resources mentioned above, please refer to the above. Figure 2 The relevant description at S202 will not be repeated here.

[0103] The resource allocation unit 702 allocates CPUs according to the number of xPU accelerator cards, generating multiple CPU resource groups. Specifically, the resource allocation unit 702 allocates CPUs available for model computation according to the ratio between the computing power of the multiple xPU accelerator cards, so that each xPU accelerator card corresponds to a CPU resource group, and the number of CPU resources corresponding to each xPU accelerator card is proportional to the computing power of that accelerator card; and sends the multiple CPU resources to the processing unit 703. It should be noted that different xPU accelerator cards have different computing power; the larger the computing power of an xPU accelerator card, the greater the amount of model computation it can handle. Therefore, by allocating CPUs according to the ratio between the computing power of the xPU accelerator cards and allocating a corresponding number of CPU resources according to the computing power of each xPU accelerator card, the rational allocation of CPU resources is achieved, thereby improving the computational efficiency of the model.

[0104] The processing unit 703 is used to bind each xPU accelerator card to its corresponding CPU resource group and the second model to generate multiple computing groups; to divide the first operator corresponding to the first model and deploy the divided operator to the CPU and each xPU accelerator card; and to start the CPU and each xPU accelerator card to compute multiple second models.

[0105] The aforementioned processing unit 703 is used to implement the above. Figure 2 The model calculation methods S203 and S204 in the text.

[0106] The aforementioned binding of the xPU accelerator card to a CPU resource group creates a computing group. This allows for faster data transfer between the xPU and its corresponding CPU. For example, in deep learning models, model parameters and data often need to be frequently transferred between the CPU and xPU, which can easily cause data congestion and latency. By binding the xPU accelerator card to the CPU resource group, a high-speed transmission channel can be established between the xPU and CPU, thereby reducing latency caused by transmission. Furthermore, by binding the xPU accelerator card and CPU resources, data storage and processing are more likely to occur in the local memory area associated with the xPU accelerator card and CPU resources, thus reducing latency caused by data transfer between different storage areas and further improving the computational efficiency of the model.

[0107] Processing unit 703 divides the first operator corresponding to the first model into segments and deploys the segmented operators to the CPU and each xPU accelerator card. Specifically, processing unit 703 divides the first operator to obtain multiple second operators, including forward computation operators, backward computation operators, optimizer operators, and accompanying operators. Processing unit 703 deploys the forward computation operators and backward computation operators in the second operators to the xPU accelerator cards. Processing unit 703 further divides the accompanying operators and optimizer operators in each second operator and deploys the segmented accompanying operators to their corresponding CPUs and multiple xPU accelerator cards, and deploys the segmented optimizer operators to their corresponding CPUs and multiple xPU accelerator cards. Processing unit 703 adjusts the segmentation strategy of the optimizer operators and accompanying operators through linear programming.

[0108] The first operator mentioned above is the training operator corresponding to the first model, and one of the multiple second operators is the computation operator of one of the multiple second models. The second operators include a forward computation operator, a backward computation operator, an optimizer operator, and an accompanying operator. The forward computation operator and the backward computation operator are used to calculate the output result and the error between the output result and the expected result based on the given input and model parameters. The optimizer operator is used to adjust the model parameters based on the error between the output result and the expected result. The accompanying operator is used to perform auxiliary operations during the model computation process.

[0109] This section explains that due to the structural and functional differences between CPUs and xPUs, deploying operators on the CPU and xPU will have different effects. By segmenting operators based on their structural characteristics and rationally deploying these smaller operators, the overall computational efficiency of the model can be improved. For example, for computationally intensive operators, since xPUs typically have a large number of computational units that can execute the same instructions simultaneously, deploying computationally intensive operators on the xPU can improve computational efficiency. Similarly, for logic processing operators, since CPUs have diverse instruction sets capable of executing various logical instructions, including judgments and comparisons, deploying logic processing operators on the xPU can improve their computational efficiency.

[0110] Optionally, before deploying the second operator to the CPU and multiple xPU accelerator cards, the processing unit 703 will first perform operator fusion on the operators corresponding to the multiple xPU accelerator cards and on the operators corresponding to the CPU through the operator fusion unit 102, thereby reducing DRAM access and improving the computational efficiency of the model.

[0111] The processing unit 703 starts the CPU and each xPU accelerator card to calculate multiple second models. Specifically, the processing unit 703 starts the CPU and each xPU accelerator card to perform parallel calculations on multiple second models. After all the second models have been calculated, the processing unit 703 merges the calculation results of the multiple second models to obtain the calculation result of the first model.

[0112] The optional processing unit 703, during the process of starting the CPU and multiple xPU accelerator cards to calculate multiple second models, also calculates and obtains the utilization rates of the first CPU and the second CPU. The first CPU refers to the CPU resources in the system used for model calculation, and the second CPU refers to the CPU resources in the system not used for the first model calculation. If the utilization rate of the second CPU is less than or equal to the utilization rate of the first CPU, some CPU resources from the second CPU are allocated to the first CPU; if the utilization rate of the second CPU is greater than the utilization rate of the first CPU, some CPU resources from the first CPU are allocated to the second CPU. It should be understood that during model calculation, CPU resources are dynamically adjusted based on the relationship between the CPU utilization rates used for model calculation, the CPU utilization rates not used for model calculation, and the total CPU utilization, thereby improving CPU utilization efficiency. Furthermore, when the CPU utilization rate not used for model calculation is low, allocating more CPU resources to model calculation can reduce the overall calculation time and improve the model's computational efficiency.

[0113] In summary, this application provides a model computing device 700. The model computing device 700 uses a model partitioning unit 701 to partition a first model based on the number of xPU accelerator cards, generating multiple second models; a resource partitioning unit 702 partitions the CPU based on the number of xPU accelerator cards, generating multiple CPU resource groups; and a processing unit 703 binds each xPU accelerator card to its corresponding CPU resource group and the second model, generating multiple computing groups; and then computes multiple second models through these multiple computing groups. In this device, the model computing device 700 rationally partitions the model and CPU based on the number of xPU accelerator cards, and then binds the partitioned models, CPU resources, and xPU accelerator cards. This not only enables parallel processing of the partitioned multiple models, leveraging the computational advantages of each xPU accelerator card, but also reduces latency caused by data transfer between different memory regions by binding CPU resources to xPU accelerator cards, thereby improving the overall computational efficiency of the model.

[0114] The above text combines Figures 2 to 6 The calculation method based on the model provided in this application is described in detail below. Figures 8 to 10 This application describes the computing device and computing device cluster provided in accordance with this application.

[0115] Figure 8 This is a schematic diagram of the structure of a computing device provided in this application. The computing device 800 may be the model computing device 700 mentioned above.

[0116] Furthermore, the computing device 800 includes a central processing unit 801, other processors 802, a communication interface 803, and a storage unit 804. The central processing unit 801, other processors 802, communication interface 803, and storage unit 804 communicate via a bus 805, and also via other means such as wireless transmission.

[0117] In a specific implementation, as one example, the central processing unit 801 includes one or more CPUs, for example... Figure 8 CPU0 and CPU1 are shown in the diagram.

[0118] In a specific implementation, as one example, the computing device 800 also includes multiple processors, for example... Figure 8 The central processing units 801 and 806 are shown. Each of these processors is a multi-core processor. Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0119] Other processors 802 consist of multiple general-purpose or special-purpose processors, such as a TPU, GPU, NPU, or a combination of hardware chips. The aforementioned hardware chips are application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs are complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), data processing units (DPUs), systems-on-chips (SoCs), or any combination thereof. The central processing unit 801 executes various types of digital storage instructions, such as software or firmware programs stored in storage unit 804, enabling the computing device 800 to provide a wide range of services.

[0120] The communication interface 803 is a wired interface (e.g., an Ethernet interface), an internal interface (e.g., a high-speed serial computer expansion bus interface), a wired interface (e.g., an Ethernet interface), or a wireless interface (e.g., a cellular network interface or a wireless LAN interface) for communicating with other servers or units.

[0121] Storage unit 804 is used to store program code, and its execution is controlled by central processing unit 801 to perform the above-mentioned tasks. Figures 2 to 6 The processing steps of the model calculation method in any embodiment. The program code includes one or more software units. The one or more software units mentioned above are... Figure 7 The model segmentation unit 701, resource segmentation unit 702, and processing unit 703 in the embodiment are specifically used for executing... Figure 2 In step S201 of the embodiment, the resource partitioning unit 702 is used to execute... Figure 2 In step S202 of the embodiment, the processing unit 703 is used to execute... Figure 2 Steps S203 and S204 in the embodiment will not be described again here.

[0122] Storage cell 804 is volatile memory, non-volatile memory, or a combination of both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), used as an external cache. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). The term "hard disk" can refer to various types of hard disk drives, such as hard disk drives (HDDs), USB flash drives, flash memory, SD cards (Secure Digital Memory Cards), and memory sticks. Hard disk drives can be classified as hard disk drives (HDDs), solid-state drives (SSDs), or mechanical hard disks (HDDs). This application does not impose any specific limitations on these terms.

[0123] The 805 bus is a Peripheral Component Interconnect Express (PCIe) bus, or an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The 805 bus is divided into address bus, data bus, and control bus.

[0124] In addition to the data bus, the 805 bus also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 805 in the diagram.

[0125] It needs to be explained that, Figure 8 This is merely one possible implementation of an embodiment of this application. In practical applications, the computing device 800 may include more or fewer components, and this is not a limitation. For content not shown or described in the embodiments of this application, please refer to the foregoing. Figures 2 to 6 The relevant descriptions in the embodiments will not be repeated here.

[0126] Figure 9 This is a schematic diagram of a computing device cluster provided in this application, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0127] like Figure 9 As shown, the computing device cluster includes at least one computing device 900. The memory 903 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing model calculation methods.

[0128] In some possible implementations, the memory 903 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the model calculation method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the model calculation method.

[0129] It should be noted that the memories 903 in different computing devices 900 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the resource migration system. Specifically, the instructions stored in the memories 903 of different computing devices 900 can implement the model partitioning unit 701, the resource partitioning unit 702, and the processing unit 703. Specifically, the model partitioning unit 701 is used to execute... Figure 2 In step S201 of the embodiment, the resource partitioning unit 702 is used to execute... Figure 2 In step S202 of the embodiment, the processing unit 703 is used to execute... Figure 2 Steps S203 and S204 in the embodiment will not be described again here.

[0130] The computing device 900 includes a processor 901, a communication interface 902, a memory 903, and a bus 904. Further descriptions of the processor 901, communication interface 902, memory 903, and bus 904 can be found in [reference needed]. Figure 8 The descriptions of the central processing unit 801, storage unit 804, communication interface 803, and bus 805 in the embodiments will not be repeated here.

[0131] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc., as described below. Figure 10 One possible implementation method is shown.

[0132] Figure 10 This is another schematic diagram of a computing device cluster provided in this application, such as... Figure 10 As shown, the two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this possible implementation, the memory 903 in computing device 900A stores instructions for implementing the model partitioning unit 701, the resource partitioning unit 702, and the processing unit 703. Simultaneously, the memory 903 in computing device 900B stores instructions for implementing the model partitioning unit 701, the resource partitioning unit 702, and the processing unit 703.

[0133] It should be understood that Figure 10 The functions of the computing device 900A shown can also be performed by multiple computing devices 900. Similarly, the functions of the computing device 900B can also be performed by multiple computing devices 900.

[0134] It needs to be explained that, Figure 10 The implementation shown may be implemented when the processing power of the computing device 900A is insufficient, or when the storage space of the computing device 900A is insufficient, or in other business scenarios. This application does not make any specific limitations.

[0135] This application also provides another type of computing device cluster. The interconnection relationships between the computing devices in this computing device cluster can be similarly referenced. Figure 9 and Figure 10 The connection method of the computing device cluster. The difference is that the memory 903 of one or more computing devices 900 in the computing device cluster can store the same instructions for executing the model calculation method.

[0136] In some possible implementations, the memory 903 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the model calculation method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the model calculation method.

[0137] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a model calculation method.

[0138] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to perform a model calculation method.

[0139] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0140] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model calculation system, characterized in that, The system is used to perform calculations on a first model, which is divided into multiple second models. The system includes multiple computing groups, each of which includes a central processing unit (CPU) and a heterogeneous processor unit (xPU) accelerator card. The number of second models is equal to the number of computing groups. Each computing group is used to perform calculations on one of the multiple second models. The number of CPU resources in each computing group is proportional to the computing power of the xPU accelerator card in that group, and the amount of computation allocated to the second model in each computing group is proportional to the computing power of the xPU accelerator card in that group.

2. The system according to claim 1, characterized in that, The number of xPU accelerator cards in each computing group is 1, and the CPU resources in each computing group include one or more CPU cores; wherein, when the CPU resources in a computing group include multiple CPU cores, the multiple CPU cores are located in one or more CPUs.

3. The system according to claim 2, characterized in that, The system also includes a segmentation processor. The segmentation processor is used to segment the first model according to the number of xPU accelerator cards to generate the plurality of second models, wherein the computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model. The partitioning processor is also used to partition the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group.

4. The system according to claim 3, characterized in that, The segmentation processor is further configured to segment the first operator to generate multiple second operators. The first operator is a computation operator of the first model, and one of the multiple second operators is a computation operator of one of the multiple second models. Each of the multiple second operators includes a forward computation operator, a backward computation operator, an optimizer operator, and an accompanying operator. The optimizer operator is used to update the model parameters based on the gradient information obtained from the backward computation operator. The accompanying operator is used to assist model computation, including one or more of the following functions: overflow detection, gradient scaling, and precision transformation. The segmentation processor is also used to deploy each second operator to a computing group.

5. The system according to claim 4, characterized in that, The deployment of the second operator on each computation group is specifically used for: The forward computation operator and the backward computation operator of the second operator corresponding to the first computation group are deployed on the xPU accelerator card in the first computation group; wherein, the first computation group is one of the plurality of computation groups, and the second operator corresponding to the first computation group is the computation operator of the second model corresponding to the first computation group; Based on the function of the optimizer operator, the optimizer operator in the second operator corresponding to the first calculation group is divided into a first optimizer operator and a second optimizer operator; Based on the function of the following operator, the following operator in the second operator corresponding to the first calculation group is divided into a first following operator and a second following operator; The first optimizer operator and the first associated operator are deployed on the CPU resources in the first computing group, and the second optimizer operator and the second associated operator are deployed on the xPU accelerator card in the first computing group.

6. The system according to claim 5, characterized in that, The system also includes an operator fusion unit. The operator fusion unit is used to fuse the operators to be deployed to the xPU accelerator cards in the first computing group before the splitting processor deploys each second operator to a computing group.

7. The system according to claim 6, characterized in that, The operator fusion unit is also used to fuse the operators to be deployed on the CPU before the splitting processor deploys each second operator to a computing group.

8. The system according to any one of claims 1 to 7, characterized in that, The system also includes a CPU utilization calculator. The CPU utilization calculator is used to calculate the utilization of a first CPU resource and a second CPU resource, wherein the first CPU resource is the CPU resource in the system used for the calculation of the plurality of second models, and the second CPU resource is the CPU resource in the system other than the first CPU resource. The CPU utilization calculator is further configured to allocate a portion of the CPU resources in the second CPU resource to the first CPU resource when the utilization rate of the second CPU resource is less than or equal to the utilization rate of the first CPU resource.

9. A model calculation method, characterized in that, The method includes: The first model is divided into multiple second models based on the number of xPU accelerator cards. The computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model. The CPU is divided according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group. Each xPU accelerator card is bound to its corresponding CPU resource group to generate multiple computing groups; The multiple second models are calculated using the multiple computing groups.

10. A model calculation device, characterized in that, The device includes: The model segmentation unit is used to segment the first model according to the number of xPU accelerator cards to generate multiple second models, wherein the computing power of each xPU accelerator card is proportional to the computing power of the corresponding second model. The resource partitioning unit is used to partition the CPU according to the number of xPU accelerator cards to generate multiple CPU resource groups, wherein the computing power of each xPU accelerator card is proportional to the number of CPU resources contained in the corresponding CPU resource group. The processing unit is used to bind each xPU accelerator card to its corresponding CPU resource group to generate multiple computing groups; The processing unit is used to calculate the plurality of second models through the plurality of computing groups.

11. A computing device, characterized in that, The computing device includes a processor and a memory, the memory being used to store instructions and the processor being used to execute the instructions such that the computing device implements the method of claim 9.

12. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each of the at least one computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method as described in claim 9.

13. A computer program product containing instructions, characterized in that, The computer program product includes instructions that, when executed on at least one computing device, cause the at least one computing device to perform the method as described in claim 9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that are executed by a computing device or a cluster of computing devices to implement the method of claim 9.

Images