Task execution method, product and device based on hybrid heterogeneous memory resources

Abstract

The application discloses a task execution method, product and equipment based on mixed heterogeneous memory resources, relates to the technical field of artificial intelligence, and comprises the following steps: determining the computing power performance of various computing resources in a server and the storage performance of various mixed heterogeneous memory resources, and respectively counting the communication delay between any two memories; sorting a plurality of memories based on the computing power performance, the memory storage performance and the communication delay; dividing a target natural language processing model into stages according to a computing layer to obtain a plurality of model stages; wherein the memory space required by each model stage is not greater than the size of any memory; deploying the inference tasks of each model stage to the sorted memories in sequence, and executing a natural language processing task by using the deployed model. The application makes full use of various hardware resources, so that efficient deployment of inference tasks can be realized under mixed heterogeneous memory resources with limited resources, thereby improving the efficiency of natural language processing task execution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a task execution method based on mixed heterogeneous memory resources, a product and equipment. BACKGROUND

[0002] With the vigorous development of natural language processing (NLP) technology, large-scale pre-training models such as GPT (Generative Pre-trained Transformer) and Bert (Bidirectional Encoder Representations from Transformers) have been widely applied in fields such as text classification, information retrieval, machine translation, sentiment analysis, medical question answering, speech recognition, and text generation. Moreover, the parameter quantity of pre-training models for natural language processing also presents a significant growth trend and will continue to grow, which greatly increases the demand for computing power and mixed heterogeneous memory resources required for model training and inference.

[0003] However, when deploying inference tasks for large models, it is usually deployed in a resource-constrained scenario on a heterogeneous device cluster (such as mixed heterogeneous memory). For example, when the memory of a single computing device in a server, such as the memory of an accelerator, is insufficient to hold the entire model, the inference task cannot be deployed. It can be seen that when deploying inference tasks, the hardware resources in the server, such as the mixed heterogeneous memory resources under the CPU (Central Processing Unit), are not fully utilized, resulting in deployment failure and low deployment efficiency and poor model inference performance. Therefore, how to efficiently deploy inference tasks for large models under mixed heterogeneous memory resources including accelerator memory, traditional memory, CXL (Compute Express Link, open interconnection standard) memory, etc. is a problem that needs to be further solved in the field. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a task execution method based on mixed heterogeneous memory resources, a product and equipment, which can efficiently deploy model inference tasks under resource-constrained mixed heterogeneous memory resources, reduce the time of natural language processing task execution, reduce power consumption, and improve the efficiency of natural language processing task execution. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a task execution method based on mixed heterogeneous memory resources, comprising:

[0006] determine the computing power performance of various computing resources and the storage performance of various mixed heterogeneous memory resources in the current server to obtain device computing power performance and memory storage performance, and respectively count the communication delay between any two memories in the mixed heterogeneous memory resources;

[0007] sort a plurality of memories in the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance, and the communication delay to obtain sorted memories;

[0008] divide the target natural language processing model into stages according to the computing layers to obtain a plurality of model stages; the number of the plurality of model stages is the same as the number of the computing layers in the target natural language processing model, and the memory space required by each model stage is not greater than the size of any memory in the mixed heterogeneous memory resources;

[0009] deploy the inference tasks of each model stage to the sorted memories in sequence to obtain a deployed model, and execute a natural language processing task by using the deployed model.

[0010] In a second aspect, the present application discloses a computer program product, comprising a computer program, which, when executed by a processor, implements the aforementioned task execution method based on mixed heterogeneous memory resources.

[0011] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the aforementioned task execution method based on mixed heterogeneous memory resources when executing a computer program stored in the memory.

[0012] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned task execution method based on mixed heterogeneous memory resources.

[0013] It can be seen that the present application first determines the computing power performance of various computing resources and the storage performance of various mixed heterogeneous memory resources in the current server to obtain device computing power performance and memory storage performance, and respectively counts the communication delay between any two memories in the mixed heterogeneous memory resources; sorts a plurality of memories in the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance, and the communication delay to obtain sorted memories; divides the target natural language processing model into stages according to the computing layers to obtain a plurality of model stages; the number of the plurality of model stages is the same as the number of the computing layers in the target natural language processing model, and the memory space required by each model stage is not greater than the size of any memory in the mixed heterogeneous memory resources; and deploys the inference tasks of each model stage to the sorted memories in sequence to obtain a deployed model, and executes a natural language processing task by using the deployed model.

[0014] The application can be applied to model inference task deployment under mixed heterogeneous memory resources. First, the computing power performance of various computing resources in the server and the storage performance of mixed heterogeneous memory resources and the communication delay between any two memories are collected, and then the plurality of memories are sorted, and the plurality of model stages obtained after the model is divided are deployed in turn to the sorted memories. Since the computing resources and memory resources in the server are considered at the same time, and the communication delay between different memories is combined, the hardware resources in the server can be fully utilized, so that efficient deployment of model inference tasks can be realized under resource constraints, while the execution time of natural language processing tasks is reduced, the power consumption is reduced, and the efficiency of natural language processing task execution is improved. In addition, since the number of model stages is the same as the number of computing layers in the natural language processing model, and the memory space required by each model stage is not greater than the size of any memory, it can be ensured that the inference tasks of each stage can be successfully deployed. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0016] Figure 1 A task execution method flow chart based on mixed heterogeneous memory resources disclosed by the present application;

[0017] Figure 2 A specific mixed heterogeneous memory resource distribution diagram disclosed by the present application;

[0018] Figure 3 A specific model stage division diagram disclosed by the present application;

[0019] Figure 4 A specific model inference task deployment diagram disclosed by the present application;

[0020] Figure 5 A specific model inference task execution diagram disclosed by the present application;

[0021] Figure 6 A specific model inference task execution diagram disclosed by the present application;

[0022] Figure 7 A task execution method flow chart based on mixed heterogeneous memory resources disclosed by the present application;

[0023] Figure 8This is a schematic diagram illustrating the deployment of a specific model inference task disclosed in this application;

[0024] Figure 9 This is a schematic diagram illustrating the execution of a specific model inference task disclosed in this application;

[0025] Figure 10 This is a schematic diagram illustrating the execution of a specific model inference task disclosed in this application;

[0026] Figure 11 This is a schematic diagram illustrating the execution of a specific model reasoning task disclosed in this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] This application discloses a task execution method based on hybrid heterogeneous memory resources. See also Figure 1 As shown, the method includes:

[0031] Step S11: Determine the computing power performance of various computing resources and the storage performance of various heterogeneous memory resources in the current server to obtain the device's computing power performance and memory storage performance.

[0032] In this embodiment, various computing resources and various heterogeneous memory resources in the current server are first collected, and then the computing power performance of various computing resources and the storage performance of various heterogeneous memory resources are obtained respectively to obtain the corresponding device computing power performance and memory storage performance.

[0033] The computing resources include, but are not limited to, a central processing unit (CPU), an acceleration computing device such as a GPU (graphics processing unit), a TPU (tensor processing unit), an FPGA (Field-Programmable Gate Array), and an AI (Artificial Intelligence) device; and the mixed heterogeneous memory resources include, but are not limited to, a local memory of the central processing unit (such as a Local dram), a memory of another central processing unit connected to the central processing unit (such as a Remoto dram), an open interconnection standard memory (such as a CXL memory), and an acceleration device memory in the acceleration computing device (such as a GPU memory).

[0034] Specifically, the device computing power performance and the memory storage performance are determined by determining the computing power performance of various computing resources in the current server and the storage performance of various mixed heterogeneous memory resources, which can include: obtaining the computing power performance of the central processing unit and the acceleration computing device in the current server to obtain the processor computing power performance and the acceleration device computing power performance; and collecting the sizes of the local memory of the central processing unit, the remote memory connected to the central processing unit, the open interconnection standard memory, and the acceleration device memory in the acceleration computing device in the server to obtain the corresponding local memory size, remote memory size, interconnection standard memory size, and acceleration device memory size. In this embodiment, the CPU and the acceleration computing device (such as a GPU) in the current server can be first collected to obtain the CPU computing power performance and the GPU computing power performance, and then the sizes of the local memory of the CPU, the remote memory connected to the CPU, the CXL memory, and the acceleration device memory in the GPU in the server are collected to obtain the corresponding local memory size, remote memory size, CXL memory size, and GPU memory size. By collecting the performance of various computing resources and various mixed heterogeneous memory resources in the server, the hardware resource situation of the current server can be understood, so that the hardware resources of the server can be fully utilized, and the inference task of the natural language processing model can be reasonably deployed.

[0035] Specifically, the local memory is connected to the operating system of the server through a first memory, the remote memory is connected to the operating system through a memory access node and a second memory, the open interconnection standard memory is connected to the operating system through an interconnection standard controller, a data transmission bus, and a proxy node, and the acceleration device memory is connected to the operating system through a device controller, a data transmission bus, and a proxy node. For example, referring to Figure 2As shown, the operating system (OS) of the server is located in the central processor (CPU), and the local memory of the central processor (CPU) is connected with the operating system through DDR1 (Double Data Rate Synchronous Dynamic Random Access Memory); and the remote memory of the central processor (CPU) is connected with the operating system through DDR2 and NUMA (Non-Uniform Memory Access) node; in addition, the CXL memory is connected with the operating system through CXL controller, data transmission bus such as PCIe (peripheral component interconnect express) bus and agent node (home agent Node); in addition, the memory of the acceleration device (such as GPU memory) is connected with the operating system through device controller, data transmission bus (PCIe bus) and agent node.

[0036] Step S12: respectively statistics the communication delay between any two memories in the mixed heterogeneous memory resource.

[0037] In this embodiment, after determining the computing performance of various computing resources and the storage performance of various mixed heterogeneous memory resources, the communication delay between any two memories in the mixed heterogeneous memory resource is respectively counted.

[0038] In this embodiment, the communication delay between any two memories in the mixed heterogeneous memory resource is respectively counted, which can specifically include: respectively counting the first communication delay between the acceleration device memory and the open interconnection standard memory, the second communication delay between the acceleration device memory and the local memory, and the third communication delay between the acceleration device memory and the remote memory in the mixed heterogeneous memory resource; wherein the first communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the open interconnection standard memory and the interconnection standard controller; the second communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the data transmission bus, the agent node, the local memory and the first memory; and the third communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the data transmission bus, the agent node, the remote memory, the memory access node and the second memory. For example, see Figure 2As shown, the average communication latency between the accelerator device memory (such as GPU memory) in the mixed heterogeneous memory resource and the open interconnection standard memory (i.e., CXL memory) = communication latency of the accelerator device memory (such as GPU memory) (40 ns) + communication latency of the device controller (40 ns) + communication latency of the open interconnection standard memory (40 ns) + communication latency of the interconnection standard controller (40 ns) = 160 ns (minimum). The average communication latency between the accelerator device memory (such as GPU memory) and the local memory = communication latency of the accelerator device memory (40 ns) + communication latency of the device controller (40 ns) + communication latency of the data transfer bus (140 ns) + communication latency of the proxy node (40 ns) + communication latency of the local memory (40 ns) + communication latency of the first memory (80 ns) = 380 ns. The average communication latency between the accelerator device memory and the remote memory = communication latency of the accelerator device memory (40 ns) + communication latency of the device controller (40 ns) + communication latency of the data transfer bus (140 ns) + communication latency of the proxy node (40 ns) + communication latency of the remote memory (40 ns) + communication latency of the memory access node (80 ns) + communication latency of the second memory (80 ns) = 460 ns. By calculating the communication latency between different memories, the communication efficiency between different memories in the current server can be understood, so as to preferentially use the memory with high communication efficiency, which is beneficial to improve the deployment efficiency of subsequent model reasoning tasks and the speed of natural language processing task execution after model deployment.

[0039] Step S13: sorting the plurality of memories in the mixed heterogeneous memory resource based on the device computing power performance, the memory storage performance, and the communication latency, to obtain sorted memories.

[0040] In this embodiment, after obtaining the device computing power performance, the memory storage performance, and the communication latency between any two memories, further, the plurality of memories in the mixed heterogeneous memory resource are sorted based on the device computing power performance, the memory storage performance, and the communication latency between any two memories, to obtain the sorted memories. That is, the sorting of the memories is based on both the hardware resources of the server and the communication efficiency between different memories.

[0041] In a specific embodiment, the order of the memories in the sorted memories is the accelerator device memory, the cache memory, and the local memory in sequence. It can be understood that, referring to Figure 2As shown, in general, the performance of the computing device with acceleration performance is the highest, and the preferred device for large model inference is the device such as GPU, TPU, etc. However, the memory capacity of the device such as GPU, TPU, etc. is relatively small, and the entire model inference task cannot be deployed, so other memories can also be considered to deploy part of the inference task, such as CXL memory, CPU local memory, etc. Considering that the device such as GPU, TPU, etc. has high acceleration computing performance, the memory of the device such as GPU, TPU, etc. can be ranked first, and then the CXL memory and the local memory of the CPU can be selected in turn.

[0042] It can be understood that the CXL memory provides a new way for the CPU to access the device memory and the device to access the CPU memory. The CPU can access the computing device memory in the same way as accessing the local memory, thereby greatly improving the efficiency of data exchange between the device and the host, thereby accelerating the efficiency of deploying the model in the hybrid heterogeneous memory resource.

[0043] In this embodiment, the plurality of memories in the hybrid heterogeneous memory resource are sorted based on the device computing power performance, the memory storage performance, and the communication delay to obtain the sorted memories. Specifically, the memory used for caching the inference tasks of different model stages can be determined from the hybrid heterogeneous memory resource based on the device computing power performance, the memory storage performance, and the communication delay to obtain the cache memory. The plurality of memories in the hybrid heterogeneous memory resource are sorted based on the cache memory to obtain the sorted memories. In this embodiment, the memory used for caching the inference tasks of different model stages is first determined from the hybrid heterogeneous memory resource based on the device computing power performance of different computing devices, the memory storage performance of different memories, and the communication delay between different memories to obtain the cache memory. Then, the plurality of memories in the hybrid heterogeneous memory resource are sorted based on the determined cache memory to obtain the sorted memories. By selecting the memory used for caching the inference tasks of different model stages to obtain the cache memory, the dynamic scheduling of the inference tasks in the plurality of memories can be realized, and the to-be-executed inference tasks can be preferentially cached during the task execution, thereby improving the speed of natural language processing task execution. In a specific implementation manner, the memory with the minimum communication delay with the memory of the acceleration device such as GPU, TPU, etc. can be used as the cache, and of course, other selection rules can also be used to select the cache memory.

[0044] Specifically, determining the memory for caching the inference tasks of different model stages from the mixed heterogeneous memory resources based on the device computing power performance, memory storage performance and communication delay to obtain the cache memory can include: determining the performance with the highest computing power performance from the processor computing power performance and the accelerator device computing power performance to obtain the accelerator device computing power performance; determining whether the acceleration computing device can deploy the entire target natural language processing model based on the accelerator device memory size corresponding to the accelerator device computing power performance; if the acceleration computing device cannot deploy the entire target natural language processing model, determining the memory with the minimum communication delay with the accelerator device memory based on the communication delay to obtain the open interconnection standard memory, and taking the open interconnection standard memory as the cache memory for caching the inference tasks of different model stages. In this embodiment, the performance with the highest computing power performance from the processor computing power performance (i.e., CPU computing power performance) and the accelerator device computing power performance (e.g., GPU computing power performance) is first determined to obtain the accelerator device computing power performance (i.e., GPU computing power performance), and then it is determined whether the corresponding acceleration computing device (i.e., GPU device) can deploy the entire target natural language processing model (e.g., GPT model based on Transformer architecture) based on the accelerator device memory size (i.e., the capacity of the memory) corresponding to the accelerator device computing power performance (i.e., GPU computing power performance); if the memory capacity of the acceleration computing device (i.e., GPU device) cannot deploy the entire target natural language processing model, the memory with the minimum communication delay with the accelerator device memory (i.e., GPU device memory) is determined based on the communication delay between different memories to obtain the open interconnection standard memory (e.g., CXL memory), and then the open interconnection standard memory (i.e., CXL memory) is taken as the cache memory for caching the inference tasks of different model stages.

[0045] In this embodiment, determining whether an accelerated computing device can deploy the entire target natural language processing model is based on the memory size of the accelerated computing device corresponding to its computing power performance. Specifically, this can include: calculating the data volume of the entire inference task corresponding to the target natural language processing model to obtain the model task data volume; determining whether the memory size of the accelerated computing device corresponding to its computing power performance is greater than the model task data volume; if the accelerated computing device memory size is greater than the model task data volume, then it is determined that the accelerated computing device can deploy the entire target natural language processing model; if the accelerated computing device memory size is not greater than the model task data volume, then it is determined that the accelerated computing device cannot deploy the entire target natural language processing model. In this embodiment, when determining whether the accelerated computing device can deploy the model, the data volume of the entire inference task corresponding to the target natural language processing model can be calculated first, and then the relationship between the calculated model task data volume and the accelerated computing device memory size can be used to determine whether the entire model can be deployed. Specifically, if the accelerated computing device memory size is greater than the model task data volume, then it is determined that the accelerated computing device (such as a GPU device) can deploy the entire model; if the accelerated computing device memory size is not greater than the model task data volume, then it is determined that the accelerated computing device (such as a GPU device) cannot deploy the entire model. Specifically, the model task data volume refers to the sum of the parameters of different layers of the model.

[0046] Step S14: Divide the target natural language processing model into stages according to the computational layers to obtain multiple model stages; wherein the number of multiple model stages is the same as the number of computational layers in the target natural language processing model, and the memory space required by each model stage is no greater than the size of any memory in the hybrid heterogeneous memory resources.

[0047] In this embodiment, see Figure 3 As shown, after sorting the multiple memory locations in the hybrid heterogeneous memory resources, the target natural language processing model can be divided into stages according to the principle of one computational layer corresponding to one model stage, resulting in eight model stages, the same number of computational layers as the target natural language processing model. It should be noted that the memory space required by each model stage is no greater than the capacity of any memory location in the hybrid heterogeneous memory resources; that is, the memory space required by each model stage after division is no larger than the size of any single memory location. This ensures that any model stage can be deployed in any memory location, such as CPU local memory, CXL memory, GPU memory, etc.

[0048] Step S15: Deploy the inference tasks of each model stage sequentially into the sorted memory to obtain the deployed model, and use the deployed model to perform natural language processing tasks.

[0049] In this embodiment, after dividing the model into stages, the inference tasks corresponding to each model stage can be deployed sequentially into the sorted memory, and then the deployed models can be used to execute the natural language processing tasks sent by the client.

[0050] Natural language processing tasks include, but are not limited to, text classification, information retrieval, machine translation, intelligent customer service Q&A, sentiment analysis, medical Q&A, voice assistants, search engines, text summarization, text classification, and text generation.

[0051] It is understandable that the model consists of multiple layers, each of which consists of parameters. Therefore, model deployment is essentially writing the parameters of the corresponding layers into memory.

[0052] In this embodiment, the deployment of the model to perform a natural language processing task can specifically include: when a natural language processing task is detected, acquiring a sequence of natural language data to be processed; sequentially inputting the data in the natural language data sequence into the target natural language processing model, and using a first dynamic pipelined task scheduling method to perform pipelined processing on each data in the natural language data sequence to obtain the natural language processing result; wherein, the first dynamic pipelined task scheduling method dynamically schedules inference tasks at different model stages to different memory locations, so that each data in the natural language data sequence is executed in a pipelined mode on different computing devices in the computing resources. See also Figure 4 As shown, the deployed model specifically will... Figure 3 The inference task, divided into 8 model stages, was deployed across 3 different memory locations; among them, the inference task corresponding to inference stage 1 was... The inference task corresponding to inference stage 2 is deployed to accelerated device memory 1 (such as GPU memory 1). The inference tasks corresponding to inference stages 3 to 8 (i.e., other inference stages) are deployed in Open Interconnect Standard Memory 1 (such as CXL Memory 1). to The tasks are deployed to the local memory of CPU 0 (i.e., CPU 0's local memory). CPU 0 is also connected to other CPUs (such as CPU 1) via UPI (Ultra Path Interconnect), and the local memory of CPU 1 can be used as remote memory for CPU 0. In other words, after the initial deployment, the first stage task is deployed in GPU memory, the second stage task is deployed in CXL memory, and the other stages are deployed in the CPU's local memory.

[0053] In this embodiment, when a natural language processing task (such as medical question answering) is detected as requiring execution, a sequence of natural language data to be processed (such as consultation information for a specific disease) is first acquired. Then, the data in this sequence is sequentially input into the target natural language processing model so that each piece of data in the sequence can be processed in a pipelined manner using dynamic pipelined task scheduling to obtain the corresponding natural language processing results. It should be noted that during data processing, inference tasks at different model stages are dynamically scheduled to different memory locations, thereby enabling each piece of data in the natural language data sequence to be executed in a pipelined mode on different computing devices (such as CPUs, GPUs, etc.) within the computing resources. For details, see [link to documentation]. Figure 4 As shown, this accelerates inference tasks in device memory 1 (such as GPU memory 1). The data 1 in the natural language data sequence is processed, and during the processing, Open Interconnect Standard Memory 1 (such as CXL Memory 1) is used to advance the next stage of the inference task, i.e., the inference task. Loading and caching are performed; then, see... Figure 5 As shown, this will accelerate the inference task that runs out of device memory 1 (such as GPU memory 1). Release the inference tasks cached in Central Processing Unit 0 (i.e., CPU0's local memory) and open the cached inference tasks in Interconnect Standard Memory 1 (i.e., CXL Memory 1). The data is loaded into accelerated device memory 1 (such as GPU memory 1) to continue processing data 1. It should be noted that while data 1 is being processed first, data 2 is also being processed simultaneously through an idle inference task located in the local memory of central processing unit 0 (i.e., CPU 0's local memory). See [link to relevant documentation]. Figure 5 As shown, the inference task is performed in local memory (i.e., the local memory of CPU0). When processing data 2, since data 1, data 2, etc., are input in chronological order, the data input earlier will be processed first. For example, data 1 will be processed first. If there is an idle inference task in local memory, and this inference task happens to be the inference task currently required for other data (such as data 2), then... If the inference task required for other data (such as data 2) is occupied by data 1, then data 1 will be processed first. After data 1 is processed, it will be released back to local memory (i.e., local memory of CPU0) before the corresponding inference task is used.

[0054] Additionally, it should be noted that, see Figure 6 As shown, when the acceleration device memory 1 (such as GPU memory 1) executes the final stage of the inference task (i.e., the inference task) ) needs to be processed, since data 2 has been processed by 2 inference tasks, the inference task of stage 3 needs to be executed at present Therefore, the acceleration device memory 1 (such as GPU memory 1) needs to load the current inference task to be executed from the open interconnection standard memory 1 (that is, CXL memory 1) , to continue processing data 2, at the same time, the open interconnection standard memory 1 (that is, CXL memory 1) will continue to load the inference task of stage 4 , so as to process data 2 in pipeline mode. While processing data 2, the idle inference task (such as inference task ) in the local memory of the central processing unit 0 (that is, the local memory of CPU0) also processes data 3 accordingly. Further, the above process is repeated until the last data in the natural language data sequence is completely processed, thereby outputting the final natural language processing result. By dynamically scheduling the flow task, each data in the natural language data sequence is processed in a flow manner, which can efficiently transfer and deploy different stage tasks in different memories, so that different data can be executed in a flow manner on the central processing unit and the acceleration device, thereby fully utilizing the hardware resources in the server and improving the execution speed of the natural language processing task.

[0055] In addition, in the process of deploying the inference tasks of each model stage to the sorted memory in sequence, the data amount of the inference tasks corresponding to any two or more adjacent model stages is counted respectively to obtain a plurality of inference task statistical results. It is judged whether each inference task statistical result is not greater than the size of any memory in the mixed heterogeneous memory resource. If it is not greater than, the corresponding two or more model stages are selected according to the execution order of the inference tasks, and the selected two or more model stages are deployed together into a single memory in the sorted memory to obtain a deployed model. That is, a plurality of adjacent inference tasks are identified and deployed into a single memory, which can reduce the number of inference task scheduling during subsequent natural language processing task execution, thereby further improving the efficiency of subsequent natural language processing task execution.

[0056] It can be seen that the embodiment of the application can be applied to model inference task deployment under mixed heterogeneous memory resources. First, the computing power performance of various computing resources in the server and the storage performance of mixed heterogeneous memory resources and the communication delay between any two memories are collected, and then the plurality of memories are sorted, and the inference tasks of the plurality of model stages obtained after the model is divided are deployed in the sorted memories in turn. Since the computing resources and memory resources in the server are considered at the same time, and the communication delay between different memories is combined, the hardware resources in the server can be fully utilized, so that efficient deployment of model inference tasks can be realized in the case of resource limitation, and the execution time of natural language processing tasks is reduced, the power consumption is reduced, and the efficiency of natural language processing task execution is improved. In addition, since the number of the plurality of model stages is the same as the number of the computing layers in the natural language processing model, and the memory space required by each model stage is not greater than the size of any memory, it can be ensured that the inference tasks of each stage can be successfully deployed.

[0057] The embodiment of the application discloses a specific task execution method based on mixed heterogeneous memory resources, as shown in Figure 7 The method comprises the following steps:

[0058] Step S21: Determine the computing power performance of various computing resources in the current server and the storage performance of various mixed heterogeneous memory resources to obtain device computing power performance and memory storage performance. The computing resources include central processing units and acceleration computing devices. The mixed heterogeneous memory resources include local memories of central processing units, remote memories connected with the central processing units, open interconnection standard memories, and acceleration device memories in the acceleration computing devices.

[0059] Step S22: Statistically analyze the communication delay between any two memories in the mixed heterogeneous memory resources.

[0060] Step S23: Sort the plurality of memories in the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance, and the communication delay to obtain sorted memories. The order of the memories in the sorted memories is acceleration device memory, cache memory, local memory, and remote memory.

[0061] Step S24: Divide the target natural language processing model into stages according to the computing layers to obtain a plurality of model stages. The number of the plurality of model stages is the same as the number of the computing layers in the target natural language processing model, and the memory space required by each model stage is not greater than the size of any memory in the mixed heterogeneous memory resources.

[0062] Step S25: Deploy the inference tasks of the model stages in the sorted memories in turn to obtain a deployed model.

[0063] Referring to Figure 8As shown, when initializing the deployment of the inference task, the inference task corresponding to inference stage 1 can be deployed into the acceleration device memory 1 (such as GPU memory 1) according to the server hardware resources and the communication delay between different memories , the inference task corresponding to inference stage 2 is deployed into the open interconnection standard memory 1 (such as CXL memory 1), the inference tasks corresponding to inference stages 3 to 8 (i.e., other inference stages) are deployed into the central processor 0 local memory (i.e., the local memory of CPU 0), and the inference tasks and are deployed into the remote memory of the central processor 0.

[0064] Step S26: When it is monitored that the natural language processing task needs to be executed, the multiple batches of natural language data sequences to be processed are obtained.

[0065] Step S27: The multiple batches of natural language data sequences are input into the target natural language processing model to perform parallel processing on the multiple batches of natural language data sequences in a second dynamic flow task scheduling manner to obtain a natural language processing result; wherein the second dynamic flow task scheduling manner is to dynamically schedule the inference tasks of different model stages into different memories to make each data in the same batch of natural language data sequences execute in a pipeline mode on different computing devices of the computing resources, and different batches of natural language data sequences execute in parallel on different computing devices of the computing resources.

[0066] In this embodiment, if the data corresponding to the natural language processing task to be processed is multiple batches of natural language data sequences, the multiple batches of natural language data sequences can be input into the target natural language processing model together to perform parallel processing on the multiple batches of natural language data sequences in a dynamic flow task scheduling manner to obtain a natural language processing result, so as to improve the speed of data processing. It should be pointed out that the specific processing process of a single batch of natural language data sequence is the same as that of Figures 4 to 6 , only the remote memory is added. Specifically, refer to Figure 8 As shown, for processing different batches of natural language data sequences, a parallel processing manner can be used, for example, when processing different batches of sequences, the acceleration device memory 1 (such as GPU memory 1) and the open interconnection standard memory 1 (i.e., CXL memory 1) can be used to process the first batch of natural language data sequences, and at the same time, the acceleration device memory 2 (such as GPU memory 2) and the open interconnection standard memory 2 (i.e., CXL memory 2) can be used to process the second batch of natural language data sequences.

[0067] ​​​In this embodiment, in the process of parallel processing of multiple batches of natural language data sequences by using the second dynamic flow task scheduling mode, the process can further include: dynamically releasing the inference task of the model stage that has been calculated in the acceleration computing device, and loading the inference task of the next model stage; when it is detected that the inference task of the last model stage is calculated by the acceleration computing device, determining the inference task of the model stage corresponding to the to-be-processed data in the next batch of natural language data sequences, and loading the inference task corresponding to the to-be-processed data into the cache memory. Specifically, referring to FIG. 2, first, the inference task in the acceleration device memory 1 (such as GPU memory 1) is processed Figure 8 , and then the inference task of the next stage, i.e., the inference task , is loaded and cached in the open interconnection standard memory 1 (such as CXL memory 1) in the processing process. Next, referring to FIG. 3, the used inference task in the acceleration device memory 1 (such as GPU memory 1) is released to the remote memory of the central processing unit 0 (i.e., the local memory of CPU 1), and the cached inference task in the open interconnection standard memory 1 (i.e., CXL memory 1) is loaded into the acceleration device memory 1 (such as GPU memory 1) to continue the processing operation on data 1. At the same time, the idle inference task (such as inference task Figure 9 ) in the local memory of the central processing unit 0 (i.e., the local memory of CPU 0) is used to perform corresponding processing operation on data 2. It can be understood that the performance of the acceleration computing device (such as GPU device) is much higher than that of other computing devices (such as CPU), so the acceleration computing device (such as GPU device) returns the calculated stage task to the CPU memory once to process other data, thereby achieving the effect of parallel processing and achieving higher data processing speed. At the same time, referring to FIG. 4, when the acceleration computing device 1 (such as GPU device 1) calculates to the inference task of the last stage, it is determined which model stage the current batch of data (i.e., data 2) in the CPU calculates to, and the inference task of the next stage (i.e., inference task ) is sent to the open interconnection standard memory 1 (i.e., CXL memory 1) for caching, and the current calculation result (i.e., intermediate calculation result) of data 2 is sent to the acceleration computing device 1 (such as GPU device 1) so that the acceleration computing device 1 (such as GPU device 1) directly receives the inference task corresponding to stage 3 from the CXL memory 1 next time

[0068] Figure 10 ​​​​and the received current calculation result (i.e. intermediate calculation result) is calculated accordingly. Because the communication efficiency between the CXL memory and the acceleration computing device (such as a GPU device) is the highest, such as higher than the memory communication efficiency of a CPU, the data processing procedure using the interaction with the CXL memory is faster.

[0069] In addition, referring to Figure 10 When the acceleration device memory 1 (such as the GPU memory 1) performs the inference task (i.e. inference task ) to the last stage (i.e. stage 10), the next data (such as data 2) needs to be processed, because the data 2 has been processed by 2 inference tasks, and the inference task of stage 3 needs to be executed at present, therefore the acceleration device memory 1 (such as the GPU memory 1) needs to load the current inference task to be executed from the open interconnection standard memory 1 (i.e. CXL memory 1) to continue processing the data 2, and at the same time, the open interconnection standard memory 1 (i.e. CXL memory 1) will continue to load the inference task of stage 4, so as to process the data 2 in the pipeline mode. And at the same time of processing the data 2, the idle inference task (such as inference task ) in the local memory of the central processing unit 0 (i.e. local memory of CPU 0) can also process the data 3 accordingly, and the above process is repeated until all data in the natural language data sequence of multiple batches are processed, so as to obtain the final natural language processing result.

[0070] Among them, the more specific processing process of the above steps S21 to S24 and S26 can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0071] It can be seen that in the embodiments of the present application, the plurality of memories in the mixed heterogeneous memory resources are first sorted based on the performance of various hardware resources in the server and the communication delay between different memories, then the inference tasks of each model stage are deployed in the sorted memories in turn, and when it is monitored that the natural language processing task needs to be executed, the multiple batches of natural language data sequences are input into the target natural language processing model, so as to adopt the dynamic flow task scheduling mode to perform parallel processing on the multiple batches of natural language data sequences, thereby obtaining the natural language processing result. Since the dynamic flow task scheduling mode dynamically schedules the inference tasks of different model stages to different memories, the data in the same batch of natural language data sequences is executed in a pipeline mode on different computing devices of the computing resources, and different batches of natural language data sequences are executed in parallel on different computing devices of the computing resources, so that the execution speed of the natural language processing task can be improved. In addition, the scheduling of the divided inference tasks between different memories and the cooperation of different computing devices through the dynamic flow task scheduling mode can achieve the deployment and execution of the model inference task under the condition of extremely few computing resources, achieve the effects of high resource utilization and high inference performance, and the data between different memories is communicated in a flow mode, which can achieve lower delay, thereby improving the efficiency of data processing and further improving the efficiency of natural language processing. In addition, the performance of various hardware resources and the communication delay between different memories can achieve accurate division of model tasks, reduce the execution time of tasks, and reduce the power consumption of task execution, thereby solving the problems of low communication efficiency of data-intensive tasks and difficult utilization of heterogeneous memory resources, and the heterogeneous memory and bandwidth can be fully utilized.

[0072] Specifically, referring to FIG. 6, Figure 11 It should be noted that the same batch of data is preferentially processed on the acceleration computing device (such as GPU), that is, when the acceleration computing device (such as GPU) finishes processing the inference task of stage 1 of data 1, it dynamically acquires the next batch of data (such as data 2) and the inference task of the current computing stage 3 from the central processing unit (CPU) memory for processing. Through the above dynamic scheduling strategy, highly efficient multi-data parallel processing can be achieved, thereby improving the efficiency of task execution.

[0073] The embodiment of the present application further provides a task execution device based on a hybrid heterogeneous memory resource. The features of the embodiment of the task execution device based on the hybrid heterogeneous memory resource can be referred to the related descriptions of the embodiment of the task execution method based on the hybrid heterogeneous memory resource, which will not be repeated here.

[0074] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor. The memory stores a computer program. The processor is configured to execute the computer program to perform the steps in any of the embodiments of the task execution method based on the hybrid heterogeneous memory resource.

[0075] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is configured to perform the steps in any of the embodiments of the task execution method based on the hybrid heterogeneous memory resource when executed.

[0076] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing computer programs.

[0077] The embodiment of the present application further provides a computer program product, which comprises a computer program. The computer program is executed by a processor to perform the steps in any of the embodiments of the task execution method based on the hybrid heterogeneous memory resource.

[0078] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium. The non-volatile computer readable storage medium stores a computer program. The computer program is executed by a processor to perform the steps in any of the embodiments of the task execution method based on the hybrid heterogeneous memory resource.

[0079] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0080] The above provides a task execution method based on a hybrid heterogeneous memory resource, a product, equipment and a storage medium. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A task execution method based on hybrid heterogeneous memory resources, characterized in that, The method comprises the following steps: determining the computing power performance of various computing resources and the storage performance of various mixed heterogeneous memory resources in the current server to obtain device computing power performance and memory storage performance; statistically determining the communication delay between any two memories in the mixed heterogeneous memory resources; sorting a plurality of memories in the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance and the communication delay to obtain sorted memories; dividing a target natural language processing model into stages according to computing layers to obtain a plurality of model stages; the number of the plurality of model stages is the same as the number of the computing layers in the target natural language processing model, and the memory space required by each model stage is not greater than the size of any memory in the mixed heterogeneous memory resources; deploying the inference tasks of each model stage to the sorted memories in sequence to obtain a deployed model, and using the deployed model to perform a natural language processing task; the computing resources include central processing units and acceleration computing devices; the mixed heterogeneous memory resources include local memories of the central processing units, remote memories connected to the central processing units, open interconnection standard memories and acceleration device memories in the acceleration computing devices; the determination of the computing power performance of various computing resources and the storage performance of various mixed heterogeneous memory resources in the current server to obtain device computing power performance and memory storage performance comprises: obtaining the computing power performance of central processing units and acceleration computing devices in the current server to obtain processor computing power performance and acceleration device computing power performance; collecting the sizes of local memories of the central processing units, remote memories connected to the central processing units, open interconnection standard memories and acceleration device memories in the acceleration computing devices in the server to obtain corresponding local memory sizes, remote memory sizes, interconnection standard memory sizes and acceleration device memory sizes; the sorting of a plurality of memories in the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance and the communication delay to obtain sorted memories comprises: determining memories for caching inference tasks of different model stages from the mixed heterogeneous memory resources based on the device computing power performance, the memory storage performance and the communication delay to obtain cache memories; sorting a plurality of memories in the mixed heterogeneous memory resources based on the cache memories to obtain sorted memories; The determining the memory for caching the inference tasks of different model stages from the mixed heterogeneous memory resources based on the device computing performance, the memory storage performance and the communication delay comprises: determining the performance with the highest computing performance from the processor computing performance and the acceleration device computing performance, to obtain the acceleration device computing performance; determining whether the acceleration computing device can deploy the entire target natural language processing model based on the acceleration device memory size corresponding to the acceleration device computing performance; if the acceleration computing device cannot deploy the entire target natural language processing model, determining the memory with the smallest communication delay with the acceleration device memory based on the communication delay, to obtain the open interconnection standard memory, and taking the open interconnection standard memory as the cache memory for caching the inference tasks of different model stages. 2.The method of claim 1, wherein, The determining whether the acceleration computing device can deploy the entire target natural language processing model based on the acceleration device memory size corresponding to the acceleration device computing performance comprises: statistically obtaining the data amount of the entire inference task corresponding to the target natural language processing model, to obtain the model task data amount size; determining whether the acceleration device memory size corresponding to the acceleration device computing performance is greater than the model task data amount size; if the acceleration device memory size is greater than the model task data amount size, determining that the acceleration computing device can deploy the entire target natural language processing model; if the acceleration device memory size is not greater than the model task data amount size, determining that the acceleration computing device cannot deploy the entire target natural language processing model. 3.The method of claim 1, wherein, The memory order in the sorted memory is the acceleration device memory, the cache memory, the local memory in turn; or, the memory order in the sorted memory is the acceleration device memory, the cache memory, the local memory, the remote memory in turn.

4. The method of claim 3, wherein, The executing the natural language processing task by using the deployed model comprises: when it is monitored that the natural language processing task needs to be executed, obtaining a natural language data sequence to be processed; inputting data in the natural language data sequence into the target natural language processing model in turn, to perform flow processing on each data in the natural language data sequence in a first dynamic flow task scheduling manner, to obtain a natural language processing result; wherein, the first dynamic flow task scheduling manner is to dynamically schedule the inference tasks of different model stages to different memories, so that each data in the natural language data sequence is executed in a pipeline mode on different computing devices in the computing resources.

5. The method of claim 3, wherein, The executing the natural language processing task by using the deployed model comprises: when it is monitored that the natural language processing task needs to be executed, obtaining a plurality of batches of natural language data sequences to be processed; inputting the plurality of batches of natural language data sequences into the target natural language processing model, to perform parallel processing on the plurality of batches of natural language data sequences in a second dynamic flow task scheduling manner, to obtain a natural language processing result; The second dynamic pipelining task scheduling mode is to dynamically schedule inference tasks of different model stages to different memories, so that each data in the same batch of natural language data sequences is executed in a pipelining mode on different computing devices of the computing resource, and different batches of natural language data sequences are executed in parallel on different computing devices of the computing resource.

6. The method of claim 5, wherein, In the process of parallel processing of multiple batches of natural language data sequences by the second dynamic pipelining task scheduling mode, the method further comprises: dynamically releasing the inference task of the model stage currently calculated in the acceleration computing device, and loading the inference task of the next model stage; when it is detected that the inference task of the last model stage is calculated by the acceleration computing device, determining the inference task of the model stage corresponding to the to-be-processed data in the next batch of natural language data sequences, and loading the inference task corresponding to the to-be-processed data into the cache memory.

7. The method of claim 1 to 6, wherein, The local memory is connected to the operating system of the server through a first memory, the remote memory is connected to the operating system through a memory access node and a second memory, the open interconnection standard memory is connected to the operating system through an interconnection standard controller, a data transmission bus and a proxy node, and the acceleration device memory is connected to the operating system through a device controller, the data transmission bus and the proxy node.

8. The method of claim 7, wherein, The communication delay between any two memories in the mixed heterogeneous memory resource is respectively counted, including: The first communication delay between the acceleration device memory and the open interconnection standard memory, the second communication delay between the acceleration device memory and the local memory, and the third communication delay between the acceleration device memory and the remote memory in the mixed heterogeneous memory resource are respectively counted; The first communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the open interconnection standard memory and the interconnection standard controller; The second communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the data transmission bus, the proxy node, the local memory and the first memory; The third communication delay is the sum of the communication delays of the acceleration device memory, the device controller, the data transmission bus, the proxy node, the remote memory, the memory access node and the second memory.

9. An electronic device, comprising: It comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the task execution method based on the mixed heterogeneous memory resource according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and when the computer program is executed by the processor, the steps of the task execution method based on the mixed heterogeneous memory resource according to any one of claims 1 to 8 are implemented.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the task execution method based on the mixed heterogeneous memory resource according to any one of claims 1 to 8.

Images