Memory scheduling method, system and product based on separate memory system

Abstract

The present invention discloses a memory scheduling method, system and product based on a separate memory system, and relates to the field of memory scheduling technology. The target memory device accessed by the current task is determined according to the demand parameters of the current task and the actual operating parameters of the separate memory system. Before the memory device is actually deployed, the target memory device to be accessed by the current task is preliminarily determined according to the demand parameters of the execution of the current task, corresponding to the demand parameters of different tasks and the actual operating parameters of the separate memory system. In order to reduce the access delay corresponding to the current task, the access cost of the current target memory device is estimated by the historical call times and access delay of the target computing accelerator corresponding to the current task, and the scheduling memory device of the current task is determined according to the access cost, so that the access cost of the scheduling memory device accessed by the target computing accelerator corresponding to each task is small, thereby improving the access execution efficiency of the target computing accelerator of the current task.

Description

Technical Field

[0001] The present invention relates to the field of memory scheduling technology, and in particular to a memory scheduling method, system and product based on a separate memory system. Background Art

[0002] The local memory of the computing accelerator and the remote extended memory are connected to the separate memory system so that all the memories in the separate memory can be directly accessed by the memory of the computing accelerator.

[0003] During the task execution process, the data corresponding to the task is placed in a memory of the separated memory system, so that each target computing accelerator can call the data in the memory for execution. When executing regular tasks, the data is deployed in a fixed memory device, and there may be a memory device placed far away from the target computing accelerator, which will introduce a large memory access delay during the memory access process of the target computing accelerator, thereby reducing the execution efficiency of the task.

[0004] Therefore, how to reduce the memory access latency of the target computing accelerator to improve the execution efficiency of the task is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a memory scheduling method, system and product based on a separate memory system to solve the problem of reduced task execution efficiency caused by low memory access latency of a target computing accelerator.

[0006] In order to solve the above technical problems, the present invention provides a memory scheduling method based on a separate memory system, comprising:

[0007] Get the required parameters of the current task;

[0008] Determining a target memory device corresponding to the separate memory system accessed by the current task according to the demand parameters of the current task and the actual operation parameters of the separate memory system;

[0009] If there are multiple target memory devices, the historical call count of the target computing accelerator corresponding to the current task is obtained, and the access cost of the target memory device is obtained by performing an estimated processing based on the historical call count of the target computing accelerator and the access latency corresponding to the target computing accelerator accessing the target memory device;

[0010] A scheduling memory device for the current task is determined in the target memory device according to the access cost.

[0011] On the one hand, the demand parameter includes at least the memory demand of the current task; determining the target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and the actual operation parameter of the separate memory system, comprises:

[0012] Obtaining the memory capacity corresponding to each separate memory device in the separate memory system;

[0013] Determine whether the memory requirement of the current task is smaller than the memory capacity corresponding to each separate memory device;

[0014] If so, a separate memory device whose memory requirement for the current task is smaller than the memory capacity is used as the target memory device.

[0015] On the other hand, the demand parameter includes at least a critical access delay of the current task; determining the target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and the actual operating parameters of the separate memory system, including:

[0016] Obtaining an actual access latency of the target computing accelerator to a current separate memory device of the separate memory system;

[0017] Determining whether an actual access delay of the target computing accelerator to access the current separate memory device is less than the critical access delay;

[0018] If so, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access latency of the target computing accelerator to the current separate memory device of the separate memory system until all the separate memory devices are traversed;

[0019] If not, obtaining the next separate memory device as a new current separate memory device, and returning to the step of obtaining the actual access latency of the target computing accelerator to access the current separate memory device of the separate memory system, until all the separate memory devices are traversed;

[0020] All first target memory devices are used as the target memory devices.

[0021] On the other hand, the access cost of the target memory device is obtained by performing an estimated processing based on the historical call times of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device, including:

[0022] Obtaining a historical number of calls to the target computing accelerator corresponding to the current task within a preset time interval;

[0023] The access cost of the target memory device is obtained by multiplying the historical call count and the access delay.

[0024] On the other hand, the access cost of the target memory device is obtained by multiplying the historical call count by the access delay, including:

[0025] When the number of the target computing accelerator is one, the number of historical calls of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device are multiplied to obtain a first historical delay; each first historical delay is added to obtain an access cost of the target memory device; wherein the number of the first historical delays is the same as the number of the target memory devices;

[0026] When there are multiple target computing accelerators, the historical call times of each target computing accelerator are multiplied by the corresponding access delay to the target memory device to obtain the corresponding second historical delay; and the second historical delays are added to obtain the access cost of the target memory device.

[0027] On the other hand, determining the scheduling memory device of the current task according to the access cost in the target memory device includes:

[0028] Sorting the access costs corresponding to the target memory devices to obtain sorted access costs;

[0029] The minimum access cost is obtained from the sorted access costs, and the target memory device corresponding to the minimum access cost is used as the scheduling memory device of the current task.

[0030] On the other hand, after determining that the current task accesses a target memory device corresponding to the separate memory system according to the demand parameter of the current task and the actual operation parameter of the separate memory system, the method further includes:

[0031] When the number of the target memory devices is zero, obtaining the local memory of the target computing accelerator corresponding to the current task;

[0032] The current task is deployed to the local memory of the target computing accelerator to execute the current task.

[0033] On the other hand, the process of obtaining the current task includes:

[0034] Get the historical call count of each task in the corresponding target computing accelerator;

[0035] The historical call times of the target computing accelerator corresponding to each task are summed up to obtain the total historical call times corresponding to each task;

[0036] Sort the sum of the number of historical calls from large to small to obtain the sorted sum of the number of historical calls;

[0037] The order of tasks corresponding to the sorted total number of historical calls is used as the order of obtaining the current task; wherein the first current task is the task corresponding to the largest total number of historical calls.

[0038] On the other hand, when the current task is an inference task, the process of determining the critical access delay includes:

[0039] Obtaining the number of parameters and key-value cache of the target model corresponding to the inference task;

[0040] Determine the model occupied memory of the target model according to the number of parameters and the number of bytes of each corresponding parameter;

[0041] Determine the total data volume of the target model according to the memory occupied by the model and the key-value cache;

[0042] The critical access latency is determined according to the total data volume and the memory bandwidth corresponding to the reasoning task.

[0043] On the other hand, the process of determining the access latency corresponding to the target computing accelerator accessing the target memory device includes:

[0044] Pre-acquire the number of clock cycles and clock cycle time required for the target memory device to respond to the target computing accelerator access instruction;

[0045] Determine first delay data according to the number of clock cycles and the clock cycle time;

[0046] Acquire a preset amount of input data; and write the preset amount of input data to the target memory device;

[0047] Recording the time required for the input data to complete the operation as the second delay data;

[0048] The first delay data and the second delay data are averaged to obtain an access delay of the target computing accelerator accessing the target memory device.

[0049] On the other hand, the demand parameter includes at least a critical access delay of the current task; determining the target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and the actual operating parameters of the separate memory system, including:

[0050] Obtaining an actual access latency of the target computing accelerator to a current separate memory device of the separate memory system;

[0051] Determining whether an actual access delay of the target computing accelerator to access the current separate memory device is less than the critical access delay;

[0052] If the actual access delay of the target computing accelerator accessing the current separate memory device is less than the critical access delay, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access delay of the target computing accelerator accessing the current separate memory device of the separate memory system until all the separate memory devices are traversed;

[0053] If the actual access latency of the target computing accelerator accessing the current separate memory device is greater than or equal to the critical access latency, then obtaining the next separate memory device as a new current separate memory device, and returning to the step of obtaining the actual access latency of the target computing accelerator accessing the current separate memory device of the separate memory system, until all the separate memory devices are traversed;

[0054] Determining whether all first target memory devices belong to actual operating memory devices of the separate memory system;

[0055] If yes, all first target memory devices are used as the target memory devices.

[0056] In order to solve the above technical problems, the present invention also provides a separate memory system, including a computing server, a computing accelerator and a separate memory device; wherein the computing server is used to implement the memory scheduling method based on the separate memory system as described.

[0057] To solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of a memory scheduling method based on a separate memory system when executed by a processor.

[0058] In order to solve the above technical problems, the present invention further provides a server, comprising:

[0059] Memory for storing computer programs;

[0060] A processor is used to implement the steps of the memory scheduling method based on the separated memory system when executing the computer program.

[0061] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the memory scheduling method based on the separated memory system are implemented.

[0062] The beneficial effects of the present invention are that, on the one hand, the target memory device to be accessed by the current task is determined according to the demand parameters of the current task and the actual operating parameters of the separated memory system. Before the memory device is actually deployed, the demand parameters of the execution of the current task are taken into account, and the factors corresponding to the actual operating parameters of the separated memory system are taken into consideration, so as to preliminarily determine the target memory device to be accessed by the current task according to the demand parameters of the execution of the current task and the actual operating parameters of the separated memory system. Through the actual operating parameters of the separated memory system, real-time deployment based on the actual demand parameters of different tasks is achieved without being placed in a fixed memory device in advance. At the same time, the access delay factors corresponding to the actual operating parameters are also taken into account to determine the corresponding target memory device with the demand parameters, so as to prevent the occurrence of a long access delay caused by the long distance of the memory accessed by the target computing accelerator. Compared with the situation where data is placed in a fixed memory device and the memory device is replaced later when the corresponding memory capacity is small, the present invention determines the target memory device in advance according to the demand parameters and the actual operating parameters before deployment, saving memory resources and computing resources involved in mobile replacement. In addition, corresponding to the data placed in the fixed memory device when there is a fault during the operation of the target computing accelerator, the present invention pre-determines the target memory device in real time through the parameters corresponding to the target computing accelerator corresponding to the actual operation parameters, thereby improving the flexibility of deployment and the reliability of the distributed memory system. On the other hand, on the basis of determining the target memory device, considering that there are more than one target memory devices, in order to reduce the access delay corresponding to the current task, the access cost of the current target memory device is estimated through the historical call times and access delay of the target computing accelerator corresponding to the current task, and the scheduling memory device of the current task is determined according to the access cost, so that the access cost corresponding to the scheduling memory device accessed by the target computing accelerator corresponding to each task is small, and the demand parameters of the current task and various factors of the target computing accelerator accessing the memory are comprehensively considered, so as to realize the dynamic optimization of the memory of the separated memory, improve the access execution efficiency of the target computing accelerator of the current task, and save the memory resources of the separated memory device.

[0063] Secondly, the task order corresponding to the total number of historical calls sorted from large to small is used as the acquisition order of each task for subsequent execution, ensuring that the urgent tasks are processed first and improving the user experience. The screening method based on the memory demand of the current task and the memory capacity corresponding to each separate memory device is traversed and compared to simplify the target memory device determination process and improve the screening efficiency of the determination process. Based on the comparison and screening between the actual access delay of the target computing accelerator to access the current separate memory device and the critical access delay, the target memory device is determined by the access delay factor, taking into account the corresponding delay problem in the access process, so as to improve the access efficiency of the target computing accelerator in the execution process. The process of determining the target memory device, based on the determination based on the first target memory device, adds the factor screening of the actual running memory device to improve the user experience. The process of determining the access cost of the target memory device is to evaluate the efficiency and cost-effectiveness of deploying resources. If the product of the number of historical calls and the access delay is large, then the deployment cost will be large. On the contrary, if this product is small, then the deployment cost will be small. Through the evaluation process, the simplicity of judgment of scheduling memory devices is achieved. The access cost determination process of the target memory device is processed by multiplying the number of historical calls with the access delay. Considering that the total historical delay data corresponding to different numbers of target computing accelerators is different, the uniqueness and flexibility of the determination process are improved. The determination process of the scheduling memory device calculates the appropriate memory optimization deployment plan under the premise of avoiding the large performance impact caused by memory expansion, so that one or more target computing accelerators corresponding to the current task can read the data of the trained model at the lowest cost, and at the same time make the sum of the execution costs of all tasks as low as possible. The current task is executed based on the local memory of the target computing accelerator corresponding to the current task to ensure that the execution process of the current task can be completed sequentially when there is no target memory device. The determination process of the critical access delay is determined by specific data such as the number of model parameters and bandwidth number to improve the accuracy of the determination process and make the subsequent deployment plan authoritative. The access delay determined by two test methods is improved in accuracy, and the corresponding interference problems in the determination process are also reduced through test methods of different dimensions.

[0064] In addition, the present invention also provides a separate memory system, a computer program product, a server and a medium, which have the same beneficial effects as the above-mentioned memory scheduling method based on the separate memory system. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 It is a schematic diagram of a conventional separated memory system architecture;

[0067] Figure 2 A flowchart of a memory scheduling method based on a separate memory system provided by an embodiment of the present invention;

[0068] Figure 3 A schematic diagram of a calling architecture of a separate memory device provided by an embodiment of the present invention;

[0069] Figure 4 A structural diagram of a memory scheduling device based on a separate memory system provided by an embodiment of the present invention;

[0070] Figure 5 A structural diagram of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] The core of the present invention is to provide a memory scheduling method, system and product based on a separate memory system to solve the problem of reduced task execution efficiency caused by low memory access latency of a target computing accelerator.

[0073] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0074] Disaggregated Memory System is a new type of computer architecture. Its core idea is to decouple memory resources from the traditional tightly coupled mode of central processing unit (CPU) memory, so that memory can be managed and dynamically allocated in a pooled manner like storage resources. It is suitable for scenarios with large memory requirements. Although disaggregated memory is used to expand the memory of the target computing accelerator, it is likely to be affected by memory access latency, resulting in low execution efficiency of the entire computing accelerator.

[0075] Figure 1 is a schematic diagram of a conventional separated memory system architecture, such as Figure 1 As shown, in this scenario, the local memory of the computing accelerator ( Figure 1Local video memory 1' to local video memory 3') and remote extended memory ( Figure 1 In the example, memory devices 1' to 3' are represented) will be connected to a separate memory system, and all memories in the separate memory system can be directly accessed as the memory of any computing accelerator. For example, when performing an inference task, the trained model for inference can be placed on a memory device of the separate memory system as model data that can be read-only accessed. All computing accelerators that need to use the model for inference can call the model for inference, which can realize memory reuse to reduce the memory consumption of the inference task and save memory resources. Placing the memory farther away from the computing accelerator will inevitably introduce a larger memory access delay, which will lead to a decrease in computing efficiency. For example, if there is a large delay between the extended memory device and a computing accelerator, although the extended memory device can be used as an extended memory of the computing accelerator, the computing efficiency of the inference task will be greatly affected, and ultimately the user experience will be reduced. The memory scheduling method based on the separate memory system provided by the present invention can solve the above technical problems.

[0076] Figure 2 A flowchart of a memory scheduling method based on a separate memory system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0077] S11: Get the required parameters of the current task;

[0078] S12: determining a target memory device corresponding to the separated memory system accessed by the current task according to the demand parameters of the current task and the actual operation parameters of the separated memory system;

[0079] S13: If there are multiple target memory devices, obtain the historical call count of the target computing accelerator corresponding to the current task, and estimate the access cost of the target memory device according to the historical call count of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device;

[0080] S14: Determine a scheduling memory device for the current task in the target memory device according to the access cost.

[0081] Specifically, the current task in step S11 can be a model training task or a model reasoning task. The corresponding tasks are not limited. The current task can be multiple tasks executed in parallel, or it can be an iterative process of processing the next task after one task is completed. Corresponding to the iterative process of multiple tasks, the current task selection process at this time can be a random selection, or it can be obtained in the order of obtaining the current task according to the priority sorting of memory requirements, which can be set according to actual conditions. During the reasoning task, multiple target computing accelerators access the same memory, that is, the model parameters are finally placed in a memory device, which serves as the final scheduling memory device.

[0082] In some embodiments, the process of obtaining the current task includes:

[0083] Get the historical call count of each task in the corresponding target computing accelerator;

[0084] The historical call times of the target computing accelerator corresponding to each task are summed up to obtain the total historical call times corresponding to each task;

[0085] Sort the sum of the number of historical calls from large to small to obtain the sorted sum of the number of historical calls;

[0086] The order of tasks corresponding to the sorted total number of historical calls is used as the order of obtaining the current tasks; among which, the first current task is the task corresponding to the largest total number of historical calls.

[0087] Specifically, when multiple tasks are performed serially, they can be acquired based on a certain order, taking into account the priority order determined by the memory occupied, training time, etc. corresponding to different tasks, to ensure the execution of tasks with higher priority. In this embodiment, the priority order is determined by the number of historical calls corresponding to the target computing accelerator for each task. The more historical calls, the more frequent and complex the corresponding tasks are, which also reflects the urgency of the tasks. Therefore, the number of historical calls corresponding to the target computing accelerator for each task is added and processed to obtain the total number of historical calls for each task, and the corresponding historical calls are comprehensively sorted by size, so that the task corresponding to the largest total number of historical calls is acquired first, and the task corresponding to the second total number of historical calls is acquired and executed step by step, until the task corresponding to the last smallest total number of historical calls is executed.

[0088] The present embodiment provides a task sequence corresponding to the total number of historical calls sorted from largest to smallest as the acquisition sequence of each task for subsequent execution, thereby ensuring that urgent tasks are processed first and improving user experience.

[0089] The demand parameters of the current task may be various parameters corresponding to the execution of the current task, so as to meet the parameters during the execution of the current task, such as memory demand, maximum tolerable access delay, etc. In step S12, the target memory device corresponding to the access of the current task to the separated memory system is determined according to the demand parameters of the current task and the actual operating parameters of the separated memory system. The separated memory system here includes a system composed of various servers, memories, and computing accelerators, as long as the parameters of each device in the separated memory system during operation are actual operating parameters. The comparison between the demand parameters and the actual operating parameters determines the target memory device accessed by the current task. It can be understood that there are multiple memory storage devices in the separated memory system, and the target memory device accessed by the current task is one or more. The accessible target memory device can be part of the memory of the separated memory system, or all of the memory, which can be set according to the actual situation. The determination process can correspond to the comparison of different parameter combinations, or it can be a comparison between a demand parameter of the current task and the actual operating parameter, which is not limited here. The target memory device can also be determined by a specific comparison algorithm or by comparing the actual operating parameters with the demand parameters after data processing. The target memory device is determined based on the real-time situation of the current task to realize real-time deployment of the target memory device corresponding to the current task, avoiding pre-placement in a certain memory device and then judging execution based on different demand parameters. This embodiment reduces the number of migration operation steps caused by temporarily storing the current task in a certain memory in advance and then moving it based on the execution process of different demand parameters, thereby saving the occupancy of memory resources and the calculation process of computing resources, and improving the execution efficiency of tasks.

[0090] In the case where there are multiple target memory devices determined in step S13, the costs of deploying to different target memory devices are also different, that is, the positions of the target computing accelerators corresponding to the current task to be deployed to the target memory device are different, resulting in different corresponding deployment costs (throughput rates). In order to determine that there is only one target memory device in the end, this embodiment further obtains the historical call times of the target computing accelerator corresponding to the current task as a scheduling memory device, and estimates the access cost of the corresponding target memory device based on the historical call times of the target computing accelerator and the access delay of the target computing accelerator to the target memory device. It should be noted here that this embodiment takes into account the access delay factor of the target computing accelerator when the corresponding target computing accelerator fails, the access delay factor generated by the subsequent task data being moved to other target memory devices, etc., and the access cost obtained by the estimated processing is used to achieve deployment under the condition of small access delay, save memory resources, and improve the flexibility of memory deployment.

[0091] The estimation process in step S13 may be based on the product of the number of calls and the access delay, or may be implemented based on a fixed algorithm based on the number of calls and the access delay, etc., without limitation.

[0092] In step S14, a scheduling memory device is determined based on the access cost, and a scheduling memory device is obtained based on the minimum access cost. Here, it is sufficient to achieve that the final number of scheduling memory devices is one. If the number of target memory devices is one, the target memory device is used as the scheduling memory device of the current task.

[0093] In addition, computing accelerators are used to improve the performance of specific computing tasks, which can surpass the computing efficiency of traditional CPUs and accelerate the computing process by processing large amounts of data in parallel. Specifically, it can be a graphics processor (Graphics Processing Unit, GPU), an application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC) or a field-programmable gate array (Field-Programmable Gate Array, FPGA), etc., which are not limited here. The current task corresponding to this embodiment can be an inference task or other computing tasks. The target computing accelerator is predetermined for each task, which can correspond to the user's settings or the execution demand of the task. The historical number of calls to the target computing accelerator can be called based on the time interval, and the time interval here can be set according to the actual situation.

[0094] The embodiment of the present invention provides a memory scheduling method based on a separate memory system, which obtains the demand parameters of the current task; determines the target memory device corresponding to the current task accessing the separate memory system according to the demand parameters of the current task and the actual operating parameters of the separate memory system; if there are multiple target memory devices, obtains the historical call times of the target computing accelerator corresponding to the current task, and estimates the access cost of the target memory device according to the historical call times of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device; determines the scheduling memory device of the current task in the target memory device according to the access cost. On the one hand, the target memory device accessed by the current task is determined according to the demand parameters of the current task and the actual operating parameters of the separate memory system. Before the actual deployment of the memory device, according to the demand parameters of the execution of the current task, the factors corresponding to the actual operating parameters of the separate memory system are taken into consideration, so as to realize the preliminary determination of the target memory device to be accessed by the current task according to the demand parameters of different tasks and the actual operating parameters of the separate memory system. Through the actual operating parameters of the separate memory system, real-time deployment based on the actual demand parameters of different tasks is realized without placing them in fixed memory devices in advance. At the same time, the access delay factor corresponding to the actual operating parameters and the target memory device corresponding to the demand parameters are also considered to prevent the target computing accelerator from accessing a long memory distance, which causes a long access delay. Compared to placing data in a fixed memory device and subsequently replacing the memory device when the corresponding memory capacity is small, the present invention determines the target memory device in advance based on the demand parameters and the actual operating parameters before deployment, saving memory resources and computing resources involved in mobile replacement. In addition, when data is placed in a fixed memory device corresponding to a fault during the operation of the target computing accelerator, the present invention determines the target memory device in real time in advance through the parameters corresponding to the target computing accelerator corresponding to the actual operating parameters, thereby improving the flexibility of deployment and the reliability of the distributed memory system. On the other hand, based on the determination of the target memory device, taking into account that there are more than one target memory devices, in order to reduce the access delay corresponding to the current task, the access cost of the current target memory device is estimated through the historical call times and access delay of the target computing accelerator corresponding to the current task, and the scheduling memory device of the current task is determined according to the access cost, so that the access cost corresponding to the scheduling memory device accessed by the target computing accelerator corresponding to each task is smaller. The demand parameters of the current task and various factors of the target computing accelerator accessing the memory are comprehensively considered to realize the dynamic optimization of the separated memory, improve the access execution efficiency of the target computing accelerator of the current task, and save the memory resources of the separated memory device.

[0095] In some embodiments, the demand parameter includes at least the memory demand of the current task; determining the target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and the actual operation parameter of the separate memory system includes:

[0096] Obtaining the memory capacity corresponding to each separate memory device in the separate memory system;

[0097] Determine whether the memory requirement of the current task is smaller than the memory capacity corresponding to each separate memory device;

[0098] If it exists, a separate memory device that meets the memory requirement of the current task and is smaller than the memory capacity is used as the target memory device.

[0099] Specifically, the memory requirement of the current task is compared with the memory capacity corresponding to each separate memory device. If there is a memory requirement that is less than the memory capacity, the separate memory device that meets the memory requirement of the current task and is less than the memory capacity will be used as the target memory device.

[0100] The present embodiment provides a screening method that performs traversal comparison based on the memory requirement of the current task and the memory capacity corresponding to each separate memory device, which simplifies the target memory device determination process and improves the screening efficiency of the determination process.

[0101] In some other embodiments, the demand parameter includes at least a critical access delay of the current task; determining a target memory device corresponding to the access of the current task to the separate memory system according to the demand parameter of the current task and actual operating parameters of the separate memory system includes:

[0102] Obtaining an actual access latency of a target computing accelerator to a current separate memory device of the separate memory system;

[0103] Determine whether the actual access delay of the target computing accelerator to the current separate memory device is less than the critical access delay;

[0104] If yes, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access latency of the target computing accelerator accessing the current separate memory device of the separate memory system until all separate memory devices are traversed;

[0105] If not, the next separate memory device is obtained as a new current separate memory device, and the process returns to the step of obtaining the actual access latency of the target computing accelerator to the current separate memory device of the separate memory system until all separate memory devices are traversed;

[0106] Set all first target memory devices as target memory devices.

[0107] Specifically, the actual access delay is based on the time required for the target computing accelerator to access the current separate memory device transmission, including any combination of sending delay (transmission delay), propagation delay, processing delay and queuing delay. The sending delay is the time required for data to be sent from the sender to the network, which can be obtained by dividing the data packet size by the link bandwidth. The propagation delay is the time required for data to be transmitted on the network, which is obtained by dividing the distance from the target computing accelerator to the current separate memory device by the transmission speed. The processing delay is the time required for the target computing accelerator to process data, including operations such as router lookup in the forwarding table and checking the integrity of the data packet. The queuing delay is the time required for data to wait for processing in the cache queue of the target computing accelerator, which depends on the network load and queue length.

[0108] In actual applications, to accurately calculate latency, professional network diagnostic tools such as the ping command are required to help users understand the actual performance of data packets on the network. The actual access latency is particularly important for application scenarios that require fast response, such as real-time communications, online games, and high-frequency trading.

[0109] The critical access latency of the current task is the maximum tolerable memory access latency, which needs to consider multiple factors, including model configuration, hardware performance, and specific application scenarios. Model configuration and parameters directly affect memory access requirements. For example, pre-trained models have a large number of parameters and require more memory to store model weights and intermediate calculation results. The memory bandwidth and computing power of the hardware are key to determining memory access latency. Inference tasks may be compute-limited or memory-bandwidth-limited, and performance can be optimized. Arithmetic intensity refers to the ratio of the number of data bytes transferred from memory per run to the number of floating-point operations performed per run. Different application scenarios have different tolerances for latency, for applications that require real-time response. User experience is also an important factor in determining the maximum tolerable memory access latency. When considering the maximum tolerable memory access latency, cost efficiency must also be considered. Higher memory access speeds may mean higher hardware costs. System performance indicators such as latency, the time it takes from sending a request to receiving a complete response, and throughput measure the ability of the inference server to process multiple requests in parallel.

[0110] By screening and comparing the actual access delay of the target computing accelerator accessing the current separated memory device with the critical access delay, since the number of target computing accelerators can be one or more, the actual access delay corresponding to all current target computing accelerators is compared with the critical access delay for judgment. If it exists, the current separated memory device is used as the first target memory device, and the next separated memory device is obtained as the new current separated memory device for iteration until all separated memory devices corresponding to the current task are traversed. If it does not exist, the next separated memory device is continued to be obtained for iteration until all separated memory devices corresponding to the current task are traversed, and all the first target memory devices screened out are used as target memory devices.

[0111] This embodiment provides a comparison and screening between the actual access delay and the critical access delay of the target computing accelerator accessing the current separate memory device, and determines the target memory device through the access delay factor, taking into account the corresponding delay problem in the access process, so as to subsequently improve the access efficiency of the target computing accelerator during the execution process.

[0112] In some other embodiments, the demand parameter includes at least a critical access delay of the current task; determining a target memory device corresponding to the access of the current task to the separate memory system according to the demand parameter of the current task and actual operating parameters of the separate memory system includes:

[0113] Obtaining an actual access latency of a target computing accelerator to a current separate memory device of the separate memory system;

[0114] Determine whether the actual access delay of the target computing accelerator to the current separate memory device is less than the critical access delay;

[0115] If the actual access delay of the target computing accelerator accessing the current separate memory device is less than the critical access delay, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access delay of the target computing accelerator accessing the current separate memory device of the separate memory system until all separate memory devices are traversed;

[0116] If the actual access latency of the target computing accelerator accessing the current separate memory device is greater than or equal to the critical access latency, then the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access latency of the target computing accelerator accessing the current separate memory device of the separate memory system until all separate memory devices are traversed;

[0117] Determining whether all first target memory devices belong to actual operating memory devices of the separate memory system;

[0118] If yes, all first target memory devices are used as target memory devices.

[0119] Specifically, the determination process corresponding to the first target memory device has been described in the above embodiment, and reference can be made to the above embodiment. In this embodiment, after all the first target memory devices are screened, the fixed running memory devices (actual running memory devices) set by the user are screened. Here, the matching of memory devices of different manufacturers and target computing accelerators and specific requirements are taken into consideration. If all the first target memory devices belong to the actual running memory devices, they are used as target memory devices. If not, the final target memory device can be determined according to the corresponding number. It can be confirmed by the corresponding screening, or it can be re-determined, etc. It is not limited here and can be set according to the actual situation.

[0120] The target memory device determination process provided in this embodiment adds the factor of actually running memory devices to the screening based on the determination of the first target memory device, thereby improving the user experience.

[0121] In some other embodiments, the demand parameters include at least the memory demand of the current task and the critical access delay of the current task; determining the target memory device corresponding to the access of the current task to the separate memory system according to the demand parameters of the current task and the actual operating parameters of the separate memory system includes:

[0122] Obtaining the memory capacity corresponding to each separate memory device in the separate memory system;

[0123] Determine whether the memory requirement of the current task is smaller than the memory capacity corresponding to each separate memory device;

[0124] If it exists, a separate memory device that meets the memory requirement of the current task and is smaller than the memory capacity is used as the target memory device;

[0125] Obtaining an actual access latency of a target computing accelerator to a current second target memory device of the separated memory system;

[0126] Determine whether an actual access delay of the target computing accelerator to the current second target memory device is less than a critical access delay;

[0127] If the actual access delay of the target computing accelerator accessing the current second target memory device is less than the critical access delay, the current second target memory device is used as the third target memory device; the next second target memory device is obtained as the new current second target memory device, and the process returns to the step of obtaining the actual access delay of the target computing accelerator accessing the current second target memory device of the separated memory system until all the second target memory devices are traversed;

[0128] If the actual access latency of the target computing accelerator accessing the current second target memory device is greater than or equal to the critical access latency, then obtaining the next second target memory device as the new current second target memory device, and returning to the step of obtaining the actual access latency of the target computing accelerator accessing the current second target memory device of the separate memory system until all second target memory devices are traversed;

[0129] Set all third target memory devices as target memory devices.

[0130] It should be noted that the determination process corresponding to the second target memory device in this embodiment is the same as the screening process by comparing the memory demand and the memory capacity in the above embodiment, and the above embodiment can be referred to. The determination process corresponding to the third target memory device is the same as the screening process by comparing the actual access delay and the critical access delay of the target computing accelerator to access the current separate memory device in the above embodiment, and the above embodiment can be referred to. This embodiment fully considers the two demand parameters of the current task, the memory demand and the critical access delay, and on the basis of the second target memory device determined based on the memory demand parameter, the critical access delay parameter is considered, and the second target memory device is continued to be judged and screened to obtain the final target memory device.

[0131] In the process of determining the target memory device provided by this embodiment, the screening process of at least two demand parameters is taken into consideration, thereby improving the flexibility of screening and laying a foundation for the subsequent determination of the scheduling memory device.

[0132] In some embodiments, the access cost of the target memory device is obtained by performing an estimated process based on the historical call count of the target computing accelerator and the access latency corresponding to the target computing accelerator accessing the target memory device, including:

[0133] Obtain the historical call count of the target computing accelerator corresponding to the current task within a preset time interval;

[0134] The access cost of the target memory device is obtained by multiplying the historical call count and the access latency.

[0135] Specifically, by obtaining the historical call count of the target computing accelerator corresponding to the current task, the access cost of the target memory device is obtained by multiplying the historical call count with the access delay, that is, the total waiting time (historical delay) is obtained based on the number of requests and the time from the issuance of each request to the receipt of a response.

[0136] It should be noted that this embodiment is based on the total waiting time as the access cost of the target memory device. In other scenarios, the frequency of current computing accelerator calls and performance parameters corresponding to different historical call times can also be considered to select under the corresponding allowable delay parameters.

[0137] The access cost determination process of the target memory device provided in this embodiment is used to evaluate the efficiency and cost-effectiveness of deploying resources. If the product of the number of historical calls and the access delay is large, the deployment cost will be large. On the contrary, if this product is small, the deployment cost will be small. Through the evaluation process, the determination of scheduling memory devices is simple.

[0138] In some embodiments, the access cost of the target memory device is obtained by multiplying the number of historical calls by the access latency, including:

[0139] When the number of target computing accelerators is one, the number of historical calls of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device are multiplied to obtain a first historical delay; each first historical delay is added to obtain an access cost of the target memory device; wherein the number of the first historical delays is the same as the number of the target memory devices;

[0140] When there are multiple target computing accelerators, the historical call times of each target computing accelerator are multiplied by the corresponding access delay of accessing the target memory device to obtain the corresponding second historical delay; and the second historical delays are added to obtain the access cost of the target memory device.

[0141] Specifically, when the number of target computing accelerators corresponding to the current task is one or more, the sum of the corresponding historical delays is different. When the number of target computing accelerators is one, the corresponding number of historical calls is one, and the access delay corresponding to the target computing accelerator accessing the target memory device is multiplied to obtain the first historical delay. Since the number of target memory devices is multiple, the number of the current corresponding first historical delays is the same as the number of target memory devices, and the corresponding total historical delay is the first historical delay corresponding to the same number of target memory devices. The delay is added to obtain the access cost of the target memory device.

[0142] In the case where there are multiple target computing accelerators, the second historical delay equal to the number of target memory devices is obtained by multiplying the historical call times of each target computing accelerator with the corresponding access delay to the target memory device. It should be noted that here only the historical delay determined by one target computing accelerator exists. Currently, there are multiple target computing accelerators, and the corresponding historical delays are collectively referred to as the second historical delay. All the second historical delays are added up to obtain the access cost of the target memory device.

[0143] The specific formula is as follows:

[0144] ;

[0145] in, represents the access latency corresponding to the kth target computing accelerator accessing the jth target memory device; k is the kth target computing accelerator, z represents the total number of target computing accelerators; 1≤k≤z; D j represents the jth target memory device, Indicates the historical call count of the kth target computing accelerator corresponding to the i-th task.

[0146] The process of determining the access cost of the target memory device provided in this embodiment multiplies the number of historical calls by the access delay, taking into account the different total historical delay data corresponding to different numbers of target computing accelerators, thereby improving the uniqueness and flexibility of the determination process.

[0147] In some embodiments, determining a scheduling memory device for a current task in a target memory device according to an access cost includes:

[0148] Sorting the access costs corresponding to the target memory devices to obtain sorted access costs;

[0149] The minimum access cost is obtained from the sorted access costs, and the target memory device corresponding to the minimum access cost is used as the scheduling memory device of the current task.

[0150] Specifically, the access costs corresponding to each target memory device are sorted to determine the sorted access costs, the minimum access cost is obtained from the sorted access costs, and the target memory device corresponding to the minimum access cost is used as the scheduling memory device for the current task. The scheduling memory device takes into account various factors so that all target computing accelerators corresponding to the current task have the lowest access cost to the device and the fastest execution efficiency.

[0151] The scheduling memory device determination process provided in this embodiment calculates a suitable memory optimization deployment plan while avoiding a significant performance impact caused by memory expansion, so that one or more target computing accelerators corresponding to the current task can read the data of the trained model at the lowest possible cost, while making the total execution cost of all tasks as low as possible.

[0152] In some embodiments, after determining that the current task accesses a target memory device corresponding to the separate memory system according to the demand parameters of the current task and the actual operating parameters of the separate memory system, the method further includes:

[0153] When the number of target memory devices is zero, obtain the local memory of the target computing accelerator corresponding to the current task;

[0154] Deploy the current task to the local memory of the target computing accelerator to execute the current task.

[0155] Specifically, if the number of current target memory devices is 0, that is, the corresponding memory device cannot be found for subsequent deployment, the local memory of the target computing accelerator corresponding to the current task is used as the scheduling deployment scheme of the current task to execute the current task.

[0156] This embodiment provides executing the current task based on the local memory of the target computing accelerator corresponding to the current task, so as to ensure that the execution process of the current task is completed sequentially when there is no target memory device.

[0157] In some embodiments, when the current task is an inference task, the process of determining the critical access latency includes:

[0158] Get the number of parameters and key-value cache of the target model corresponding to the inference task;

[0159] Determine the model memory occupied by the target model according to the number of parameters and the corresponding number of bytes of each parameter;

[0160] Determine the total amount of data for the target model based on the model's memory usage and key-value cache;

[0161] Determine the critical access latency based on the total data volume and the memory bandwidth corresponding to the inference task.

[0162] Specifically, when the current task is an inference task, the number of parameters of the target model corresponding to the inference task is obtained. Different numbers of parameters correspond to different memories. The model memory occupied by the target model is determined by multiplying the number of parameters and the number of bytes corresponding to each parameter. The size of the key-value cache depends on the context window size and sequence length of the model. The amount of data required to load a token (the total amount of data of the target model) is the sum of the memory occupied by the model and the key-value cache. The critical access delay is obtained by dividing the total amount of data and the memory bandwidth corresponding to the inference task. It should be noted that the critical access delay is calculated based on ideal conditions. It assumes that the hardware resources are fully utilized by the current task, there are no other system processes or tasks competing for resources, and the hardware performance reaches its theoretical maximum.

[0163] Of course, the critical access delay may also be determined by human factors, such as a value set by human experience.

[0164] The critical access delay determination process provided in this embodiment is determined by specific data such as the parameter quantity and bandwidth number of the model, so as to improve the accuracy of the determination process and make the subsequent deployment plan authoritative.

[0165] In some embodiments, a process of determining an access latency corresponding to a target computing accelerator accessing a target memory device includes:

[0166] Pre-acquire the number of clock cycles and clock cycle time required for the target memory device to respond to the target computing accelerator access instruction;

[0167] Determine first delay data according to the number of clock cycles and the clock cycle time;

[0168] Obtaining a preset amount of input data; and writing the preset amount of input data to a target memory device;

[0169] Record the time required for the input data to complete the operation and use it as the second delay data;

[0170] The first delay data and the second delay data are averaged to obtain an access delay of the target computing accelerator accessing the target memory device.

[0171] It should be noted that the access delay can be determined through performance testing tools, hardware performance testing, or using a memory delay calculator, etc., which are not limited here. The storage process can be stored in a matrix format to save storage memory.

[0172] The first delay data is determined by multiplying the number of clock cycles and the clock cycle time. For example, for random access memory (RAM) delay, the formula can be used: RAM delay = (CAS delay * t CLK ) / 1000, where Column Address Strobe Latency (CAS) is the number of clock cycles required for the memory to respond to a command, t CLK is the clock cycle time, usually measured in nanoseconds.

[0173] Using hardware performance testing, a certain amount of input data (preset amount of input data) is written to the target memory device. The time required for the input data to complete the operation is recorded as the second delay data. It should be noted that in this embodiment, the delay data obtained by two test methods are used. In order to determine the accuracy of the delay time, the first delay data and the second delay data are averaged to obtain the access delay.

[0174] In addition, the determination process may be conducted by a single test method or by a combination of multiple test methods, without limitation.

[0175] This embodiment provides two testing methods to determine the access delay to improve accuracy, and also reduces the corresponding interference problem in the determination process through testing methods of different dimensions.

[0176] Further, Figure 3 A schematic diagram of a call architecture of a separate memory device provided by an embodiment of the present invention, such as Figure 3 As shown, it is divided into three modules, and their names and functions are as follows:

[0177] 1. Data center information collection module: used to periodically collect information on artificial intelligence (AI) reasoning tasks, task execution and calling status, and related information within the data center. The collected data is used as input for the separate memory deployment solution calculation module to calculate the memory deployment solution.

[0178] 2. Separate memory deployment solution calculation module: calculates the separate memory deployment solution based on the information collected by the data center information collection module, and sends it to the separate memory deployment delivery module.

[0179] 3. Separate memory deployment and distribution module: deploys the received separate memory deployment plan into the data center to complete the dynamic optimization of neural network inference memory for separate memory.

[0180] The specific steps for each module are as follows:

[0181] 1. Data center information collection module:

[0182] When a new inference task needs to be deployed, or the time since the last information collection exceeds a certain interval (for example, 1 day, 12 hours, which can be set according to needs), this module will collect real-time information such as inference tasks and memory status of the data center. The specific information collected includes:

[0183] a. Reasoning task information:

[0184] i. The memory requirements of all inference task models that need to be deployed in the data center.

[0185] ii. The computing accelerator deployment location for each inference task (tasks may be deployed to one or more computing accelerators, depending on the customer's settings and the execution requirements of the inference task).

[0186] iii. The number of times each inference task is called on the computing accelerator on which it is deployed during the last collection interval (if it is a newly deployed inference task, the number of calls is 0).

[0187] iv. Maximum tolerable memory access latency for each inference task (manually set).

[0188] b. Data center information:

[0189] i. Available separate memory device identifiers (including non-local memory devices and computing accelerator local memory).

[0190] ii. Memory capacity of discrete memory devices.

[0191] iii. The memory access latency from all available discrete memory devices to all computing accelerators in the data center. This latency can be obtained using the memory access latency test tool and can be stored in a matrix format. The collected information will be sent to the discrete memory deployment solution calculation module to calculate the optimal deployment solution.

[0192] 2. Separate memory deployment solution computing module:

[0193] After receiving the information collected by the data center information collection module, this module triggers the calculation of the separated memory deployment plan and outputs the separated memory deployment plan.

[0194] This module will output the memory deployment location of the inference task model for each inference task that needs to be deployed. Specifically, each inference task model is deployed to a separate memory device. The result of this deployment location will be sent to the separate memory deployment and delivery module for actual deployment.

[0195] 3. Separate memory deployment and delivery module:

[0196] Based on the calculation results of the split memory deployment solution calculation module, this module will automatically open up memory space on the corresponding split memory device in the data center. Subsequently, the module will migrate the trained model to the new split memory device and restart the inference task with the changed model storage location, thus completing the memory optimization and deployment.

[0197] It should be noted that the specific calculation process of the calculation module of the separated memory deployment solution is as follows:

[0198] For each inference task, a separate memory device is determined for placing the model. For example, if the given tasks are [T1, T2, ...], the output is [(T1, D1), (T2, D2), ...], where (T1, D1) represents deploying the inference task model of T1 to the separate memory device of D1. The goal of this algorithm is to ensure that one or more computing accelerators corresponding to each inference task can read the data of the trained model at the lowest possible cost while meeting the maximum tolerable memory access latency of each inference task, while ensuring that the total execution cost of all inference tasks is as small as possible.

[0199] The specific algorithm steps are as follows:

[0200] Given that the current inference tasks that need to be optimized include [T1, T2, ..., Tn] (which may include newly arrived tasks or not, depending on what triggers the information collection of the data center information collection module), the memory requirement of the required inference task model is [q1, q2, ..., qn], and the maximum tolerable memory access latency of each inference task is [l1, l2, ..., ln]. The deployment locations are [[ ][ ]……[ ]], the number of times these tasks were called during the last collection interval on the corresponding computing accelerator is [[ ][ ]……[ ]]. The available separate memory devices are [D1, D2, ..., Dm], and the corresponding memory capacities are [d1, d2, ..., dm], where m represents the mth separate memory device, and its memory capacity corresponds to the memory capacity under the mth separate memory device. XPU i Accessing separate memory device D j The access delay is .

[0201] Step 1: According to the total number of calls of each inference task on all computing accelerators, sort the inference tasks from large to small [T1, T2, ..., Tn] to obtain a sorted inference task list.

[0202] Step 2: For the sorted inference task list, iterate and extract the inference tasks T to be assigned one by one i , the deployment location corresponding to the obtained inference task Ti[ ], z is T i The number of compute accelerator deployment locations.

[0203] Step 3: Iterate over all separate memory devices [D1, D2, ..., Dm], and for one of the separate memory devices D j , 1≤j≤m; if its memory capacity dj<=memory demand qi, or there is an inference task T i The corresponding deployment location computes the accelerator to the separate memory device D j Access latency , then continue to iterate the next separate memory device, otherwise record the calculation in D j Top T i The deployment cost is:

[0204] ;

[0205] Finally, get the smallest , we can get the reasoning task T i The deployment location of the inference model is Dx, and (T i , Dx) to the result list, and change the memory capacity of the deployment location to the new dx=dx-qi. If no , then the reasoning task T i There is no suitable separate memory deployment solution for the task. The task can only use the traditional method to deploy the inference task model to the local memory of each computing accelerator.

[0206] Step 4: If T i If it is the last reasoning task in the sorted reasoning task list, go to step 5, otherwise return to step 2 and continue to iterate the next reasoning task in the sorted reasoning task list.

[0207] Step 5: Output the final result list as the output of the entire algorithm.

[0208] For example, in a data center, there are three inference tasks [T1, T2, T3] being executed, which are deployed to [A100_1, A100_2, A100_3], [A100_4, A_100_5], and [A100_6] respectively. After the data center information collection module collects relevant information, it triggers the dynamic optimization of the inference memory and finds that the number of calls of the three inference tasks in the last monitoring cycle is [100, 50, 3], [1000, 20], and

[50] respectively. After the calculation module of the separated memory deployment solution calls the optimization algorithm, the algorithm will first meet the needs of T2, and according to the algorithm calculation, it is better to deploy the model of the inference task to the local memory location of A100_4. Then, according to the algorithm, the inference task of calculating T1 will be deployed to a location close to A100_1 and A100_2, but relatively far from A100_3. The inference task of calculating T3 will be deployed to the local memory of A100_6. Subsequently, the separate memory deployment and delivery module migrates the trained [T1, T2, T3] model to the corresponding separate memory device, and restarts the inference task with the changed model storage location, finally completing this memory optimization.

[0209] Furthermore, the present invention also provides a separate memory system, including a computing server, a computing accelerator and a separate memory device; wherein the computing server is used to implement a memory scheduling method based on the separate memory system.

[0210] For an introduction to a separate memory system provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above memory scheduling method based on the separate memory system.

[0211] Furthermore, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of a memory scheduling method based on a separate memory system when executed by a processor.

[0212] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above memory scheduling method based on the separated memory system.

[0213] The above describes in detail various embodiments corresponding to the memory scheduling method based on the separated memory system. On this basis, the present invention also discloses a memory scheduling device based on the separated memory system corresponding to the above method. Figure 4 The structure diagram of a memory scheduling device based on a separate memory system provided by an embodiment of the present invention. Figure 4 As shown, the memory scheduling device based on the separated memory system includes:

[0214] The acquisition module 11 is used to obtain the required parameters of the current task;

[0215] A first determination module 12, configured to determine a target memory device corresponding to the separate memory system accessed by the current task according to a demand parameter of the current task and an actual operation parameter of the separate memory system;

[0216] The estimation processing module 13 is used to obtain the historical call times of the target computing accelerator corresponding to the current task if there are multiple target memory devices, and to perform estimation processing according to the historical call times of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device to obtain the access cost of the target memory device;

[0217] The second determining module 14 is used to determine the scheduling memory device of the current task in the target memory device according to the access cost.

[0218] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0219] For an introduction to a memory scheduling device based on a separate memory system provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above memory scheduling method based on a separate memory system.

[0220] Figure 5 A structural diagram of a server provided in an embodiment of the present invention, such as Figure 5 As shown, including:

[0221] A memory 21, used for storing computer programs;

[0222] The processor 22 is used to implement the steps of the memory scheduling method based on the separated memory system when executing the computer program.

[0223] Among them, the processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one hardware form of a digital signal processor (DSP), an FPGA, and a programmable logic array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may also include an AI processor, which is used to process computing operations related to machine learning.

[0224] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the memory scheduling method based on the separated memory system disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, data involved in the memory scheduling method based on the separated memory system, etc.

[0225] In some embodiments, the server may further include a display screen 23 , an input / output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0226] Those skilled in the art can understand that Figure 5 The structure shown in the figure does not constitute a limitation on the server, and may include more or fewer components than shown in the figure.

[0227] The processor 22 implements the memory scheduling method based on the separate memory system provided by any of the above embodiments by calling the instructions stored in the memory 21 .

[0228] For an introduction to a server provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above memory scheduling method based on the separated memory system.

[0229] Furthermore, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor 22, the steps of the memory scheduling method based on the separated memory system are implemented.

[0230] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), RAM, disk or optical disk and other media that can store program code.

[0231] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above memory scheduling method based on the separate memory system.

[0232] The above is a detailed introduction to a memory scheduling method, system and product based on a separate memory system provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0233] It should also be noted that, in this specification, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A memory scheduling method based on a separate memory system, characterized in that: include: Get the required parameters of the current task; Determining a target memory device corresponding to the separate memory system accessed by the current task according to the demand parameters of the current task and the actual operation parameters of the separate memory system; If there are multiple target memory devices, the historical call count of the target computing accelerator corresponding to the current task is obtained, and the access cost of the target memory device is obtained by performing an estimated processing based on the historical call count of the target computing accelerator and the access latency corresponding to the target computing accelerator accessing the target memory device; Determining a scheduling memory device for the current task in the target memory device according to the access cost; The process of obtaining the current task includes: Get the historical call count of each task in the corresponding target computing accelerator; The historical call times of multiple target computing accelerators corresponding to each task are summed up to obtain the total historical call times corresponding to each task; Sort the sum of the number of historical calls from large to small to obtain the sorted sum of the number of historical calls; The task order corresponding to the sorted total number of historical calls is used as the acquisition order of the current task.

2. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: The demand parameters include at least the memory requirement of the current task; Determining, according to the requirement parameters of the current task and the actual operation parameters of the separate memory system, a target memory device corresponding to the separate memory system accessed by the current task, includes: Obtaining the memory capacity corresponding to each separate memory device in the separate memory system; Determine whether the memory requirement of the current task is smaller than the memory capacity corresponding to each separate memory device; If so, a separate memory device whose memory requirement for the current task is smaller than the memory capacity is used as the target memory device.

3. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: The demand parameter at least includes a critical access delay of the current task; determining a target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and actual operating parameters of the separate memory system, including: Obtaining an actual access latency of the target computing accelerator to a current separate memory device of the separate memory system; Determining whether an actual access delay of the target computing accelerator to access the current separate memory device is less than the critical access delay; If so, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access latency of the target computing accelerator to the current separate memory device of the separate memory system until all the separate memory devices are traversed; If not, obtaining the next separate memory device as a new current separate memory device, and returning to the step of obtaining the actual access latency of the target computing accelerator to access the current separate memory device of the separate memory system, until all the separate memory devices are traversed; All first target memory devices are used as the target memory devices.

4. The memory scheduling method based on a separate memory system according to any one of claims 1 to 3, characterized in that: The access cost of the target memory device is obtained by performing an estimated processing according to the historical call times of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device, including: Obtaining a historical number of calls to the target computing accelerator corresponding to the current task within a preset time interval; The access cost of the target memory device is obtained by multiplying the historical call count and the access delay.

5. The memory scheduling method based on the separated memory system according to claim 4, characterized in that: The access cost of the target memory device is obtained by multiplying the historical call count by the access delay, including: When the number of the target computing accelerator is one, the number of historical calls of the target computing accelerator and the access delay corresponding to the target computing accelerator accessing the target memory device are multiplied to obtain a first historical delay; each first historical delay is used as an access cost of each target memory device; wherein the number of the first historical delays is the same as the number of the target memory devices; When there are multiple target computing accelerators, the historical call times of each target computing accelerator are multiplied by the corresponding access delay to the target memory device to obtain the corresponding second historical delay; and the second historical delays are added to obtain the access cost of the target memory device.

6. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: Determining the scheduling memory device of the current task according to the access cost in the target memory device includes: Sorting the access costs corresponding to the target memory devices to obtain sorted access costs; The minimum access cost is obtained from the sorted access costs, and the target memory device corresponding to the minimum access cost is used as the scheduling memory device of the current task.

7. The memory scheduling method based on a separate memory system according to any one of claims 1 to 3, characterized in that: After determining that the current task accesses a target memory device corresponding to the separate memory system according to the demand parameter of the current task and the actual operation parameter of the separate memory system, the method further includes: When the number of the target memory devices is zero, obtaining the local memory of the target computing accelerator corresponding to the current task; The current task is deployed to the local memory of the target computing accelerator to execute the current task.

8. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: The first current task is the task corresponding to the largest total number of historical calls.

9. The memory scheduling method based on the separated memory system according to claim 3, characterized in that: When the current task is an inference task, the process of determining the critical access delay includes: Obtaining the number of parameters and key-value cache of the target model corresponding to the inference task; Determine the model occupied memory of the target model according to the number of parameters and the number of bytes of each corresponding parameter; Determine the total data volume of the target model according to the memory occupied by the model and the key-value cache; The critical access latency is determined according to the total data volume and the memory bandwidth corresponding to the reasoning task.

10. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: The process of determining the access latency corresponding to the target computing accelerator accessing the target memory device includes: Pre-acquire the number of clock cycles and clock cycle time required for the target memory device to respond to the target computing accelerator access instruction; Determine first delay data according to the number of clock cycles and the clock cycle time; Acquire a preset amount of input data; and write the preset amount of input data to the target memory device; Recording the time required for the input data to complete the operation as the second delay data; The first delay data and the second delay data are averaged to obtain an access delay of the target computing accelerator accessing the target memory device.

11. The memory scheduling method based on the separated memory system according to claim 1, characterized in that: The demand parameter at least includes a critical access delay of the current task; determining a target memory device corresponding to the separate memory system accessed by the current task according to the demand parameter of the current task and actual operating parameters of the separate memory system, including: Obtaining an actual access latency of the target computing accelerator to a current separate memory device of the separate memory system; Determining whether an actual access delay of the target computing accelerator to access the current separate memory device is less than the critical access delay; If the actual access delay of the target computing accelerator accessing the current separate memory device is less than the critical access delay, the current separate memory device is used as the first target memory device; the next separate memory device is obtained as the new current separate memory device, and the process returns to the step of obtaining the actual access delay of the target computing accelerator accessing the current separate memory device of the separate memory system until all the separate memory devices are traversed; If the actual access latency of the target computing accelerator accessing the current separate memory device is greater than or equal to the critical access latency, then obtaining the next separate memory device as a new current separate memory device, and returning to the step of obtaining the actual access latency of the target computing accelerator accessing the current separate memory device of the separate memory system, until all the separate memory devices are traversed; Determining whether all first target memory devices belong to actual operating memory devices of the separate memory system; If yes, all first target memory devices are used as the target memory devices.

12. A split memory system, characterized in that: It includes a computing server, a computing accelerator and a separate memory device; wherein the computing server is used to implement the memory scheduling method based on the separate memory system as described in any one of claims 1 to 11.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the memory scheduling method based on a separate memory system as described in any one of claims 1 to 11 are implemented.

14. A server, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the memory scheduling method based on a separate memory system as described in any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the memory scheduling method based on a separate memory system as described in any one of claims 1 to 11 are implemented.

Images