Memory scheduling method and device, chip, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence chips and memory scheduling, in particular to a memory scheduling method and device, a chip, electronic equipment and a storage medium. The invention provides a memory scheduling method for a central processing unit (CPU)-NPU-RCU heterogeneous chip. The method comprises the following steps: firstly, identifying a current task type, judging whether a memory scheduling condition is met or not, and if the memory scheduling condition is met, evaluating the memory access demand of each unit by monitoring the memory access request number and type of a CPU (Central Processing Unit), an NPU (Network Processing Unit) and an RCU (Remote Control Unit) in a system operation cycle. When a memory access read request is detected, the estimated delay tolerance is calculated according to the memory access request number of each unit, and then the memory access priorities of the CPU, the NPU and the RCU in the future system operation cycle are determined. And formulating a memory scheduling scheme based on the priority, and allocating the memory among the CPU, the NPU and the RCU according to the memory scheduling scheme. Therefore, allocation and utilization of memory resources in the heterogeneous chip are optimized, and the overall performance of the system is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of artificial intelligence chips and memory scheduling, and in particular to a memory scheduling method, device, chip, electronic device and storage medium. Background Art

[0002] With the rapid development of AI (Artificial Intelligence) and machine learning technologies, heterogeneous chips have gradually become the mainstream architecture for processing complex computing tasks such as edge AI tasks. Taking the CPU-NPU-RCU heterogeneous chip as an example, the CPU (Central Processing Unit) has strong processing logic control capabilities, but relatively weak parallel computing capabilities. It is generally responsible for executing instructions in programs, processing various data, and managing system resources; NPU (Neural Networks Process Units) is a processing unit designed specifically for neural network computing. With its efficient parallel computing capabilities and special optimization of neural network algorithms, it has shown significant performance advantages in processing such tasks. When processing tasks such as deep learning, the performance of NPU far exceeds that of CPU; RCU (Reconfigurable Computing Unit) is a computing unit with programmable logic devices. It can flexibly configure hardware logic according to different computing tasks to achieve efficient computing. This reconfigurability enables RCU to achieve higher energy efficiency than CPU and NPU when processing specific types of computing tasks.

[0003] Specifically, the CPU-NPU-RCU heterogeneous chip combines the logic control capabilities of the CPU, the neural network computing capabilities of the NPU, and the reconfigurable computing capabilities of the RCU. The CPU is responsible for processing logic control and overall coordination, the NPU is responsible for efficient computing of deep learning tasks, and the RCU flexibly configures hardware logic according to different computing tasks to achieve efficient computing. This heterogeneous chip can fully utilize the advantages of different types of units, thereby improving the overall performance and energy efficiency of the system.

[0004] In the existing CPU-NPU-RCU collaborative work, the memory scheduling mechanism usually lacks sufficient perception and prediction capabilities. Traditional memory scheduling methods often use simple fixed rules to allocate memory tasks. These methods do not take into account the computing characteristics of the NPU and the real-time requirements of the tasks. Therefore, in practical applications, these methods often lead to low memory resource utilization and limited overall system performance. Especially when processing edge AI tasks, due to the complexity and real-time requirements of the tasks, traditional memory scheduling methods often cannot dynamically adjust the memory scheduling strategy according to the real-time requirements of the tasks, which affects the inference speed and accuracy, and also causes unnecessary power consumption waste.

[0005] Specifically, the memory scheduling in the existing CPU-NPU-RCU heterogeneous chips has the following problems:

[0006] 1. Lack of dynamic adjustment capability: Traditional memory scheduling methods usually allocate memory based on static rules and cannot be dynamically adjusted according to the real-time needs of tasks and the operating status of the system. This results in memory resources not being fully utilized during task execution, resulting in memory access conflicts and delays.

[0007] 2. Lack of global optimization perspective: Existing memory scheduling methods often only focus on the performance optimization of a single task or a single processing unit, but lack consideration of optimizing system performance from a global perspective. This results in the overall system performance being limited when multiple tasks or multiple processing units are running at the same time.

[0008] In summary, the memory scheduling mechanism in the existing CPU-NPU-RCU heterogeneous chips has many limitations and cannot meet the requirements of high performance, low power consumption and real-time performance of complex computing tasks such as edge AI tasks.

[0009] How to optimize the allocation and utilization of memory resources in CPU-NPU-RCU heterogeneous chips and improve the overall performance of the system is an urgent problem to be solved. Summary of the invention

[0010] In order to solve the problems in the related art, the embodiments of the present disclosure provide a memory scheduling method, device, chip, electronic device and storage medium.

[0011] In a first aspect, a memory scheduling method is provided in an embodiment of the present disclosure, and the method is applied to a CPU-NPU-RCU heterogeneous chip, wherein the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU, and a reconfigurable computing unit RCU, and the CPU, the NPU, and the RCU exchange data through a shared memory, and the method includes:

[0012] Get the type of the currently executed task;

[0013] Determine whether the type of the task meets the condition for shared memory scheduling. If the type of the task meets the condition for shared memory scheduling, then:

[0014] Acquire the number and type of memory access requests of the CPU, the NPU, and the RCU within a system operation cycle when executing the task, wherein the system operation cycle includes a preset number of clock cycles;

[0015] When it is determined that the memory access request type of any one of the CPU, the NPU, and the RCU in the current system operation cycle is a memory read request, the estimated delay tolerance of the CPU is determined according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task to the end of the current system operation cycle, the estimated delay tolerance of the NPU is determined according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of the task to the end of the current system operation cycle, and the estimated delay tolerance of the RCU is determined according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of the task to the end of the current system operation cycle;

[0016] Calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in a future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle;

[0017] Determining a memory scheduling scheme within the future system operation cycle according to the memory access priority;

[0018] The shared memory is scheduled and allocated among the CPU, the NPU and the RCU according to the memory scheduling scheme.

[0019] According to an embodiment of the present disclosure, the types of tasks include: neural network reasoning training tasks and scientific computing tasks, and determining whether the types of tasks meet the conditions for shared memory scheduling includes:

[0020] When the type of the task is a neural network reasoning training task or a scientific computing task, the type of the task meets the conditions for shared memory scheduling; otherwise, the type of the task does not meet the conditions for shared memory scheduling.

[0021] According to an embodiment of the present disclosure, determining the estimated latency tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes:

[0022] Obtain the number of memory access requests of the CPU in each system operation cycle from the start of the task to the end of the current system operation cycle; accumulate and calculate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU;

[0023] Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes:

[0024] Obtaining the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulating the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; taking the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU;

[0025] Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes:

[0026] Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

[0027] According to an embodiment of the present disclosure, the calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU includes:

[0028] When the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU, and the RCU in the future system operation cycle are calculated as follows:

[0029] Determine the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU and the first auxiliary parameter;

[0030] Determine the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter;

[0031] Determine the memory access priority of the RCU in the future system operation cycle according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and a third auxiliary parameter;

[0032] Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

[0033] According to an embodiment of the present disclosure, determining the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU, and the first auxiliary parameter includes:

[0034] The memory access priority of the CPU in the future system operation cycle is calculated by the following formula:

[0035]

[0036] The determining the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter includes:

[0037] The memory access priority of the NPU in the future system operation cycle is calculated by the following formula:

[0038]

[0039] The determining, according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU, and the third auxiliary parameter, the memory access priority of the RCU in the future system operation cycle includes:

[0040] The memory access priority of the RCU in the future system operation cycle is calculated by the following formula:

[0041]

[0042] Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCU is the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCU is the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCU is the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

[0043] According to an embodiment of the present disclosure, determining the memory scheduling scheme in the future system operation cycle according to the memory access priority includes:

[0044] If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access demand of the RCU;

[0045] If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU is greater than a first preset threshold, then: processing memory access requests of units corresponding to the values ​​greater than the first preset threshold in a batch processing manner within the future system operation cycle;

[0046] If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

[0047] According to an embodiment of the present disclosure, processing the memory access requests of the CPU, the NPU, and the RCU based on an improved first-ready, first-come, first-served algorithm during the future system operation cycle includes:

[0048] Acquire memory access requests in a ready state among the memory access requests of the CPU, the NPU, and the RCU; sort the memory access requests in a ready state in the order of request time to form an initial ready memory access request queue;

[0049] Performing a priority evaluation on each memory access request in the initial ready memory access request queue according to a preset priority rule, and sorting each memory access request in the initial ready memory access request queue in order of priority according to a priority evaluation result to form a final ready memory access request queue;

[0050] Each memory access request in the final ready memory access request queue is processed according to the queue order after the final sorting.

[0051] According to an embodiment of the present disclosure, after forming the final ready memory access request queue and before processing each memory access request in the final ready memory access request queue according to the queue order after the final sorting, the method further includes:

[0052] Obtain the memory access requests that meet the low priority condition in the final ready memory access request queue, and dynamically adjust the memory access request control parameters of the units corresponding to the memory access requests that meet the low priority condition, so that the units corresponding to the memory access requests that meet the low priority condition send memory access requests according to the dynamically adjusted memory access request control parameters; wherein the memory access request control parameters include: the concurrent number of memory access requests and the interval time of memory access requests.

[0053] According to an embodiment of the present disclosure, the scheduling and allocating the shared memory among the CPU, the NPU, and the RCU according to the memory scheduling scheme includes:

[0054] Get the current state parameters when executing the task, the current state parameters include: current reasoning speed v current 、Current power consumption P current , current precision loss L current , the execution progress, CPU load, NPU load and RCU load of each layer in the task;

[0055] The state variables are determined according to the preset target state parameters and the current state parameters, wherein the target state parameters include: the expected inference speed v desired , power consumption limit P max , lower limit of accuracy loss L min and the expected execution progress of each layer in the task; the state variables include: the current reasoning speed v current With the expected inference speed v desired The deviation of the current power consumption P current With the power consumption upper limit P max The deviation, the current accuracy loss L current With the lower limit of precision loss L min and the deviation of the execution progress of each layer in the task from the expected execution progress of each layer in the task;

[0056] Based on the state variable, the Lyapunov function V(x) is constructed by the following formula:

[0057] V(x)=α(v current -v desired ) 2 +β(P current -P max ) 2 +γ(L curent -L min ) 2 +∑ i δ i (Progress i -TargetProgress i ) 2 ;

[0058] where x is a vector containing the state variables, α, β, γ, and δ i is the weight coefficient, Progress i is the execution progress of the i-th layer in the task, TargetProgress i is the expected execution progress of the i-th layer in the task;

[0059] Calculating the derivative of the Lyapunov function V(x) according to the rate of change of the current state parameter; the rate of change of the current state parameter is calculated according to the CPU load, the NPU load and the RCU load;

[0060] The scheduling strategy of each layer in the task is dynamically adjusted according to the sign and size of the derivative.

[0061] In a second aspect, a memory scheduling device is provided in an embodiment of the present disclosure, wherein the memory scheduling device is arranged in a CPU-NPU-RCU heterogeneous chip, wherein the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU and a reconfigurable computing unit RCU, wherein the CPU, the NPU and the RCU exchange data through a shared memory, and the memory scheduling device includes:

[0062] A perception module is configured to obtain the type of a currently executed task; determine whether the type of the task satisfies a condition for shared memory scheduling, and if the type of the task satisfies the condition for shared memory scheduling, then: obtain the number and type of memory access requests of the CPU, the NPU, and the RCU in a system operation cycle when executing the task, wherein the system operation cycle includes a preset number of clock cycles;

[0063] The prediction module is configured to determine the estimated delay tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task execution to the end of the current system operation cycle, when determining that the memory access request type of any one of the CPU, the NPU, and the RCU in the current system operation cycle is a memory read request, determine the estimated delay tolerance of the CPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of the task execution to the end of the current system operation cycle, and determine the estimated delay tolerance of the NPU .... The estimated delay tolerance of the RCU is determined based on the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of the task execution to the end of the current system operation cycle; the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU in a future system operation cycle are calculated based on the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle;

[0064] The scheduling module is configured to determine a memory scheduling scheme in the future system operation cycle according to the memory access priority; and schedule and allocate the shared memory among the CPU, the NPU and the RCU according to the memory scheduling scheme.

[0065] According to an embodiment of the present disclosure, determining the estimated latency tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes:

[0066] Obtain the number of memory access requests of the CPU in each system operation cycle from the start of the task to the end of the current system operation cycle; accumulate and calculate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU;

[0067] Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes:

[0068] Obtaining the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulating the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; taking the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU;

[0069] Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes:

[0070] Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

[0071] According to an embodiment of the present disclosure, the calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU includes:

[0072] When the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU, and the RCU in the future system operation cycle are calculated as follows:

[0073] Determine the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU and the first auxiliary parameter;

[0074] Determine the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter;

[0075] Determine the memory access priority of the RCU in the future system operation cycle according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and the first auxiliary parameter;

[0076] Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

[0077] According to an embodiment of the present disclosure, determining the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU, and the first auxiliary parameter includes:

[0078] The memory access priority of the CPU in the future system operation cycle is calculated by the following formula:

[0079]

[0080] The memory access priority of the NPU in the future system operation cycle is calculated by the following formula:

[0081]

[0082] The memory access priority of the RCU in the future system operation cycle is calculated by the following formula:

[0083]

[0084] Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCUis the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCU is the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCU is the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

[0085] According to an embodiment of the present disclosure, determining the memory scheduling scheme in the future system operation cycle according to the memory access priority includes:

[0086] If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access demand of the RCU;

[0087] If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU is greater than a first preset threshold, then: processing memory access requests of units corresponding to the values ​​greater than the first preset threshold in a batch processing manner within the future system operation cycle;

[0088] If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

[0089] In a third aspect, a chip is provided in an embodiment of the present disclosure, comprising the memory scheduling device described in any one of the second aspect; or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any one of the methods described in the first aspect.

[0090] In a fourth aspect, an electronic device is provided in an embodiment of the present disclosure, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any method described in the first aspect.

[0091] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the method described in any one of the first aspects is implemented.

[0092] In a sixth aspect, a computer program product is provided in an embodiment of the present disclosure, comprising a computer program, wherein when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0093] According to the technical solution provided by the embodiment of the present disclosure, a memory scheduling method for CPU-NPU-RCU heterogeneous chips is proposed. The method first identifies the current task type and determines whether it meets the conditions for shared memory scheduling. If it does, the memory access requirements of each unit are evaluated by monitoring the number and type of memory access requests of the CPU, NPU and RCU during the system operation cycle. When a memory access read request is detected, the estimated delay tolerance is calculated according to the number of memory access requests of each unit, and then the memory access priority of the CPU, NPU and RCU in the future system operation cycle is determined. Based on these priorities, a memory scheduling scheme is formulated, and shared memory is dynamically allocated between the CPU, NPU and RCU according to the memory scheduling scheme. This optimizes the allocation and utilization of memory resources in heterogeneous chips, improves memory access efficiency, and ensures that each processing unit can efficiently access shared resources on demand, thereby improving the overall performance of the system.

[0094] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0096] Figure 1 A flowchart of a memory scheduling method according to an embodiment of the present disclosure is shown;

[0097] Figure 2 A flowchart of a method for calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU according to an embodiment of the present disclosure is shown;

[0098] Figure 3 A flow chart showing a method for processing memory access requests of the CPU, the NPU and the RCU based on an improved first-ready, first-come, first-served algorithm in the future system operation cycle according to an embodiment of the present disclosure;

[0099] Figure 4 A flowchart of another method for processing memory access requests of the CPU, the NPU and the RCU based on an improved first-ready, first-come, first-served algorithm in the future system operation cycle according to an embodiment of the present disclosure is shown;

[0100] Figure 5 A flow chart showing a method for scheduling and allocating the shared memory among the CPU, the NPU and the RCU according to the memory scheduling scheme according to an embodiment of the present disclosure is shown;

[0101] Figure 6 A structural block diagram of a memory scheduling device according to an embodiment of the present disclosure is shown;

[0102] Figure 7 A structural block diagram of a CPU-NPU-RCU heterogeneous chip according to an embodiment of the present disclosure is shown;

[0103] Figure 8 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0104] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0105] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0106] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0107] As mentioned above, in the prior art, CPU-NPU-RCU heterogeneous chips usually allocate memory based on static rules when performing memory scheduling, which may result in the inability to fully utilize memory resources during task execution, resulting in memory access conflicts and delays. In addition, existing memory scheduling methods often only focus on the performance optimization of a single task or a single processing unit, but lack consideration of optimizing system performance from a global perspective. This results in the overall system performance being limited when multiple tasks or multiple processing units are running simultaneously.

[0108] In order to solve the above problems in the prior art, the inventors of the present disclosure have conducted in-depth research on the memory scheduling mechanism in the CPU-NPU-RCU heterogeneous chip, and innovatively proposed a method for dynamically adjusting the memory scheduling of the CPU-NPU-RCU heterogeneous chip based on the task type and the historical and real-time memory access request situation, so as to achieve efficient memory resource allocation and access. Specifically, for tasks suitable for memory scheduling, the memory access requirements of each unit are evaluated by monitoring the number and type of memory access requests of the CPU, NPU and RCU during the system operation cycle. When a memory access read request is detected, the estimated delay tolerance is calculated according to the number of memory access requests of each unit, and then the memory access priority of the CPU, NPU and RCU in the future system operation cycle is determined. Based on these priorities, a memory scheduling scheme is formulated, and memory is dynamically allocated between the CPU, NPU and RCU according to the memory scheduling scheme. This scheduling method based on historical and real-time data and dynamic adjustment can more effectively utilize memory resources and improve the overall performance and efficiency of the CPU-NPU-RCU heterogeneous chip.

[0109] Figure 1 A flowchart of a memory scheduling method according to an embodiment of the present disclosure is shown. The method is applied to a CPU-NPU-RCU heterogeneous chip, which integrates three different types of processing units, namely: a central processing unit CPU, a neural network processing unit NPU, and a reconfigurable computing unit RCU. Each processing unit has its own unique computing capabilities and characteristics, and they work together to complete complex computing tasks.

[0110] For the CPU, tasks suitable for execution on the CPU include:

[0111] 1. Task scheduling and management. The CPU can be responsible for the scheduling and management of the overall neural network computing tasks. Determine the execution order of each subtask, coordinate the work of the NPU and RCU, and monitor the progress of tasks and resource usage. For example, when there are multiple neural network computing tasks at the same time, the CPU can reasonably allocate them according to the priority and resource requirements of the tasks.

[0112] 2. Data preprocessing and postprocessing. In neural network calculations, data preprocessing and postprocessing usually include data cleaning, normalization, format conversion and other operations. These tasks generally do not require a large amount of parallel computing and can be completed efficiently by the CPU. For example, converting image data from the original format to an input format acceptable to the neural network, or interpreting and visualizing the output results of the neural network.

[0113] 3. Small-scale logic judgment and control tasks. For some simple logic judgment and control tasks, such as judging whether the input data meets specific conditions and selecting appropriate neural network models for different usage scenarios, the CPU can process them quickly.

[0114] For NPU, tasks suitable for execution on NPU include:

[0115] 1. Large-scale matrix operations and convolution calculations. NPU is specially optimized for large-scale parallel calculations such as matrix multiplication and convolution in neural networks. These calculations are the core part of neural network training and reasoning, and NPU can complete them efficiently. For example, in image recognition tasks, NPU can quickly process the convolution operation of the image and extract the features of the image.

[0116] 2. Training and reasoning of deep learning models. NPU can accelerate the training and reasoning process of deep learning models. Through parallel computing and hardware optimization, it can greatly shorten the training time and improve the reasoning speed. For example, in natural language processing tasks, NPU can quickly process large-scale text data and perform language model training and reasoning.

[0117] 3. Parallel processing of large-scale data. For large-scale data sets, NPU can use its parallel computing capabilities to process quickly. For example, in big data analysis, NPU can process multiple data samples at the same time to improve the efficiency of data analysis.

[0118] For RCU, tasks suitable for execution in the reconfigurable RCU include:

[0119] 1. Customized computing in specific fields. If the neural network computing task involves unadapted operators (neural networks that have not been optimized through the chip tool chain) or power science (current, voltage, electrical load, electrical parameters) computing requirements, customized hardware acceleration can be performed through the reconfigurable RCU.

[0120] 2. Dynamically adjusted computing tasks. For some tasks that require dynamic adjustment of computing logic according to different situations, the reconfigurability of RCU can enable it to quickly adapt to changes. For example, in real-time video processing, the structure and parameters of the neural network can be adjusted according to different scenarios and requirements, and RCU can quickly implement these adjustments.

[0121] 3. Hardware-accelerated auxiliary tasks. RCU can be used as an auxiliary module of NPU to provide hardware acceleration for some specific computing tasks. For example, during the neural network training process, RCU can accelerate certain specific computing steps.

[0122] 4. Matrix decomposition acceleration. In the process of neural network training and reasoning, large-scale matrix operations are often involved. For example, in the fully connected layer and convolutional layer, matrix multiplication is the main computing operation. RCU can decompose the matrix, split the large matrix into multiple small matrices, and perform calculations in parallel, thereby accelerating the speed of matrix multiplication. For example, for a large matrix multiplication operation A×B, matrices A and B can be decomposed into multiple small matrices respectively, and then the multiplication of these small matrices can be calculated in parallel on the RCU, and finally the results are combined to obtain the final result.

[0123] 5. Sparse matrix calculation acceleration. The weight matrices in many neural networks are sparse, that is, most of the elements are zero. In this case, the traditional matrix multiplication algorithm will waste a lot of computing resources on zero elements. RCU can take advantage of the characteristics of sparse matrices and only calculate non-zero elements, thereby greatly improving computing efficiency. For example, in a sparse convolutional neural network, RCU can quickly identify sparse convolution kernels and only calculate non-zero convolution kernel elements, thereby speeding up the convolution operation.

[0124] 6. Feature extraction acceleration. In fields such as image processing and pattern recognition, feature extraction is an important computational step. For example, in image recognition, useful features such as edges, textures, colors, etc. need to be extracted from the image. RCU can quickly implement various feature extraction algorithms through hardware acceleration. For example, in real-time video surveillance, RCU can quickly extract features from video images for tasks such as target detection and tracking, thereby improving the real-time performance and accuracy of the system.

[0125] 7. Data compression acceleration. Data compression is an important part in big data processing and storage. It can reduce data storage space and transmission bandwidth and improve data processing efficiency. RCU can quickly implement various data compression algorithms such as Huffman coding, Lempel-Ziv coding, arithmetic coding, etc. through hardware acceleration. For example, in a data center, RCU can quickly compress large-scale data sets, reduce storage costs and transmission time, and improve data processing efficiency.

[0126] The CPU, the NPU and the RCU exchange data through shared memory. Shared memory is an efficient communication method that allows multiple processing units to directly access the same memory space, eliminating the delay in data transmission between different processing units, significantly improving the speed of data exchange, and speeding up the execution of overall computing tasks, thereby achieving fast data transmission and sharing.

[0127] In a specific implementation, the present disclosure uses SRAM (Static Random-Access Memory) as shared memory to maximize the interactive speed of memory access.

[0128] like Figure 1 As shown, the method includes the following steps S110 to S170:

[0129] In step S110 , the type of the currently executed task is obtained.

[0130] In step S120, it is determined whether the type of the task meets the conditions for shared memory scheduling. If the type of the task meets the conditions for shared memory scheduling, the following step S130 is executed, otherwise other default or specific memory scheduling strategies may be adopted.

[0131] When starting the memory scheduling scheme provided by the present disclosure, it is an important part of the scheme to determine whether the currently executed task is suitable for memory scheduling. The inventors of the present disclosure found in their research on the memory scheduling mechanism that not all tasks require or are suitable for memory scheduling. Some tasks may have specific memory access modes, or have specific requirements for the latency and bandwidth of memory access. For tasks suitable for memory scheduling, such as tasks that need to frequently exchange data between the CPU, NPU, and RCU, reasonable scheduling can significantly improve the efficiency of memory access, reduce memory access conflicts and delays, and thus improve the performance of the overall system. For tasks that are not suitable for memory scheduling, if this scheduling method is forced to be used, unnecessary overhead may be introduced, such as additional scheduling delays, complex synchronization mechanisms, etc. By judging the task type, these unnecessary overheads can be avoided, the simplicity and efficiency of the system can be improved, and at the same time, it is ensured that only those tasks that can truly benefit from memory scheduling use this scheduling method, thereby optimizing the use of system resources.

[0132] According to the embodiments of the present disclosure, the types of tasks include but are not limited to: neural network reasoning training tasks and scientific computing tasks. In addition, they also include, for example: simple data processing and preprocessing tasks, low-latency control system tasks, and fixed-mode operations in media processing tasks.

[0133] Among them, neural network inference training tasks involve a large number of matrix operations, convolution operations, activation functions, etc. The access patterns of these operations in memory are often irregular and vary with different network structures. During the training process, steps such as weight update and gradient calculation need to read and write memory frequently, and these operations are usually not simple sequential access, but involve a large number of random access and jump access. In addition, neural network models usually take up a lot of memory space, especially when processing large networks or batch data. During the training process, as the number of iterations increases, the memory demand may change dynamically, such as gradient accumulation, storage of intermediate results, etc. In addition, neural network inference training tasks have high requirements for computing performance and memory bandwidth to ensure training speed and model accuracy. Complex memory scheduling strategies can more effectively utilize memory resources, reduce memory bottlenecks, and improve overall performance.

[0134] Scientific computing tasks usually involve large-scale data sets and complex computing processes, which require processing a large number of memory access requests. The size and complexity of the data set may lead to the diversity and unpredictability of memory access patterns. In addition, the algorithms in scientific computing may involve a large number of matrix operations, linear algebra operations, etc., which often lead to poor locality of memory access. In addition, in scientific computing tasks, computing operations and memory access are often intertwined, and memory resources need to be carefully managed to ensure the continuity and efficiency of computing.

[0135] For other types of tasks, such as simple data processing and preprocessing tasks, since the data processing tasks are relatively simple, such as basic cleaning, filtering, or format conversion, and the data set size is not large, the pattern and demand for memory access may be relatively fixed and predictable. In this case, a simple memory allocation and access strategy may be sufficient, and no complex scheduling algorithm is needed.

[0136] Based on the above analysis, when determining whether the type of the task meets the conditions for shared memory scheduling, the present disclosure determines whether to continue the execution of the memory scheduling scheme by judging whether the type of the task is a neural network reasoning training task or a scientific computing task. When the type of the task is a neural network reasoning training task or a scientific computing task, the type of the task meets the conditions for shared memory scheduling, otherwise, the type of the task does not meet the conditions for shared memory scheduling. In specific implementation, the judgment conditions can be flexibly set according to the actual needs of the project. For example, in addition to neural network reasoning training tasks and scientific computing tasks, when it is a task of a certain other type, the next step of the memory scheduling scheme can also be executed.

[0137] When determining whether the type of the task is a neural network reasoning training task or a scientific computing task, the task type can be directly determined by parsing the type identifier or metadata provided when the task is submitted or started, or by other feasible methods.

[0138] The present disclosure determines whether the currently executed task is suitable for shared memory scheduling. For tasks suitable for shared memory scheduling, such as tasks that need to frequently exchange data between the CPU, NPU and RCU, reasonable scheduling can significantly improve the efficiency of memory access, reduce memory access conflicts and delays, and thus improve the performance of the overall system; in addition, this judgment mechanism is more flexible, allowing the system to dynamically select the most appropriate memory scheduling strategy based on the type of task currently being executed. This helps to adapt to different application scenarios and performance requirements, making the system more robust and efficient. As a result, it helps to achieve optimal resource allocation, performance improvement, enhanced flexibility and avoidance of unnecessary overhead, which is of great significance for improving the overall performance of CPU-NPU-RCU heterogeneous chips.

[0139] In step S130, the number of memory access requests and the type of memory access requests of the CPU, the NPU and the RCU in a system operation cycle when executing the task are obtained, wherein the system operation cycle includes a preset number of clock cycles.

[0140] The clock cycle represents a complete cycle of the processor's clock signal. The length of the clock cycle is usually determined by the processor's clock frequency. In the present disclosure, the clock cycle specifically refers to the CPU's clock cycle. For example, if the CPU's clock frequency is 2 GHz, then its clock cycle is 1 / 2 GHz = 0.5 nanoseconds.

[0141] The length of the system operation cycle can be set according to the needs and characteristics of the system. It can be a fixed value or a dynamically changing value (for example, adjusted according to the complexity of the current task or the load of the system). During the system operation cycle, the processor will perform a series of operations, including memory access requests, calculation operations, etc.

[0142] When implementing step S130, it can be performed in the following manner:

[0143] First, determine the length of the system operation cycle, that is, a preset number of clock cycles. This length can be determined based on the characteristics of the system, the nature of the task, and the performance requirements. For example, a fixed number of clock cycles (such as 1000 clock cycles) can be selected as a system operation cycle.

[0144] Then, a memory access request monitoring module is implemented at the interface between the CPU, NPU and RCU and the shared memory. This module is responsible for monitoring and recording the memory access requests issued by each processing unit during each system operation cycle. For each memory access request, the monitoring module needs to record its type (such as read request, write request) and quantity. This can be achieved by setting counters or registers in the monitoring module, and the corresponding counters or registers will be updated whenever a memory access request occurs. In order to track the occurrence time of memory access requests more accurately, a timestamp can be recorded when each request occurs. This timestamp can be associated with the counter of the system operation cycle for subsequent analysis.

[0145] Afterwards, at the end of each system operation cycle after the task starts executing, the monitoring module needs to summarize the collected memory access request data and types for subsequent calculation and analysis. The summarized data needs to be transmitted to a central processing unit (such as a CPU) or a dedicated scheduler for further processing and analysis. This can be achieved through an internal bus or a dedicated communication interface. After receiving the summarized data, the central processing unit or scheduler will process the data, including calculating the number and type of memory access requests for each processing unit, as well as the cumulative number of memory access requests from the start of the task execution to the end of the current system operation cycle.

[0146] In the specific implementation, in order to achieve efficient memory access request monitoring and recording, hardware-level support may be required, such as integrating special monitoring modules and counters in chip design. In addition to hardware support, corresponding software cooperation is also required to achieve data aggregation, transmission and processing. This may require modifying the memory management part of the operating system or adding special drivers to interact with the hardware monitoring module. In addition, when implementing memory access request monitoring and recording, its impact on system performance needs to be considered. Therefore, it is necessary to weigh the relationship between monitoring accuracy and performance overhead to ensure effective memory scheduling without affecting the overall system performance.

[0147] In step S140, when it is determined that the memory access request type of any one of the CPU, the NPU, and the RCU in the current system operation cycle is a memory read request, the estimated delay tolerance of the CPU is determined according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle, the estimated delay tolerance of the NPU is determined according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle, and the estimated delay tolerance of the RCU is determined according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle.

[0148] In the specific implementation of "determining whether the memory access request type of any one of the CPU, NPU, and RCU in the current system operation cycle is a memory access read request", the following two methods can be used:

[0149] Method 1: Single memory access read request trigger: According to the actual situation of the specific system, it can be set to trigger the calculation of latency tolerance as long as a read request is detected in any processing unit (CPU, NPU or RCU). This setting is suitable for applications that are very sensitive to latency.

[0150] Method 2: Triggering by a preset number of memory read requests: In order to reduce unnecessary computing overhead and avoid frequent memory scheduling adjustments, a threshold (such as N memory read requests) can be set. The calculation of the estimated latency tolerance is triggered only when the number of memory read requests of a processing unit in the current system operation cycle reaches or exceeds this threshold. This threshold can be adjusted according to actual applications and system performance requirements.

[0151] The estimated latency tolerance refers to the tolerance of each processing unit (CPU, NPU, RCU) to memory access latency during the current system operation cycle. It is calculated based on the ratio of the number of memory access requests during the current system operation cycle to the total number of memory access requests accumulated from the start of task execution to the end of the current system operation cycle, reflecting the urgency and relative importance of the memory access requirements of the processing unit during the current system operation cycle.

[0152] Specifically, when calculating the estimated delay tolerance, according to an embodiment of the present disclosure, determining the estimated delay tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes:

[0153] Obtain the number of memory access requests of the CPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU.

[0154] Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes:

[0155] Obtaining the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulating the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; taking the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU;

[0156] Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes:

[0157] Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

[0158] The present disclosure innovatively proposes the concept of estimated delay tolerance, and defines the estimated delay tolerance of each processing unit as the ratio of the number of memory access requests in the current system operation cycle to the total number of accumulated memory access requests. The estimated delay tolerance of each processing unit reflects the memory access urgency and delay tolerance of each processing unit in the current system operation cycle. Then, according to the estimated delay tolerance of each processing unit, the priority of memory access can be arranged more reasonably and memory resources can be allocated more fairly. If the ratio of the estimated delay tolerance is higher, it means that the memory access demand in the current cycle is more urgent and the tolerance for memory access delay is lower. In this way, for processing units with lower delay tolerance, their memory access needs can be met first to reduce waiting time and improve overall system performance.

[0159] In step S150, the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU in a future system operation cycle are calculated according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle.

[0160] Figure 2A flowchart of a method for calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU according to an embodiment of the present disclosure is shown. Figure 2 As shown, when calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU, the following steps S151-154 can be performed:

[0161] In step S151, when the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU and the RCU in the future system operation cycle are calculated as follows.

[0162] The "memory access priority" involved in this disclosure is an indicator used to determine the priority of different processing units (such as CPU, NPU, RCU, etc.) in accessing shared memory resources. It reflects the priority of each processing unit relative to other units in accessing shared memory during a specific system operation cycle. The higher the value, the higher the priority of the processing unit in accessing shared memory, and the more likely it is to obtain memory access rights.

[0163] In the case where the latency tolerance of NPU and CPU is low (i.e., both are lower than the preset minimum latency tolerance threshold, such as 5%), the disclosure takes into account that NPU and CPU may be facing high latency or performance bottlenecks, and RCU usually handles real-time or near-real-time tasks, such as input / output operations, network communications, etc., and may need faster access to memory to complete its tasks, and has higher requirements for memory access latency. At this time, giving RCU a higher memory access priority can, on the one hand, more effectively utilize system resources and avoid resource waste, and on the other hand, make full use of its characteristics to improve the overall performance of the system.

[0164] In step S152, the memory access priority of the CPU in the future system operation cycle is determined according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU and the first auxiliary parameter.

[0165] In step S153, the memory access priority of the NPU in the future system operation cycle is determined according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter.

[0166] In step S154, the memory access priority of the RCU in the future system operation cycle is determined according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and the third auxiliary parameter.

[0167] Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

[0168] Specifically, the memory access priority of the CPU in the future system operation cycle can be calculated by the following formula:

[0169]

[0170] The memory access priority of the NPU in the future system operation cycle is calculated by the following formula:

[0171]

[0172] The memory access priority of the RCU in the future system operation cycle is calculated by the following formula:

[0173]

[0174] Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCU is the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCU is the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCUis the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

[0175] The present disclosure calculates the memory access priority of each processing unit based on the current number of memory access requests, the total number of shared memory access requests, the estimated delay tolerance and the pre-determined auxiliary parameters, comprehensively considering the current demand of the processing unit, the overall memory access load and its sensitivity to delay, so that the memory access priority of the CPU, NPU and RCU can be dynamically adjusted according to the current situation, ensuring that the memory access demand of the key processing unit is prioritized in the case of delay sensitivity, and avoiding performance bottlenecks caused by memory access delays. At the same time, when the memory access resources are abundant, resources can be allocated more fairly, and the parallel processing efficiency of the overall system can be improved. In addition, the memory access priority is calculated based on the current memory access demand and the delay tolerance, which can more accurately reflect the actual needs of each processing unit. This helps to avoid resource waste and over-allocation and improve the utilization of shared memory. In addition, the introduction of auxiliary parameters makes the calculation of memory access priority more flexible and can be adjusted according to different tasks or application scenarios. This helps heterogeneous chips adapt to a wider range of task types and demand changes, and improves their versatility and practicality.

[0176] Among them, when determining the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter, they are determined by the batch size during task execution, which is based on the consideration of the direct impact on memory access performance. The impact is specifically manifested as follows: During task execution, the batch size (Batch Size) determines the number of samples used each time the model weights are updated, which directly affects the memory access requirements and modes, for example: a larger batch size usually means that more memory is required to store input data, intermediate results and model parameters.

[0177] In step S160, a memory scheduling scheme in the future system operation cycle is determined according to the memory access priority.

[0178] According to an embodiment of the present disclosure, the memory scheduling scheme in the future system operation cycle is determined according to the memory access priority. Based on different situations of memory access priority, the present disclosure provides the following three methods:

[0179] Method 1: If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access requirements of the RCU.

[0180] Method 2: If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is greater than a first preset threshold, then: the memory access requests of the units corresponding to the values ​​greater than the first preset threshold are processed in a batch manner during the future system operation cycle.

[0181] Method 3: If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

[0182] For the above method 1, when the memory access priority of the CPU and NPU is 0, and the memory access priority of the RCU is 100%, it means that in the current and future system operation cycles, the memory access demand of the RCU is extremely urgent, and the memory access demand of the CPU and NPU is relatively low or can be processed delayed. Therefore, it is chosen to process only the memory access request of the RCU in the future system operation cycle, that is, the RCU has exclusive memory access to ensure that the RCU can obtain sufficient memory access resources to meet its high-performance computing or real-time processing needs.

[0183] For method 2 above, when the memory access priority value of any unit in the CPU, NPU or RCU is greater than the first preset threshold, it indicates that the memory access demand of the unit is relatively urgent in the current and future system operation cycles. In order to efficiently process these memory access requests, a batch processing method is selected, that is, the memory access requests of high-priority units are processed in batches. The batch processing method can merge multiple memory access requests into one batch for processing, thereby reducing the memory access overhead and improving the efficiency of memory access. In this way, the system can give priority to meeting the memory access needs of high-priority units and ensure the timely execution of critical tasks.

[0184] For the above method 3, when the numerical difference between any two of the memory access priorities of the CPU, NPU and RCU is less than the second preset threshold, it means that the memory access requirements of these three units are relatively balanced in the current and future system operation cycles, and no unit has a particularly prominent memory access requirement. In this case, the system chooses to use an improved first-ready, first-come, first-served algorithm to process memory access requests. The so-called "improved first-ready, first-come, first-served algorithm" means that on the basis of retaining the simplicity of the traditional algorithm, a priority mechanism is introduced. While processing memory access requests in the order in which they arrive, it also considers dynamically adjusting the processing order according to the priority of memory access requests. While ensuring that lower priority memory access requests obtain relatively equal memory access opportunities, it also takes into account that higher priority memory access requests can be processed in a timely manner, thereby ensuring the reasonable allocation and efficient use of memory resources, while better adapting to different types of computing tasks and load conditions.

[0185] Figure 3 A flowchart of a method for processing memory access requests of the CPU, the NPU, and the RCU based on an improved first-ready, first-come, first-served algorithm in the future system operation cycle according to an embodiment of the present disclosure is shown. Figure 3 As shown, the processing of the memory access requests of the CPU, the NPU and the RCU based on the improved first-ready, first-come, first-served algorithm in the future system operation cycle includes the following steps S161 to S163:

[0186] In step S161, memory access requests in a ready state among the memory access requests of the CPU, the NPU and the RCU are obtained; the memory access requests in a ready state are sorted in the order of request time to form an initial ready memory access request queue.

[0187] In step S162, priority evaluation is performed on each memory access request in the initial ready memory access request queue according to a preset priority rule, and based on the priority evaluation result, each memory access request in the initial ready memory access request queue is sorted in order of priority to form a final ready memory access request queue.

[0188] In step S163, each memory access request in the final ready memory access request queue is processed according to the queue order after the final sorting.

[0189] In one specific implementation, steps S161 to S163 can be performed as follows:

[0190] First, create a memory access request queue to store memory access requests from the CPU, NPU, and RCU. At the beginning of each system operation cycle or at a specific time point, the system traverses the queue and filters out memory access requests that are currently in the ready state. The ready state means that the request is ready to be processed, that is, the required resources (such as memory) are available, and there are no other preconditions that prevent its execution.

[0191] Then, the selected memory access requests in the ready state are sorted according to the time sequence of their arrival in the system. This can be achieved by including a timestamp field in the memory access request data structure, which records the time when the request arrived in the system. After sorting, an initial queue of ready memory access requests is obtained, which is arranged in the order of request time.

[0192] After that, each request in the initial ready memory access request queue is evaluated for priority according to the preset priority rules. The priority rules can be determined according to system requirements and design, and factors that may be considered include the urgency of the request, the priority of the task to which it belongs, the size of the requested data, the identity of the requester (such as whether it is a critical task or user), etc. Based on the priority evaluation results, the requests in the initial ready memory access request queue are reordered to form the final ready memory access request queue. The requests in this queue are arranged from high to low priority.

[0193] Finally, the system processes each memory access request in the order of the final ready memory access request queue. Processing requests can include allocating memory resources, performing memory access operations, updating system status, etc. When processing each request, it is necessary to ensure the correct allocation and release of resources to avoid deadlocks, resource leaks and other problems. After processing is completed, the system can update the status of the memory access request queue, remove the processed requests from the queue, and add new ready requests to the queue.

[0194] Figure 4 A flowchart of another method for processing memory access requests of the CPU, the NPU, and the RCU based on an improved first-ready, first-served algorithm in the future system operation cycle according to an embodiment of the present disclosure is shown. In addition to the above steps S161 to 163, after the final ready memory access request queue is formed, before the memory access requests in the final ready memory access request queue are processed according to the queue order after the final sorting, the following step S164 is also included:

[0195] In step S164, the memory access requests that meet the low priority condition in the final ready memory access request queue are obtained, and the memory access request control parameters of the units corresponding to the memory access requests that meet the low priority condition are dynamically adjusted so that the units corresponding to the memory access requests that meet the low priority condition send memory access requests according to the dynamically adjusted memory access request control parameters.

[0196] Among them, the memory access request regulation parameters include: the concurrent number of memory access requests and the interval time of memory access requests. For the concurrent number of memory access requests: this parameter limits the number of requests for simultaneous memory access. By setting a reasonable upper limit on the number of concurrent requests, it is possible to avoid too many memory access requests occupying the memory bandwidth at the same time, thereby reducing the impact on high-priority tasks and ensuring the stability and performance of the system. For the interval time of memory access requests: this parameter determines the time interval between two adjacent memory access requests. By inserting an appropriate interval time, the frequency of memory access can be slowed down, providing more memory access opportunities for high-priority tasks. This helps to balance the memory access requirements between tasks of different priorities and ensure fair allocation of system resources.

[0197] The present disclosure can manage memory access more flexibly by obtaining memory access requests that meet low priority conditions in the final ready memory access request queue and dynamically adjusting the memory access request control parameters (such as the number of concurrent requests and the interval time) of the units corresponding to these requests. This dynamic adjustment can be performed according to the current system load, resource availability and performance requirements, thereby optimizing resource allocation and improving overall system efficiency. In addition, in the case where there are many low-priority memory access requests or system resources are tight, by adjusting the number of concurrent requests and the interval time of the memory access requests, resource contention and conflict caused by multiple units accessing the memory at the same time can be effectively avoided. This helps to reduce the delay of memory access, improve memory access efficiency, and ensure that critical tasks or high-priority tasks can obtain the required memory resources in a timely manner. In addition, dynamically adjusting the memory access request control parameters can make the system more flexible to respond to different workloads and performance requirements. For example, when the system load is light, the number of concurrent requests can be increased to improve the memory access speed; when the system load is heavy, the number of concurrent requests can be reduced or the interval time can be extended to reduce memory pressure and ensure stable operation of the system. In addition, by dynamically adjusting the memory access request control parameters, the system can better control the flow and rhythm of memory access, thereby avoiding system crashes or performance degradation caused by too many memory access requests. This mechanism helps to enhance the stability and reliability of the system and ensure that the system can provide services continuously and stably. In addition, by optimizing the scheduling and processing of memory access requests, the system can use memory resources more effectively. After dynamic adjustment, low-priority memory access requests are arranged and processed more reasonably, avoiding unnecessary memory occupation and waste, and improving memory utilization and overall system performance.

[0198] In step S170, the shared memory is scheduled and allocated among the CPU, the NPU, and the RCU according to the memory scheduling scheme.

[0199] When executing the memory scheduling scheme determined by steps S110 to 160 above, the present disclosure also considers power consumption factors, constructs a Lyapunov function, and dynamically adjusts the scheduling of each layer in the task according to the value of the Lyapunov function. For example, if a certain layer is executed on the NPU and causes power consumption to rise too quickly, close to the upper limit of power consumption, and has little effect on the reasoning accuracy, part of the calculation of the layer may be transferred to the CPU; or if the execution delay of a certain layer affects the real-time reasoning speed, its priority may be adjusted to speed up its execution.

[0200] Figure 5 A flow chart of a method for scheduling and allocating the shared memory among the CPU, the NPU, and the RCU according to the memory scheduling scheme according to an embodiment of the present disclosure is shown. Figure 5 As shown, the scheduling and allocating of the shared memory among the CPU, the NPU and the RCU according to the memory scheduling scheme includes the following steps S171-175:

[0201] In step S171, the current state parameters when executing the task are obtained.

[0202] The current state parameters include: current inference speed v current 、Current power consumption P current , current precision loss L current , the execution progress of each layer in the task, the CPU load, the NPU load and the RCU load.

[0203] In step S172, the state variables are determined according to the preset target state parameters and the current state parameters.

[0204] The target state parameters include: expected inference speed v desired , power consumption limit P max , lower limit of accuracy loss L min and the expected execution progress of each layer in the task; the state variables include: the current reasoning speed v current With the expected inference speed v desired The deviation of the current power consumption P current With the power consumption upper limit P max The deviation, the current accuracy loss L current With the lower limit of precision loss L min and the deviation of the execution progress of each layer in the task from the expected execution progress of each layer in the task;

[0205] In step S173, a Lyapunov function V(x) is constructed based on the state variable by the following formula:

[0206] V(x)=α(vcurrent -v desired ) 2 +β(P current -P max ) 2 +γ(L current -L min ) 2 +∑ i δ i (Progress i -TargetProgress i ) 2 ;

[0207] where x is a vector containing the state variables, α, β, γ, and δ i is the weight coefficient, Progress i is the execution progress of the i-th layer in the task, TargetProgress i is the expected execution progress of the i-th layer in the task.

[0208] In step S174, the derivative of the Lyapunov function V(x) is calculated according to the rate of change of the current state parameter.

[0209] The change rate of the current state parameter includes: the change rate of inference speed, the change rate of power consumption and the change rate of precision loss. The change rate of the current state parameter is based on the CPU load Load CPU , the NPU load Load NPU and the RCU Load RCU Calculated. For example, the rate of change of inference speed can be expressed as:

[0210]

[0211] Similarly, the rate of change of power consumption and the rate of change of accuracy loss can be defined.

[0212] In step S175, the scheduling strategy of each layer in the task is dynamically adjusted according to the sign and size of the derivative.

[0213] Assumption: If the derivative is less than 0, it means that the target state is approaching, the current scheduling strategy is valid, and can continue to be executed; if the derivative is greater than 0, it means that the target state is moving away and the scheduling strategy needs to be adjusted.

[0214] If the power consumption is too high, it means that power consumption is the main problem. You can consider transferring some computing tasks from the NPU to the CPU, or reducing the calculation accuracy of certain layers to reduce power consumption. If the inference speed is too slow, it means that the inference speed is the main problem. You can prioritize the key layers or increase the parallelism of the CPU and NPU to increase the processing speed. If the accuracy loss is too large, it means that the accuracy loss is the main problem. You can try to restore the calculation accuracy of certain layers or improve the accuracy by retraining the model.

[0215] By constructing the function and monitoring its derivative, the present disclosure can ensure that the system state always tends to the preset target state parameters, thereby enhancing the overall stability of the system. In addition, by dynamically adjusting the scheduling strategies of each layer in the task according to the rate of change of the current state parameters, computing resources can be allocated and utilized more effectively. This helps to avoid resource overload or idleness and improve the overall performance and efficiency of the system. In addition, by comprehensively considering the current inference speed, power consumption, precision loss and other parameters, and dynamically adjusting according to the target state parameters, power consumption can be minimized while ensuring a certain performance level.

[0216] Figure 6 The structural block diagram of a memory scheduling device according to an embodiment of the present disclosure is shown. The memory scheduling device is arranged in a CPU-NPU-RCU heterogeneous chip, the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU and a reconfigurable computing unit RCU, and the CPU, the NPU and the RCU exchange data through a shared memory. Figure 6As shown, the memory scheduling device includes: a perception module, which is configured to obtain the type of the currently executed task; determine whether the type of the task meets the conditions for shared memory scheduling, and if the type of the task meets the conditions for shared memory scheduling, then: obtain the number of memory access requests and the type of memory access requests of the CPU, the NPU and the RCU in the system operation cycle when executing the task, wherein the system operation cycle includes a preset number of clock cycles; a prediction module, which is configured to determine the estimated delay tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task to the end of the current system operation cycle when it is determined that the memory access request type of any one of the CPU, the NPU and the RCU in the current system operation cycle is a memory read request, and determine the estimated delay tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task to the end of the current system operation cycle, and determine the estimated delay tolerance of the CPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task to the end of the current system operation cycle. The method comprises the following steps: determining an estimated delay tolerance of the NPU according to the total number of memory access requests of the NPU at the end of the current system operation cycle, and determining an estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of the task to the end of the current system operation cycle; calculating the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU in a future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle; a scheduling module is configured to determine a memory scheduling scheme in the future system operation cycle according to the memory access priority; and scheduling and allocating the shared memory among the CPU, the NPU and the RCU according to the memory scheduling scheme.

[0217] According to an embodiment of the present disclosure, determining the estimated latency tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes:

[0218] Obtain the number of memory access requests of the CPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU.

[0219] Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes:

[0220] Obtain the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; and use the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU.

[0221] Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes:

[0222] Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

[0223] According to an embodiment of the present disclosure, the calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU includes:

[0224] When the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU, and the RCU in the future system operation cycle are calculated as follows:

[0225] The memory access priority of the CPU in the future system operation cycle is determined according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the CPU and the first auxiliary parameter.

[0226] The memory access priority of the NPU in the future system operation cycle is determined according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter.

[0227] The memory access priority of the RCU in the future system operation cycle is determined according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and the first auxiliary parameter.

[0228] Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

[0229] According to an embodiment of the present disclosure, determining the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU, and the first auxiliary parameter includes:

[0230] The memory access priority of the CPU in the future system operation cycle is calculated by the following formula:

[0231]

[0232] The memory access priority of the NPU in the future system operation cycle is calculated by the following formula:

[0233]

[0234] The memory access priority of the RCU in the future system operation cycle is calculated by the following formula:

[0235]

[0236] Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCU is the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCUis the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCU is the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

[0237] According to an embodiment of the present disclosure, determining the memory scheduling scheme in the future system operation cycle according to the memory access priority includes:

[0238] If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access demand of the RCU.

[0239] If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is greater than the first preset threshold, then: the memory access requests of the units corresponding to the values ​​greater than the first preset threshold are processed in a batch manner during the future system operation cycle.

[0240] If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

[0241] The present disclosure also provides a chip, comprising a memory scheduling device as described in any one of the above device embodiments; or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above method embodiments.

[0242] Figure 7 FIG. 1 shows a structural block diagram of a CPU-NPU-RCU heterogeneous chip according to an embodiment of the present disclosure. Figure 7 As shown, the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU and a reconfigurable computing unit RCU, as well as a memory scheduling device as described in any of the above device embodiments.

[0243] Figure 8 1 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 8As shown, the electronic device includes a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method described in any one of the above method embodiments.

[0244] The present disclosure also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0245] The present disclosure also provides a computer program product, including a computer program, which implements any method described in the present disclosure when the computer program is executed by a processor.

[0246] According to the technical solution provided by the embodiment of the present disclosure, a memory scheduling method for CPU-NPU-RCU heterogeneous chips is proposed. The method first identifies the current task type and determines whether it meets the conditions for shared memory scheduling. If it does, the memory access requirements of each unit are evaluated by monitoring the number and type of memory access requests of the CPU, NPU and RCU during the system operation cycle. When a memory access read request is detected, the estimated delay tolerance is calculated according to the number of memory access requests of each unit, and then the memory access priority of the CPU, NPU and RCU in the future system operation cycle is determined. Based on these priorities, a memory scheduling scheme is formulated, and shared memory is dynamically allocated between the CPU, NPU and RCU according to the memory scheduling scheme. This optimizes the allocation and utilization of memory resources in heterogeneous chips, improves memory access efficiency, and ensures that each processing unit can efficiently access shared resources on demand, thereby improving the overall performance of the system.

[0247] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A memory scheduling method, characterized in that: The method is applied to a CPU-NPU-RCU heterogeneous chip, wherein the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU and a reconfigurable computing unit RCU, wherein the CPU, the NPU and the RCU exchange data through a shared memory, and the method includes: Get the type of the currently executed task; Determine whether the type of the task meets the condition for shared memory scheduling. If the type of the task meets the condition for shared memory scheduling, then: Acquire the number and type of memory access requests of the CPU, the NPU, and the RCU within a system operation cycle when executing the task, wherein the system operation cycle includes a preset number of clock cycles; When it is determined that the memory access request type of any one of the CPU, the NPU, and the RCU in the current system operation cycle is a memory read request, the estimated delay tolerance of the CPU is determined according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task to the end of the current system operation cycle, the estimated delay tolerance of the NPU is determined according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of the task to the end of the current system operation cycle, and the estimated delay tolerance of the RCU is determined according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of the task to the end of the current system operation cycle; Calculating the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in a future system operation cycle according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle; Determining a memory scheduling scheme within the future system operation cycle according to the memory access priority; The shared memory is scheduled and allocated among the CPU, the NPU and the RCU according to the memory scheduling scheme.

2. The method according to claim 1, characterized in that The types of tasks include: neural network reasoning training tasks and scientific computing tasks, and determining whether the types of tasks meet the conditions for shared memory scheduling includes: When the type of the task is a neural network reasoning training task or a scientific computing task, the type of the task meets the conditions for shared memory scheduling; otherwise, the type of the task does not meet the conditions for shared memory scheduling.

3. The method according to claim 1, characterized in that Determining the estimated latency tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes: Obtain the number of memory access requests of the CPU in each system operation cycle from the start of the task to the end of the current system operation cycle; accumulate and calculate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU; Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes: Obtaining the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulating the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; taking the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU; Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes: Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

4. The method according to claim 1, characterized in that The calculating, according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU, the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle includes: When the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU, and the RCU in the future system operation cycle are calculated as follows: Determine the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU and the first auxiliary parameter; Determine the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter; Determine the memory access priority of the RCU in the future system operation cycle according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and a third auxiliary parameter; Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

5. The method according to claim 4, characterized in that The determining the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the CPU and the first auxiliary parameter includes: The memory access priority of the CPU in the future system operation cycle is calculated by the following formula: The determining the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter includes: The memory access priority of the NPU in the future system operation cycle is calculated by the following formula: The determining, according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU, and the third auxiliary parameter, the memory access priority of the RCU in the future system operation cycle includes: The memory access priority of the RCU in the future system operation cycle is calculated by the following formula: Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCU is the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCU is the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCU is the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

6. The method according to claim 1, characterized in that Determining the memory scheduling scheme in the future system operation cycle according to the memory access priority includes: If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access demand of the RCU; If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU is greater than a first preset threshold, then: processing memory access requests of units corresponding to the values ​​greater than the first preset threshold in a batch processing manner within the future system operation cycle; If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

7. The method according to claim 6, characterized in that The processing of the memory access requests of the CPU, the NPU and the RCU based on an improved first-ready, first-come, first-served algorithm during the future system operation cycle includes: Acquire memory access requests in a ready state among the memory access requests of the CPU, the NPU, and the RCU; sort the memory access requests in a ready state in the order of request time to form an initial ready memory access request queue; Performing a priority evaluation on each memory access request in the initial ready memory access request queue according to a preset priority rule, and sorting each memory access request in the initial ready memory access request queue in order of priority according to a priority evaluation result to form a final ready memory access request queue; Each memory access request in the final ready memory access request queue is processed according to the queue order after the final sorting.

8. The method according to claim 7, characterized in that After forming the final ready memory access request queue and before processing each memory access request in the final ready memory access request queue according to the queue order after the final sorting, the method further includes: Obtain the memory access requests that meet the low priority condition in the final ready memory access request queue, and dynamically adjust the memory access request control parameters of the units corresponding to the memory access requests that meet the low priority condition, so that the units corresponding to the memory access requests that meet the low priority condition send memory access requests according to the dynamically adjusted memory access request control parameters; wherein the memory access request control parameters include: the concurrent number of memory access requests and the interval time of memory access requests.

9. The method according to claim 1, characterized in that: The scheduling and allocating the shared memory among the CPU, the NPU, and the RCU according to the memory scheduling scheme includes: Get the current state parameters when executing the task, the current state parameters include: current reasoning speed v current 、Current power consumption P current , current precision loss L current , the execution progress, CPU load, NPU load and RCU load of each layer in the task; The state variables are determined according to the preset target state parameters and the current state parameters, wherein the target state parameters include: the expected inference speed v desired , power consumption limit P max , lower limit of accuracy loss L min and the expected execution progress of each layer in the task; the state variables include: the current reasoning speed v current With the expected inference speed v desired The deviation of the current power consumption P current With the power consumption upper limit P max The deviation, the current accuracy loss L current With the lower limit of precision loss L min and the deviation of the execution progress of each layer in the task from the expected execution progress of each layer in the task; Based on the state variable, the Lyapunov function V(x) is constructed by the following formula: V(x)=α(v crrrent -v desired )+β(P current -P max )+γ(L current -L min )+∑ i d i (Progress i -TargetProgress i ); where x is a vector containing the state variables, α, β, γ, and δ i is the weight coefficient, Progress i is the execution progress of the i-th layer in the task, TargetProgress i is the expected execution progress of the i-th layer in the task; Calculating the derivative of the Lyapunov function V(x) according to the rate of change of the current state parameter; the rate of change of the current state parameter is calculated according to the CPU load, the NPU load and the RCU load; The scheduling strategy of each layer in the task is dynamically adjusted according to the sign and size of the derivative.

10. A memory scheduling device, characterized in that: The memory scheduling device is arranged in a CPU-NPU-RCU heterogeneous chip, wherein the CPU-NPU-RCU heterogeneous chip includes a central processing unit CPU, a neural network processing unit NPU and a reconfigurable computing unit RCU, wherein the CPU, the NPU and the RCU exchange data through a shared memory, and the memory scheduling device includes: A perception module is configured to obtain the type of a currently executed task; determine whether the type of the task satisfies a condition for shared memory scheduling, and if the type of the task satisfies the condition for shared memory scheduling, then: obtain the number and type of memory access requests of the CPU, the NPU, and the RCU in a system operation cycle when executing the task, wherein the system operation cycle includes a preset number of clock cycles; The prediction module is configured to determine the estimated delay tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of the task execution to the end of the current system operation cycle, when determining that the memory access request type of any one of the CPU, the NPU, and the RCU in the current system operation cycle is a memory read request, determine the estimated delay tolerance of the CPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of the task execution to the end of the current system operation cycle, and determine the estimated delay tolerance of the NPU .... The estimated delay tolerance of the RCU is determined based on the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of the task execution to the end of the current system operation cycle; the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU in a future system operation cycle are calculated based on the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU and the estimated delay tolerance of the RCU; wherein the future system operation cycle includes one or more system operation cycles following the end of the current system operation cycle; The scheduling module is configured to determine a memory scheduling scheme in the future system operation cycle according to the memory access priority; and schedule and allocate the shared memory among the CPU, the NPU and the RCU according to the memory scheduling scheme.

11. The memory scheduling device according to claim 10, characterized in that: Determining the estimated latency tolerance of the CPU according to the number of memory access requests of the CPU in the current system operation cycle and the total number of memory access requests of the CPU from the start of execution of the task to the end of the current system operation cycle includes: Obtain the number of memory access requests of the CPU in each system operation cycle from the start of the task to the end of the current system operation cycle; accumulate and calculate the number of memory access requests of the CPU in each system operation cycle to obtain the total number of memory access requests of the CPU; and use the ratio of the number of memory access requests of the CPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the CPU; Determining the estimated delay tolerance of the NPU according to the number of memory access requests of the NPU in the current system operation cycle and the total number of memory access requests of the NPU from the start of execution of the task to the end of the current system operation cycle includes: Obtaining the number of memory access requests of the NPU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulating the number of memory access requests of the NPU in each system operation cycle to obtain the total number of memory access requests of the NPU; taking the ratio of the number of memory access requests of the NPU in the current system operation cycle to the total number of memory access requests of the CPU as the estimated delay tolerance of the NPU; Determining the estimated delay tolerance of the RCU according to the number of memory access requests of the RCU in the current system operation cycle and the total number of memory access requests of the RCU from the start of execution of the task to the end of the current system operation cycle includes: Obtain the number of memory access requests of the RCU in each system operation cycle from the start of execution of the task to the end of the current system operation cycle; accumulate the number of memory access requests of the RCU in each system operation cycle to obtain the total number of memory access requests of the RCU; and use the ratio of the number of memory access requests of the RCU in the current system operation cycle to the total number of memory access requests of the RCU as the estimated delay tolerance of the RCU.

12. The memory scheduling device according to claim 10, characterized in that: The calculating, according to the estimated delay tolerance of the CPU, the estimated delay tolerance of the NPU, and the estimated delay tolerance of the RCU, the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU in the future system operation cycle includes: When the estimated delay tolerance of the CPU and the estimated delay tolerance of the NPU are both lower than the preset minimum delay tolerance threshold, the memory access priority of the CPU and the memory access priority of the NPU in the future system operation cycle are both 0, and the memory access priority of the RCU is 100%; otherwise, the memory access priorities of the CPU, the NPU, and the RCU in the future system operation cycle are calculated as follows: Determine the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated latency tolerance of the CPU and the first auxiliary parameter; Determine the memory access priority of the NPU in the future system operation cycle according to the number of memory access requests of the NPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the NPU and the second auxiliary parameter; Determine the memory access priority of the RCU in the future system operation cycle according to the number of memory access requests of the RCU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the RCU and the first auxiliary parameter; Among them, the first auxiliary parameter, the second auxiliary parameter and the third auxiliary parameter are predetermined according to the batch size when the task is executed, and the total number of memory access requests of the shared memory is: the sum of the number of memory access requests of the CPU, the number of memory access requests of the NPU and the number of memory access requests of the RCU in the current system operation cycle.

13. The memory scheduling device according to claim 12, characterized in that: The determining the memory access priority of the CPU in the future system operation cycle according to the number of memory access requests of the CPU in the current system operation cycle, the total number of memory access requests of the shared memory, the estimated delay tolerance of the CPU and the first auxiliary parameter includes: The memory access priority of the CPU in the future system operation cycle is calculated by the following formula: The memory access priority of the NPU in the future system operation cycle is calculated by the following formula: The memory access priority of the RCU in the future system operation cycle is calculated by the following formula: Among them, P CPU is the memory access priority of the CPU in the future system operation cycle, P NPU is the memory access priority of the NPU in the future system operation cycle, P RCU is the memory access priority of the RCU in the future system operation cycle; CPU is the number of memory access requests of the CPU during the current system operation cycle, R NPU is the number of memory access requests of the NPU during the current system operation cycle, R RCU is the number of memory access requests of the RCU in the current system operation cycle; D CPU is the estimated latency tolerance of the CPU, D NPU is the estimated latency tolerance of the NPU, D RCU is the estimated delay tolerance of the RCU; A1 is the first auxiliary parameter, A2 is the second auxiliary parameter, and A3 is the third auxiliary parameter.

14. The memory scheduling device according to claim 10, characterized in that: Determining the memory scheduling scheme in the future system operation cycle according to the memory access priority includes: If the memory access priority of the CPU and the memory access priority of the NPU are both 0, and the memory access priority of the RCU is 100%, then: only the memory access request of the RCU is processed in the future system operation cycle to meet the memory access demand of the RCU; If the value of any one of the memory access priority of the CPU, the memory access priority of the NPU, and the memory access priority of the RCU is greater than a first preset threshold, then: processing memory access requests of units corresponding to the values ​​greater than the first preset threshold in a batch processing manner within the future system operation cycle; If the numerical difference between any two of the memory access priority of the CPU, the memory access priority of the NPU and the memory access priority of the RCU is less than a second preset threshold, then: during the future system operation cycle, the memory access requests of the CPU, the NPU and the RCU are processed based on an improved first-ready, first-come, first-served algorithm.

15. A chip, characterized in that: A memory scheduling device comprising any one of claims 10 to 14; or, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 9.

16. An electronic device, characterized in that: The method comprises a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 9.

17. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images