Scheduling engine, scheduling method, electronic device, storage medium and program product

Abstract

The present disclosure relates to the field of computer technology, and specifically discloses a scheduling engine, a scheduling method, an electronic device, a storage medium, and a program product. The scheduling engine includes a task manager, a central manager, and a task execution body. The task manager is used to receive reasoning tasks; the central manager is used to divide multiple memory blocks from the processor memory of an artificial intelligence processor, and press the memory block information of the multiple memory blocks into a memory block information queue. When the task manager completes the preprocessing of the reasoning task, the reasoning task is pressed into the task queue corresponding to one of the task execution bodies, and based on the memory space required for the token of the reasoning task and the memory block information in the memory block information queue, the target memory block for storing the token is allocated to the reasoning task; the task execution body is used to execute the reasoning task, and different task execution bodies are associated with different cores of the artificial intelligence processor, and the reasoning task is executed in the associated core. The model reasoning speed can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a scheduling engine, a scheduling method, an electronic device, a storage medium, and a program product. Background Art

[0002] Artificial intelligence processors, also known as AI processors, include cores, memory, NoC, interfaces and other modules, which can provide powerful computing support for model reasoning. However, in the actual process of calling artificial intelligence processors, if these resources are used unreasonably, it will inevitably affect the reasoning speed of the model. Summary of the invention

[0003] In view of this, the implementation method of the present disclosure provides a scheduling engine, a scheduling method based on the scheduling engine, an electronic device, a computer-readable storage medium and a computer program product, which can optimize the resource usage in the artificial intelligence processor and improve the model reasoning speed.

[0004] In one aspect, the present disclosure provides a scheduling engine, the scheduling engine comprising a task manager, a central manager and at least one task execution body, wherein:

[0005] The task manager is used to receive the reasoning task and pre-process the reasoning task;

[0006] The central manager is used to pre-divide a plurality of memory blocks from the processor memory of the artificial intelligence processor, and push the memory block information of the plurality of memory blocks into a memory block information queue, and when the task manager completes the preprocessing of the reasoning task, push the reasoning task into the task queue corresponding to one of the task execution bodies, and allocate a target memory block for storing the token to the reasoning task based on the memory space required for the token of the reasoning task and the memory block information in the memory block information queue, and after the reasoning task is executed, push the memory block information of the target memory block into the memory block information queue again;

[0007] The task execution body is used to execute the reasoning task in the corresponding task queue, wherein different task execution bodies are associated with different cores of the artificial intelligence processor, and each task execution body executes the reasoning task in the associated core.

[0008] Another aspect of the present disclosure provides a scheduling method based on a scheduling engine, wherein the scheduling engine includes at least one task execution body, and different task execution bodies are associated with different cores of an artificial intelligence processor; the method includes:

[0009] Receiving an inference task and preprocessing the inference task;

[0010] Pushing the inference task into a task queue corresponding to one of the task execution bodies of the scheduling engine, and allocating a target memory block for storing the token to the inference task based on the memory space required by the token of the inference task and the memory block information of multiple memory blocks in the memory block information queue, wherein the multiple memory blocks are pre-divided from the processor memory of the artificial intelligence processor;

[0011] When the task execution body completes executing the reasoning task in the associated kernel, the memory block information of the target memory block is pushed back into the memory block information queue.

[0012] Another aspect of the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method described above is implemented.

[0013] Another aspect of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0014] Another aspect of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0015] In the technical solutions of some embodiments of the present disclosure, on the one hand, by setting a task execution body associated with the core of the artificial intelligence processor in the scheduling engine, the core can be scheduled by scheduling the task execution body, and the scheduling strategy of the core can be optimized by optimizing the scheduling strategy of the task execution body. In this way, the core idle period can be reduced, the resource utilization of the core can be improved, and the model reasoning speed can be improved; on the other hand, since the technical solution of the present disclosure allocates memory space for reasoning tasks in units of memory blocks, when the memory space size of a single memory block is set appropriately, the problem of too large or too small memory space allocated to the reasoning task can be effectively avoided, that is, the waste of processor memory is reduced, and the model reasoning speed can be improved. In summary, the technical solution of the present disclosure can optimize the use of resources in the artificial intelligence processor and improve the model reasoning speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0017] Figure 1 A schematic diagram of a module of an artificial intelligence processor provided by some embodiments of the present disclosure is shown;

[0018] Figure 2 A schematic diagram of scheduling of an artificial intelligence processor provided by some embodiments of the present disclosure is shown;

[0019] Figure 3 A schematic diagram of a module of a scheduling engine provided by an embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram of a memory block information queue provided by some embodiments of the present disclosure is shown;

[0021] Figure 5 A schematic diagram of a processor memory provided by an embodiment of the present disclosure is shown;

[0022] Figure 6 A schematic diagram of a token storage location of a target memory block provided by an embodiment of the present disclosure is shown;

[0023] Figure 7 An embodiment of the present disclosure provides Figure 3 A detailed diagram of the scheduling engine in ;

[0024] Figure 8 A schematic diagram of a scheduling method provided by an embodiment of the present disclosure is shown;

[0025] Fig. 9 A flow diagram of task scheduling provided by an embodiment of the present disclosure is shown;

[0026] Fig.10 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the implementation method of the present disclosure clearer, the technical scheme in the implementation method of the present disclosure will be clearly and completely described below in conjunction with the drawings in the implementation method of the present disclosure. Obviously, the implementation method described is a part of the implementation method of the present disclosure, not all of the implementation methods. Based on the implementation method in the present disclosure, all other implementation methods obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0028] Artificial intelligence processors may include but are not limited to CPU (Central Processing Unit), GPU (Graphics Processing Unit) and NPU (Neural Processing Unit). In practical applications, one or more of the CPU, GPU, NPU and other processors may be used as artificial intelligence processors. When multiple processors are used as artificial intelligence processors as a whole, the multiple processors may be integrated to obtain an artificial intelligence processor. Alternatively, CPU, GPU and other processors may be deployed independently of each other and connected through high-speed interfaces to obtain an artificial intelligence processor.

[0029] Combined with reference Figure 1 , which is a module diagram of the artificial intelligence processor 11 provided in some embodiments of the present disclosure. Figure 1 In the embodiment, the artificial intelligence processor 11 includes at least one core 111, a processor memory 112, an on-chip interconnection network 113 and at least one interface 114. The core 111 is also called a Core, which is used for large-scale parallel computing, the processor memory 112 is used to store input data, output data and intermediate computing results in the model reasoning process, the on-chip interconnection network 113 is responsible for communication between the cores 111, and the interface 114 is used to connect the artificial intelligence processor 11 with other components.

[0030] Combined with reference Figure 2 , which is a scheduling diagram of the artificial intelligence processor 11 provided in some embodiments of the present disclosure. Figure 2 In the embodiment, the upper layer application in the host (such as the scheduling engine in the present disclosure) can call the artificial intelligence processor 11 by calling the libraries of the artificial intelligence processor 11. The upper layer application and the libraries can be located in the CPU, and the GPU can be used as the artificial intelligence processor.

[0031] The libraries can include various high-performance operators, based on which different layers of the model can be constructed. The upper-layer application can complete model reasoning by calling these operators. In the process of calling these operators, the hardware resources of the artificial intelligence processor 11 can be used. For example, take the linear operator in the libraries library as an example. The linear operator can be expressed as Y=X1·X2, which is used to complete the multiplication of two matrices. When the upper-layer application in the host calls the linear operator, it can first call the malloc function to open up multiple memory spaces in the artificial intelligence processor 11 for caching X1, X2, Y and other data. Then, call the copy function to copy the data in the host to the memory of the artificial intelligence processor 11, and call the operator to pass data pointers such as X1, X2, Y, etc. to start the matrix multiplication operation; finally, copy Y from the artificial intelligence processor 11 to the host.

[0032] However, when the hardware resources of the artificial intelligence processor 11 are actually used for model reasoning, if these hardware resources are used unreasonably, the reasoning speed of the model will inevitably be affected. For example, in the process of some model reasoning, a large amount of intermediate result data (such as the tokens generated in the reasoning process of the language model, etc.) will be generated, and the demand for memory is relatively large, but the processor memory 112 in the artificial intelligence processor 11 is limited. In this case, if the use of the processor memory 112 is not reasonable enough, the reasoning speed of the model will inevitably be affected. For ease of understanding, take the tokens generated in the reasoning process of the language model as an example. In order to save the tokens of an inference task in a continuous memory space, in some technologies, a larger continuous memory space will be pre-divided to store the tokens of the reasoning task separately, and during the execution of an inference task, the continuous memory space will always be reserved for the reasoning task (that is, other reasoning tasks cannot use it). However, the token lengths of different reasoning tasks may be different. When executing some reasoning tasks with shorter token lengths, the pre-divided continuous memory space may have a large amount of surplus, and the surplus memory space cannot be used by other reasoning tasks, which leads to a waste of memory resources, that is, unreasonable use of the processor memory 112. This will inevitably reduce the reasoning speed of the model when the processor memory 112 in the artificial intelligence processor 11 is limited.

[0033] For another example, unreasonable scheduling of the cores 111 in the artificial intelligence processor 11 causes the idle periods of some cores 111 to be too long, which results in a waste of computing resources and is bound to reduce the inference speed of the model.

[0034] In view of this, the present disclosure first provides a scheduling engine 30, which can solve the above problems. Figure 3, which is a module diagram of a scheduling engine 30 provided in one embodiment of the present disclosure. Figure 3 In the example, the scheduling engine 30 includes a task manager 31, a central manager 32 and at least one task execution body 33 for executing reasoning tasks.

[0035] The task manager 31 is used to receive reasoning tasks and pre-process the reasoning tasks. Among them, the reasoning task can refer to the task in which the model analyzes and infers the input content to obtain the reasoning result. For example, a question is input into the model, and the model analyzes and infers the question and outputs the answer to the question. For another example, a specified keyword is input into the model, and the model outputs a text content that matches the keyword based on the keyword. The task manager 31 pre-processes the reasoning task, mainly converting the reasoning task into an input that can be processed by the artificial intelligence processor 11.

[0036] The central manager 32 is used to pre-divide multiple memory blocks from the processor memory 112 of the artificial intelligence processor 11, and push the memory block information of the multiple memory blocks into the memory block information queue. Specifically, the memory block information includes but is not limited to the size, offset address, number, etc. of each memory block. Some memory block information (such as offset address, number, etc.) can uniquely identify the memory block. Therefore, managing the memory block information is managing the memory block. Figure 4 , is a schematic diagram of a memory block information queue provided by some embodiments of the present disclosure. Figure 4 It can be seen that the memory block information queue may include memory block information corresponding to multiple memory blocks in the processor memory 112, for example, memory block information 1 is memory block information of memory block 1, and memory block information 2 is memory block information of memory block 2. Moreover, in some embodiments, among the multiple memory blocks corresponding to the memory block information queue, the physical addresses of at least some of the memory blocks may be discontinuous.

[0037] When the task manager 31 completes the preprocessing of the reasoning task, the central manager 32 can push the reasoning task into the task queue corresponding to one of the task executors 33, and based on the memory space required for the token of the reasoning task and the memory block information in the memory block information queue, allocate a target memory block for storing the token to the reasoning task, and after the reasoning task is executed, push the memory block information of the target memory block back into the memory block information queue.

[0038] Specifically, the central manager 32 can determine the number of target memory blocks that need to be allocated to the reasoning task based on the memory space required by the token of the reasoning task and the memory space size of a single memory block, and then pop out the corresponding number of memory block information from the memory block information queue. The memory blocks corresponding to these popped-up memory block information are the target memory blocks allocated to the reasoning task for storing the token. Figure 4 For example, assuming that three target memory blocks are needed to save the token, memory block information 1, memory block information 2 and memory block information 3 can be popped out from the memory block information queue. The memory blocks corresponding to these three memory block information are the target memory blocks allocated to the reasoning task.

[0039] After the reasoning task is executed, as the target memory blocks are released, the central manager 32 can push the memory block information of these target memory blocks back into the memory block information queue, so that when a new reasoning task is received, the target memory blocks can be allocated to the new reasoning task based on the memory block information.

[0040] The task execution body 33 is used to execute the reasoning task in the corresponding task queue, wherein different task execution bodies 33 are associated with different cores 111 of the artificial intelligence processor 11, and each task execution body 33 executes the reasoning task in the associated core.

[0041] Specifically, in this embodiment, each task execution body 33 can be associated with one or more cores 111 of the artificial intelligence processor 11. Since each task execution body 33 performs reasoning tasks in the associated core 111, it can be understood that the task execution body 33 is an abstraction and representation of the core 111. The central manager 32 schedules the task execution body 33 to perform reasoning tasks, which is essentially scheduling different cores 111 in the artificial intelligence processor 11. In this way, the task execution body 33 can be scheduled based on a certain scheduling strategy to minimize the idle window period of the core 111, thereby improving the reasoning speed of the model.

[0042] In this embodiment, at least some of the task execution bodies 33 can be used to execute different types of reasoning tasks. After the task manager 31 receives the reasoning task, the central manager 32 can push the reasoning task into the task queue of the corresponding task execution body 33 based on the task type of the reasoning task, so as to schedule the corresponding task execution body 33 to execute the reasoning task. Each task execution body 33 can execute the reasoning tasks in the corresponding task queue in sequence according to the order in which the reasoning tasks are pushed into the task queue.

[0043] In summary, in the technical solutions of some embodiments of the present disclosure, on the one hand, by setting a task execution body 33 associated with the core 111 of the artificial intelligence processor 11 in the scheduling engine 30, the core 111 can be scheduled by scheduling the task execution body 33, and the scheduling strategy of the core 111 can be optimized by optimizing the scheduling strategy of the task execution body 33. In this way, the idle period of the core 111 can be reduced, the resource utilization of the core 111 can be improved, and the model reasoning speed can be improved; on the other hand, since the technical solution of the present disclosure allocates memory space for reasoning tasks in units of memory blocks, when the memory space size of a single memory block is set appropriately, the problem of too large or too small memory space allocated to the reasoning task can be effectively avoided, that is, the waste of the processor memory 112 is reduced, and the model reasoning speed can be improved. In summary, the technical solution of the present disclosure can optimize the resource usage in the artificial intelligence processor 11 and improve the model reasoning speed.

[0044] In some embodiments, before using the scheduling engine 30, the number of cores of the artificial intelligence processor 11 and the total processor memory allowed to be used by each task executor 33 can be pre-configured. For any task executor 33, the total processor memory allowed to be used by the task executor 33 can be the total memory of the core 111 associated with the task executor 33. The central manager 32 is specifically used to call the target number of task executors 33 matching the number of cores to simulate the execution of reasoning tasks based on the pre-configured number of cores and the total processor memory allowed to be used by each task executor 33, so as to determine the first processor memory required to be consumed by the task information data, input data and output data of the reasoning tasks executed by each task executor 33, and determine the second processor memory remaining after subtracting the corresponding first processor memory from the total processor memory of each task executor 33, and divide a plurality of memory blocks from the second processor memory, wherein the second processor memory is used to store the intermediate result data of the reasoning task, and the intermediate result data includes the token of the reasoning task. The task information data may include model weights, etc.

[0045] Specifically, the above-mentioned target number and the number of cores may be the same. For example, if the number of configured cores is 3, then three task executors 33 are called to simulate the execution of reasoning tasks. The task executor 33 simulates the execution of reasoning tasks, which means that the task executor 33 executes the reasoning tasks in an unloaded state. The reasoning tasks simulated and executed by different task executors 33 may be different, that is, the first processor memory consumed by the task information data, input data and output data of the reasoning tasks executed by each task executor 33 may be different. For any task executor 33, the task executor 33 can determine the first processor memory required for the necessary data such as the input data, output data, task information data of the reasoning task during the process of simulating the execution of the reasoning task. By removing these first processor memories that must be used from the total processor memory of the task executor 33, the remaining second processor memory corresponding to the task executor 33 can be obtained, and then these second processor memories can be divided into multiple memory blocks.

[0046] For example, refer to Figure 5 , a schematic diagram of a processor memory provided for one embodiment of the present disclosure. Figure 5 In the figure, it is assumed that processor memories AC, DF, and GI are the first processor memories allowed to be used by task execution bodies 1, 2, and 3, respectively, and processor memories AB, DE, and GH are the processor memories required to be used by the input data, output data, and other necessary data of task execution bodies 1, 2, and 3, respectively. Then, processor memories BC, EF, and HI can be used as second processor memories, and these second processor memories can be divided into multiple memory blocks.

[0047] In the above embodiment, by calling the task execution body 33 to simulate the execution of the reasoning task, the memory space required for the necessary data such as the input data and output data of the reasoning task can be relatively accurate. Furthermore, the remaining second processor memory corresponding to each task execution body 33 can be determined relatively accurately, avoiding the problem of the second processor memory being too large or too small.

[0048] In some embodiments, the central manager 32 is specifically used to divide the second processor memory into multiple memory blocks according to the pre-configured memory block size. Among them, the memory block size can be the memory space required to consume a preset number of tokens. For example, the memory block size can be the memory space required to consume 1 token or 3 tokens. In this way, each memory block obtained by division can store a specified number of tokens. For example, when the pre-configured memory block size is the memory space required to consume 3 tokens, each memory block obtained by division can store 3 tokens respectively. The memory block size can be configured according to actual needs. Generally, the smaller the configured memory block size, the smaller the memory space of each memory block obtained by segmentation, and when allocating the target memory block for the reasoning task, it can be more accurate and the waste rate of the processor memory 112 is lower.

[0049] In some embodiments, considering that the memory space of the processor memory 112 may be reduced, if the remaining second processor memory corresponding to each task execution body 33 is divided into memory blocks, then there may be insufficient memory space for input data, output data, model weights, tokens and other data. For example, assume that the memory space of the target memory block released by the previous reasoning task is exactly equal to the memory space of the target memory block required by the next reasoning task. If the next memory space is smaller than the previous memory space, it means that the memory space has been reduced. In this case, since there is no extra memory space, it may cause the next reasoning task to fail. In view of this, the central manager 32 is specifically used to divide the third processor memory from the second processor memory, and divide a plurality of memory blocks from the third processor memory, wherein the memory space of the third processor memory is smaller than the memory space of the second processor memory. For example, 90% of the second processor memory can be used as the third processor memory, and a plurality of memory blocks can be divided from the third processor memory. In this way, when the memory space is reduced, the remaining 10% of the second processor memory can be used to ensure the smooth execution of the reasoning task.

[0050] In some embodiments, it is considered that the memory space used to store the token needs to be continuous, but among the multiple memory blocks obtained by division, the physical addresses of at least some of the memory blocks are discontinuous, that is, there is fragmented memory space, which does not meet the storage space requirements of the token. In view of this, the central manager 32 is specifically used to map the target memory block to a virtual memory with a continuous logical address after allocating the target memory block. Specifically, a memory mapping table can be established in advance. The memory mapping table includes a mapping relationship between the physical address and the logical address of each memory block, and these logical addresses are continuous. In this way, after the target memory block is divided, based on the memory mapping table, the target memory block with a discontinuous physical address can be mapped to a virtual memory with a continuous logical address. In this way, a continuous virtual memory for storing tokens can be obtained, which improves the utilization rate of the fragmented memory space.

[0051] In some embodiments, the memory block information queue includes numbers of each memory block, the preconfigured memory block size is the memory space required to be consumed by the first number of tokens, and the inference task includes a second number of tokens;

[0052] When both the first number and the second number are greater than 1, the central manager 32 is specifically used to specify the number of the target memory block in the memory block information queue and the storage location of each token of the reasoning task in the target memory block when allocating the target memory block for the token of the reasoning task.

[0053] Specifically, when the first number is greater than 1, it means that each target memory block can store multiple tokens. Therefore, when allocating target memory blocks for tokens of reasoning tasks, one can be allocated to each token. . Where m represents the number of the target memory block allocated to the token, and n represents the position of the token in the mth target memory block, that is, the position of the nth token in the target memory block, where the value range of n is , That is, the number of tokens that can be saved in a target memory block. For ease of understanding, refer to Figure 6 , is a schematic diagram of the token storage location of the target memory block provided by an embodiment of the present disclosure. Assume Figure 6 The target memory block shown is numbered 33 and can store 6 tokens. Then the storage location of the first token stored in the target memory block is , the storage location of the second token is , and so on.

[0054] For a clearer explanation, please refer to Table 1, which lists the storage locations of tokens of different reasoning tasks in the target memory block as an example.

[0055] Table 1 The number of the target memory block and the location where the token is stored in the target memory block

[0056]

[0057] In Table 1, assuming that b_0 represents the first reasoning task, b_1 represents the second reasoning task, token represents the token, and block represents the target memory block. Then the first row means that the first 64 tokens of the first reasoning task are stored in the target memory block block_0, that is, the target memory block block_0 can store 64 tokens; the third row means that the last 64 tokens of the first reasoning task are stored in the target memory block block_z; the fourth row means that the first 64 tokens of the second reasoning task are stored in the target memory block block_y.

[0058] In the above embodiment, when both the first number and the second number are greater than 1, the storage location of each token can be accurately determined by specifying the number of the target memory block in the memory block information queue and the storage location of each token of the reasoning task in the target memory block.

[0059] Further, in the case where the first number is 1, the central manager 32 is specifically used to specify the number of the target memory block in the memory block information queue when allocating the target memory block for the token of the reasoning task;

[0060] Specifically, when the first number is 1, it means that each target memory block can only be used to store one token. In this case, it is not necessary to specify the storage location of the token in the target memory block. That is, according to the number of tokens of the reasoning task, a corresponding number of target memory blocks that are continuous in logical addresses are allocated, and the numbers of each target memory block are specified.

[0061] The scheduling strategy of the task execution body 33 is described below.

[0062] In some embodiments, when the reasoning tasks performed by at least some of the task executors 33 are different, the central manager 32 is specifically used to push the reasoning task into the task queue of the target task executor 33 used to execute the reasoning task when the task manager 31 completes the preprocessing of the reasoning task, so as to schedule the target task executor 33 to execute the reasoning task.

[0063] Specifically, for any task execution body 33, the task queue of the task execution body 33 includes a running queue, a switching queue and a waiting queue, wherein:

[0064] The reasoning tasks in the waiting queue are the reasoning tasks to be executed by the task execution body 33;

[0065] The reasoning tasks in the run queue are reasoning tasks being executed by the task executor 33 that has been saved in the artificial intelligence processor 11. In this embodiment, the run queue includes a formal run queue and a temporary run queue, and the formal run queue and the temporary run queue are ping-pong configurations. When executing the reasoning tasks in the formal run queue, the newly arrived reasoning tasks that need to be pressed into the run queue can be pressed into the temporary run queue. After the reasoning tasks in the formal run queue are executed, the temporary run queue can be used to replace the formal run queue. In this way, the reasoning tasks can be executed continuously, reducing the length of time that the core 111 in the artificial intelligence processor 11 is in the idle period, and at the same time, avoiding the waste of memory space.

[0066] The reasoning tasks in the exchange queue are the reasoning tasks in the running queue that need to be transferred to the outside of the artificial intelligence processor 11 when the processor memory of the artificial intelligence processor 11 is insufficient. Specifically, in this embodiment, for any task execution body 33, when the memory of the kernel 111 associated with the task execution body 33 is insufficient or the currently available memory blocks are insufficient, some of the reasoning tasks in the running queue of the task execution body 33 that do not need to be executed temporarily can be transferred from the GPU (i.e., the artificial intelligence processor) to the memory of the CPU where the scheduling engine 30 is located for storage. After the reasoning tasks in the running queue are transferred to the CPU, the memory resources occupied by these reasoning tasks can be released, so that other reasoning tasks in the running queue can have enough memory space, so as to avoid the problem of a large number of reasoning tasks being backlogged in the artificial intelligence processor 11 due to a shortage of memory space. In other words, for any task execution body 33, when there is a reasoning task in the exchange queue of the task execution body 33, it means that the memory of the kernel 111 associated with the task execution body 33 is insufficient or the currently available memory blocks are insufficient. That is, there is a backlog of reasoning tasks in the artificial intelligence processor 11.

[0067] For any of the above-mentioned running queues, exchange queues and waiting queues, under normal circumstances, reasoning tasks can be executed according to the principle of first-come-first-served, that is, reasoning tasks are executed in sequence according to the order in which the reasoning tasks are pushed into the queue.

[0068] Based on the above description, when pushing the inference task into the task queue of the target task executor 33, the central manager 32 is specifically used to push the inference task into the waiting queue of the target task executor 33, and when the exchange queue of the target task executor 33 is empty, it is determined whether the sum of the memory space of the currently unallocated memory blocks is greater than the memory space required for the first inference task in the waiting queue. If so, the target memory block for storing the token is allocated to the first inference task, and the first inference task that completes the target memory block allocation is pushed into the running queue of the target task executor 33.

[0069] Specifically, if the exchange queue of the target task execution body 33 is empty, it can indicate that there is no task backlog in the task queue of the target task execution body 33, that is, there are sufficient memory resources in the core 111 associated with the target task execution body 33 to execute the reasoning task. In this case, it can be further determined whether the memory space of the unused memory block in the artificial intelligence processor 11 is greater than the memory space required by the token of the reasoning task. If so, it means that the memory block in the artificial intelligence processor 11 is sufficient and the reasoning task can be executed normally. Therefore, the target memory block can be allocated to the first reasoning task in the waiting queue, and the first reasoning task that completes the target memory block allocation can be pushed into the running queue of the target task execution body 33.

[0070] On the contrary, if the exchange queue of the target task execution body 33 is not empty, it may indicate that there is a backlog of tasks in the task queue of the target task execution body 33. In this case, the reasoning task may continue to be placed in the waiting queue of the target task execution body 33 until the exchange queue of the target task execution body 33 is empty.

[0071] In some embodiments, when the reasoning task in the waiting queue of the target task execution body 33 is pressed into the running queue, although the target memory block will be allocated to the reasoning task, the number of the target memory block can be estimated. In the process of executing the reasoning task, the memory block actually required by the reasoning task may be larger than the allocated target memory block. The problem caused by this is that in the actual operation of the reasoning task, there may be insufficient memory blocks, which in turn causes the reasoning task to be backlogged in the artificial intelligence processor 11 due to the problem of insufficient memory blocks.

[0072] In order to avoid the above problems, in some embodiments, the central manager 32 can also be used to take out the last inference task in the formal running queue when both the exchange queue and the formal running queue of the target task executor 33 are not empty, and determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the last inference task. If so, the last inference task can be pushed into the temporary running queue of the target task executor 33. If not, the last inference task can be pushed into the exchange queue of the target task executor 33 to release the target memory block allocated for the target task executor.

[0073] Specifically, if the sum of the memory spaces of the currently unused memory blocks is greater than the memory space required by the token of the last inference task in the formal run queue, it means that the memory blocks in the artificial intelligence processor 11 are sufficient. In this case, the last inference task can be pushed into the temporary run queue of the target task execution body 33. Since the inference tasks in the formal run queue will be executed after the execution is completed, the inference tasks in the temporary run queue will be executed next. Therefore, pushing the last inference task in the formal run queue into the temporary run queue will not destroy the execution order between the inference tasks.

[0074] If the sum of the memory spaces of the currently unused memory blocks is less than the memory space required by the token of the last reasoning task in the formal run queue, it means that there is a problem of insufficient memory blocks during the actual operation of the reasoning task. In this case, the last reasoning task in the formal run queue can be pushed into the exchange queue of the target task execution body 33 to release the target memory block allocated for the last reasoning task and the kernel memory consumed by the input and output data of the last reasoning task, so as to provide sufficient memory space for other reasoning tasks in the formal run queue and ensure the smooth execution of other reasoning tasks. After the execution of other reasoning tasks is completed, the memory blocks allocated for these reasoning tasks and the kernel memory consumed by these reasoning tasks will be released. In this case, the reasoning tasks placed in the exchange queue can be put back into the run queue.

[0075] Specifically, in some embodiments, after the last inference task is pushed into the exchange queue of the target task execution body 33, the central manager 32 is also used to determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the first inference task in the exchange queue after the execution of the inference tasks in the running queue is completed. If so, it means that the current memory space in the artificial intelligence processor 11 is sufficient, and the first inference task in the exchange queue can be pushed into the running queue of the target task execution body 33. In this way, it is ensured that the inference task can be executed in time.

[0076] In some embodiments, when the sum of the memory spaces of the unused memory blocks in the artificial intelligence processor 11 is less than the memory space required by the token of the reasoning task, the method of the present disclosure may further include:

[0077] Searching for the third reasoning task in the waiting queue;

[0078] If the sum of the memory spaces of the unallocated memory blocks is greater than or equal to the memory space required by the token of the third reasoning task, the third reasoning task is executed first.

[0079] In this way, the kernel 111 associated with the target task execution body 33 can be prevented from being in an idle window period for a long time, and at the same time, waste of memory space can be avoided.

[0080] In some embodiments, in order to improve the utilization rate of the artificial intelligence processor 11, the task manager 31 is also used to package one or more reasoning tasks that need to be executed by the target task executor 33 into a task bundle, and the central manager 32 is specifically used to push the task bundle into the task queue of the target task executor 33.

[0081] Specifically, in the task bundle, each reasoning task may have information such as task number, estimated required memory block size, request data, input offset, output number, output probability, output offset, etc. It should be noted that when the task manager 31 packages the reasoning tasks, the memory block size required for each reasoning task may be set, but the target memory block is not allocated. The target memory block may be allocated in real time when the reasoning task enters the execution queue of the target task body.

[0082] In some embodiments, the task manager 31 is also used to mark the task arrival time of each reasoning task, and the target task executor 33 is also used to perform timeout processing on the reasoning task if the reasoning task has not been completed when the task arrival time is reached.

[0083] Specifically, if the execution of the reasoning task has not been completed when the task reaches the time limit, it can be indicated that the reasoning task is abnormal. In this case, the reasoning task can be timed out, for example, the execution of the reasoning task is forcibly interrupted. In this way, the reasoning task is prevented from occupying the resources of the kernel 111 and the target storage block associated with the target task execution body 33 for a long time, thereby affecting the execution of other reasoning tasks.

[0084] Combined with reference Figure 7 , which is provided for one embodiment of the present disclosure Figure 3 Detailed schematic diagram of the scheduling engine 30 in FIG. Figure 7 In the example, the task manager 31 in the scheduling engine 30 includes a preprocessing module 311 and a task group management module 312, wherein the preprocessing module 311 is used to convert the input text in the reasoning task into data that can be processed by the artificial intelligence processor 11. The task group management module 312 is used to package one or more reasoning tasks into a task bundle and initialize each reasoning task (such as marking the reasoning task number, identifying the required target memory block size, the arrival time of the reasoning task, etc.). It should be noted that during the initialization process, the target memory block size required for the reasoning task can be identified (estimated), but the target memory block is not allocated. The allocation of the target memory block can wait until before entering the execution queue, so as to avoid the reasoning task occupying the target memory block for a long time but not actually using the target memory block, thereby preventing waste of resources.

[0085] The central manager 32 includes a memory management module 332, a task queue management module 322 and a scheduling module 323. The memory management module 332 is used to divide the memory block information queue and allocate the target memory block for the reasoning task, the task queue management module 322 is used to initialize the execution queue, waiting queue and exchange queue for each task execution body 33; the scheduling module 323 is used to schedule the reasoning task in the execution queue, waiting queue and exchange queue of the task execution body 33 according to the actual situation.

[0086] The task execution body 33 includes a memory reporting module 331, a memory management module 332 and an operation module 333. Among them, the memory reporting module 331 is used to report the remaining second processor memory of each target execution body to the central manager 32 when dividing the memory block information queue, so as to facilitate the central manager 32 to divide the memory block information queue based on the second processor memory. The memory management module 332 is used to allocate the memory of the kernel 111, the corresponding CPU memory (the memory used for data exchange when the memory of the kernel 111 is tight) and the functions related to the implementation of memory operations (such as the functions required for memory data exchange between the kernel 111 and the CPU) during the execution of the inference task. The operation module 333 is used to call the corresponding operator in the libraries according to the model structure to complete the corresponding operation of each kernel 111.

[0087] The present disclosure also provides a scheduling method based on the scheduling engine 30. The scheduling method can be applied to the scheduling engine 30, or to an electronic device running the scheduling engine 30. The electronic device includes but is not limited to a tablet computer, a laptop computer, a desktop computer, etc. Figure 8 , which is a flow chart of a scheduling method provided in one embodiment of the present disclosure. Figure 8 In the scheduling method, the following steps are included:

[0088] Step S801: receiving an inference task and preprocessing the inference task.

[0089] Step S802, push the inference task into the task queue corresponding to one of the task executors 33 of the scheduling engine 30, and allocate a target memory block for storing the token to the inference task based on the memory space required for the token of the inference task and the memory block information of multiple memory blocks in the memory block information queue, wherein the multiple memory blocks are pre-divided from the processor memory 112 of the artificial intelligence processor 11.

[0090] Step S803 , when the task execution body 33 completes executing the inference task in the associated core 111 , the memory block information of the target memory block is pushed back into the memory block information queue.

[0091] In the technical solutions of some embodiments of the present disclosure, on the one hand, by setting a task execution body 33 associated with the core of the artificial intelligence processor in the scheduling engine 30, the core can be scheduled by scheduling the task execution body 33, and the scheduling strategy of the core can be optimized by optimizing the scheduling strategy of the task execution body 33. In this way, the core idle period can be reduced, the resource utilization of the core can be improved, and the model reasoning speed can be improved; on the other hand, since the technical solution of the present disclosure allocates memory space for reasoning tasks in units of memory blocks, when the memory space size of a single memory block is set appropriately, the problem of too large or too small memory space allocated to the reasoning task can be effectively avoided, that is, the waste of processor memory is reduced, and the model reasoning speed can be improved. In summary, the technical solution of the present disclosure can optimize the resource usage in the artificial intelligence processor 11 and improve the model reasoning speed.

[0092] In some embodiments, the plurality of memory blocks are obtained based on the following method:

[0093] Based on the pre-configured number of cores and the total processor memory allowed to be used by each task execution body 33, calling the target number of task execution bodies 33 matching the number of cores to simulate and execute the reasoning task, so as to determine the first processor memory that needs to be consumed by the task information data, input data and output data of the reasoning task executed by each task execution body 33;

[0094] Determine the second processor memory remaining after subtracting the corresponding first processor memory from the total processor memory of each task execution body 33, and divide the second processor memory into multiple memory blocks, wherein the second processor memory is used to store intermediate result data of the reasoning task, and the intermediate result data includes the token of the reasoning task.

[0095] In some embodiments, among the multiple memory blocks obtained by division, physical addresses of at least some of the memory blocks are discontinuous, and the method of the present disclosure further includes:

[0096] After the target memory block is allocated, the target memory block is mapped as a virtual memory with continuous logical addresses.

[0097] In some embodiments, the plurality of memory blocks are obtained by dividing the memory of the second processor according to a preconfigured memory block size. The memory block information queue includes the serial number of each memory block, the preconfigured memory block size is the memory space required to be consumed by the first number of tokens, and the reasoning task includes the second number of tokens; the target memory block for storing the tokens is allocated to the reasoning task based on the memory space required by the tokens of the reasoning task and the memory block information of the plurality of memory blocks in the memory block information queue, including:

[0098] When the first number is 1, the target memory block is allocated to the token of the reasoning task by specifying the number of the target memory block in the memory block information queue;

[0099] When both the first number and the second number are greater than 1, the target memory block is allocated to the token of the reasoning task by specifying the number of the target memory block in the memory block information queue and the storage position of each token of the reasoning task in the target memory block; and / or

[0100] A plurality of memory blocks are divided from the memory of the second processor, including:

[0101] A third processor memory is divided from the second processor memory, and a plurality of memory blocks are divided from the third processor memory, wherein the memory space of the third processor memory is smaller than the memory space of the second processor memory.

[0102] In some embodiments, at least some of the reasoning tasks performed by the task execution bodies 33 are different;

[0103] Pushing the inference task into a task queue corresponding to one of the task execution bodies 33 of the scheduling engine 30 includes:

[0104] When the preprocessing of the reasoning task is completed, the reasoning task is pushed into the task queue of the target task execution body 33 for executing the reasoning task, so as to schedule the target task execution body 33 to execute the reasoning task.

[0105] In some embodiments, for any task execution body 33, the task queue of the task execution body 33 includes a running queue, an exchange queue and a waiting queue, wherein the reasoning tasks in the running queue are the reasoning tasks being executed by the task execution body 33 that has been saved in the artificial intelligence processor 11, the reasoning tasks in the exchange queue are the reasoning tasks in the running queue that need to be transferred to outside the artificial intelligence processor 11 when the processor memory of the artificial intelligence processor 11 is insufficient, and the reasoning tasks in the waiting queue are the reasoning tasks to be executed by the task execution body 33;

[0106] Pushing the inference task into the task queue of the target task execution body 33 for executing the inference task includes:

[0107] The inference task is pushed into the waiting queue of the target task execution body 33, and when the exchange queue of the target task execution body 33 is empty, it is determined whether the sum of the memory space of the currently unallocated memory blocks is greater than the memory space required by the first inference task in the waiting queue. If so, a target memory block for storing the token is allocated to the first inference task, and the first inference task that completes the target memory block allocation is pushed into the running queue of the target task execution body 33;

[0108] And, the running queue includes a formal running queue and a temporary running queue; when the exchange queue and the formal running queue of the target task execution body 33 are not empty, the last reasoning task in the formal running queue is taken out, and it is determined whether the sum of the memory spaces of the currently unused memory blocks is greater than the memory space required by the token of the last reasoning task. If so, the last reasoning task is pushed into the temporary running queue of the target task execution body 33; if not, the last reasoning task is pushed into the exchange queue of the target task execution body 33 to release the target memory block allocated for the last reasoning task;

[0109] Also, after the last inference task is pushed into the exchange queue of the target task executor 33, when the inference tasks in the running queue are executed, determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the first inference task in the exchange queue. If so, push the first inference task in the exchange queue into the running queue of the target task executor 33.

[0110] The principle of the scheduling method is similar to that of the scheduling engine 30 and has the same beneficial effects, which will not be described in detail here.

[0111] The following combination Fig. 9 , describes the task scheduling process between different task queues of the same task execution body 33. Specifically, for the task queue of any task execution body 33, during the task scheduling process, it can be first determined whether the exchange queue is empty.

[0112] When the exchange queue is empty, you can execute Fig. 9The process on the left continues to determine whether the waiting queue is empty. If it is empty, it means that no new reasoning tasks are pushed into the waiting queue, and the reasoning tasks in the running queue can continue to be executed. If it is not empty, the first task bundle in the waiting queue can be taken out, and it can be determined whether the token length of the first task bundle is out of bounds. If so, the first task bundle is marked as an ignored task bundle, and the task bundle is taken out and pushed into the ignored queue. Among them, the ignored queue is a queue other than the above-mentioned running queue, exchange queue and waiting queue, which is used to store task bundles that need to be ignored. After the task bundle is pushed into the ignored queue, the first task bundle in the waiting queue changes. At this time, it can continue to return to determine whether the waiting queue is empty, and continue to determine whether the token length of the new first task bundle is out of bounds. If not, continue to determine whether the sum of the memory space of the unallocated memory blocks is greater than the memory space required by the token of the new task bundle. If not, end the process. If yes, continue to determine whether the token length of all task bundles in the waiting queue exceeds the threshold. If yes, end the process. If no, continue to determine whether the padding data is out of bounds. If yes, end the process. If no, allocate the target memory block to the inference task in the new task bundle, and after completing the allocation of the target memory block, push the new task bundle into the run queue. Specifically, push it into the temporary run queue in the run queue.

[0113] If the exchange queue is not empty, it means that there is a backlog of tasks in the run queue. You can execute Fig. 9 The process on the right side continues to determine whether the formal run queue is empty. If so, the temporary run queue replaces the formal run queue; if not, the last task bundle in the formal run queue is taken out, and it is determined whether the memory space of the unallocated memory block is greater than the memory space required by the token of the last task bundle. If so, the target memory block is allocated to the last task bundle (if it has been allocated, it can be ignored), and the task bundle with the allocated target memory block is placed in the temporary run queue, and it returns to re-determine whether the formal run queue is empty. If the memory space of the unallocated memory block is less than the memory space required by the token of the last task bundle, the last task bundle can be pushed into the exchange queue and the process ends.

[0114] After pushing the last task bundle into the exchange queue, you can continue to determine whether the run queue is free (i.e., whether the reasoning tasks in the run queue have been completed). If not, continue to execute the reasoning tasks in the run queue. If yes, take out the first task bundle in the exchange queue and determine whether the sum of the memory space of the unallocated memory blocks is greater than the memory space required by the token of the task bundle. If yes, allocate the target memory block to the task bundle and push the task bundle with the allocated target memory block into the run queue. After pushing the task bundle into the run queue, the task bundle in the exchange queue changes. At this time, you can continue to determine whether the exchange queue is empty. If not, take out the new first task bundle in the exchange queue and continue to determine whether the target memory block can be allocated to the new first task. If the exchange queue is empty, it means that the backlog of reasoning tasks has been executed.

[0115] Combined with reference Fig.10 , is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device includes a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the above method is implemented.

[0116] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0117] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0118] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0120] An embodiment of the present disclosure further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0121] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A scheduling engine, characterized in that: The scheduling engine includes a task manager, a central manager and at least one task execution body for executing reasoning tasks, wherein: The task manager is used to receive the reasoning task and pre-process the reasoning task; The central manager is used to call each of the task executors to simulate and execute the reasoning task based on the total processor memory of the artificial intelligence processor that each of the task executors is allowed to use in advance, determine the first processor memory that each of the task executors needs to consume when executing the reasoning task, and determine the second processor memory remaining after subtracting the corresponding first processor memory from the total processor memory of each of the task executors, divide the multiple memory blocks from the second processor memory, push the memory block information of the multiple memory blocks into the memory block information queue, and when the task manager completes the preprocessing of the reasoning task, push the reasoning task into the task queue corresponding to one of the task executors, and allocate a target memory block for storing the token for the reasoning task based on the memory space required for the token of the reasoning task and the memory block information in the memory block information queue, and after the reasoning task is executed, push the memory block information of the target memory block back into the memory block information queue, the second processor memory is used to store the intermediate result data of the reasoning task, and the intermediate result data includes the token of the reasoning task; The task execution body is used to execute the reasoning task in the corresponding task queue, wherein different task execution bodies are associated with different cores of the artificial intelligence processor, and each task execution body executes the reasoning task in the associated core.

2. The scheduling engine according to claim 1, characterized in that: The central manager is specifically used to call a target number of task execution bodies that matches the pre-configured number of cores to simulate and execute reasoning tasks based on the pre-configured number of cores, so as to determine the first processor memory corresponding to each of the task execution bodies.

3. The scheduling engine according to claim 1 or 2, characterized in that: Among the multiple memory blocks obtained by division, the physical addresses of at least some of the memory blocks are discontinuous, and the central manager is specifically used to map the target memory block to a virtual memory with a continuous logical address after allocating the target memory block.

4. The scheduling engine according to claim 2, characterized in that: The central manager is specifically used to divide the second processor memory into the multiple memory blocks according to a pre-configured memory block size, and the memory block information queue includes the serial number of each of the memory blocks, the pre-configured memory block size is the memory space required to be consumed by the first number of tokens, and the reasoning task includes the second number of tokens; When the first number is 1, the central manager is specifically used to specify the number of the target memory block in the memory block information queue when allocating the target memory block to the token of the reasoning task; When the first number and the second number are both greater than 1, the central manager is specifically used to specify the number of the target memory block in the memory block information queue and the storage location of each token of the reasoning task in the target memory block when allocating the target memory block to the token of the reasoning task.

5. The scheduling engine according to claim 2, characterized in that: The central manager is specifically used to divide a third processor memory from the second processor memory, and to divide the multiple memory blocks from the third processor memory, and the memory space of the third processor memory is smaller than the memory space of the second processor memory.

6. The scheduling engine according to claim 1, characterized in that: The reasoning tasks performed by at least some of the task executors are different. The central manager is specifically used to push the reasoning tasks into the task queue of the target task executor used to execute the reasoning tasks when the task manager completes the preprocessing of the reasoning tasks, so as to schedule the target task executor to execute the reasoning tasks.

7. The scheduling engine according to claim 6, characterized in that: For any of the task execution bodies, the task queue of the task execution body includes a running queue, a switching queue and a waiting queue, wherein the reasoning tasks in the running queue are the reasoning tasks that have been saved in the artificial intelligence processor and are being executed by the task execution body; the reasoning tasks in the switching queue are the reasoning tasks in the running queue that need to be transferred to outside the artificial intelligence processor when the processor memory of the artificial intelligence processor is insufficient; the reasoning tasks in the waiting queue are the reasoning tasks to be executed by the task execution body; When pushing the inference task into the task queue of the target task executor, the central manager is specifically used to push the inference task into the waiting queue of the target task executor, and when the exchange queue of the target task executor is empty, determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the first inference task in the waiting queue; if so, allocate a target memory block for storing the token to the first inference task, and push the first inference task that has completed the target memory block allocation into the running queue of the target task executor.

8. The scheduling engine according to claim 7, characterized in that: The running queue includes a formal running queue and a temporary running queue; The central manager is also used to take out the last inference task in the formal running queue when both the exchange queue and the formal running queue of the target task executor are not empty, and determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the last inference task. If so, push the last inference task into the temporary running queue of the target task executor; if not, push the last inference task into the exchange queue of the target task executor to release the target memory block allocated for the last inference task.

9. The scheduling engine according to claim 8, characterized in that: After pushing the last inference task into the exchange queue of the target task executor, the central manager is also used to determine whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the first inference task in the exchange queue after the inference tasks in the running queue are executed. If so, the first inference task in the exchange queue is pushed into the running queue of the target task executor.

10. The scheduling engine according to claim 6, characterized in that: The task manager is further used to package one or more of the reasoning tasks that need to be executed by the target task executor into a task bundle, and the central manager is specifically used to push the task bundle into the task queue of the target task executor.

11. The scheduling engine according to claim 6, characterized in that: The task manager is further used to mark the task arrival time of each of the reasoning tasks, and the target task executor is further used to perform timeout processing on the reasoning task if the reasoning task has not been completed when the task arrival time is reached.

12. A scheduling method based on a scheduling engine, characterized in that: The scheduling engine includes at least one task execution body for executing reasoning tasks, and different task execution bodies are associated with different cores of the artificial intelligence processor; the method includes: Receiving a reasoning task and preprocessing the reasoning task; The reasoning task is pushed into the task queue corresponding to one of the task executors of the scheduling engine, and based on the memory space required by the token of the reasoning task and the memory block information of multiple memory blocks in the memory block information queue, a target memory block for storing the token is allocated to the reasoning task, wherein the multiple memory blocks are pre-divided from the processor memory of the artificial intelligence processor based on the following method: based on the total processor memory of the artificial intelligence processor allowed to be used by each of the pre-configured task executors, each of the task executors is called to simulate the execution of the reasoning task, and the first processor memory required for the task information data, input data and output data of each of the task executors when executing the reasoning task is determined, and the second processor memory remaining after subtracting the corresponding first processor memory from the total processor memory of each of the task executors is determined, and the multiple memory blocks are divided from the second processor memory; the second processor memory is used to store the intermediate result data of the reasoning task, and the intermediate result data includes the token of the reasoning task. When the task execution body completes executing the reasoning task in the associated kernel, the memory block information of the target memory block is pushed back into the memory block information queue.

13. The method according to claim 12, characterized in that The calling of each of the task execution bodies to simulate and execute the reasoning task includes: Based on the preconfigured number of cores, a target number of task execution bodies matching the number of cores are called to simulate and execute reasoning tasks, so as to determine the first processor memory corresponding to each of the task execution bodies.

14. The method according to claim 12 or 13, characterized in that Among the multiple memory blocks obtained by division, physical addresses of at least some of the memory blocks are discontinuous, and the method further includes: After the target memory block is allocated, the target memory block is mapped to a virtual memory with a continuous logical address.

15. The method according to claim 13, characterized in that The multiple memory blocks are obtained by dividing from the memory of the second processor according to a preconfigured memory block size, the memory block information queue includes the serial number of each of the memory blocks, the preconfigured memory block size is the memory space required to be consumed by the first number of tokens, and the inference task includes the second number of tokens; The method of allocating a target memory block for storing the token for the reasoning task based on the memory space required by the token of the reasoning task and the memory block information of the plurality of memory blocks in the memory block information queue comprises: When the first number is 1, allocating the target memory block to the token of the reasoning task by specifying the number of the target memory block in the memory block information queue; When both the first number and the second number are greater than 1, allocating the target memory block to the token of the reasoning task by specifying the number of the target memory block in the memory block information queue and the storage position of each token of the reasoning task in the target memory block; and / or The dividing the plurality of memory blocks from the memory of the second processor comprises: A third processor memory is divided from the second processor memory, and the plurality of memory blocks are divided from the third processor memory, wherein the memory space of the third processor memory is smaller than the memory space of the second processor memory.

16. The method according to claim 12, characterized in that The reasoning tasks performed by at least some of the task execution bodies are different; The step of pressing the inference task into a task queue corresponding to one of the task execution bodies of the scheduling engine includes: When the preprocessing of the reasoning task is completed, the reasoning task is pushed into the task queue of the target task execution body for executing the reasoning task, so as to schedule the target task execution body to execute the reasoning task.

17. The method according to claim 16, characterized in that For any of the task execution bodies, the task queue of the task execution body includes a running queue, a swap queue and a waiting queue, wherein the reasoning tasks in the running queue are reasoning tasks that have been saved in the artificial intelligence processor and are being executed by the task execution body, the reasoning tasks in the swap queue are reasoning tasks in the running queue that need to be transferred outside the artificial intelligence processor when the processor memory of the artificial intelligence processor is insufficient, and the reasoning tasks in the waiting queue are reasoning tasks to be executed by the task execution body; The step of pressing the inference task into a task queue of a target task execution body for executing the inference task comprises: Push the inference task into the waiting queue of the target task execution body, and when the exchange queue of the target task execution body is empty, determine whether the sum of the memory spaces of the currently unallocated memory blocks is greater than the memory space required by the first inference task in the waiting queue; if so, allocate a target memory block for storing the token to the first inference task, and push the first inference task that has completed the target memory block allocation into the running queue of the target task execution body; Furthermore, the run queue includes a formal run queue and a temporary run queue; when both the exchange queue and the formal run queue of the target task execution body are not empty, the last reasoning task in the formal run queue is taken out, and it is determined whether the sum of the memory spaces of the currently unused memory blocks is greater than the memory space required by the token of the last reasoning task; if so, the last reasoning task is pushed into the temporary run queue of the target task execution body; if not, the last reasoning task is pushed into the exchange queue of the target task execution body to release the target memory block allocated for the last reasoning task; Furthermore, after the last inference task is pushed into the exchange queue of the target task execution body, when the inference tasks in the running queue are executed, it is determined whether the sum of the memory space of the currently unused memory blocks is greater than the memory space required by the token of the first inference task in the exchange queue. If so, the first inference task in the exchange queue is pushed into the running queue of the target task execution body.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method according to any one of claims 12 to 17 is implemented.

19. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 12 to 17 is implemented.

20. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 12 to 17 is implemented.

Images