A Multi-DNN Inference Scheduling Method and System for Accelerating Edge Servers with In-Memory Computing

Abstract

This application relates to a multi-DNN inference scheduling method and system for in-memory computing-accelerated edge servers. The method involves receiving inference requests; reading the model parameters of a target DNN model from a cached DNN model file based on the inference request; and generating scheduling parameters based on these parameters. The scheduling parameters include the number of copies of the target DNN model, the batch size for processing the target DNN model per run, and a model packaging strategy for splitting the target DNN model. Based on the scheduling parameters, the inference request is constructed into a task. A PIM accelerator is invoked to process the task according to a runtime function pre-compiled for the target DNN model, generating the corresponding inference result. This application overcomes the limitation of current PIM accelerators that only optimize single DNN models, enabling concurrent scheduling of multiple types of DNN inference tasks to meet the diverse needs of edge IoT devices.

Description

Technical Field

[0001] This application relates to the field of in-memory computing acceleration technology, and in particular to a multi-DNN inference scheduling method and system for in-memory computing acceleration edge servers. Background Technology

[0002] A Processing-in-Memory (PIM) accelerator is a hardware accelerator that integrates computing units and storage modules. Its core advantages are low power consumption and high computational density. It is suitable for DNN (Deep Neural Networks) inference tasks in edge computing scenarios and mainly consists of three parts: a tile grid, an on-chip buffer, and an I / O interface. In multi-access edge computing (MEC) scenarios, PIM accelerators are deployed in edge servers. The edge servers receive inference requests for different DNN models from IoT devices and call the PIM accelerator to run the DNN models for inference.

[0003] A single DNN optimization technique for PIM accelerators has been proposed. Although this technique is designed for PIM, it can only optimize a single DNN model and cannot handle concurrent requests from multiple models, resulting in low hardware utilization.

[0004] Currently, there is no effective solution for the diverse DNN inference needs of PIM-accelerated edge servers. Summary of the Invention

[0005] Based on this, it is necessary to provide a multi-DNN inference scheduling method, DNN inference scheduling system, computer device, and computer-readable storage medium for in-memory computing-accelerated edge servers that meets the diverse DNN inference needs and satisfies the above-mentioned technical problems.

[0006] In a first aspect, this application provides a multi-DNN inference scheduling method for in-memory computing-accelerated edge servers, including:

[0007] Receive inference request;

[0008] According to the inference request, the model parameters of the target DNN model are read from the cached DNN model file, and scheduling parameters are generated based on the model parameters of the target DNN model; wherein, the scheduling parameters include: the number of copies of the target DNN model, the batch size of the target DNN model in a single processing, and the model packaging strategy for splitting the target DNN model;

[0009] Based on the scheduling parameters, the inference request is constructed as a task;

[0010] Based on the runtime function pre-compiled for the target DNN model, the PIM accelerator is invoked to process the task and generate inference results corresponding to the task.

[0011] In one embodiment, based on the inference request, model parameters of the target DNN model are read from a cached DNN model file, and scheduling parameters are generated based on the model parameters of the target DNN model, including:

[0012] From the set of DNN models to be scheduled, multiple DNN model packages are divided according to the PIM accelerator resources and the computational resource consumption of each target DNN model; wherein each DNN model package contains a Leader model, or contains a Leader model and at least one Follower model, and the processing time of the Follower model in each DNN model package does not exceed the processing time of the corresponding Leader model.

[0013] Based on the resource constraints of the PIM accelerator and the request frequency of each Leader model, the number of copies of each Leader model is determined.

[0014] Based on the request frequency of each target DNN model in each DNN model sub-package, the batch size of each target DNN model processed in a single DNN model sub-package is adjusted.

[0015] In one embodiment, the set of DNN models to be scheduled is divided into multiple DNN model packages based on the PIM accelerator resources and the computational resource consumption of each target DNN model, including:

[0016] From the set of DNN models to be scheduled, select the target DNN model with the largest computational resource consumption as the current Leader model, and remove the current Leader model from the set of DNN models;

[0017] Determine whether there are any remaining target DNN models in the current set of DNN models to be scheduled that meet preset conditions. The preset conditions include: the remaining PIM accelerator resources after allocating the current Leader model can cover the resources required by the remaining target DNN models, and the processing time of the remaining target DNN models does not exceed the processing time of the current Leader model.

[0018] If there are remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled, then the remaining target DNN models that meet the preset conditions are taken as Follower models, and the Follower models are removed from the set of DNN models. Then, the process returns to the step of determining whether there are remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled.

[0019] If there are no remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled, then the current Leader model is packaged into the current DNN model package, or the current Leader model and at least one selected Follower model are packaged into the current DNN model package.

[0020] In one embodiment, after dividing the set of DNN models to be scheduled into packets based on the PIM accelerator resources and the computational resource consumption of each target DNN model, the method further includes:

[0021] In the current set of DNN models to be scheduled, select the next Leader model, and then divide the DNN models into packets based on the next Leader model to obtain the next DNN model packet.

[0022] In one embodiment, after receiving the inference request, the method further includes:

[0023] The inference requests are categorized and cached according to the DNN model they target;

[0024] Based on the cached inference requests, the request frequency of each DNN model is statistically analyzed.

[0025] In one embodiment, constructing the inference request as a task based on the scheduling parameters includes:

[0026] Based on the batch size of the target DNN model in a single processing, extract the corresponding number of inference requests targeting the same target DNN model;

[0027] According to the model packaging strategy, the extracted inference requests for one or more target DNN models are packaged to obtain the tasks corresponding to each DNN model.

[0028] Based on the number of copies of the target DNN model, determine the number of times the target DNN model is copied when the task is processed in the PIM accelerator.

[0029] In one embodiment, before invoking the PIM accelerator to process the task according to a runtime function pre-compiled for the target DNN model and generating an inference result corresponding to the task, the method further includes:

[0030] Based on the model parameters of the target DNN model, a first runtime function and a second runtime function are compiled and generated.

[0031] The first runtime function is used to write the target DNN model into the PIM accelerator;

[0032] The second runtime function is used to start the target DNN model in the PIM accelerator for inference.

[0033] Secondly, this application also provides a DNN inference scheduling system, including: a collector, a constructor and a dispatcher coupled in sequence;

[0034] The collector is used to receive and cache inference requests, and to cache DNN model files of multiple DNN models.

[0035] The constructor is configured to read the model parameters of the target DNN model from the cached DNN model file according to the inference request, and generate scheduling parameters according to the model parameters of the target DNN model; wherein, the scheduling parameters include: the number of copies of the target DNN model, the batch size of the target DNN model for processing in a single operation, and the model packaging strategy for splitting the target DNN model into packages; and construct the inference request into a task according to the scheduling parameters;

[0036] The dispatcher is used to compile and generate runtime functions for the target DNN model, so as to call the PIM accelerator to process the task and generate inference results corresponding to the task.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0039] The aforementioned multi-DNN inference scheduling method, DNN inference scheduling system, computer equipment, and computer-readable storage medium for in-memory computing-accelerated edge servers determine the corresponding target DNN model based on the inference request. Based on the model parameters of the target DNN model, it determines the number of copies of the target DNN model, the batch size for processing the target DNN model per run, and the model packaging strategy for splitting the target DNN model. Based on these scheduling parameters, the inference request is constructed as a task to package multiple target DNN models into one task. This allows the PIM accelerator to process multiple different target DNN models simultaneously, achieving spatial multiplexing of the PIM accelerator and improving its hardware resource utilization. Addressing the concurrent inference requirements of multiple types of DNN models in edge computing, this method overcomes the current limitation of PIM accelerators that only optimize single DNN models, enabling concurrent scheduling of multiple types of DNN inference tasks to meet the diverse needs of edge IoT devices. Attached Figure Description

[0040] Figure 1 This is an application environment diagram of a DNN inference scheduling system in one embodiment;

[0041] Figure 2 This is a schematic diagram of the structure of a DNN inference scheduling system in one embodiment;

[0042] Figure 3 This is a flowchart illustrating a multi-DNN inference scheduling method for in-memory computing-accelerated edge servers in one embodiment.

[0043] Figure 4 This is a flowchart illustrating a scheduling parameter calculation method in one embodiment;

[0044] Figure 5 This is a schematic diagram of the PIMShare scheduling architecture in one embodiment;

[0045] Figure 6 This is a schematic diagram of the PIMShare runtime flow in one embodiment;

[0046] Figure 7 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0049] Terminology Explanation:

[0050] PIM Accelerator (Processing-in-Memory Accelerator): A hardware accelerator that integrates computing units and storage modules, suitable for DNN inference tasks in edge scenarios. It mainly consists of three parts: Tile grid, on-chip buffer, and I / O interface.

[0051] Tile: The core in-memory computing component of the PIM accelerator, interconnected in a grid format, is the basic unit for implementing DNN inference computation. A tile stores a matrix X and receives a vector as input. Output the matrix-vector multiplication result. Different DNN models require different numbers of tiles to complete inference due to structural differences (i.e., the tile usage of the DNN model). ).

[0052] On-chip buffer: A built-in storage unit in the PIM accelerator used to temporarily store three types of key data during the DNN inference process: input data, output data, and intermediate results of inter-layer computations. Different DNN models require different on-chip buffer spaces to store input and output data during inference. ) and intermediate calculation results ( ).

[0053] DNN duplication: A parallel optimization technique for PIM accelerators. A DNN model occupying T tiles is duplicated to generate d independent model instances, occupying dT tiles for parallel processing. This technique can increase the parallelism of model inference by d times, thereby processing d inference requests for the same DNN model simultaneously and improving the throughput of inference request processing.

[0054] The method and system provided in this application can be applied to... Figure 1 In the application environment shown, terminal 100 communicates with server 200 via a network. Data storage system 300 can store data that server 200 needs to process. Data storage system 300 can be integrated onto server 200, or it can be placed on the cloud or other network servers.

[0055] Terminal 100 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 200 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0056] Server 200 is equipped with a DNN inference scheduling system. The DNN inference scheduling system receives inference requests initiated by multiple terminals 100, generates inference results, and returns the inference results to the corresponding terminals 100.

[0057] Figure 2 This is a schematic diagram of the DNN inference scheduling system in this embodiment, as shown below. Figure 2 As shown, the system includes a collector 201, a constructor 202, and a dispatcher 203 coupled in sequence. Figure 3 This is a flowchart illustrating the multi-DNN inference scheduling method for in-memory computing-accelerated edge servers in this embodiment. This method can run on the DNN inference scheduling system, such as... Figure 3 As shown, the process includes the following steps:

[0058] Step S101: Receive inference request.

[0059] Collector 201 interfaces with terminal devices to receive and cache inference requests, as well as cache DNN model files for multiple DNN models. Its output is the inference request. For example, different IoT devices, including mobile phones, surveillance cameras, and microphones, are connected to the same edge server. The edge server is equipped with a PIM accelerator, which receives inference requests for different DNN models from the IoT devices, such as a mobile phone requesting a text classification DNN, a camera requesting an object detection DNN, and a microphone requesting a speech recognition DNN.

[0060] Specifically, collector 201 maintains a DNN database, which stores DNN model files for multiple DNN models. DNN model files are in a standard format, such as ONNX files. Each DNN model file contains the model parameters of its respective DNN model. The parameters include: number of model layers PIM Accelerator Tile Usage And the usage of the PIM accelerator's on-chip buffer. and .

[0061] In some embodiments, after receiving an inference request, the collector 201 categorizes and caches the inference requests according to the DNN model they target; based on the cached inference requests, it counts the request frequency of each DNN model. Specifically, the collector 201 maintains a request buffer to categorize and cache inference requests for the same DNN model, avoiding request mixing. Furthermore, the collector 201 monitors the request frequency of each DNN model in real time and counts the requests for each DNN model. Requests per second This provides data support for subsequent scheduling.

[0062] In some embodiments, the collector 201 performs classification caching and status monitoring of inference requests, and its output is a set of inference requests categorized and organized according to the DNN model. By merging inference requests for the same DNN model and avoiding redundant model copying operations, the number of writes generated during model loading can be significantly reduced, solving the time overhead problem caused by frequent writes and extending the device lifespan of the PIM accelerator.

[0063] Step S102: Based on the inference request, read the model parameters of the target DNN model from the cached DNN model file, and generate scheduling parameters based on the model parameters of the target DNN model; wherein, the scheduling parameters include: the number of copies of the target DNN model, the batch size of the target DNN model in a single processing, and the model packaging strategy for splitting the target DNN model.

[0064] The constructor 202 generates scheduling parameters based on the inference requests and DNN model files cached by the collector 201, in order to configure and package the tasks. Its output is a task containing at least one DNN inference request.

[0065] Specifically, the constructor generates three sets of scheduling parameters based on the model parameters of the target DNN model:

[0066] Number of copies of the target DNN model: This refers to the number of copies of each DNN model on the PIM accelerator.

[0067] Batch size of the target DNN model in a single processing run: This refers to the number of requests processed per DNN model on the PIM accelerator in a single run.

[0068] Model packaging strategy for splitting the target DNN model: This refers to a collection of DNN models. It is split into multiple sub-packages.

[0069] Step S103: Construct the inference request into a task according to the scheduling parameters.

[0070] Based on the batch size of a single processing of the target DNN model, constructor 202 retrieves a corresponding number of inference requests for the same target DNN model; according to the model packaging strategy, it packages the retrieved inference requests for one or more target DNN models to obtain tasks corresponding to each DNN model; based on the number of copies of the target DNN model, it determines the number of times the target DNN model is copied when the task is processed in the PIM accelerator.

[0071] Specifically, when constructing tasks based on scheduling parameters, constructor 202 packages each model. A corresponding task is constructed. A task refers to the set of DNN inference requests that a PIM accelerator can complete within a time slice. In some embodiments, when constructing a task, the constructor 202 first initializes an empty task, and then... DNN model in extract The corresponding reasoning request is added to the task and set. The number of copies processed on the PIM accelerator is [number]. .

[0072] Step S104: Based on the runtime function pre-compiled for the target DNN model, call the PIM accelerator to process the task and generate the inference result corresponding to the task.

[0073] Dispatcher 203 will pre-compile and generate a first runtime function and a second runtime function based on the model parameters of the target DNN model; based on the first runtime function, write the target DNN model into the PIM accelerator; based on the second runtime function, start the target DNN model in the PIM accelerator to perform inference and generate inference results.

[0074] Specifically, dispatcher 203 pre-compiles a runtime function library, which is provided for each DNN model. Two runtime functions were compiled, namely the first runtime function. Second runtime function .in, Used to Write it into the Tile of the PIM accelerator. Used to launch the PIM accelerator The reasoning.

[0075] For a given task, dispatcher 203 initiates a PIM_Process process for handling. PIM_Process calls the corresponding DNN model within the task. and The PIM accelerator processes DNN inference requests within the task. After generating the task inference result, the dispatcher 203 packages the task inference result into the user request response and sends it to the terminal device that initiated the inference request.

[0076] In steps S101 to S104 above, the corresponding target DNN model is determined according to the inference request. Based on the model parameters of the target DNN model, the number of copies of the target DNN model, the batch size for processing the target DNN model in a single operation, and the model packaging strategy for splitting the target DNN model are determined. Based on these scheduling parameters, the inference request is constructed as a task to package multiple target DNN models into one task. The PIM accelerator is then used to process multiple different target DNN models simultaneously, achieving spatial multiplexing of the PIM accelerator and improving its hardware resource utilization. Addressing the concurrent inference requirements of multiple types of DNN models in edge computing, this approach overcomes the current limitation of PIM accelerators that only optimize single DNN models, enabling concurrent scheduling of multiple types of DNN inference tasks to meet the diverse needs of edge IoT devices.

[0077] Current DNN inference scheduling for PIM-accelerated edge servers, besides the single-DNN optimization techniques mentioned above, also includes a multi-DNN scheduling technique adapted to GPUs. However, the resource partitioning and partial loading strategies of this technique are incompatible with the hardware characteristics of PIM, which requires complete model loading. This can easily lead to resource monopolies and latency issues, and it also ignores the significant time overhead and device wear caused by frequent model writes. To address this problem, this application provides the following embodiments for improvement.

[0078] In this embodiment, from the set of DNN models to be scheduled, multiple DNN model packages are divided according to the PIM accelerator resources and the computational resource consumption of each target DNN model. Each DNN model package contains one Leader model, or contains one Leader model and at least one Follower model. The processing time of the Follower model in each DNN model package does not exceed the processing time of the corresponding Leader model. Based on the resource constraints of the PIM accelerator and the request frequency of each Leader model, the number of copies of each Leader model is determined. Based on the request frequency of each target DNN model in each DNN model package, the batch size of each target DNN model processed in the corresponding DNN model package is adjusted.

[0079] Figure 4 A flowchart illustrating the process of DNN model subdivision is provided, such as... Figure 4 As shown, the process includes the following steps:

[0080] Step S201: Select the target DNN model with the largest computational resource consumption from the set U of DNN models to be scheduled as the current Leader model, and remove the current Leader model from the set U of DNN models.

[0081] When executing this process for the first time, the set of DNN models to be scheduled, U, can be initialized to M. The data required for this embodiment includes: the number of tiles on the PIM accelerator. PIM accelerator on-chip cache space DNN model set And the request frequency per second of each DNN model. .

[0082] Step S202: Based on the resource constraints of the PIM accelerator and the request frequency of the current Leader model, determine the number of copies of the current Leader model.

[0083] Calculate the following two key values:

[0084] (1) The maximum number of replications of the Leader model under PIM resource constraints ;

[0085] (2) The Leader model can meet its request frequency. Minimum number of copies .

[0086] Pick and The minimum value in the set is used as the final number of replications for the Leader model. This setting avoids resource waste and redundant write operations due to excessive replication, while also preventing request backlog due to insufficient replication.

[0087] Step S203: Determine whether there are any remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled, U.

[0088] The preset conditions include: the remaining PIM accelerator resources after allocating the current Leader model are sufficient to cover the resources required by the remaining target DNN models, and the processing time of the remaining target DNN models does not exceed the processing time of the current Leader model. If there is a remaining target DNN model in the current set of DNN models to be scheduled that meets the preset conditions, then step S204 is executed. If there is no remaining target DNN model in the current set of DNN models to be scheduled that meets the preset conditions, then step S205 is executed.

[0089] Specifically, within the current set U of DNN models to be scheduled, models are traversed from smallest to largest to select those that can serve as Followers. There can be one or more Follower models. If a DNN model can utilize the remaining resources of the Leader model for processing, and its processing time is no longer than that of the Leader model, then that DNN model is selected as a Follower model and removed from U. This setup improves the hardware resource utilization of the PIM accelerator by filling the resource gaps between larger models with smaller ones, while also ensuring that Follower models do not experience excessive latency due to concurrency with larger models.

[0090] Step S204: Select the remaining target DNN models that meet the preset conditions as Follower models and remove them from U. After this step, return to step S203.

[0091] Step S205: Package the current Leader model into the current DNN model sub-package, or package the current Leader model and at least one selected Follower model into the current DNN model sub-package.

[0092] Some Leader models may not have Follower models and can be packaged as a separate task.

[0093] Step S206: Determine whether the current set of DNN models to be scheduled, U, is not empty. If yes, return to step S201; if no, proceed to step S207.

[0094] In the set of DNN models to be scheduled, after dividing the current DNN model into packets based on the PIM accelerator resources and the computational resource consumption of each target DNN model, the next Leader model is selected from the current set of DNN models to be scheduled. DNN model packets are then generated based on the next Leader model. Steps S201 to S205 are repeated until all DNN models in the DNN model set U are removed. This results in several tasks, each containing a combination of Leader and Follower models, or only a Leader model.

[0095] Step S207: Based on the request frequency of each target DNN model in the current DNN model package, adjust the batch size of each target DNN model processed in the current DNN model package.

[0096] Based on the request frequency of the DNN model Adjust the DNN model Batch size processed within a task This ensures that the time to collect a batch of requests does not exceed the processing time of that batch. This step combines the resource allocation of the DNN model (i.e., the number of replications) to determine the batch size, avoiding excessive latency issues caused by independent optimization.

[0097] In steps S201 to S207 above, the configuration is copied by generating the model. Batch configuration Together with the model packaging strategy p, aiming to improve hardware resource utilization, ensure low processing latency, and reduce write operations, these three sets of scheduling parameters are collaboratively optimized, and task construction is guided by the optimized scheduling parameters. This setting fully considers the hardware constraint that the PIM accelerator must fully load the entire DNN model before inference can begin, avoiding the resource allocation schemes that GPU-adapted scheduling techniques might generate due to ignoring this characteristic, thus ensuring that the scheduling strategy is compatible with the PIM hardware architecture. Collaborative optimization of DNN model resource allocation and batch size breaks the inherent logic of GPU-adapted scheduling techniques that optimize these two independently, effectively avoiding the problem of large-size DNN models monopolizing resources, and ensuring that even small models executing concurrently can meet strict latency requirements.

[0098] In some embodiments, before determining the number of copies of the current Leader model based on the resource constraints of the PIM accelerator and the request frequency of the current Leader model, the method further includes: classifying and caching inference requests according to the DNN model they target; and calculating the request frequency of each DNN model based on the cached inference requests. This configuration enables batch processing of inference requests for the same DNN model, avoiding duplicate model loading and reducing the number of writes to the PIM accelerator.

[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0100] In one embodiment, Figure 5 A schematic diagram of the PIMShare scheduling architecture is provided. This PIMShare architecture serves as an example of a DNN inference scheduling system, which can run as software on a computer device (e.g., a host CPU) to schedule multiple DNN inference requests. PIMShare includes a Collector, a Constructor, and a Dispatcher. The PIMShare operation flow is as follows:

[0101] ① Initialization: IoT devices submit their DNN models (ONNX format) to the collector and register their DNN inference request streams. The collector saves the ONNX files of the DNN models from the IoT devices, registers the DNN inference request streams, and monitors the request frequency per second for each DNN model in real time. PIMShare can adapt to the needs of multi-DNN inference in edge scenarios and can receive different DNN models from different IoT devices.

[0102] ② Compile: The dispatcher reads the DNN model ONNX file and compiles it to generate the corresponding runtime functions. and .

[0103] ③ Collect Request: The collector gathers DNN inference requests from IoT devices in real time and categorizes and caches requests for the same DNN model. This step enables batch processing of inference requests for the same DNN model, avoiding duplicate model loading and reducing the number of writes to the PIM accelerator.

[0104] ④ Construct Task: The constructor copies the configuration based on the model. Batch configuration The model packaging strategy p retrieves requests from the collector cache and constructs tasks. The constructor packages multiple DNNs into one task, utilizing the PIM accelerator to process multiple different DNN models simultaneously, achieving spatial multiplexing of the PIM accelerator and improving the hardware resource utilization of the PIM accelerator.

[0105] ⑤PIM Process: The dispatcher runs the PIM process, starts the PIM accelerator to process the tasks constructed by the constructor in step 4, and packages the inference results and returns them to the IoT device that initiated the request.

[0106] This embodiment designs a multi-DNN inference request scheduler, PIMShare, adapted to the PIM accelerator, to efficiently schedule the PIM accelerator to process inference tasks of multiple different DNN models in edge scenarios. The Leader-Follower algorithm proposed in this embodiment effectively improves the hardware resource utilization of the PIM accelerator, enhances the efficiency of the PIM accelerator in processing multi-DNN inference tasks, reduces the number of model loading writes, and extends the device lifespan of the PIM accelerator.

[0107] The following section will use three IoT devices as examples to further illustrate the PIMShare scheduling architecture described above.

[0108] Figure 6 This is a schematic diagram of the PIMShare workflow in this embodiment, as shown below. Figure 6 As shown, suppose that three IoT devices each initiate an attack on one of the three DNN models. The inference request. A PIM accelerator is deployed on the edge server; the number of Tile resources for the PIM accelerator... The processing flow is as follows:

[0109] Step S301, Collector saves Record the ONNX file and related information. For more information, please refer to [reference needed]. Figure 5 :

[0110] Number of floors: ;

[0111] Tile occupancy on the PIM accelerator: ;

[0112] PIM accelerator on-chip buffer usage .

[0113] Step S302: The dispatcher reads the ONNX file and compiles three runtime function libraries for the three DNN models: , .

[0114] In step S303, the IoT device initiates a DNN inference request, and the collector collects and caches the requests. The collector monitors the request frequency per second for each model in real time. .

[0115] In step S304, the constructor uses the Leader-Follower algorithm to calculate three sets of scheduling parameters. These are: model replication configuration. Batch configuration Model Packaging Strategy .in, Because it had the largest tile usage, it was selected as the Leader model, along with the Follower model. Package them into the same task. The Leader model is used as a standalone task, without a Follower model.

[0116] Step S305, configure according to batch The constructor retrieves 3 lines The reasoning request, 2 The inference request is packaged into Task 1, and 3 results are extracted. The reasoning request is packaged separately as Task 2.

[0117] Step S306: The dispatcher calls the PIM accelerator to process the task. When the dispatcher processes task 1, it calls... Will and Write it into the Tile of the PIM accelerator, then call it. implement and The DNN inference. Similarly, when the dispatcher processes task 2, it calls... Will Write it into the Tile of the PIM accelerator, then call it. implement The DNN inference is performed. Finally, the inference results are packaged and sent to IoT devices.

[0118] Compared with related technologies, the above steps S301 to S306 have the following advantages:

[0119] (1) The scene adaptability has been fully upgraded to match the inference needs of multiple DNN models at the edge: It is designed specifically for concurrent requests of a large number of different types of DNN inference tasks in edge computing scenarios. It solves the problem that the current single DNN optimization technology for PIM accelerators can only handle a single DNN model and cannot cope with multiple DNN models. It can flexibly adapt to the diverse inference needs of IoT devices.

[0120] (2) Hardware resource utilization is greatly improved, releasing the computing potential of PIM accelerator: By scheduling PIM accelerator to process multiple DNN inference tasks concurrently, the high computing density advantage of PIM accelerator is fully utilized, and the idle hardware resources that cannot be occupied by a single DNN model are efficiently utilized, changing the status quo of long-term idle resources caused by the existing single DNN optimization technology for PIM accelerator.

[0121] (3) The resource allocation scheme is feasible and adapted to the characteristics of PIM hardware architecture: When allocating resources, the Leader-Follower scheduling optimization algorithm fully considers the hardware constraint that the PIM accelerator must fully load the entire DNN model before it can start inference. This avoids the situation where GPU-adapted scheduling technology generates an unworkable resource allocation scheme due to ignoring this characteristic, and ensures that the scheduling strategy is compatible with the PIM hardware architecture.

[0122] (4) Joint optimization of resource allocation and batch size to ensure latency requirements of multiple models: The Leader-Follower scheduling optimization algorithm coordinates the optimization of DNN model resource allocation and batch size, breaking the inherent logic of GPU-adapted scheduling technology that optimizes the two independently. This effectively avoids the problem of large-size DNN models monopolizing resources and ensures that small models executed concurrently can also meet strict latency requirements.

[0123] (5) Significantly reduce write operations, reduce time overhead and hardware wear: By merging inference requests for the same model and avoiding redundant model copying operations, the number of writes generated during model loading is greatly reduced, solving the time overhead problem caused by frequent writes and extending the lifespan of the PIM accelerator.

[0124] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown below. Figure 7As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores DNN model data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a multi-DNN inference scheduling method for in-memory computing-accelerated edge servers.

[0125] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0126] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0127] Receive inference request;

[0128] Based on the inference request, the model parameters of the target DNN model are read from the cached DNN model file, and scheduling parameters are generated based on the model parameters of the target DNN model. The scheduling parameters include: the number of copies of the target DNN model, the batch size of the target DNN model in a single processing, and the model packaging strategy for splitting the target DNN model.

[0129] Based on the scheduling parameters, the inference request is constructed as a task;

[0130] Based on the runtime function pre-compiled for the target DNN model, the PIM accelerator is invoked to process the task and generate inference results corresponding to the task.

[0131] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0132] Based on the inference request, read the model parameters of the target DNN model from the cached DNN model file, and generate scheduling parameters based on the model parameters of the target DNN model, including:

[0133] From the set of DNN models to be scheduled, multiple DNN model packages are divided according to the PIM accelerator resources and the computational resource consumption of each target DNN model. Each DNN model package contains a Leader model, or contains a Leader model and at least one Follower model. The processing time of the Follower model in each DNN model package does not exceed the processing time of the corresponding Leader model.

[0134] Based on the resource constraints of the PIM accelerator and the request frequency of each Leader model, the number of copies of each Leader model is determined.

[0135] Based on the request frequency of each target DNN model in each DNN model package, adjust the batch size of each target DNN model processed in the corresponding DNN model package.

[0136] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0137] Within the set of DNN models to be scheduled, based on the PIM accelerator resources and the computational resource consumption of each target DNN model, multiple DNN model packages are divided, including:

[0138] From the set of DNN models to be scheduled, select the target DNN model with the largest computational resource consumption as the current Leader model, and remove the current Leader model from the set of DNN models;

[0139] Determine whether there are any remaining target DNN models in the current set of DNN models to be scheduled that meet the preset conditions. The preset conditions include: the remaining PIM accelerator resources after allocating the current Leader model can cover the resources required by the remaining target DNN models, and the processing time of the remaining target DNN models does not exceed the processing time of the current Leader model.

[0140] If there are remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled, then the remaining target DNN models that meet the preset conditions will be used as Follower models, and the Follower models will be removed from the DNN model set. Then, return to the step of determining whether there are remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled.

[0141] If there are no remaining target DNN models that meet the preset conditions in the current set of DNN models to be scheduled, then the current Leader model is packaged into the current DNN model package, or the current Leader model and at least one selected Follower model are packaged into the current DNN model package.

[0142] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0143] After dividing the current DNN model into packets based on the PIM accelerator resources and the computational resource consumption of each target DNN model in the set of DNN models to be scheduled, the method also includes:

[0144] In the current set of DNN models to be scheduled, select the next Leader model, and then divide the DNN models into packets based on the next Leader model to obtain the next DNN model packet.

[0145] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0146] After receiving the inference request, the method also includes:

[0147] Inference requests are categorized and cached according to the DNN model they target;

[0148] Based on the cached inference requests, the request frequency of each DNN model is calculated.

[0149] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0150] Based on the scheduling parameters, the inference request is constructed into a task, including:

[0151] Based on the batch size of the target DNN model in a single processing session, retrieve the corresponding number of inference requests that target the same DNN model.

[0152] Based on the model packaging strategy, the extracted inference requests for one or more target DNN models are packaged to obtain the tasks corresponding to each DNN model.

[0153] Based on the number of copies of the target DNN model, determine the number of times the target DNN model is copied when the task is processed in the PIM accelerator.

[0154] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0155] Before invoking the PIM accelerator to process the task and generate the inference result corresponding to the task, based on the runtime function pre-compiled for the target DNN model, the method also includes:

[0156] Based on the model parameters of the target DNN model, the first runtime function and the second runtime function are compiled and generated.

[0157] The first runtime function is used to write the target DNN model into the PIM accelerator.

[0158] The second runtime function is used to start the target DNN model in the PIM accelerator for inference.

[0159] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the steps of a multi-DNN inference scheduling method for in-memory compute-accelerated edge servers. The multi-DNN inference scheduling method for in-memory compute-accelerated edge servers refers to the execution steps of the computer device described above, and will not be repeated here.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0161] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-DNN inference scheduling method for in-memory computing acceleration edge server, characterized in that, The method comprises the following steps: receiving an inference request; reading model parameters of a target DNN model from a cached DNN model file according to the inference request, and generating scheduling parameters according to the model parameters of the target DNN model; wherein the scheduling parameters comprise: the number of copies of the target DNN model, the batch size of single processing of the target DNN model, and a model packaging strategy for packaging the target DNN model; constructing the inference request into a task according to the scheduling parameters; calling a PIM accelerator to process the task according to a runtime function generated by pre-compiling the target DNN model, and generating an inference result corresponding to the task.

2. The multi-DNN inference scheduling method of an in-memory computing acceleration edge server according to claim 1, wherein, reading model parameters of a target DNN model from a cached DNN model file according to the inference request, and generating scheduling parameters according to the model parameters of the target DNN model, comprises: dividing a plurality of DNN model packages from a set of DNN models to be scheduled according to PIM accelerator resources and the size of the calculation resource occupation of each target DNN model; wherein each DNN model package contains a Leader model, or contains a Leader model and at least one Follower model, and the processing time of the Follower model in each DNN model package does not exceed the processing time of the corresponding Leader model; determining the number of copies of each Leader model based on the resource constraints of the PIM accelerator and the request frequency of each Leader model; adjusting the batch size of single processing of each target DNN model in the corresponding DNN model package based on the request frequency of each target DNN model in each DNN model package.

3. The multi-DNN inference scheduling method of an in-memory computing acceleration edge server according to claim 2, wherein, dividing a plurality of DNN model packages from a set of DNN models to be scheduled according to PIM accelerator resources and the size of the calculation resource occupation of each target DNN model, comprises: selecting a target DNN model with the largest calculation resource occupation from the set of DNN models to be scheduled as a current Leader model, and removing the current Leader model from the set of DNN models; determining whether there is a remaining target DNN model in the set of DNN models to be scheduled that meets a preset condition, wherein the preset condition comprises: the PIM accelerator resources remaining after the current Leader model is allocated can cover the resources required by the remaining target DNN model, and the processing time of the remaining target DNN model does not exceed the processing time of the current Leader model; if there is a remaining target DNN model in the set of DNN models to be scheduled that meets the preset condition, the remaining target DNN model that meets the preset condition is taken as a Follower model, and the Follower model is removed from the set of DNN models, and the step of determining whether there is a remaining target DNN model in the set of DNN models to be scheduled that meets the preset condition is returned to; If there is no remaining target DNN model satisfying the preset condition in the current set of DNN models to be scheduled, the current Leader model is packaged into the current DNN model package, or the current Leader model and the selected at least one Follower model are packaged into the current DNN model package.

4. The multi-DNN inference scheduling method of an in-memory computing acceleration edge server according to claim 3, wherein, After the current DNN model package is divided from the set of DNN models to be scheduled according to the PIM accelerator resources and the computation resource occupation sizes of the target DNN models, the method further includes: In the current set of DNN models to be scheduled, a next Leader model is selected, and DNN model packaging is performed according to the next Leader model to obtain a next DNN model package.

5. The multi-DNN inference scheduling method of an in-memory computing acceleration edge server according to claim 1, wherein, After receiving the inference request, the method further includes: The inference request is classified and cached according to the DNN model to which the inference request is directed; The request frequency of each DNN model is counted according to the cached inference request.

6. The multi-DNN inference scheduling method of an in-memory computing accelerated edge server according to claim 1, wherein, The inference request is constructed into a task according to the scheduling parameter, including: According to the batch size of a single processing of the target DNN model, inference requests of a corresponding number and directed to the same target DNN model are taken out; According to the model packaging strategy, the taken-out inference requests directed to one or more target DNN models are packaged to obtain a task corresponding to each DNN model package; According to the replication number of the target DNN model, the number of times the target DNN model is replicated when the task is processed in the PIM accelerator is determined.

7. The multi-DNN inference scheduling method of an in-memory computing accelerated edge server according to claim 1, wherein, Before the PIM accelerator processes the task according to the runtime function generated by pre-compiling the target DNN model to generate an inference result corresponding to the task, the method further includes: According to the model parameters of the target DNN model, a first runtime function and a second runtime function are compiled and generated; The first runtime function is used to write the target DNN model into the PIM accelerator; The second runtime function is used to start the target DNN model in the PIM accelerator for inference.

8. A DNN inference scheduling system, comprising: including: a collector, a constructor, and a dispatcher coupled in sequence; wherein, the collector is configured to receive and cache inference requests, and cache DNN model files of a plurality of DNN models; the constructor is configured to read model parameters of a target DNN model from the cached DNN model files according to the inference request, and generate a scheduling parameter according to the model parameters of the target DNN model; wherein, the scheduling parameter includes: a replication number of the target DNN model, a batch size of a single processing of the target DNN model, and a model packaging strategy of packaging the target DNN model; and the inference request is constructed into a task according to the scheduling parameter; the dispatcher is configured to compile and generate a runtime function for the target DNN model to call the PIM accelerator to process the task, and generate an inference result corresponding to the task. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images