Method and apparatus for scheduling inference task

The method and device for scheduling inference tasks address the limitations of conventional AI model execution by managing resource lists and switching tasks based on availability, ensuring efficient, reliable, and adaptable AI model performance across heterogeneous hardware environments.

WO2024237706A1PCT designated stage expired Publication Date: 2024-11-21OPT AI CO LTD

Patent Information

Application Number
PCT/KR2024/006661
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-05-16
Filing Date
2024-05-16
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Conventional methods for performing multiple AI models in embedded devices face limitations in hardware and platform expandability, and lack consideration for resource availability and failure mode impact analysis, leading to inefficiencies and reliability issues.

Method used

A method and device for scheduling inference tasks that generate lists of memory resources and processing units, receive status information, and determine whether preset task conditions are met, allowing for efficient execution of AI models by managing resources and switching to alternative tasks when conditions are not satisfied.

Benefits of technology

This approach enables efficient performance of multiple AI models without reducing response speed or accuracy, ensures high reliability, supports heterogeneous hardware environments, predicts and manages Service Level Objective (SLO) violations, minimizes energy consumption, and optimizes performance in real-time conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2024006661_21112024_PF_FP_ABST
    Figure KR2024006661_21112024_PF_FP_ABST
Patent Text Reader

Abstract

One embodiment of the present invention provides a method for scheduling an inference task performed by at least one processor. The method comprises the steps of: generating a memory resource list and a processing unit list for an inference task on the basis of a configuration file and generating a task thread for the inference task on the basis of the processing unit list; identifying an available first required resource in the memory resource list and an available second required resource in the processing unit list, and receiving state information about the first required resource and the second required resource; and determining whether a preset task condition is satisfied in consideration of the state information about the first required resource and the second required resource, and performing the inference task or an alternative task to the inference task according to whether the task condition is satisfied.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for scheduling inference tasks

[0001] The present invention relates to a method and device for scheduling an inference task, and more particularly, to a method and device for scheduling an inference task by taking into account status information of available resources.

[0002] As performance expectations from AI technology increase, the demand for simultaneous execution of multiple AI models continues to grow. For example, autonomous driving requires recognition of various types of objects, such as lanes, signals, pedestrians, and signs, and vehicle control accordingly. Because each recognition technology can have different requirements, such as response speed and accuracy, most systems develop separate AI models and execute them at appropriate intervals as needed.

[0003] To efficiently execute these multiple AI models, an AI model execution coordination algorithm that includes hardware resource management functions and scheduling functions for model execution timing / order is essential.

[0004] In particular, industries such as mobility, aviation, and defense that require both advanced cognitive and judgment capabilities and high reliability require a higher level of software optimization than other industries, and functional safety design must be considered from the design stage as software requires high reliability.

[0005] Various technologies have been developed to perform multiple AI models on conventional embedded devices.

[0006] However, conventional techniques for executing multiple AI models have many limitations in hardware and platform scalability, and have the problem that failure mode and effects analysis (FMEA) is not taken into account.

[0007] Prior art 1 has been presented regarding a task distribution method that divides a task into multiple unit tasks and processes each of them. Prior art 1 presents a technique for dividing a requested task into multiple unit tasks and assigning each of the divided unit tasks to multiple cores so that they can be processed in parallel on multiple cores.

[0008] However, prior art 1 has limitations in that it does not consider the availability of resources such as memory resources and hardware lists, as well as their respective states, and does not consider failure modes and impact analysis.

[0009] Accordingly, there is a need for a method to schedule work by considering the availability of resources and analysis of failure modes and impacts.

[0010] In order to solve the above-described technical problem, the present invention provides a scheduling method and device that generates a list of memory resources and a list of processing units suitable for an inference task, receives status information of necessary resources among these, and performs an inference task after determining whether a preset target value is satisfied.

[0011] The technical problems to be solved by the present invention are not limited to the technical problems described above, and other technical problems of the present invention can be derived from the following description.

[0012] In order to solve the above-described technical problem, one embodiment of the present invention provides a method for scheduling an inference task performed by at least one processor. The method for scheduling an inference task includes the steps of: generating a memory resource list and a processing unit list for the inference task based on a configuration file; generating a work thread for the inference task based on the processing unit list; checking available first necessary resources from the memory resource list and available second necessary resources from the processing unit list in response to a request for starting the inference task, and receiving status information of the first necessary resources and the second necessary resources; and determining whether a preset task condition is satisfied by considering the status information of the first necessary resources and the second necessary resources, and performing the inference task or an alternative task of the inference task depending on whether the task condition is satisfied.

[0013] In addition, another embodiment of the present invention provides a device for scheduling an inference task. The device includes at least one processor, a memory electrically connected to the processor and storing at least one code to be executed by the processor. The memory stores a code that, when executed through the processor, causes the processor to generate a memory resource list and a processing unit list for the inference task based on a configuration file, generate a work thread for the inference task based on the processing unit list, check an available first necessary resource from the memory resource list and an available second necessary resource from the processing unit list in response to a start request for the inference task, receive status information of the first necessary resource and the second necessary resource, determine whether a preset task condition is satisfied by considering the status information of the first necessary resource and the second necessary resource, and cause the inference task to be performed or an alternative task of the inference task to be performed depending on whether the task condition is satisfied.

[0014] According to the solution to the problem of the present invention described above, various AI models can be efficiently performed without a decrease in response speed and response accuracy can be maintained.

[0015] In addition, according to the solution to the problem of the present invention described above, highly reliable AI calculations can be performed by making maximum use of limited hardware resources.

[0016] In addition, according to the solution to the problem of the present invention described above, it is possible to support multiple AI models even in a heterogeneous hardware environment, thereby enabling operations on various hardware, thereby providing optimized performance.

[0017] In addition, according to the solution to the problem of the present invention described above, the stability of the system can be strengthened by predicting and managing violations of the Service Level Objective (SLO), and automatically switching to an alternative operation when a problem occurs.

[0018] In addition, according to the solution to the problem of the present invention described above, the execution of the AI ​​model can be scheduled to maintain optimal performance in accordance with real-time changing conditions.

[0019] In addition, according to the solution to the problem of the present invention described above, energy consumption can be minimized through efficient resource management.

[0020] In addition, according to the solution to the problem of the present invention described above, it is possible to evenly distribute the resources of the entire system while satisfying the requirements of each user in a multi-tenant environment.

[0021] The effects of the present invention are not limited to the effects described above, and include all effects understood from the following description.

[0022] FIG. 1 is a diagram illustrating an inference task scheduling device according to an embodiment of the present invention.

[0023] FIG. 2 is a drawing showing a detailed configuration of another example of the inference task scheduling device illustrated in FIG. 1.

[0024] FIG. 3 is a drawing illustrating in detail the initialization task and task thread creation process during the operation process of the inference task scheduling device illustrated in FIG. 1 and FIG. 2.

[0025] FIG. 4 is a drawing illustrating in detail the process of validating resource status and providing resource data during the operation process of the inference task scheduling device illustrated in FIGS. 1 and 2.

[0026] FIG. 5 is a drawing illustrating in detail the inference execution process during the operation process of the inference task scheduling device illustrated in FIGS. 1 and 2.

[0027] FIG. 6 is a flowchart illustrating the sequence of an inference task scheduling method according to another embodiment of the present invention.

[0028] FIGS. 7 to 9 are flowcharts illustrating additional steps included in the inference task scheduling method illustrated in FIG. 6.

[0029] Hereinafter, the present invention will be described in detail with reference to the attached drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. In addition, the attached drawings are only intended to facilitate understanding of the embodiments of the invention disclosed herein, and the technical ideas disclosed herein are not limited by the attached drawings. All terms, including technical and scientific terms, used herein should be interpreted as having meanings generally understood by a person of ordinary skill in the art to which the present invention pertains. Terms defined in the dictionary should be interpreted as having additional meanings consistent with the relevant technical literature and the present invention, and shall not be interpreted in an extremely ideal or restrictive sense unless otherwise defined.

[0030] In order to clearly explain the present invention in the drawings, parts irrelevant to the description have been omitted, and the size, shape, and appearance of each component shown in the drawings may be modified in various ways. Identical / similar parts throughout the specification are given identical / similar drawing reference numerals.

[0031] The suffixes "form" and "function" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. Furthermore, in describing the embodiments disclosed herein, detailed descriptions of related known technologies have been omitted if they are deemed likely to obscure the gist of the embodiments disclosed herein.

[0032] Throughout the specification, when a part is said to be "connected (connected, in contact with, or coupled)" to another part, this includes not only cases where it is "directly connected (connected, in contact with, or coupled)" but also cases where it is "indirectly connected (connected, in contact with, or coupled)" with another member in between. Furthermore, when a part is said to "include (have or provide)" a certain component, this does not mean that it excludes other components, but rather that it may "include (have or provide)" other components, unless otherwise specifically stated.

[0033] As used herein, ordinal terms such as "first," "second," etc., are used solely to distinguish one component from another and do not limit the order or relationship of the components. For example, the first component of the present invention may be referred to as the "second component," and similarly, the second component may also be referred to as the "first component." As used herein, singular expressions should be construed to include plural expressions, unless explicitly stated otherwise.

[0034] FIG. 1 is a diagram illustrating an inference scheduling device (100) of the present invention.

[0035] Referring to FIG. 1, the scheduling device (100) may include a communication module (110), a processor (120) that performs operations according to code stored in a memory (130), and a memory (130) that stores the code.

[0036] The scheduling device (100) may be implemented as a computer or a portable terminal capable of connecting to a server or other terminal via a network. Here, the computer includes, for example, a notebook, desktop, or laptop equipped with a web browser, and the portable terminal may include, for example, a wireless communication device that guarantees portability and mobility, and may include all types of handheld-based wireless communication devices such as various types of communication-based terminals, smartphones, tablet PCs, etc. In addition, the portable terminal may be an edge device or on-device AI having at least one processor capable of artificial intelligence model inference work. The network may be implemented as a wired network such as a local area network (LAN), a wide area network (WAN), or a value-added network (VAN), or all types of wireless networks such as a mobile radio communication network or a satellite communication network.

[0037] The communication module (110) may include a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through a wired or wireless connection with other network devices. The communication module transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to technical standards or communication methods for mobile communication used in the mobile communication module (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), etc.).

[0038] The processor (120) may include various types of devices that control and process data. The processor (120) may refer to a data processing device built into hardware that has a physically structured circuit to perform functions expressed by codes or commands included in a program. In one example, the processor (120) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

[0039] The memory (130) can store at least one of information and data input to the communication module (110), information and data required for functions performed by the processor (120), and data generated according to the execution of the processor (120).

[0040] Memory (130) should be interpreted as a general term for a non-volatile storage device that maintains stored information even when no power is supplied and a volatile storage device that requires power to maintain stored information. In addition to a volatile storage device that requires power to maintain stored information, memory (130) may include cloud storage, SSD, magnetic storage media, or flash storage media, but the scope of the present invention is not limited thereto.

[0041] The memory (130) is electrically connected to the processor (120) and stores at least one code that is executed by the processor (120). The memory (130) stores a code that causes the processor (120) to perform the following functions and procedures when executed by the processor (120).

[0042] The memory (130) stores code that causes the processor (120) to create a memory resource list and a processing unit list for the inference task based on a configuration file, and to create a task thread for the inference task based on the processing unit list. The memory (130) stores code that causes the processor (120) to check available first necessary resources from the memory resource list and available second necessary resources from the processing unit list in response to a request to start the inference task, and to receive status information of the first necessary resource and the second necessary resource. The memory (130) stores code that causes the processor (120) to determine whether preset task conditions are met by considering the status information of the first necessary resource and the second necessary resource, and to perform the inference task or an alternative task of the inference task depending on whether the task conditions are met.

[0043] The memory (130) may further store code that causes the processor (120) to generate a memory resource list and a processing unit list for the inference task based on a configuration file, and to perform an initial task for the inference task based on the configuration file before generating a task thread for the inference task based on the processing unit list. Here, the initial task is to set an initial value for each of different types of processing units available for the inference task. The processing unit list may include a plurality of heterogeneous processing units, and the task thread may be generated for each of the plurality of heterogeneous processing units.

[0044] The above task conditions may include first and second conditions. The first condition includes constraints related to the operation of the entire system, and each constraint may include a risk level, a constraint item, and a threshold value. The second condition includes constraints required for each task model to be performed, and may use conditions including maximum delay time, memory usage, the number of recent consecutive executions, and the number of recent consecutive results missing.

[0045] The memory (130) may store a code that causes the processor (120) to compare the importance of the preset inference task with the expected risk when performing the inference task and determine whether the first condition is satisfied. The memory (130) may further store a code that causes the processor (120) to compare the status information of the first required resource and the second required resource with the target information set for the inference task and determine whether the second condition is satisfied. The risk may be set according to at least one of the temperature and current consumption of a specific area expected when performing the inference task, and the set target information may include at least one of the amount of memory resource usage and the time required to complete performing the inference task.

[0046] The memory (130) may further store code that causes the processor (120) to determine that the inference task does not satisfy the first condition and to search for a preset first alternative task for the inference task if the importance is lower than the risk level. The memory (130) may further store code that causes the processor (120) to determine whether the searched first alternative task satisfies the first condition and, based on the determination result, to replace the searched first alternative task with the inference task or to perform an error report.

[0047] The memory (130) may further store code that causes the processor (120) to perform the reasoning task if the reasoning task satisfies the second condition, and to search for a preset second alternative task for the reasoning task if the reasoning task does not satisfy the second condition. The memory (130) may further store code that causes the processor (120) to determine whether the searched second alternative task satisfies the first condition and the second condition, and to replace the searched second alternative task with the reasoning task or to perform an error report based on the determination result.

[0048] FIG. 2 is a diagram illustrating another example configuration of a scheduling device according to one embodiment of the present invention. The scheduling device (200) described below has substantially the same configuration as the scheduling device (100) described above with reference to FIG. 1. In other words, the operations and functions of the service providing unit (210), the inference performing unit (220), and the inference allocation unit (230) described below can be performed by the memory and processor illustrated in FIG. 1. Therefore, any redundant description will be omitted.

[0049] Referring to FIG. 2, the scheduling device (200) may include a service providing unit (210), an inference performing unit (220), and an inference allocation unit (230).

[0050] The service provider (210) reads a configuration file to initialize the entire program and the inference execution unit (220) and inference allocation unit (230), and can accept a user's request and return the inference result to the user. Here, the configuration file includes global information, resource information, model information, and scheduling information.

[0051] Global information in the configuration file can include the full list of models, available hardware resources (e.g., NPU0, CPU0, CPU1, CPU, DSP0, etc.), and global SLO constraints (e.g., CPU usage, memory usage, and power consumption).

[0052] The resource information in the configuration file is a list of input and output data used by each model, and can include the resource main ID, resource name, resource data type, and resource data tensor shape. Here, each model can include an importance and alternative model to predict whether or not an SLO violation will occur. If the importance and alternative model are not set, the importance is set to 1, and the alternative model is set to error reporting. The importance of error reporting is set to the maximum the system can have, and the resource consumption value is set to 'none', so that the model can be assumed to never violate the SLO.

[0053] The model information in the configuration file may include the model name, model backend type (implementation name on which the model will be executed), model expected execution time, model binary file path, model configuration parameters, SLO information for each model (e.g., delay limit time and number of executions per second, etc.), alternative model name in case of SLO failure, input resource main ID list, input resource sub ID generation rule, output resource ID list, and output resource sub ID generation rule.

[0054] Scheduling information in the configuration file can include scheduling algorithm names (e.g., static, PREMA, MAEL, DAIMOS, etc.), a list of default scheduling execution models and execution conditions (e.g., model0: 30fps, model1: 30fps), and algorithm parameters.

[0055] The service provider (210) may include a service interface (211), a schedule maker (212), and a resource manager (213).

[0056] The service interface (211) can communicate with the user terminal (400) and other software modules. Here, data provided externally via the communication connection can be registered for use in internal AI inference. Furthermore, the inference result can be requested internally. If the result is not ready, a wait time of a timeout value can be applied. Furthermore, a request can be made to perform inference manually without waiting for the inference result, which is automatically performed by the scheduler.

[0057] When the service interface (211) and other software modules are connected for communication, they can all use a single Message class. Message classes include a Message ID, which is a variable indicating the type of message, a Sub ID, which is a unique value that distinguishes messages with the same ID, a type, which is a message type defined by the user, and a datum, which includes an array of actual data, where the datum has different contents depending on the Message ID.

[0058] The resource manager (213) can perform new allocations for requested resources, register data provided from the outside, return resources for unused data, determine availability and status of requested data resources, and transfer objects. The resource manager (213) can have the authority to manage all data provided from the outside and all data generated internally, and can only transmit the address of data when a request is received from another operator or a user within the system. The resource manager (213) can externally designate a constructor and a destructor so that it can manage all data provided from the outside, statically allocated, or dynamically allocated. Here, the policy for managing each data is determined based on information in the configuration file.

[0059] The inference execution unit (220) may include a model interface (221) and an operation performer (222).

[0060] The model interface (221) operates on AI operators or operators with high operators, and can perform operations using hardware accelerators or CPUs. Furthermore, a hardware-optimized implementation is implemented separately as a library, and each interface class can load and use the implementation based on settings read from a configuration file, or call and use another implementation implemented directly within the interface.

[0061] The operation performer (222) performs operations according to multiple threads. Each thread may have an input / output queue, and may use a priority queue or a general queue data structure. Each thread may perform operations on a single piece of hardware. For example, if multiple inferences can be performed in parallel through an internal queue on a single piece of hardware, one operation performer is set as NPU0-Primary, another operation performer is set as NPU0-Secondary, and if the scheduler determines that inference work is possible, the scheduler uses the operation performer of NPU0-Secondary. However, depending on the characteristics of the hardware, one piece of hardware may include two or more threads, but the scope of the present invention is not limited thereto.

[0062] Each thread of the operator (222) reads task information from the input queue, receives a model instance and a resource instance, performs inference, and then returns a result message to the output queue. Here, the task information may include a task ID, a usage model ID, a related data resource ID, time information (e.g., execution start time, etc.), execution device ID information, related parameters required for execution, and profiling information generated during the execution process. If there is no task information in the input queue of each thread, each thread may be recognized as in a waiting state and recheck the input queue after a certain period of time. If the task information input to the input queue grows to a certain amount or more, inference may not be performed, but one piece of task information input to the input queue may be deleted and an error message may be returned to the output queue.

[0063] The operation performer (222) represents each piece of hardware and is in charge of operations on the corresponding hardware. Depending on the characteristics of the scheduler, one piece of hardware can be used by two or more operation performers (222). For example, if multiple inferences can be performed in parallel through an internal queue on one piece of hardware, one operation performer is set as NPU0-Primary, the other as NPU0-Secondary, and if the scheduler (231) determines that it is possible, the NPU0-secondary inferrer is used.

[0064] The operator (222) reads task information from its assigned queue (or priority queue), receives model instances and resource instances, performs inference, and returns the resulting data instances to the output queue. If the input queue is empty, it is recognized as a waiting state and rechecks the input queue after a certain amount of time. If the input queue grows beyond a certain size, inference is not performed, one piece of information from the input queue is discarded, and an error message is sent to the output queue.

[0065] The inference allocation unit (230) may include a scheduler (231), an SLO manager (232), and a schedule executor (233).

[0066] The scheduler (231) can dynamically or statically determine the operators to be performed on individual hardware at each point in time based on the data in the configuration file. If the task is determined statically, the initial configuration data can be read and used based on the list of structures in which task information is stored. If the task is determined dynamically, the use algorithm and parameters can be read to dynamically generate a task information list. Even after the task information list is generated, if the task information list changes according to an alternative task request from the SLO manager (232), the scheduler (231) can handle part or all of the scheduling algorithm in the hardware depending on the hardware. Here, an abstracted interface to be handled by the SoC-dependent scheduler (not shown) can be created, and the configuration values ​​can be passed to allow each implementation to manage the scheduling algorithm.

[0067] The SLO manager (232) can determine whether each task information is suitable for pre-set constraints by predicting the inference time, power consumption, memory usage, memory bandwidth consumption, and task success rate of the current hardware. The SLO manager (232) can suggest an alternative task to replace it when an error occurs. Here, the SLO determination is performed by the SLO manager (232) based on the task's execution history, hardware temperature, usage, memory usage, and data status given as input. If the execution result of another task affects the SLO, the result of the other task can be used as the input of the current task.

[0068] The SLO manager (232) and scheduler (231) can operate on the same thread in the form of a function call. If the SLO judgment result determines that the task execution is not suitable, the SLO manager (232) can cause an alternative task selector (not shown) to search for an alternative task.

[0069] The alternative service selector explores alternative services through the following steps:

[0070] If an error is expected but a task must be performed, the task is performed, and then the degree of violation of the SLO constraint is re-evaluated in the subsequent tasks, and alternative tasks are explored. If an alternative task is set due to an SLO non-satisfaction (violation), the alternative task is performed and a record of the alternative task performance is kept. If the alternative task can no longer be performed based on the history of alternative task performance, the search for a next alternative task is performed. Here, if it is determined that the task is still impossible to perform, the search is stopped, the degree of violation of the SLO constraint is re-evaluated, and the operation of exploring alternative tasks is performed again.

[0071] Additionally, the selection of alternative tasks due to SLO non-satisfaction (violation) follows the following algorithm.

[0072] "def SLO_Alternative_Selection(ModelID):

[0073] prev_alternatives = set()

[0074] cur_model_id = ModelID

[0075] while True:

[0076] if cur_model_id in prev_alternatives:

[0077] return cur_model_id

[0078] prev_alternatives.insert(cur_model_id)

[0079] current_warn_lev = 0

[0080] for gc in global_constraints: # 전역 조건 수행 검사

[0081] expected = get_resource(ModelID, gc.type)

[0082] current = get_current_resource(gc.type)

[0083] if gc.threshold < expected + current:

[0084] current_warn_lev = max(current_warn_lev, gc.warn_lev)

[0085] if cur_model_id.priority < current_warn_lev

[0086] cur_model_id = cur_model_id.alternative

[0087] continue

[0088] if model_SLO_check(cur_model_id) == False:

[0089] cur_model_id = cur_model_id.alternative

[0090] continue

[0091] return cur_model_id"

[0092] Let's explain the above algorithm in detail. If the current task ID is in the existing alternative task review history, it is determined to be forcibly executed and the current task ID is returned. If it is not in the review history, the current task ID is added to the alternative task review history list. Then, the current numerical values ​​for the global constraints are retrieved. Global conditions can be hardware temperature, voltage, current consumption, CPU usage, memory usage, and other data obtained from the device system.

[0093] This function predicts the consumption incurred while performing a task based on the current task ID. The consumption prediction is calculated based on the results of the most recent N executions. The calculation can be performed using a constant value for computational efficiency, statistical methods, or predictive methods based on machine learning (ML) algorithms. Different margins can be used during prediction depending on the risk level of the constraints. The table below provides examples.

[0094] Case 1 Case 2 Risk 1, 2 Recent average value Consumption prediction Risk 3, 4 Maximum value within 66% confidence interval Consumption prediction * 1.1 Risk 5 or higher Maximum value within 90% confidence interval Consumption prediction * 1.2

[0095] The predicted consumption value calculated in the process of obtaining the current numerical value for the global constraint and the usage obtained in the process of returning the current task ID are added and compared with the threshold value of the global condition.

[0096]

[0097] In the above formula, cond.level is the preset risk of the global condition, u(x) is the unit step function, c(cond) is the numerical value for the constraint in the current system, and e(cond) is the predicted consumption value of the constraint when performing the current task.

[0098] The current system's risk is set to the highest risk among the global conditions set to be greater than the threshold.

[0099]

[0100] For example, in a system assuming the above constraints are used, if the current CPU temperature is 84.5°C and the temperature increase estimated with a confidence interval of 66% when performing a given task is 0.4 to 1.0°C, the predicted value after performing the task is calculated to be 85.5°C, and the risk level is set to 3. In the same situation, if the current consumption is 10mA and the predicted value is 0.1mA, the risk level is increased to 5.

[0101] For example, in a system assuming the above constraints are used, if the current CPU temperature is 84.5°C and the temperature increase estimated with a confidence interval of 66% when performing a given task is 0.4 to 1.0°C, the predicted value after performing the task is calculated to be 85.5°C, and the risk level is set to 3. In the same situation, if the current consumption is 10mA and the predicted value is 0.1mA, the risk level is increased to 5.

[0102] If the importance of the current model is lower than the risk of the current system based on the global conditions calculated in the previous step, the current model is considered to have violated the constraint. The alternative task ID is selected as the current task ID, and the process of adding the current task ID is repeated. Next, whether the current model violates the constraints of each task is examined. The risk of each task is evaluated using the formula in Step 4, which is set to a random constant depending on the system. If a violation occurs, the alternative task ID is selected as the current task ID, and the process is repeated from step 1. If the model is determined to be usable without any violations, the current task ID is returned.

[0103] The schedule executor (233) can read the next task determined by the scheduler (231) and, based on the value of the task information, transfer the task information to the input queue of the appropriate operation executor (222) in the inference execution unit (220). In addition, it can receive data from the result queue of each operation executor (222) to update internal data so that it can be used by the scheduler (231). Here, profiling information (e.g., execution time, end time, hardware resource usage, etc.) can be recorded and managed so that it can be used by the SLO manager (232). The schedule executor (233) can communicate with the operation executor (222) asynchronously using a message queue in one thread. The schedule executor (233) can retrieve the input task from the scheduler (231) and transfer it to the corresponding device operation executor (222), and can receive data from the result queue of each operation executor (222) to update internal data so that it can be used by the scheduler (231). The scheduler (233) can asynchronously use a message queue to communicate with the task operator in a single thread. Here, the scheduler (231), model loader (215), and SLO manager (232) can operate in the same thread in the form of function calls. The model loader (215) can be included in the service execution unit (210).

[0104] Figures 3 to 5 are drawings illustrating in detail the operation of the scheduling device (200) illustrated in Figure 2. The operation of the scheduling device (200) described below can also be performed in the same manner by the processor (120) and memory (130) of the scheduling device (100).

[0105] FIG. 3 is a drawing illustrating in detail the initialization operation and thread creation process during the operation of the scheduling device (200) illustrated in FIG. 2.

[0106] The service interface (211) receives an initialization signal from the outside (S101). Here, the initialization signal may be in the form of a library function call. Thereafter, the service interface (211) reads a configuration file from a memory or other configuration analyzer (214) (S102). The configuration analyzer (214) returns the data of the interpreted configuration file to the service interface (211) (S103). The configuration analyzer (214) may be included in the service execution unit (210).

[0107] The service interface (211) transmits the configuration file data to the model loader (215) included in the inference execution unit so that each model can be initialized (S104). The model loader (215) interprets each model information in the configuration file data to initialize each model (S105), and initializes the implementation (backend) using the model interface (221) class as an interface according to each backend type (S106).

[0108] The model interface (221) initializes the backend using the binary file path and setting parameter values ​​given to the model setting values ​​of each model information in the setting file data (S107). The model interface (221) returns the initialized model instance to the model loader (215) (S108), and the model loader (215) returns the initialized model instance to the service interface (211) (S109). Thereafter, the schedule maker (212) is initialized (S110) (the initialization steps of the schedule maker (212) are the same as steps S105 to S108).

[0109] The schedule maker (212) interprets the scheduling-related parameters obtained from the configuration file (S111) and initializes the implementation of the scheduler (231) (S112). Here, the scheduler (231) determines whether the scheduling conditions interpreted during initialization are valid (S113). Based on the results obtained in S111 or S113, the schedule maker (212) returns an error code to the service interface (211) (S114).

[0110] The resource manager (213) creates a list of available resources based on the memory resource list of the configuration file data (S115), and returns an error code to the service interface (211) after resource initialization (S116).

[0111] The operator (222) creates an operation thread according to the list of available hardware resources in the above-mentioned configuration file data (S117). Thereafter, the service interface (211) creates a main service task thread (S118).

[0112] FIG. 4 is a drawing illustrating in detail the process of checking the validity of resource status and providing resource data during the operation of the scheduling device (200) illustrated in FIG. 2.

[0113] Referring to Fig. 4, it includes internal processing steps (S211 to S214) according to external data reception after data reception and registration, and return steps (S221 to S226) according to data request from a user terminal (400). In addition, only one of the steps may be operated, or both steps may be operated simultaneously.

[0114] Referring to FIG. 4, the service interface (211) receives resource data resource value from the outside (S211) and calls the resource registration function of the resource manager (213) (S212). The resource manager (213) checks the validity of the resource ID and resource data, and if it is determined to be valid, it stores it internally (S213). The resource manager (213) returns the result of the validity determination of the resource ID and the resource data to the service interface (211). Here, if the resource data is determined to be valid, STATUS_OK is returned, and if there is an error, an error status value is returned accordingly (S214).

[0115] The service interface (211) receives a request for resource data from the user terminal (400) (S221) and calls the resource data request function of the resource manager (213) (S222). The resource manager (213) checks the validity of the resource ID, Sub ID, and resource data, and if it is determined to be valid, generates return data (S223). If the return data is not ready, the resource manager (213) waits for a preset timeout period, and if the return data is valid, regenerates the return data (S224). The resource manager (213) returns the return data to the service interface (211). Here, if the return data is invalid, pre-stored error data is returned to the service interface (211) (S225). The service interface (211) transmits the data received from the resource manager (213) to the user terminal (400).

[0116] FIG. 5 is a drawing illustrating in detail the inference execution process during the operation of the scheduling device (200) illustrated in FIG. 2.

[0117] Referring to Fig. 5, it may include an inference step (S301 to S303) by an explicit request of a user terminal (400) and an automatic inference step (S311 to S327) by a scheduler (231). In addition, only one of the steps may operate, or both steps may operate simultaneously.

[0118] The service interface (211) receives an explicit inference request from the user terminal (400) (S301) and transmits an inference request message to the schedule executor (233) (S302). The schedule executor (233) requests inference from the scheduler (231) (S303). The schedule executor (233) transmits a timestamp value of the current time to the scheduler (231) to request an inference task to be performed (S311). The scheduler (231) requests the SLO manager (232) to review whether the inference task can be performed without violating the SLO (S312). The SLO manager (232) receives confirmation of whether resources can be used from the resource manager (213) (S313) and receives current status information of required resources from the resource manager (213) (S314).

[0119] If the above-mentioned required resources are available, the SLO manager (232) verifies whether the task can be performed without violating the SLO (S315). If the SLO manager (232) determines that the task can be performed without violating the SLO, it selects the task. If it determines that the task cannot be performed, it selects an alternative task from an alternative service selector (not shown). If an alternative task exists, steps S312 to S315 are repeated to determine whether the task can be performed (S316).

[0120] The SLO manager (232) returns the final selection task information to the scheduler (231) (S317), and the scheduler (231) returns the final selection task information to the schedule executor (233) (S318). If the final selection task information is valid, the schedule executor (233) requests the operation executor (222) to perform the task. In addition, it transmits the task information to the thread of the operation executor (222) according to the execution device ID information of the task information. Here, if the task is impossible, the process returns to step S312 after a certain period of time (S319).

[0121] The operator (222) interprets the received task information and initializes profiling information. Here, the operator identifies an instance of the model interface (221) based on the execution device ID specified in the task information and sets related parameters (S320). Thereafter, the operator (222) requests the resource manager (213) for ID information used as input in the task information (S321).

[0122] The resource manager (31) returns resource information to the operator (222) (S322), and the operator (222) calls an inference execution function to an instance of the identified model interface (221) (S323).

[0123] The model interface (221) registers the inference result data with the resource manager (213) (S324), and the resource manager (213) verifies the registration result of the model interface (221) (S325). The model interface (221) returns an error code containing whether an error occurred during the inference process or resource registration process to the operator (222) (S326). The operator (222) transmits a task termination code together with the error code to the resource manager (213) (S327).

[0124] An example of the operation process algorithm of the scheduling device (200) described with reference to FIGS. 2 to 5 will be described. The program example below corresponds to the preprocessing and postprocessing algorithms of the yolo model for object detection and recognition services. Here, the configuration file data includes the following:

[0125] "[Global]

[0126] num_model = 3

[0127] model = yolo_preprocess, yolo_model,yolo_model_alt, yolo_postprocess

[0128] num_hw = 2

[0129] hardwares = CPU0, NPU0

[0130] SLO_CPU_USAGE = 0.7

[0131] [Resources]

[0132] resource_ids = 101, 201, 202, 301

[0133] resource_101 = input_image, uint8_t, 3, 1280,720,3

[0134] resource_201 = preprocessed_image, uint8_t, 4, 1,3,640,640

[0135] resource_202 = yolo_inferenced, float32_t, 3, 1,25200, 85

[0136] resource_301 = object_lists, 100, 6

[0137] [yolo_preprocess]

[0138] backend = yolo_preprocessor # implementation class name that inherits model_instance

[0139] estimated_time = 3.5 # milliseconds unit.

[0140] input_ids = 101

[0141] input_subid_type = incremental # timestamp, call_count, framecount 등

[0142] output_ids = 201

[0143] output_subid_type = same # timestamp, call_count, framecount, same 등

[0144] [yolo_model]

[0145] backend = tflite_coral

[0146] estimated_time = 10.4 # milliseconds unit.

[0147] model_binary = / var / data / yolo.tflite

[0148] SLO_types = time, fps

[0149] SLO_limits = 15, 30

[0150] SLO_alternative = yolo_model_alt

[0151] input_ids = 201

[0152] input_subid_type = framecount# timestamp, call_count, framecount 등

[0153] output_ids = 202

[0154] output_subid_type = same # timestamp, call_count, framecount, same 등

[0155] [yolo_model_alt]

[0156] backend = tflite_cpu

[0157] estimated_time = 5.4 # milliseconds unit.

[0158] model_binary = / var / data / yolo_cpu.tflite

[0159] SLO_types = time, fps

[0160] SLO_limits = 15, 30

[0161] SLO_alternative = Error_report

[0162] input_ids = 201

[0163] input_subid_type = framecount # timestamp, call_count, framecount 등

[0164] output_ids = 202

[0165] output_subid_type = same # timestamp, call_count, framecount, same 등

[0166] [yolo_postprocess]

[0167] backend = yolo_postprocessor # model_instance를 상속받은 구현체 클래스 명

[0168] estimated_time = 5.5 # milliseconds unit.

[0169] model_parameter = 0.01, 0.35, 0.2

[0170] input_ids = 202

[0171] input_subid_type = framecount # timestamp, call_count, framecount 등

[0172] output_ids = 301

[0173] output_subid_type = same # timestamp, call_count, framecount, same, etc.

[0174] [scheduling]

[0175] algorithm = naive

[0176] yolo_preprocess = fps, 30

[0177] yolo_model = fps, 30, yolo_preprocess

[0178] yolo_postprocess = yolo_model"

[0179] This embodiment includes an initialization process and an inference execution process. The program is initialized based on the above configuration file data. One Inferrer class is created for CPU0 and one for NPU0. Three model interface implementations, yolo_preprocess, yolo_model, yolo_model_alt, and yolo_postprocess, are initialized. Each uses CPU, NPU, CPU, and CPU, and this content is defined in each implementation. In addition, it is assumed that the Scheduler uses a naive scheduler. It is assumed that this algorithm executes sequentially in a round-robin manner. The input image is transmitted from the outside every 33ms. The subid of the input resource uses the frame count. The result value is then requested just before the input image is transmitted from the outside.

[0180] The inference execution process executes an inference task request on a thread of the scheduler. Based on the settings in the above configuration file, it returns yolo_preprocess, the first algorithm without dependencies. Since yolo_preprocess has no SLO constraints, the SLO manager only checks the validity of the input resource, and if the resource is valid, it returns the task to be executed. Here, the input resource is valid if it has a subid that matches the framecount rule (subid = (subid_prev + 1) % count_max) among the ID values ​​of 101.

[0181] If the input resource is invalid, "No work available" is returned, and the process repeats from the step where the next work request is executed by the scheduler's thread after a certain amount of time. The scheduler passes the work information to the CPU task operator so that the task is executed by the operator.

[0182] Since yolo_preprocess has been executed, the next task, yolo_model, is returned. Note that depending on the scheduler algorithm implementation, other results may be produced, such as no tasks being returned until preprocess is finished.

[0183] Return to the step where yolo_model is returned and repeat until the result of yolo_preprocess, which is the input resource of yolo_model, is available. SLO prediction is performed when yolo_model is executable. If executable, task information is passed to the task operator of NPU0. Here, if there is a possibility of violation of either execution time or FPS, yolo_model_alt checks whether it can be executed on CPU. If it is not executable, Error_report is performed. Yolo_postprocess is performed in the same way as the previous process.

[0184] FIG. 6 is a flowchart illustrating a sequence of a method for scheduling an inference task according to another embodiment of the present invention, and FIGS. 7 to 9 are flowcharts illustrating additional steps included in the method for scheduling an inference task.

[0185] The contents of FIGS. 6 to 9, which are the inference task scheduling methods (hereinafter referred to as “scheduling methods”) to be described below, can be performed by the scheduling device (100) or the scheduling device (200) described above with reference to FIGS. 1 to 5. Therefore, the contents of the embodiments of the present invention described above with reference to FIGS. 1 to 5 can be equally applied to the embodiments to be described below, and any overlapping contents with the above description will be omitted below. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps can be performed almost simultaneously.

[0186] Referring to FIG. 6, the scheduling method includes a step of creating a processing unit list and creating a work thread (S1000), a step of checking required resources and receiving required resource status information (S2000), and a step of determining whether a work condition is met and performing an inference work or alternative work (S3000).

[0187] The step of generating a processing unit list and creating a worker thread (S1000) is a step of generating a memory resource list and a processing unit list for an inference task based on a configuration file, and generating a worker thread for the inference task based on the processing unit list. Here, the processing unit list may include a plurality of heterogeneous processing units, and the worker thread may be generated for each of the plurality of heterogeneous processing units.

[0188] The step of checking required resources and receiving required resource status information (S2000) is a step of checking the first required resource available among the memory resource list and the second required resource available among the processing unit list in response to a request to start the inference task, and receiving status information of the first required resource and the second required resource.

[0189] The step of determining whether a task condition is met and performing an inference task or an alternative task (S3000) is a step of determining whether a preset task condition is met by considering the status information of the first required resource and the second required resource, and performing the inference task or an alternative task of the inference task depending on whether the task condition is met. Here, the task condition may include a first condition and a second condition.

[0190] The step of generating a list of processing units and creating a working thread (S1000) may further include a step of performing an initial task for the inference task based on the configuration file. Here, the initial task may be to set an initial value for each of the different types of processing units available for the inference task.

[0191] Referring to FIG. 7, the step of determining whether a task condition is met and performing an inference task or alternative task (S3000) includes a step of determining whether a first condition is met (S3100) and a step of determining whether a second condition is met (S3200).

[0192] The first condition satisfaction determination step (S3100) is a step for comparing the importance of the preset inference task with the expected risk level when performing the inference task to determine whether the first condition is satisfied. Here, the risk level may be determined based on at least one of the temperature and current consumption of a specific area expected when performing the inference task.

[0193] The second condition satisfaction determination step (S3200) is a step for comparing the status information of the first required resource and the second required resource with the target information set for the inference task to determine whether the second condition is satisfied. Here, the set target information may include at least one of the following: memory resource usage amount and the time required to complete the inference task.

[0194] Referring to FIG. 8, the step of determining whether the first condition is met (S3100) includes a step of searching for a first alternative task (S3110) and a step of replacing the first alternative task with an inference task or reporting an error (S3120).

[0195] The first alternative task exploration step (S3110) is a step in which, if the importance is lower than the risk level, the inference task is determined to not satisfy the first condition, and a preset first alternative task is explored for the inference task. The first alternative task is replaced with an inference task or an error is reported step (S3120) is a step in which, depending on the judgment result, the explored first alternative task is replaced with an inference task or an error is reported.

[0196] Referring to FIG. 9, the second condition satisfaction determination step (S3200) includes a second alternative task search step (S3210) and a second alternative task replacement with an inference task or error reporting step (S3220).

[0197] The second alternative task search step (S3210) is a step for performing the reasoning task if the reasoning task satisfies the second condition, and for searching for a preset second alternative task for the reasoning task if the reasoning task does not satisfy the second condition. The second alternative task replacement with the reasoning task or error reporting step (S3220) is a step for determining whether the searched second alternative task satisfies the first and second conditions, and replacing the searched second alternative task with the reasoning task or performing an error report based on the determination result.

[0198] The scheduling method of the embodiments of the present invention described so far can also be implemented in the form of a recording medium including computer-executable instructions, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media can include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0199] Those skilled in the art will appreciate that the present invention can be easily modified into other specific forms without changing the technical spirit or essential characteristics of the present invention based on the above description. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention. The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present application.

[0200] The form for carrying out the invention is the same as the best form for carrying out the invention described above.

[0201] The present invention is a technology for generating a list of memory resources and a list of processing units suitable for an inference task, receiving status information of necessary resources among these, and performing an inference task after determining whether a preset target value is satisfied, and thus has industrial applicability since it can be used in the AI ​​industry.

[0202] [Explanation of symbols]

[0203] 100: Inference task scheduling device

[0204] 110: Communication module

[0205] *120: Processor

[0206] 130: Memory

[0207] 200: Inference task scheduling device

[0208] 210: Service Provider Department

[0209] 211: Service Interface

[0210] 212: Schedule Maker

[0211] 213: Resource Manager

[0212] 214: Composition Analyzer

[0213] 215: Model Loader

[0214] 220: Inference Performance Unit

[0215] 221: Model Interface

[0216] 222: Operator

[0217] 230: Inference Allocation Unit

[0218] 231: Scheduler

[0219] 232: SLO Manager

[0220] 233: Schedule Executor

Claims

1. A method for scheduling an inference task performed by at least one processor, a) A step of generating a memory resource list and a processing unit list for the inference task based on a configuration file, and generating a work thread for the inference task based on the processing unit list; b) a step of checking the first required resource available among the memory resource list and the second required resource available among the processing unit list in response to a request to start the inference task, and receiving status information of the first required resource and the second required resource; c) A method for scheduling an inference task, comprising the step of determining whether a preset task condition is satisfied by considering the status information of the first required resource and the second required resource, and performing the inference task or performing an alternative task of the inference task depending on whether the task condition is satisfied.

2. In paragraph 1, Prior to step a) above, Further comprising a step of performing an initial task for the inference task based on the above setting file, The above initial work is, A method for scheduling an inference task, the method comprising setting an initial value for each of different types of processing units available for the above inference task.

3. In paragraph 1, The above processing unit list is: Contains multiple heterogeneous processing units, The above working thread is, A method for scheduling inference tasks, wherein the inference tasks are generated for each of the above-mentioned multiple heterogeneous processing units.

4. In paragraph 1, The above working conditions are, Including the first condition and the second condition, Step c) above, c-1) A step of comparing the importance of the preset inference task with the expected risk when performing the inference task to determine whether the first condition is satisfied; and c-2) A step of comparing the status information of the first required resource and the second required resource with the target information set for the inference task to determine whether the second condition is satisfied, A method for scheduling an inference task, wherein the above risk level is set based on at least one of the temperature and current consumption of a specific area expected when performing the above inference task.

5. In paragraph 4, The above set target information is: A method for scheduling an inference task, the method including at least one of memory resource usage and time required to complete an inference task.

6. In paragraph 4, Step c-1) above, If the importance is lower than the risk, the step of determining that the inference task does not satisfy the first condition and searching for a preset first alternative task for the inference task; and A method for scheduling an inference task, comprising the steps of determining whether the first alternative task searched satisfies the first condition, and replacing the first alternative task searched with the inference task or performing an error report based on the determination result.

7. In paragraph 4, Step c-2) above, A step of performing the reasoning task if the above reasoning task satisfies the second condition, and searching for a second alternative task preset for the reasoning task if the above reasoning task does not satisfy the second condition; A method for scheduling an inference task, comprising the steps of determining whether the searched second alternative task satisfies the first condition and the second condition, and replacing the searched second alternative task with the inference task or performing an error report based on the determination result.

8. At least one processor; A memory electrically connected to the processor and storing at least one code to be executed by the processor, The above memory, when executed through the above processor, An inference task scheduling device, wherein the processor generates a memory resource list and a processing unit list for the inference task based on a configuration file, generates a task thread for the inference task based on the processing unit list, checks available first necessary resources from the memory resource list and available second necessary resources from the processing unit list in response to a start request for the inference task, receives status information of the first necessary resources and the second necessary resources, determines whether a preset task condition is satisfied by considering the status information of the first necessary resources and the second necessary resources, and stores code that causes the inference task to be performed or an alternative task of the inference task to be performed depending on whether the task condition is satisfied.

9. In paragraph 8, The above memory is, Further storing code that causes the processor to generate a memory resource list and a processing unit list for the inference task based on the configuration file, and to perform an initial task for the inference task based on the configuration file before generating a task thread for the inference task based on the processing unit list, The above initial work is, An inference task scheduling device that sets an initial value for each of different types of processing units available for the above inference task.

10. In paragraph 8, The above processing unit list is: Contains multiple heterogeneous processing units, The above working thread is, It is generated by each of the above multiple heterogeneous processing units, Inference task scheduling device.

11. In paragraph 8, The above working conditions are, Including the first condition and the second condition, The above memory is, Code causing the processor to compare the importance of the preset inference task with the expected risk when performing the inference task to determine whether the first condition is satisfied; and Stores code that causes the processor to compare the status information of the first required resource and the second required resource with the target information set for the inference task to determine whether the second condition is satisfied; An inference task scheduling device, wherein the above risk level is set based on at least one of the temperature and current consumption of a specific area expected when performing the above inference task.

12. In paragraph 11, The above set target information is: An inference task scheduling device, comprising at least one of memory resource usage and time required to complete an inference task.

13. In paragraph 11, The above memory is, If the importance is lower than the risk, the code causes the processor to determine that the reasoning task does not satisfy the first condition and to search for a first alternative task set for the reasoning task; and A speculative task scheduling device, wherein the processor further stores code that determines whether the searched first alternative task satisfies the first condition and, based on the determination result, causes the searched first alternative task to be replaced with the speculative task or to perform an error report.

14. In paragraph 11, The above memory is, Code that causes the processor to perform the reasoning task if the reasoning task satisfies the second condition, and to search for a preset second alternative task for the reasoning task if the reasoning task does not satisfy the second condition; and A speculation task scheduling device, wherein the processor further stores code that causes the processor to determine whether the searched second alternative task satisfies the first condition and the second condition, and to replace the searched second alternative task with the speculation task or perform an error report based on the determination result.

Citation Information

Patent Citations

  • Decoupling the number of logical threads from thenumber of simultaneous physical threads in a processor

    KR100856144B1

  • Computer micro-jobs

    KR101377195B1

  • Apparatus for processing multi-threading framework supporting dynamic load-balancing and multi-thread processing method using by it

    KR1020090061177A

  • Semantic computing-based dynamic job scheduling system for distributed processing

    KR1020100118357A

  • Hardware-Aware Progressive Training Of Machine Learning Models

    US20230108177A1

Cited By

  • Agent resource scheduling and isolating system oriented to multiple tenants and multiple platforms

    CN121704986A