Method of operating a computer device and computer-readable storage medium storing instructions

Abstract

An operating method of a computing device including a central processing unit configured to execute a machine learning model and a plurality of sub-processors, and a computer readable medium storing instructions. For example, the operating method can include dynamically assigning one of a blocking mode or a non-blocking mode to each of a plurality of tasks of the machine learning model to establish a policy, and executing the machine learning model and its tasks based on the policy.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Korean Patent Application No. 10-2020-017718, filed on December 3, 2020, with the Korean Intellectual Property Office, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to electronic devices and computing devices, and more specifically to methods of operating a computing device configured to execute a machine learning model, and to a computer-readable storage medium for storing instructions. Background Technology

[0004] Machine learning algorithms can include a variety of algorithms, such as neural networks, decision trees, and support vector machines (SVMs). Machine learning models can be based on the results of machine learning learning and can include algorithms that draw or achieve results by performing tasks (e.g., reasoning, classifying, or recognizing inputs). For example, the input can be image data or audio samples. Examples of the tasks performed include recognizing objects in an image, classifying an object, or recognizing speech.

[0005] It is possible to enable machine learning models to perform tasks in real time. To achieve this, it may be necessary to improve the execution speed of the machine learning model. Summary of the Invention

[0006] Embodiments of this disclosure provide a method of operating a computing device that executes machine learning models at an increased speed, and provide a computer-readable storage medium for storing instructions.

[0007] According to some embodiments, a method of operating a computing device is provided. The computing device may include a central processing unit and a plurality of subprocessors configured to execute a machine learning model. The method of operation may include dynamically assigning one of a blocking mode or a non-blocking mode to each of a plurality of tasks of the machine learning model to establish a policy, and executing the machine learning model and its tasks based on the policy.

[0008] According to some embodiments, a method of operating a computing device is provided. The computing device may include a central processing unit and a plurality of subprocessors configured to execute a machine learning model. The method of operating may include dynamically assigning each of a plurality of tasks of the machine learning model to one of the central processing unit or one of the plurality of subprocessors, and dynamically assigning one of a blocking mode or a non-blocking mode to each of the plurality of tasks of the machine learning model to establish a policy, and executing the machine learning model and its tasks based on the policy.

[0009] According to some embodiments, a non-transitory computer-readable storage medium is provided that stores instructions which, when executed by a central processing unit and a plurality of subprocessors, cause the following operations: dynamically assigning each of a plurality of tasks of a machine learning model to one of the central processing unit or one of the plurality of subprocessors, dynamically assigning one of a blocking mode or a non-blocking mode to each of the plurality of tasks of the machine learning model to establish a policy, and executing the machine learning model and its tasks based on the policy. Attached Figure Description

[0010] The above and other objects and features of this disclosure will become apparent from the detailed description of embodiments thereof with reference to the accompanying drawings.

[0011] Figure 1 A computing device according to some embodiments of the present disclosure is shown.

[0012] Figure 2 Methods of operating a computing device according to some embodiments of the present disclosure are illustrated.

[0013] Figure 3 An example of a task for a machine learning model is shown.

[0014] Figure 4 An example of a task that executes a machine learning model in blocking mode is shown.

[0015] Figure 5 An example of a task that executes a machine learning model in non-blocking mode is shown.

[0016] Figure 6 An example of a machine learning model manager building strategy driven by the processor of a computing device is shown.

[0017] Figure 7 An example is shown where a machine learning model manager, executed by the processor, assigns a task to one of a blocking mode or a non-blocking mode.

[0018] Figure 8 An example is shown where a machine learning model manager, executed by a processor, dynamically assigns processors and patterns to each task.

[0019] Figure 9 An example is shown where a machine learning model manager, executed by a processor, dynamically changes its strategy to perform tasks.

[0020] Figure 10 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.

[0021] Figure 11 An example of an autonomous driving system is shown.

[0022] Figure 12 It shows in Figure 11 An example of how an autonomous vehicle in an autonomous driving system changes the strategy of its machine learning model. Detailed Implementation

[0023] Below, some embodiments of this disclosure are described in detail to the extent that they enable those skilled in the art to implement the inventive concepts provided in this disclosure.

[0024] Figure 1 A computing device 100 according to some embodiments of the present disclosure is shown. (Refer to...) Figure 1 The computing device 100 may include a bus 110, a processor 120, a memory 130, a modem 140, a storage device 150, and a user interface 160.

[0025] Bus 110 can provide a channel between components of computing device 100 to allow the components of computing device 100 to communicate with each other. For example, bus 110 can provide a channel between components of computing device 100 based on various known standards (e.g., PCIe (Peripheral Component Interconnect Fast) and SCSI (Small Computer System Interface)) or based on standards defined by the manufacturer of computing device 100.

[0026] In some embodiments, components of the computing device 100 other than the processor 120 and bus 110 (e.g., memory 130, modem 140, storage device 150, and user interface 160) may be configured to communicate with the processor 120 based on various different standards. The bus 110 may include a driver that supports communication conversion between the communication standards of the other components 130, 140, 150, 160 of the computing device 100 and the communication standard of the processor 120.

[0027] Processor 120 can control computing device 100 and its operation. Processor 120 can drive an operating system used to control computing device 100 and its operation. Processor 120 can execute various applications based on the operating system. For example, processor 120 can utilize an application processor (AP).

[0028] Processor 120 may include processor bus 121, first processor 122, second processor 123, third processor 124, fourth processor 125, processor interface 126, and processor memory 127. Processor bus 121 can provide a channel between components of processor 120, allowing the components of processor 120 to communicate with each other. For example, processor bus 121 may provide a channel between components of processor 120 based on various known standards (e.g., AMBA (Advanced Microcontroller Bus Architecture)) or based on a standard defined by the manufacturer of processor 120.

[0029] The first processor 122 can be the main processor of the processor 120. For example, the first processor 122 can be implemented using a central processing unit (CPU). The second processor 123, the third processor 124, and the fourth processor 125 can be sub-processors of the processor 120.

[0030] For example, the second processor 123 can be implemented using a neural processing unit (NPU). The third processor 124 can be implemented using a graphics processing unit (GPU). The fourth processor 125 can be implemented as a digital signal processor (DSP). An example embodiment of a processor 120 with multiple subprocessors of different types is described below. Figure 1 The number and type of subprocessors of processor 120 are not limited to the present invention. For example, in some embodiments there may be multiple subprocessors of one type (e.g., multiple NPUs), in some embodiments there may be no subprocessors of one type (e.g., no GPU), and so on.

[0031] The processor interface 126 can provide an interface between the components of the processor 120 and the bus 110 to allow the components of the processor 120 to communicate with any other components 130, 140, 150, 160 of the computing device 100 via the bus 110.

[0032] Processor memory 127 can be used to store data or instruction code used by the first processor 122, the second processor 123, the third processor 124, and / or the fourth processor 125. For example, processor memory 127 can be implemented using cache memory or working memory. Processor memory 127 may include static random access memory (SRAM).

[0033] Memory 130 may be used to store data or instruction code of an operating system and / or applications that can be executed by processor 120. Memory 130 may be the main memory of computing device 100. For example, memory 130 may be implemented using dynamic random access memory (DRAM) or storage class memory (SCM).

[0034] Memory 130 may be configured to communicate with processor 120 based on DRAM standards. In some embodiments, memory 130 may be implemented as directly coupled to processor 120 and may be implemented as providing storage space to any other components 140, 150, 160 of computing device 100 via processor 120. Memory 130 may be implemented together with a memory controller configured to provide access to memory 130. In some embodiments, the memory controller may be integrated within processor 120.

[0035] Modem 140 can provide a channel for wired or wireless communication with external devices. In some embodiments, modem 140 can provide a channel for wired or wireless communication with external devices based on various wireless communication standards (e.g., Wi-Fi, LTE (Long Term Evolution), 5G (Fifth Generation), Bluetooth, and NFC (Near Field Communication)) and / or various wired communication standards (e.g., USB (Universal Serial Bus), cable, and Fibre Channel). In some embodiments, modem 140 can be integrated with processor 120.

[0036] Storage device 150 can be implemented to store data and instruction code of the operating system and / or applications, so that the data and code can be retained even when power is interrupted or removed. Storage device 150 can be implemented using various non-volatile storage devices (e.g., solid-state drives (SSDs), optical disc drives (ODDs), and / or hard disk drives (HDDs)).

[0037] Storage device 150 can communicate with bus 110 (and corresponding components of computing device 100) based on various standards (e.g., NVMe (Non-Volatile Memory Fast), SAS (Serial Attached SCSI), and SATA (Serial Advanced Technology Attachment)). Alternatively or additionally, storage device 150 can be implemented using removable memory based on standards that support detachment and attachment (e.g., USB).

[0038] User interface 160 can receive information from a user and can provide information to a user. For example, user interface 160 may include user input devices implemented to receive information from a user, such as a keyboard, mouse, microphone, touchpad, touch panel, camera, and sensors. Additionally or alternatively, user interface 160 may include user output devices implemented to provide information to a user, such as a monitor, speaker, beam projector, and printer.

[0039] The processor 120 can drive the machine learning model manager as one of the applications. The data and instruction code of the machine learning model manager can be stored in the storage device 150 and can be loaded onto the memory 130 for use by the processor 120. The machine learning model manager can be implemented to dynamically execute various machine learning models.

[0040] A machine learning model can be based on the results of machine learning learning and can include algorithms that render or achieve results by performing tasks (e.g., reasoning, classifying, or recognizing inputs). For example, the input can be image data or audio samples. Examples of tasks performed include recognizing objects in an image, classifying an object, or recognizing speech. The data and instruction code of the machine learning model can be stored in storage device 150 and can be loaded onto memory 130 for execution by the machine learning model manager.

[0041] A machine learning model can include multiple tasks. A task can be a unit that can be executed independently by different processors (e.g., different processors among the first to fourth processors 122 to 125). To accelerate the execution of the machine learning model, the machine learning model manager can dynamically assign tasks to the first to fourth processors 122 to 125 and can dynamically assign task patterns to perform the work.

[0042] In some embodiments, the machine learning model manager may be stored in storage device 150 by a non-transitory computer-readable storage medium such as removable memory or via modem 140. Storage device 150 may be implemented using a non-transitory computer-readable storage medium.

[0043] Reference Figure 1 As described herein, the second processor 123, the third processor 124, and the fourth processor 125 are integrated into the processor 120 together with the first processor 122, but it should be understood that this disclosure is not limited thereto. For example, at least one or all of the second processor 123, the third processor 124, and the fourth processor 125 may be located outside the processor 120 and may communicate with the processor 120 via the bus 110.

[0044] Figure 2 A method of operating a computing device 100 according to some embodiments of the present disclosure is illustrated. (Refer to...) Figure 1 and Figure 2 In operation S110, the machine learning model manager, driven by the processor 120, can load a machine learning model from the storage device 150. For example, the machine learning model manager can load the data and instruction code of the machine learning model and execute the instructions of the machine learning model.

[0045] In operation S120, computing device 100 can receive execution requests to execute machine learning models. For example, the machine learning model manager can receive execution requests via modem 140 or via user interface 160. Alternatively, the machine learning model manager can receive execution requests from one or more applications driven by processor 120 via application programming interface (API).

[0046] In operation S130, the machine learning model manager of computing device 100 can dynamically assign blocking or non-blocking modes to the tasks of the machine learning model to establish the strategy of the machine learning model. The strategy can instruct the machine learning model manager on the method (or mode) for executing each task of the machine learning model. For example, the machine learning model manager can allow at least one task of the machine learning model to switch between blocking and non-blocking modes depending on given conditions. (See reference...) Figure 6 Examples that describe the given conditions in more detail.

[0047] In operation S140, the machine learning model manager of computing device 100 can execute a machine learning model to perform a task based on the strategy set in operation S130. This task can be indicated by performing tasks on the input data provided to the machine learning model to guide the process of plotting or achieving output.

[0048] Figure 3 An example of a task for the machine learning model MLM is shown. (See reference...) Figure 3 The tasks of a Machine Learning Model (MLM) can be implemented hierarchically. An MLM model can include tasks T1 through T8, from the first to the eighth. Task T1 can be performed on the input data IN. Tasks T2 and T3 can be performed on the results of Task T1. Task T4 can be performed on the results of Tasks T2 and T3. Task T5 can be performed on the results of Task T4.

[0049] Task T6 can be executed on the result data of Task T5. Task T7 can also be executed on the result data of Task T5. Task T8 can be executed on the result data of both Task T6 and Task T7. The result data of Task T8 can be output as OUT data, and can be the result of a machine learning model (MLM). Tasks T1 through T8 can be executed by different processors that are independent of each other.

[0050] Figure 4 This demonstrates execution in blocking mode. Figure 3 Examples of tasks T1-T8 of the machine learning model MLM are shown. Figure 4 In the diagram, the horizontal axis represents time "T", and the vertical axis represents the threads of processor 120. (See reference...) Figure 1 and Figure 4 In blocking mode, tasks T1 to T8, from the first to the eighth, can be executed sequentially by a first thread TH1.

[0051] For example, the second task T2 and the third task T3 are executed simultaneously on the result data of the first task T1. While the first thread TH1 of the processor 120 is executing the second task T2 on the result data of the first task T1, the first thread TH1 can block the execution of the third task T3. When the second task T2 is completed, the first thread TH1 can execute the third task T3 on the result data of the first task T1.

[0052] Similarly, tasks T6 and T7 are executed concurrently on the result data of task T5. While the first thread TH1 of processor 120 is executing task T6 on the result data of task T5, the first thread TH1 can block the execution of task T7. When task T6 is completed, the first thread TH1 can execute task T7 on the result data of task T5.

[0053] Figure 5 This demonstrates execution in non-blocking mode. Figure 3 Examples of tasks T1-T8 of the machine learning model MLM are shown. Figure 5 In the diagram, the horizontal axis again represents time "T," and the vertical axis represents the threads of processor 120. (See reference...) Figure 1 and Figure 5 In non-blocking mode, tasks T1 to T8, from the first to the eighth, can be executed sequentially by the first thread TH1 and the second thread TH2.

[0054] Thread TH1 and thread TH2 can each execute tasks that can be executed in parallel. (See again...) Figure 3 In the hierarchical structure shown, tasks T2 and T3 can be executed in parallel, as can tasks T6 and T7. Tasks T2 and T3 can be executed in parallel because neither task T2 nor T3 depends on the result of the other of tasks T2 and T3 for execution. Similarly, tasks T6 and T7 can be executed in parallel because neither task T6 nor T7 depends on the result of the other of tasks T6 and T7 for execution. Thread TH1 can execute tasks T1, T2, T4, T5, T6, and T8 sequentially.

[0055] The results of the first task T1 are used to execute the second task T2 and the third task T3. While the first thread TH1 of the processor 120 is executing the second task T2 on the results of the first task T1, the results of the first task T1 can be sent to the second thread TH2.

[0056] The second thread TH2 can execute the third task T3 on the result data of the first task T1, and can also send the result data of the third task T3 back to the first thread TH1. The first thread TH1 can execute the fourth task T4 on the result data of the second task T2 and the result data of the third task T3 sent from the second thread TH2.

[0057] Similarly, the results of the fifth task T5 are used to execute the sixth task T6 and the seventh task T7. While the first thread TH1 of the processor 120 is executing the sixth task T6 on the results of the fifth task T5, the first thread TH1 can also send the results of the fifth task T5 to the second thread TH2.

[0058] The second thread TH2 can execute the seventh task T7 on the result data of the fifth task T5, and can also send the result data of the seventh task T7 to the first thread TH1. The first thread TH1 can execute the eighth task T8 on the result data of the sixth task T6 and the result data of the seventh task T7 sent from the second thread TH2.

[0059] Compare Figure 4 and Figure 5 Generally, a Machine Learning Model (MLM) operating in non-blocking mode can be considered to execute faster than one operating in blocking mode. However, communication between the first thread TH1 and the second thread TH2 can take time, which can be as long as several milliseconds (ms) or longer. Depending on various factors such as the characteristics of the processor 120 and the workload, the communication speed between the first thread TH1 and the second thread TH2, as well as the task execution speed of each of the first thread TH1 and the second thread TH2, can become even slower.

[0060] Therefore, when the machine learning model MLM is actually executed at computing device 100, which of the non-blocking and blocking modes provides faster execution speed can be variable. A machine learning model manager driven at processor 120 of computing device 100 according to some embodiments of this disclosure can improve the actual execution speed of the machine learning model MLM by setting each task of the machine learning model MLM to either non-blocking or blocking mode.

[0061] Figure 6 An example of a machine learning model manager establishing a strategy, driven by the processor 120 of computing device 100, is shown. (Refer to...) Figure 1 and Figure 6In operation S210, the machine learning model manager can obtain model information. This model information may include information about the machine learning model loaded by the machine learning model manager. For example, the model information may include factors affecting the execution time (or speed) of the work or the execution time (or speed) of each task in the information associated with the machine learning model.

[0062] For example, model information may include information about the type of machine learning model. As an example, the type of machine learning model information may include information indicating whether the machine learning model is a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer. Additionally or alternatively, as an example, the type of machine learning model information may include information indicating whether the machine learning model (MLM) is a neural network, a decision tree, or a support vector machine (SVM).

[0063] Model information can include information about the tasks to be performed by the machine learning model. For example, as an example, the task information of the machine learning model can include information about whether the task to be performed by the machine learning model is classification, object recognition, or automatic speech recognition (ASR).

[0064] Model information may include the name or filename of the machine learning model. For example, a machine learning model manager may include (or store) information on the statistical execution time (or velocity) of each machine learning model in a job or task for each machine learning model by name or filename. The machine learning model manager may reference the statistical execution time (or velocity) based on the machine learning model's name or filename.

[0065] Model information can include the hierarchical structure of the tasks in the machine learning model. For example, a machine learning model manager can obtain information about the hierarchy, such as the number of tasks to be executed sequentially, the number of tasks that can be executed in parallel (or simultaneously), and the positions of those tasks in the hierarchy.

[0066] Model information may include hardware information. For example, model information may include information about at least one of the first to fourth processors 122 to 125 associated with the machine learning model. For example, model information may include information about the preferred or required processor type for each task, such as, for example, information about at least one preferred or required processor among CPU, NPU, GPU, and DSP. The information about the preferred or required processor may be determined statistically based on the execution history of the task and the first to fourth processors 122 to 125, or it may be detected based on different characteristics of the task and different characteristics of the first to fourth processors 122 to 125.

[0067] Model information may include information (e.g., size or structure information) of a control program (e.g., an NPU control program (NCP)) that can be executed at each of the first to fourth processors 122 to 125.

[0068] In some embodiments, before the machine learning model manager executes the machine learning model, the machine learning model manager may pre-set at least some or all of the model information collected in operation S210. Additionally or alternatively, when the machine learning model manager executes the machine learning model, at least some or all of the model information collected in operation S210 may be obtained from the machine learning model.

[0069] During operation S220, the machine learning model manager can obtain system information. This system information may include information about the system that loads the machine learning model manager and the machine learning model, such as information about the computing device 100. In some embodiments, regarding the execution of the machine learning model manager and the machine learning model, the system information may include hardware factors affecting the time (or speed) of the machine learning model performing its work or the time (or speed) of performing each task.

[0070] System information may include information about the input data of the machine learning model. System information may include at least one of the performance, currently available resources, and / or current workload of each of the first to fourth processors 122 to 125. In addition, system information may include information (e.g., number and type) of tasks (or tasks with pending requests) being processed by each of the first to fourth processors 122 to 125.

[0071] System information may include information on the capabilities, capacity, and / or limitations of each of the first to fourth processors 122 to 125. For example, capacity information may include the number of trillion operations per second (TOPS), the number of TOPS per power, and the number of operations supported by each of the first to fourth processors 122 to 125.

[0072] Limitation information may include information about the number of bits limited during the execution of each operation in the first through fourth processors 122 through 125. For example, a particular processor may only support operations with up to three digits. Another particular processor may only support floating-point operations.

[0073] System information may include historical information about each of the first through fourth processors 122 through 125 that operate in association with the machine learning model. For example, historical information may include information about the execution time of each of the first through fourth processors 122 through 125 in blocking or non-blocking modes when performing various tasks. For example, based on historical information, the machine learning model manager may determine whether blocking mode or non-blocking mode is actually faster when a particular processor performs a particular task.

[0074] System information may include information about the operating mode of each of the first to fourth processors 122 to 125. For example, the operating mode information may include information about the power mode of each of the first to fourth processors 122 to 125. The power mode information may indicate a normal mode and one of two or more progressive power-saving modes.

[0075] The operating mode information may include thermal management information for each of the first to fourth processors 122 to 125. The thermal management information may include information indicating whether the performance of each of the first to fourth processors 122 to 125 is limited by thermal management.

[0076] Information about the operating mode may include information about the DVFS (Dynamic Voltage and Frequency Scaling) level of each of the first to fourth processors 122 to 125. DVFS enables each of the first to fourth processors 122 to 125 to operate at low power by dynamically adjusting the voltage level and frequency depending on the workload of each of the first to fourth processors 122 to 125.

[0077] In some embodiments, before the machine learning model manager executes the machine learning model, the machine learning model manager may obtain or manage at least some or all of the system information collected in operation S220 in advance. Additionally or alternatively, when the machine learning model manager executes the machine learning model, at least some or all of the model information collected in operation S220 may be obtained or managed in real time.

[0078] In operation S230, the machine learning model manager can obtain demand information. This demand information may include instructions on whether the application needs to perform tasks in a blocking or non-blocking manner via the application programming interface (API).

[0079] In addition, the requirement information may include the result of at least one different task preceding at least one task in the task. For example, the requirement information may include information indicating whether the result of a previous task should be used to execute subsequent tasks in a blocking or non-blocking manner during the execution of a task.

[0080] In operation S240, the machine learning model manager can dynamically assign one of a blocking mode or a non-blocking mode to each task based on model information, system information, and requirement information.

[0081] For example, the machine learning model manager can assign a blocking mode or a non-blocking mode to each task when loading tasks. When executing a specific task, the machine learning model manager can set (or update) the corresponding model to blocking mode or non-blocking mode based on at least one of model information, system information, and requirement information.

[0082] Figure 7 This illustrates an example of a machine learning model manager, executed by processor 120, assigning one of blocking or non-blocking modes to a task. (See reference...) Figure 1 and Figure 7 In operation of S310, the machine learning model manager can select one of the tasks of the machine learning model.

[0083] In operation S320, the machine learning model manager can assign one of the first to fourth processors 122 to 125 to the selected task. For example, the machine learning model manager can assign a processor to the selected task based on information about the preferred or required processor for the execution of the selected task, as well as the available resources or workload of the corresponding processor.

[0084] In operation S330, the machine learning model manager can assign either a blocking mode or a non-blocking mode to the selected task. For example, the machine learning model manager can assign a blocking mode or a non-blocking mode to the selected task based on its performance, load, or different characteristics when the selected task is executed on the allocated processor, and based on model information, system information, and requirement information.

[0085] In operation S340, the machine learning model manager can determine whether the selected task is the last task. When the selected task is not the last task (from the "No" branch of operation S340), in operation S350, the machine learning model manager can select the next task and return to operation S320. When the selected task is the last task (from the "Yes" branch of operation S340), the machine learning model manager can terminate the assignment associated with the task.

[0086] Figure 8 An example is shown where a machine learning model manager, executed by processor 120, dynamically allocates processors and patterns to each task. (See reference...) Figure 1 and Figure 8In operation S410, the machine learning model manager, driven by the processor 120, can load machine learning models from the storage device 150. For example, the machine learning model manager can load the data and instruction code of the machine learning model and execute the instructions of the machine learning model.

[0087] In operation S420, computing device 100 can receive execution requests for executing machine learning models. For example, the machine learning model manager can receive execution requests via modem 140 or via user interface 160. Alternatively, the machine learning model manager can receive execution requests from an application driven by processor 120 via application programming interface (API).

[0088] In operation S430, as part of the process of establishing a strategy for building a machine learning model, the machine learning model manager of computing device 100 can dynamically allocate first to fourth processors 122 to 125 to the tasks of the machine learning model. The strategy can instruct the machine learning model manager on the method (or pattern) for executing each task of the machine learning model.

[0089] For example, the machine learning model manager can assign one of the first to fourth processors 122 to 125 to each task based on model information, system information, or demand information including the preferred or required processor for task execution, the available resources or workload of the first to fourth processors 122 to 125, the history of the actual execution of the task on the first to fourth processors 122 to 125, etc.

[0090] In operation S440, as part of the process of establishing a machine learning model, the machine learning model manager of computing device 100 can dynamically assign blocking and non-blocking modes to the tasks of the machine learning model. Operation S440 can be performed according to the description given in reference operation S130.

[0091] In operation S450, the machine learning model manager of computing device 100 can execute a machine learning model based on a policy to perform a task. Operation S450 can be executed according to the description given in reference operation S140.

[0092] Figure 9 An example is shown where a machine learning model manager, executed by processor 120, dynamically changes strategies to perform tasks. (See reference...) Figure 1 and Figure 9 In operation S510, the machine learning model manager can monitor model information. In operation S520, the machine learning model manager can monitor system information. In operation S530, the machine learning model manager can monitor demand information.

[0093] In operation of S540, the machine learning model manager can determine whether a policy change is needed. For example, when changes in model information reach a first threshold or more, when changes in system information reach a second threshold or more, or when demand information changes, the machine learning model manager can determine that a policy change is necessary.

[0094] In response to the absence of a policy change (from the "No" branch of operation S540), the machine learning model manager can maintain the policy in operation S550. In response to the absence of a policy change (from the "Yes" branch of operation S540), the machine learning model manager can change the assignment of at least one task to change the policy in operation S560.

[0095] For example, a policy change could include changing whether each task is assigned to any of the first to fourth processors 122 to 125. A policy change could also include changing whether each task is assigned to either a blocking mode or a non-blocking mode.

[0096] For example, a machine learning model manager can be configured to repeatedly perform work using a machine learning model, and can also be configured to determine whether a policy change is needed before performing the work. As another example, a machine learning model manager can be configured to determine whether a policy change is needed before each task of the work is performed, or between tasks with a given number of units.

[0097] Figure 10 This is a block diagram illustrating an electronic device 1000 according to the present disclosure. (Refer to...) Figure 10 The electronic device 1000 may include a main processor 1100, a touch panel 1200, a touch driver integrated circuit (TDI) 1202, a display panel 1300, a display driver integrated circuit (DDI) 1302, a system memory 1400, a storage device 1500, an audio processor 1600, a communication block 1700, an image processor 1800, and a user interface 1900. In some embodiments, the electronic device 1000 may be one of a variety of electronic devices, such as an autonomous driving device (or vehicle), a personal computer, a laptop computer, a server, a workstation, a portable communication terminal, a personal digital assistant (PDA), a portable media player (PMP), a digital camera, a smartphone, a tablet computer, and a wearable device. It should be understood that this disclosure is not limited to these types of devices.

[0098] The main processor 1100 can control the overall operation of the electronic device 1000. The main processor 1100 can control / manage the operation of the components of the electronic device 1000. The main processor 1100 can handle various operations for the purpose of operating the electronic device 1000.

[0099] The main processor 1100 may include a reference Figure 1 The processor 120 is described. For example, see reference 120. Figure 1 As described, the main processor 1100 may include different heterogeneous subprocessors and a central processing unit. Alternatively, the different heterogeneous subprocessors may be arranged outside the main processor 1100. The subprocessors can each perform tasks related to machine learning models under the control of the central processing unit.

[0100] Touch panel 1200 can be configured to sense touch input from a user. Touch panel 1200 can be under the control of touch driver integrated circuit 1202 or controlled by touch driver integrated circuit 1202. Display panel 1300 can be configured to display image information and can be under the control of display driver integrated circuit 1302 or controlled by display driver integrated circuit 1302.

[0101] System memory 1400 may store data used in the operation of electronic device 1000. For example, system memory 1400 may include volatile memory (e.g., static random access memory (SRAM), dynamic RAM (DRAM), or synchronous DRAM (SDRAM)), and / or system memory 1400 may include non-volatile memory (e.g., phase change RAM (PRAM), magnetoresistive RAM (MRAM), resistive RAM (ReRAM), or ferroelectric RAM (FRAM)).

[0102] Storage device 1500 can store data regardless of whether it is powered on. For example, storage device 1500 may include at least one of various non-volatile memories, such as flash memory, PRAM, MRAM, ReRAM, and / or FRAM. For example, storage device 1500 may include embedded memory and / or removable memory of electronic device 1000.

[0103] Audio processor 1600 can be configured to process audio signals using audio signal processor 1610. Audio processor 1600 can receive audio input via microphone 1620 and / or provide audio output via speaker 1630. Communication block 1700 can exchange signals with external devices / systems via antenna 1710. The transceiver 1720 and modulator / demodulator (MODEM) 1730 of communication block 1700 can process signals exchanged with external devices / systems based on at least one of the following wireless communication protocols: for example, Long Term Evolution (LTE), Global Microwave Access Interoperability (WiMax), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Bluetooth, Near Field Communication (NFC), Wi-Fi, and / or Radio Frequency Identification (RFID).

[0104] Image processor 1800 can be configured to receive light through lens 1810. Image device 1820 and image signal processor (ISP) 1830 included in image processor 1800 can generate image information about external objects based on the received light. User interface 1900 may include an interface capable of exchanging information with a user and may differ from touch panel 1200, display panel 1300, audio processor 1600, and image processor 1800. User interface 1900 may include a keyboard, mouse, printer, projector, various sensors, human body communication devices, etc. In some embodiments, one or more of touch panel 1200, display panel 1300, audio processor 1600, image processor 1800, and user interface 1900 may be omitted.

[0105] Electronic device 1000 may further include a power management IC (PMIC) 1010, a battery 1020, and a power connector 1030. The power management IC 1010 may be configured to generate internal power from power supplied by the battery 1020 and / or from power supplied by the power connector 1030, and may provide this internal power to a main processor 1100, a touch panel 1200, a touch driver integrated circuit (TDI) 1202, a display panel 1300, a display driver integrated circuit (DDI) 1302, a system memory 1400, a storage device 1500, an audio processor 1600, a communication block 1700, an image processor 1800, and a user interface 1900. In some embodiments, one or more of the components identified above may receive power from a source other than the power management IC 1010.

[0106] Each component of the electronic device 1000 may include a safety monitoring device SM. The safety monitoring device SM may also be connected to each channel between the components of the electronic device 1000. The electronic device 1000 may be implemented as an in-vehicle autonomous driving system or an infotainment system. The safety monitoring device SM may be implemented based on ISO 26262 or ASIL (Automotive Safety Integrity Level) standards.

[0107] Figure 11 An example of an autonomous driving system 2000 is shown. (See reference...) Figure 11 The autonomous driving system 2000 may include an autonomous vehicle 2100, external vehicles 2200, and pedestrians 2300.

[0108] The autonomous vehicle 2100 may include reference Figure 1 The described computing device 100 and / or reference Figure 10 The described electronic device 1000. The computing device 100 or electronic device 1000 can perform the functions described above. Figures 1 to 9The machine learning model manager is described. It can load machine learning models and execute those models.

[0109] Machine learning models can be used for tasks such as autonomous driving. As examples of autonomous driving tasks, machine learning models can be used for tasks such as object recognition, tracking the motion of recognized objects, and optical flow.

[0110] Figure 12 It shows in Figure 11 Examples of autonomous driving systems 2000 and autonomous vehicles 2100 that change the strategy of machine learning models. Figure 12 An example is shown where the requirements change due to the results of a previous task.

[0111] Reference Figure 11 and Figure 12 In operation S610, the machine learning model manager can perform an object recognition task to detect objects in either blocking or non-blocking mode. In operation S620, in response to the result of the object recognition task, the machine learning model manager can determine whether the nearest object is a pedestrian.

[0112] When the nearest object is a pedestrian (from the "Yes" branch of operation S620), in operation S630, the machine learning model manager can set the motion tracking and optical flow tasks of the identified object to non-blocking mode. In other words, in operation S630, the strategy can be changed because the motion tracking and optical flow tasks of the identified object may have been pre-set to blocking mode.

[0113] When the nearest object is not a pedestrian (from the "No" branch of operation S620), in operation S640, the machine learning model manager can set the motion tracking and optical flow tasks of the identified object to blocking mode. For example, the machine learning model manager can first execute the motion tracking task of the identified object, and then execute the optical flow task of the identified object. In other words, the strategy can be changed in operation S640 because the motion tracking and optical flow tasks of the identified object may have been pre-set to non-blocking mode.

[0114] As described above, the machine learning model manager of the autonomous vehicle 2100 can accelerate autonomous driving by setting subsequent tasks to non-blocking mode based on whether the nearest object to the autonomous vehicle 2100 is a pedestrian. Therefore, the stability of the autonomous vehicle 2100 can be improved.

[0115] For example, in a machine learning model performing autonomous driving tasks, the machine learning model manager can determine that faster execution speed can be achieved by performing the object's motion tracking and optical flow tasks in blocking mode, rather than in non-blocking mode. In this case, even if the result of the previous task requires non-blocking mode, the machine learning model manager can establish a strategy for the machine learning model to perform the object's motion tracking and optical flow tasks in blocking mode.

[0116] In the above embodiments, the components according to this disclosure are described using the terms "first," "second," "third," etc. However, the terms "first," "second," "third," etc., are used to distinguish the components from each other and do not limit this disclosure. For example, the terms "first," "second," "third," etc., do not imply any form of order or numerical meaning.

[0117] In the above embodiments, components according to embodiments of the present disclosure are described using blocks. Blocks can be implemented using various hardware devices, such as integrated circuits, application-specific integrated circuits (ASCIs), field-programmable gate arrays (FPGAs), and complex programmable logic devices (CPLDs), firmware driven in the hardware device, software such as applications, or a combination of hardware devices and software. Blocks may include circuits implemented using semiconductor elements in integrated circuits and / or circuits registered as intellectual property (IP).

[0118] According to this disclosure, the tasks of a machine learning model can be dynamically allocated to blocking or non-blocking modes. Furthermore, the tasks of the machine learning model can be dynamically allocated to the processor. Therefore, a method for operating a computing device to execute a machine learning model at an increased speed, and a computer-readable storage medium for storing instructions, are provided.

[0119] Although this disclosure has been described with reference to exemplary embodiments thereof, it will be apparent to those skilled in the art that various changes and modifications may be made therein without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. A method of operating a computing device, the computing device including a central processing unit configured to execute a machine learning model and a plurality of subprocessors, the method comprising: Dynamically assign one of a blocking mode or a non-blocking mode to each of the multiple tasks of the machine learning model to establish a strategy; as well as The machine learning model and its tasks are executed based on the strategy described above. Wherein, at least one of the plurality of tasks of the machine learning model switches between the blocking mode and the non-blocking mode in response to the established policy. The blocking mode is configured to execute the multiple tasks sequentially using a single thread, and The non-blocking mode is configured to execute the multiple tasks sequentially through multiple threads, wherein tasks that can be executed in parallel are assigned to different threads for parallel execution.

2. The method according to claim 1, wherein, Dynamically assigning one of the blocking mode or the non-blocking mode to each of the plurality of tasks of the machine learning model to establish the policy includes: Based on at least one of the information from the machine learning model, the system information of the computing device, and / or the demand information, one of the blocking mode or the non-blocking mode is assigned to the at least one task.

3. The method according to claim 2, wherein, The information about the machine learning model includes information about the type of the machine learning model, and The machine learning model is an information identifier of the type of convolutional neural network (CNN), recurrent neural network (RNN), or converter.

4. The method according to claim 2, wherein, The information of the machine learning model includes information about the work to be performed by the machine learning model, and The machine learning model's work includes information identification and classification, object recognition, or automatic speech recognition (ASR).

5. The method according to claim 2, wherein, The information about the machine learning model includes the name of the machine learning model or the file name of the machine learning model.

6. The method according to claim 2, wherein, The information of the machine learning model includes the hierarchical structure of the task of the machine learning model.

7. The method according to claim 2, wherein, The information of the machine learning model includes information about at least one processor among the central processing unit and the plurality of subprocessors associated with each of the plurality of tasks of the machine learning model.

8. The method according to claim 2, wherein, The plurality of subprocessors include a neural processing unit (NPU), and The information in the machine learning model includes information about the neural processing unit control program (NCP) of the neural processing unit.

9. The method according to claim 2, wherein, The system information of the computing device includes information about the input data of the machine learning model.

10. The method according to claim 2, wherein, The system information of the computing device includes at least one of the available resources and workloads of each of the central processing unit and the plurality of subprocessors.

11. The method according to claim 2, wherein, The system information of the computing device includes information on the capabilities and / or limitations of each of the central processing unit and the plurality of subprocessors.

12. The method according to claim 2, wherein, The system information of the computing device includes historical information associated with the machine learning model of the computing device.

13. The method according to claim 2, wherein, The system information of the computing device includes information about the operating mode of the computing device.

14. The method according to claim 2, wherein, The requirement information includes requirement information received via the application programming interface (API).

15. The method according to claim 2, wherein, The requirement information includes the results of at least one different task preceding the at least one task in the task of the machine learning model.

16. The method according to claim 1, wherein, Dynamically assigning one of the blocking mode or the non-blocking mode to each of the plurality of tasks of the machine learning model to establish the policy includes: The central processing unit or one of the multiple subprocessors is dynamically assigned to each of the multiple tasks of the machine learning model.

17. A method of operating a computing device, the computing device including a central processing unit configured to execute a machine learning model and a plurality of subprocessors, the method comprising: Each of the multiple tasks of the machine learning model is dynamically assigned to one of the central processing unit or one of the multiple subprocessors, and one of the blocking mode or non-blocking mode is dynamically assigned to each of the multiple tasks of the machine learning model to establish a strategy. as well as The machine learning model and its tasks are executed based on the strategy described above. In this context, the first task of the machine learning model, in response to the established policy, switches between the blocking mode and the non-blocking mode. The blocking mode is configured to execute the multiple tasks sequentially using a single thread, and The non-blocking mode is configured to execute the multiple tasks sequentially through multiple threads, wherein tasks that can be executed in parallel are assigned to different threads for parallel execution.

18. The method according to claim 17, in, The first task is assigned to a first processor selected from the central processing unit and the plurality of subprocessors, and the second task of the machine learning model is assigned to a second processor, different from the first processor, selected from the central processing unit and the plurality of subprocessors.

19. A non-transitory computer-readable storage medium storing instructions that, when executed by a central processing unit and a plurality of subprocessors, cause operations including: Each of the multiple tasks of the machine learning model is dynamically assigned to one of the central processing unit or one of the multiple subprocessors, and one of the multiple tasks of the machine learning model is dynamically assigned to a blocking mode or a non-blocking mode to establish a strategy. as well as The machine learning model and its tasks are executed based on the strategy described above. Wherein, at least one of the plurality of tasks of the machine learning model switches between the blocking mode and the non-blocking mode in response to the established policy. The blocking mode is configured to execute the multiple tasks sequentially using a single thread, and The non-blocking mode is configured to execute the multiple tasks sequentially through multiple threads, wherein tasks that can be executed in parallel are assigned to different threads for parallel execution.

Images