Model operation method and related equipment

Abstract

The embodiment of the invention provides a model operation method and related equipment. The model operation method comprises the following steps: determining first resource pre-occupation values of a first to-be-operated model corresponding to a plurality of computing units in the same equipment; according to each first resource pre-occupation value and the residual resource values corresponding to the plurality of calculation units, selecting a calculation unit capable of operating the first to-be-operated model from the plurality of calculation units as a target calculation unit; and operating the first to-be-operated model by using the target calculation unit. The model operation method and the related equipment can improve the computing power utilization rate when single equipment is used for model operation.

Description

Technical Field

[0001] The implementation methods in the present application relate to the field of model operation technology, and in particular to a model operation method and related equipment. Background Art

[0002] Computing power can usually be used to measure the efficiency and speed of electronic devices in performing computing tasks. For electronic devices with computing power, they are equipped with computing units that can perform logical operations, such as CPUs. Computing power is a comprehensive reflection of the performance of computing units. With the development of technologies such as machine learning and artificial intelligence, electronic devices are used to run related models to achieve some specific tasks.

[0003] The related technology usually uses the designated computing unit in the electronic device to perform the model running task. However, when the computing power of the designated computing unit is insufficient to run the model, it is necessary to wait for the computing power of the designated computing unit to recover to a level sufficient to run the model before using the designated computing unit to run the model. It can be seen that the related technology has the problem of low computing power utilization. Summary of the invention

[0004] In view of this, multiple embodiments of the present application are dedicated to providing a model operation method and related equipment, which can improve the computing power utilization of a single device when running a model.

[0005] One embodiment of the present application provides a model running method, including: determining that a first model to be run corresponds to first resource pre-occupancy values ​​of multiple computing units in the same device; selecting a computing unit that can run the first model to be run from the multiple computing units as a target computing unit based on each first resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units; and using the target computing unit to run the first model to be run.

[0006] Optionally, the multiple computing units include at least two of the following: a central processing unit, a graphics processing unit, a computer vision task processing unit, and a video image synthesis task processing unit.

[0007] Optionally, the step of selecting a computing unit that can run the first model to be run from the multiple computing units as a target computing unit according to each first resource pre-occupancy value and the remaining resource values ​​respectively corresponding to the multiple computing units includes: comparing the remaining resource value of each computing unit with the corresponding first resource pre-occupancy value in turn according to the calling priority between the multiple computing units to determine a remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value; determining the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run; or, in order of size between the remaining resource values, starting from the computing unit corresponding to the largest remaining resource value, comparing the remaining resource value of each computing unit with the corresponding first resource pre-occupancy value in turn to determine a remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value; determining the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run.

[0008] Optionally, the method also includes: upon receiving a first task to be executed, determining a first model that the first task to be executed indicates to run; wherein the first model includes at least the first model to be run and the second model to be run; updating the remaining resource value of the target computing unit according to the resource pre-occupancy value of the first model to be run corresponding to the target computing unit; determining the second resource pre-occupancy value of the second model to be run corresponding to the multiple computing units; and selecting a computing unit that can run the second model to be run from the multiple computing units according to each second resource pre-occupancy value and the remaining resource values ​​respectively corresponding to the multiple computing units.

[0009] Optionally, the method also includes: when the computing unit that can run the second model to be run is the target computing unit, and the remaining resource value of the target computing unit is less than the corresponding second resource pre-occupancy value, recording the second model to be run in the queue to be run corresponding to the target computing unit; wherein the target computing unit is used to run the corresponding models in sequence according to the order of entry or the model priority in the queue to be run; or, when the computing unit that can run the second model to be run is not the target computing unit, triggering the computing unit that can run the second model to be run to run the second model to be run.

[0010] Optionally, the method also includes: upon receiving a second task to be executed, determining a second model that the second task to be executed indicates to run; in the process of running the first model using at least one of the multiple computing units, determining a designated computing unit for running the second model from the multiple computing units, and preloading the second model; after detecting that the first model has finished running, unloading the first model and running the second model using the designated computing unit.

[0011] Optionally, the method further includes: loading a resident model, and determining remaining resource values ​​corresponding to the plurality of computing units respectively based on a resource pre-occupancy value corresponding to the resident model.

[0012] One embodiment of the present application provides a model running device, including: a resource pre-occupancy value determination module, used to determine the first resource pre-occupancy values ​​of a first model to be run that respectively correspond to multiple computing units in the same device; a computing unit allocation module, used to select a computing unit that can run the first model to be run from the multiple computing units as a target computing unit based on each first resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units; a model running module, used to use the target computing unit to run the first model to be run.

[0013] One embodiment of the present application provides a computer device, which includes a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the model operation method as described above.

[0014] One embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, the model operation method as described above can be implemented.

[0015] One embodiment of the present application provides a computer program product, which is used to implement the model operation method as described above.

[0016] In multiple implementations provided in the present application, it is determined that the first resource pre-occupancy values ​​of the first model to be run respectively correspond to the first resource pre-occupancy values ​​of multiple computing units in the same device; according to each first resource pre-occupancy value and the remaining resource values ​​respectively corresponding to the multiple computing units, a computing unit that can run the first model to be run is selected from the multiple computing units as the target computing unit; and the first model to be run is run using the target computing unit. In this way, the remaining resource values ​​of multiple computing units in the same device can be taken as a comprehensive consideration factor, and the target computing unit for running the first model to be run is determined accordingly. The related technology only runs the model by specifying the computing unit, and there is a problem that the computing power of the specified computing unit is insufficient, while the computing power of other computing units is sufficient but not fully utilized. In comparison, the present application can improve the computing power utilization rate of a single device when the model is running. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a system architecture for implementing a model operation method provided for one embodiment of the present application.

[0018] Figure 2 A flowchart of a model operation method provided for one embodiment of the present application.

[0019] Figure 3 A schematic diagram of system scheduling in the process of implementing a model operation method provided in one embodiment of the present application.

[0020] Figure 4 A schematic diagram of system scheduling in the process of implementing the model operation method provided in another embodiment of the present application.

[0021] Figure 5 A schematic diagram of a module of a model running device provided for one embodiment of the present application.

[0022] Figure 6 A schematic diagram of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0024] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0025] As the main component that provides computing power in electronic devices, computing units can be used to perform a variety of computing tasks, such as model running tasks. In a related technology, based on the limited computing power of electronic devices (such as edge computing devices), the way to use electronic devices to perform model running tasks is to use a designated computing unit (such as a GPU) in the electronic device to perform model running tasks. When the number of model running tasks to be executed is large, the models to be run in multiple model running tasks are usually stored as a queue, and the designated computing unit is used to process each model to be run in the queue in sequence. Obviously, there is a problem of low model running efficiency, and there is also a problem of not fully utilizing the computing power of electronic devices. In another related technology, based on the limited computing power of electronic devices (such as edge computing devices), the way to use electronic devices to perform model running tasks is to perform sparse processing on each model to be run to reduce the volume of each model to be run. Since the amount of resources of the computing unit required to be occupied by the model to be run after sparse processing is reduced, the model running efficiency can be improved, but there will be a problem that the model running effect does not reach the expected effect of the model running task, and there is also a problem of not fully utilizing the computing power of the electronic device. In another related technology, in order to improve the model operation efficiency, the related technology uses multiple electronic devices in a distributed setting to evenly process each model operation task. By evenly distributing each model operation task to each electronic device, the model operation efficiency can be improved. However, this solution needs to be implemented based on a distributed architecture. The hardware deployment of the distributed architecture is more difficult than the previous solution. In addition, the balanced model operation tasks based on the distributed architecture cannot fully utilize the computing power of a single electronic device.

[0026] Therefore, it is necessary to provide a model operation method, which can determine the first resource pre-occupancy values ​​of the first model to be run corresponding to the multiple computing units in the same device; select the computing unit that can run the first model to be run from the multiple computing units as the target computing unit according to the first resource pre-occupancy values ​​and the remaining resource values ​​corresponding to the multiple computing units; and use the target computing unit to run the first model to be run. In this way, the remaining resource values ​​of multiple computing units in the same device can be used as a comprehensive consideration to determine the target computing unit for running the first model to be run. The related technology only runs the model by specifying the computing unit, and there is a problem that the computing power of the specified computing unit is insufficient while the computing power of other computing units is sufficient but not fully utilized. In comparison, the present application can improve the computing power utilization rate of a single device when the model is running. Compared with the solution of implementing model operation with distributed prices, this case saves hardware deployment costs. In addition, since the present application can select a computing unit from multiple computing units as the target computing unit that can run the first model to be run based on the remaining resource value, the resource allocation method is optimized.

[0027] See also Figure 1 In the multiple implementations provided in the present application, the model operation method can be applied to an electronic device with certain computing capabilities and network access capabilities. The electronic device can be a desktop computer, a laptop computer, a tablet computer, or a server.

[0028] Specifically, the electronic device includes at least a task engine and an inference scheduling engine. The electronic device can be applied to a variety of application scenarios, such as the field of robot control, the field of autonomous driving, the field of sky eye monitoring and analysis, etc. Exemplarily, when the electronic device is applied to the field of robot control, the task engine is used to generate corresponding control tasks and manage robot control tasks (such as grasping tasks, recognition tasks, etc.) according to the received sensor data, environmental information and other data, and publish the robot control tasks to be executed to the inference scheduling engine. The robot control tasks contain one or more models to be run. The specific model types and number of models contained in the robot control tasks depend on actual needs, and this application does not limit this. The inference scheduling engine includes a task management module, a scheduling allocation module, and an inference calculation module. Specifically, the task management module is used to manage tasks and parallelize loading / unloading models in the order of task release; the scheduling allocation module is used to assign computing units corresponding to the tasks; and the inference calculation module is used to use the computing units to perform model operation operations.

[0029] In the process of executing the model running method, the task engine is used to publish tasks to the inference scheduling engine; the task management module is used to allocate a computing unit to the first model to be run in the task based on the task trigger scheduling allocation module; the task trigger scheduling module is used to determine that the first model to be run corresponds to the first resource pre-occupancy values ​​of multiple computing units in the same device, and according to each first resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units, a computing unit that can run the first model to be run is selected from the multiple computing units as the target computing unit; the inference computing module is used to use the target computing unit to run the first model to be run.

[0030] See also Figure 2 . The embodiment of the present application provides a model operation method. The model operation method can be applied to a model operation device. The model operation device can be an electronic device with certain computing capabilities and network access capabilities. Of course, the model operation device can also refer to a software program running in an electronic device. In this embodiment, the model operation method can include the following steps.

[0031] Step S110: Determine first resource pre-occupancy values ​​of a plurality of computing units in the same device that are respectively corresponding to the first to-be-run model.

[0032] Step S120: According to the first resource pre-occupancy values ​​and the remaining resource values ​​respectively corresponding to the plurality of computing units, a computing unit that can run the first model to be run is selected from the plurality of computing units as a target computing unit.

[0033] Step S130: Utilize the target computing unit to run the first model to be run.

[0034] In this embodiment, in order to solve the problem of insufficient computing power when a single device is running a model, the model running device can start the task engine and the reasoning scheduling engine, and initialize the parameters of the two, and then publish the model running task to the reasoning scheduling engine through the task engine. The model running task includes at least a first model to be run. The first models to be run included in different model running tasks may be different. The first model to be run may be any type of model, and the specific type of the first model to be run depends on the scenario that triggers the model running task. For example, in the robotic arm grasping scenario, the role of the first in the first model to be run is to limit uniqueness rather than order or priority. For example, the first model to be run can be an object detection and recognition model, a speech recognition model, a reinforcement learning model, a multimodal perception and fusion model, etc.

[0035] Furthermore, the first resource pre-occupancy values ​​of the first model to be run corresponding to the multiple computing units can be determined through the inference scheduling engine. The computing unit refers to an electronic device with computing capabilities, and the multiple computing units are in the same device. The relationship between each first resource pre-occupancy value and each computing unit is a one-to-one correspondence. The first resource pre-occupancy value refers to the amount of resources pre-occupied by the first model to be run for the corresponding computing unit. The first resource occupancy value can be expressed in any form such as a number, a percentage, or a symbol. For example, the multiple computing units include a CPU and a GPU, and the first resource pre-occupancy values ​​of the first model to be run corresponding to the CPU and the GPU are 40% and 60%, respectively.

[0036] Since the first resource pre-occupancy value of each computing unit occupied by the first model to be run is determined, the computing unit suitable for running the first model to be run can be selected as the target computing unit in combination with the remaining resource values ​​corresponding to the multiple computing units; wherein the remaining resource value refers to the quantified result of the current remaining available resources of the corresponding computing unit, and the remaining resource value can also be understood as the computing power utilization rate. The remaining resource value is a dynamically changing value, and the remaining resource value can be expressed in any form such as a number, a percentage, or a symbol. In addition, optionally, the remaining resource value of the target computing unit can be a value greater than the corresponding first resource pre-occupancy value, or it can be the largest value among the remaining resource values, or it can be the value with the smallest difference from the corresponding first resource pre-occupancy value, and the embodiments of the present application are not limited to this.

[0037] For example, multiple computing units include a CPU and a GPU, and the first resource pre-occupancy values ​​of the first model to be run corresponding to the CPU and GPU are 40% and 60% respectively, and the remaining resource values ​​of the CPU and GPU are 80% and 50% respectively. Since the remaining resource value of the CPU is greater than the first resource pre-occupancy value of the first model to be run corresponding to the CPU, and the remaining resource value of the GPU is less than the first resource pre-occupancy value of the first model to be run corresponding to the GPU, it can be seen that the remaining resource value of the GPU is insufficient to run the first model to be run, and the remaining resource value of the CPU is sufficient to run the first model to be run, so the CPU can be determined as the target computing unit.

[0038] After the target computing unit is determined, the resources in the target computing unit can be used to run the first model to be run to obtain the required model running results. The model running results can be used as input for the next model running process, or can be returned as a request response result of the host computer, or can be stored as a task execution result. The embodiments of the present application do not limit this. In one example, the running results obtained by running the first model to be run using the resources in the target computing unit are used as a basis for generating robot control instructions. For example, the robot control instructions can be used to control the robot to move forward, backward, turn left or turn right.

[0039] The model running method provided in the implementation mode of the present application can use the remaining resource values ​​of multiple computing units in the same device as a comprehensive consideration to determine the target computing unit for running the first model to be run. The related technology only runs the model by specifying the computing unit, which has the problem that the computing power of the specified computing unit is insufficient while the computing power of other computing units is sufficient but not fully utilized. In comparison, the present application can improve the computing power utilization of a single device when running a model.

[0040] In some embodiments, the plurality of computing units include at least two of the following: a central processing unit, a graphics processing unit, a computer vision task processing unit, and a video image synthesis task processing unit.

[0041] In this embodiment, in order to make full use of various computing units in the model running device to improve the model running efficiency and improve the computing power utilization rate of the model running device, the computing units of the model running device may include: central processing unit (CPU), graphics processing unit (GPU), computer vision task processing unit (PVA), video image synthesis task processing unit (VIC). Among them, the central processing unit is used to execute instructions in the computer program, and can specifically perform arithmetic operations, logical operations, control and input / output operations; the graphics processor is used to process computing tasks of graphics and images, and can specifically perform tasks such as graphics rendering, video processing and machine learning; the computer vision task processing unit is a hardware module for accelerating computer vision tasks, and can perform image processing and visual analysis tasks such as target detection, image recognition, and edge computing; the video image synthesis task processing unit is used to process video image synthesis tasks, and its functions are: image synthesis, image scaling, format conversion, color correction and conversion, and hardware acceleration.

[0042] In addition, the model running device may also have one or more of the following computing units: a unit for optimizing matrix multiplication and tensor operations in deep learning tasks (Tensor Processing Unit, TPU), a unit for processing computing tasks related to neural networks (Neural Processing Unit, NPU), a unit for accelerating hardware (Field-Programmable Gate Array, FPGA), etc. In the model running device, the number of each computing unit can be one or more, and the embodiments of the present application do not limit this.

[0043] In some embodiments, the model running device selects a computing unit that can run the first model to be run from the multiple computing units as the target computing unit based on the first resource pre-occupancy values ​​and the remaining resource values ​​corresponding to the multiple computing units, including: comparing the remaining resource value of each computing unit with the corresponding first resource pre-occupancy value in turn according to the calling priority between the multiple computing units to determine a remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value; determining the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run; or, in order of size between the remaining resource values, starting from the computing unit corresponding to the largest remaining resource value, comparing the remaining resource value of each computing unit with the corresponding first resource pre-occupancy value in turn to determine a remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value; determining the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run.

[0044] In this embodiment, in order to further optimize the computing power allocation method, in one case, the model running device can compare the remaining resource value of each computing unit with the corresponding first resource pre-occupancy value in turn based on the calling priority between multiple computing units, so as to determine the remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value, and determine the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run. Among them, the calling priority is used to indicate the priority calling order of the computing unit. From another level, the calling priority can also indicate the strength of the hardware capabilities between the computing units.

[0045] For example, multiple computing units include a CPU and a GPU, and the calling priority between the CPU and the GPU is that the GPU is called before the CPU. When the first resource pre-occupancy values ​​of the first model to be run corresponding to the CPU and the GPU are 40% and 60% respectively, and the remaining resource values ​​of the CPU and the GPU are 80% and 80% respectively, the remaining resource value of the GPU is preferentially compared with the corresponding first resource pre-occupancy value. Since the remaining resource value of the GPU is greater than the corresponding first resource pre-occupancy value, the GPU can be determined as the target computing unit, even if the remaining resource value corresponding to the CPU is also greater than the corresponding first resource pre-occupancy value, but since the calling priority of the GPU is higher than that of the CPU, the GPU is used as the target computing unit. This embodiment provides a function of limiting the calling priority between computing units. In actual application, the calling priority can be personalized based on the specific capabilities of the computing units in the hardware to optimize the computing power allocation method.

[0046] In order to further optimize the computing power allocation method, in another case, the model running device can compare the remaining resource values ​​of each computing unit with the corresponding first resource pre-occupancy value in order of size between the remaining resource values, starting from the computing unit corresponding to the largest remaining resource value, to determine the remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value, and determine the computing unit corresponding to the target remaining resource value as the target computing unit that can run the first model to be run.

[0047] For example, multiple computing units include a CPU and a GPU. The first model to be run corresponds to first resource pre-occupancy values ​​of 40% and 60% for the CPU and GPU respectively, and the remaining resource values ​​of the CPU and GPU are 80% and 60% respectively. Since the remaining resource value of the CPU is greater than the remaining resource value of the GPU, the remaining resource value of each computing unit can be compared with the corresponding first resource pre-occupancy value in the order of CPU first and then GPU. Since the remaining resource value of the CPU is greater than the corresponding first resource pre-occupancy value, the CPU can be determined as the target computing unit.

[0048] See also Figure 3 and Figure 4 In some implementations, the model running device may load a resident model, and determine the remaining resource values ​​corresponding to the plurality of computing units respectively based on the resource pre-occupancy value corresponding to the resident model.

[0049] In this embodiment, in order to achieve specific tasks in specific scenarios, such as robot handling tasks, in addition to running the first model to be run in a short-term task (such as an obstacle avoidance task), it is also necessary to load a resident model. The resident model is a model that runs continuously during the robot operation and provides real-time support. The resident model needs to be run frequently and multiple times to obtain real-time running results to achieve real-time control of the robot. The resident model is usually used for continuous perception, decision-making, and control to ensure that the robot can respond in a timely manner to environmental changes.

[0050] Based on this, the model running device can load the resident model. Since the resident model needs to occupy the resources of one or more computing units, the remaining resource values ​​corresponding to multiple computing units can be determined based on the resource pre-occupancy value corresponding to the resident model. The remaining resource values ​​can be used to run models in short-term tasks, such as the first model to be run.

[0051] The number of resident models depends on actual needs. For example, resident models can be state estimation models that continuously track the state of the robot and its surroundings, sensor fusion models that fuse data from multiple sensors (such as vision, lidar, ultrasound, inertial sensors, etc.) into global perception data, obstacle avoidance and path planning models, motion control models, visual processing and object detection models, human-computer interaction models, health monitoring and self-diagnosis models, multi-robot collaboration models, environment modeling and map update models, etc.

[0052] See also Figure 3 . In some embodiments, the model running device may determine the first model that the first task to be executed indicates to run upon receiving the first task to be executed; wherein the first model includes at least the first model to be executed and the second model to be executed; update the remaining resource value of the target computing unit according to the resource pre-occupancy value of the first model to be executed corresponding to the target computing unit; determine the second resource pre-occupancy value of the second model to be executed corresponding to the multiple computing units; and select a computing unit that can run the second model to be executed from the multiple computing units according to each second resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units.

[0053] In this embodiment, in order to realize the parallel operation of multiple models and maximize the utilization of device computing power at the same time, the inference scheduling engine in the model running device can determine the first model indicated to be run by the first task to be executed when receiving the first task to be executed issued by the task engine in the model running device. The first task to be executed is a model running task, which is used to instruct the inference scheduling engine to run one or more first models. The first model can be a general term for the models that need to be run in the first task to be executed, or it can specifically refer to a certain model. In this embodiment, the first model can include a first model to be run and a second model to be run. In other embodiments, the first model can include any number of models, and the embodiments of the present application are not limited to this.

[0054] Furthermore, after initializing the engine parameters, the inference scheduling engine can determine the first resource pre-occupancy values ​​of the first model to be run corresponding to multiple computing units in the same device, and select a computing unit that can run the first model to be run from the multiple computing units as the target computing unit based on the first resource pre-occupancy values ​​and the remaining resource values ​​corresponding to the multiple computing units.

[0055] Furthermore, the inference scheduling engine can update the remaining resource value of the target computing unit based on the resource pre-occupancy value of the target computing unit corresponding to the first model to be run, determine the second resource pre-occupancy value of the second model to be run corresponding to multiple computing units, and select a computing unit that can run the second model to be run from the multiple computing units based on each second resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units.

[0056] In addition, optionally, if the first model also includes a third model to be run, the inference scheduling engine can update the remaining resource value of the target computing unit based on the resource pre-occupancy value of the second model to be run corresponding to the target computing unit, determine the third resource pre-occupancy value of the third model to be run corresponding to multiple computing units, and select a computing unit that can run the third model to be run from multiple computing units based on each third resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units.

[0057] For example, multiple computing units include CPU, GPU, PVA, and VIC; the remaining resource values ​​corresponding to CPU, GPU, PVA, and VIC are 60%, 90%, 100%, and 100% respectively; the first resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 70%, 30%, empty, and 50% respectively; the second resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 70%, 100%, empty, and 90% respectively; the third resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 60%, 30%, 50%, and empty respectively. If the target computing unit is a GPU, when matching the computing unit for the second model to be run, the remaining resource values ​​corresponding to CPU, GPU, PVA, and VIC are 60%, 60%, 100%, and 100% respectively. If the computing unit used to run the second model to be run is VIC, when matching the computing unit for the third model to be run, the remaining resource values ​​corresponding to the CPU, GPU, PVA, and VIC are 60%, 60%, 10%, and 100%, respectively. In one example, the CPU can be determined as the computing unit used to run the third model to be run.

[0058] Furthermore, the inference scheduling engine can sequentially load the first model to be run and the second model to be run. The specific implementation method is to load the first model to be run onto the target computing unit, and load the second model to be run onto the computing unit used to run the second model to be run. Then, the first model to be run is run using the target computing unit, and the second model to be run is run using the computing unit that can run the second model to be run. After the operation is completed, the first model to be run and the second model to be run can be unloaded to release computing power, thereby facilitating the operation of subsequent models to be run.

[0059] In some embodiments, the model running device may, when the computing unit that can run the second model to be run is the target computing unit and the remaining resource value of the target computing unit is less than the corresponding second resource pre-occupancy value, record the second model to be run in the queue to be run corresponding to the target computing unit; wherein the target computing unit is used to run the corresponding models in sequence according to the order of entry or the model priority in the queue to be run; or, when the computing unit that can run the second model to be run is not the target computing unit, trigger the computing unit that can run the second model to be run to run the second model to be run.

[0060] In this embodiment, in order to optimize the resource allocation method, in one case, if the computing unit that can run the second model to be run is the target computing unit, the model running device can record the second model to be run into the queue to be run corresponding to the target computing unit, so that the target computing unit executes the model run according to the model order in the queue to be run.

[0061] For example, multiple computing units include CPU, GPU, PVA, VIC; the remaining resource values ​​corresponding to CPU, GPU, PVA, and VIC are 60%, 90%, 100%, and 100% respectively; the first resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 100%, empty, empty, and empty respectively; the second resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 70%, empty, empty, and empty respectively; the third resource pre-occupancy values ​​corresponding to CPU, GPU, PVA, and VIC are 90%, empty, empty, and empty respectively. Based on this, since the corresponding first model to be run, the second model to be run, and the third model to be run can be recorded in the queue to be run of the target computing unit, the target computing unit can run each model to be run in sequence according to the order of recording, or can run each model to be run in sequence according to the model priority between the models to be run. As an example, the model priority of each model to be run is from high to low: the first model to be run, the second model to be run, and the third model to be run. Since the model priority can identify the importance or urgency of the corresponding model to be run, the first model to be run, the second model to be run, and the third model to be run in the queue to be run are executed in sequence according to the model priority, so that the models to be run with higher importance or urgency can be executed first, thereby obtaining their execution results more efficiently, so that the execution results can be applied to other tasks more promptly.

[0062] In order to optimize the resource allocation method, in another case, if the computing unit that can run the second model to be run is a non-target computing unit, when the remaining resources of the target computing unit are greater than or equal to the corresponding second resource pre-occupancy value, the computing unit that can run the second model to be run is triggered to run the second model to be run; when the remaining resources of the target computing unit are less than the corresponding second resource pre-occupancy value, the second model to be run is recorded in the queue to be run corresponding to the corresponding computing unit, so that the corresponding computing unit executes the model operation according to the model order in the queue to be run.

[0063] See also Figure 4 In some embodiments, the model running device may determine the second model that the second task to be executed indicates to run upon receiving the second task to be executed; in the process of running the first model using at least one of the multiple computing units, determine a designated computing unit for running the second model from the multiple computing units and preload the second model; after detecting that the first model has finished running, unload the first model and run the second model using the designated computing unit.

[0064] In this embodiment, in order to improve the model operation efficiency and realize task parallelization, the task engine in the model operation device can be used to manage multiple tasks to be executed, and publish multiple tasks to be executed to the reasoning scheduling engine in sequence. After the reasoning scheduling engine initializes the engine parameters, it receives the first task to be executed, and then determines that the first model to be executed corresponds to the first resource pre-occupancy value of multiple computing units in the same device, and selects the computing unit that can run the first model to be executed from the multiple computing units as the target computing unit according to each first resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units. And, according to the resource pre-occupancy value corresponding to the target computing unit of the first model to be executed, the remaining resource value of the target computing unit is updated, and the second resource pre-occupancy value corresponding to the second model to be executed of the multiple computing units is determined, and according to each second resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units, the computing unit that can run the second model to be executed is selected from the multiple computing units. And, the first model to be executed and the second model to be executed are loaded in sequence.

[0065] When the inference scheduling engine receives the second task to be executed, it can determine the second model that the second task to be executed indicates to run; wherein, the second model can refer to a specific model or a general term for multiple models, and the embodiment of the present application does not limit this. Similar to the first model, the specific type of the second model depends on the actual needs, which will not be repeated here. Then, in the process of using at least one computing unit among multiple computing units to run the first model, a designated computing unit for running the second model can be determined from multiple computing units, and the second model can be preloaded. At the same time, the first model to be run is run using the target computing unit, and the second model to be run is run using the computing unit that can run the second model to be run. Then the inference scheduling engine waits for synchronization, and when it is detected that the first model to be run and the second model to be run are completed, and the second model is loaded, it is determined that the synchronization is successful. Then, the first model to be run and the second model to be run can be unloaded and the second model to be run can be run using the designated computing unit; wherein, the designated computing unit can be any computing unit with running capability among multiple computing units.

[0066] See also Figure 5 . One embodiment of the present application also provides a model running device. The model running device may include: a resource pre-occupancy value determination module, used to determine the first resource pre-occupancy values ​​of the first model to be run corresponding to multiple computing units in the same device; a computing unit allocation module, used to select a computing unit that can run the first model to be run from the multiple computing units as a target computing unit based on each first resource pre-occupancy value and the remaining resource values ​​corresponding to the multiple computing units; a model inference module, used to use the target computing unit to run the first model to be run.

[0067] In this embodiment, the specific functions and effects achieved by the model running device can be explained with reference to other embodiments of the present application and will not be repeated here.

[0068] The embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor implements the model operation method as described above.

[0069] The embodiment of the present application also provides a computer program product comprising instructions, and when the computer program product is executed by a processor, the model operation method as described above is implemented.

[0070] See also Figure 6The present description embodiment may provide a computer device, the computer device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions are executed by the one or more processors, so that the one or more processors implement the model operation method as described above.

[0071] In some embodiments, the computer device may include a processor, a non-volatile storage medium, an internal memory, a communication interface, a display device, and an input device connected by a system bus. The non-volatile storage medium may store an operating system and related computer programs.

[0072] The user information or user account information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, etc.) involved in multiple implementation methods of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0073] It should be understood that the specific examples in this article are only intended to help those skilled in the art to better understand the embodiments of the present application, rather than to limit the scope of the present invention.

[0074] It can be understood that in the various implementations of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of the present application.

[0075] It can be understood that the various embodiments described in this application can be implemented individually or in combination, and the embodiments of this application are not limited to this.

[0076] Unless otherwise stated, all technical and scientific terms used in the embodiments of the present application have the same meaning as those generally understood by those skilled in the art of the technical field of the present application. The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the scope of the present application. The term "and / or" used in the present application includes any and all combinations of one or more related listed items. The singular forms of "a kind of", "above" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0077] It can be understood that the processor of the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (DigitalSignal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0078] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasable PROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0079] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0084] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A model operation method, characterized in that: include: Determine first resource pre-occupancy values ​​of a plurality of computing units in the same device corresponding to the first to-be-run model respectively; According to each first resource pre-occupancy value and the remaining resource values ​​respectively corresponding to the plurality of computing units, selecting a computing unit that can run the first model to be run from the plurality of computing units as a target computing unit; The first model to be run is run using the target computing unit.

2. The method according to claim 1, characterized in that The multiple computing units include at least two of the following: a central processing unit, a graphics processing unit, a computer vision task processing unit, and a video image synthesis task processing unit.

3. The method according to claim 1, characterized in that The step of selecting a computing unit that can run the first model to be run from the multiple computing units as a target computing unit according to the first resource pre-occupancy values ​​and the remaining resource values ​​respectively corresponding to the multiple computing units includes: According to the calling priorities among the multiple computing units, the remaining resource value of each computing unit is compared with the corresponding first resource pre-occupancy value in turn to determine a remaining resource value greater than the first resource pre-occupancy value as a target remaining resource value; and the computing unit corresponding to the target remaining resource value is determined as a target computing unit that can run the first model to be run; or, According to the order of size between the remaining resource values, starting from the computing unit corresponding to the largest remaining resource value, the remaining resource value of each computing unit is compared with the corresponding first resource pre-occupancy value in turn to determine the remaining resource value greater than the first resource pre-occupancy value as the target remaining resource value; the computing unit corresponding to the target remaining resource value is determined as the target computing unit that can run the first model to be run.

4. The method according to claim 1, characterized in that The method further comprises: In the case of receiving a first task to be executed, determining a first model indicated by the first task to be executed to run; wherein the first model includes at least the first model to be executed and a second model to be executed; updating the remaining resource value of the target computing unit according to the resource pre-occupancy value of the first to-be-run model corresponding to the target computing unit; Determine second resource pre-occupancy values ​​corresponding to the plurality of computing units of the second to-be-run model; A computing unit that can run the second model to be run is selected from the multiple computing units according to the second resource pre-occupancy values ​​and the remaining resource values ​​respectively corresponding to the multiple computing units.

5. The method according to claim 4, characterized in that The method further comprises: When the computing unit that can run the second model to be run is the target computing unit, and the remaining resource value of the target computing unit is less than the corresponding second resource pre-occupancy value, the second model to be run is recorded in the queue to be run corresponding to the target computing unit; wherein the target computing unit is used to run the corresponding models in sequence according to the recording order or model priority in the queue to be run; or, In a case where the computing unit capable of executing the second model to be executed is not the target computing unit, the computing unit capable of executing the second model to be executed is triggered to execute the second model to be executed.

6. The method according to claim 4, characterized in that The method further comprises: Upon receiving a second task to be performed, determining a second model indicated by the second task to be performed; In the process of running the first model using at least one of the plurality of computing units, determining a designated computing unit for running the second model from the plurality of computing units, and preloading the second model; After detecting that the first model has finished running, the first model is uninstalled and the second model is run by using the designated computing unit.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The resident model is loaded, and the remaining resource values ​​respectively corresponding to the plurality of computing units are determined based on the resource pre-occupancy values ​​corresponding to the resident model.

8. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the model operation method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by the processor, it can implement the model operation method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product is used to implement the model running method according to any one of claims 1 to 7.

Images